Library changes for Blender 4.0 #109244

Open
opened 2023-06-22 15:45:25 +02:00 by Ray molenkamp · 21 comments
Member

Here is the list of all changes to our libraries for the Blender 4.0 release.

there is no temp branch yet for the 4.0 lib work

Summary

  • Windows: a number of deps that previously were build with Mingw64/GCC 3.x are now build with msys2/MSVC #105502
  • LibXvid was dropped in favor of ffmpeg's build in support for this format #105502

Preliminary Libraries

  • nothing yet

New Libraries

  • nothing yet

Changed Libraries

To Do

WIP

Done

Done 23-09-26

  • OpenImageIO 2.4.15

    • Update CMake deps builder (#112529)
    • Add Package file to SVN (63492)
    • Add precompiled libraries
      • Windows (63491)
      • Linux (63489.)
      • macOS Intel (63487)
      • macOS Arm (63487)
    • Update install_linux_packages script (N/A, keep using version of distro)
  • OpenSSL 3.1.2

    • Update CMake deps builder (#112529)
    • Add Package file to SVN (63492)
    • Add precompiled libraries
      • Windows (63491)
      • Linux (63489)
      • macOS Intel (63487)
      • macOS Arm (63487)
    • Update install_linux_packages script (N/A, keep using version of distro)
  • Python 3.10.13

    • Update CMake deps builder (#112529)
    • Add Package file to SVN (63492)
    • Add precompiled libraries
      • Windows (63491)
      • Linux (63489)
      • macOS Intel (63487)
      • macOS Arm (63487)
    • Update install_linux_packages script (N/A, keep using version of distro)
  • Sndfile 1.2.2

    • Update CMake deps builder (#112529)
    • Add Package file to SVN (63492)
    • Add precompiled libraries
      • Windows (63491)
      • Linux (63489)
      • macOS Intel (63487)
      • macOS Arm (63487)
    • Update install_linux_packages script (N/A, keep using version of distro)
  • WebP 1.3.2

    • Update CMake deps builder (#112529)
    • Add Package file to SVN (63492)
    • Add precompiled libraries
      • Windows (63491)
      • Linux (63489)
      • macOS Intel (63487)
      • macOS Arm (63487)
    • Update install_linux_packages script (N/A, keep using version of distro)
  • SDL 2.28.2

    • Update CMake deps builder (1594f7fb95)
    • Add Package file to SVN (63492)
    • Add precompiled libraries
      • Windows (63491)
      • Linux (63461)
      • macOS Intel (63487)
      • macOS Arm (63487)
    • Update install_linux_packages script (N/A, keep using version of distro)
  • OSL - rebuild for #110974

    • Update CMake deps builder (#110974)
    • Add Package file to SVN (N/A)
    • Add precompiled libraries
      • Windows (63491)
      • Linux (63461)
      • macOS Intel (63455)
      • macOS Arm (63455)
    • Update install_linux_packages script (N/A)
  • USD 23.05 - with patch to support OpenGL core profile

    • Update CMake deps builder (#110474)
    • Add Package file to SVN (N/A)
    • Add precompiled libraries
      • Windows (63491)
      • Linux (63461)
      • macOS Intel (63460)
      • macOS Arm (63460)
    • Update install_linux_packages script (N/A)

Done - before we tracked dates

  • embree 4.1.0 - no version change, there is an additional patch to resolve a crasher.

    • Update CMake deps builder (ece704af45)
    • Add Package file to SVN (N/A patch change only)
    • Add precompiled libraries
      • Windows (63448)
      • Linux (63461)
      • macOS Intel (N/A, unaffected by patch)
      • macOS Arm (N/A, unaffected by patch)
    • Update install_linux_packages script (N/A, not building it)
  • FFMpeg - no version change, rebuild without libxvid

    • Update CMake deps builder (#105502)
    • Add Package file to SVN (N/A)
    • Add precompiled libraries
      • Windows (63413)
      • Linux(63426)
      • macOS Intel (63460)
      • macOS Arm (63460)
    • Update install_linux_packages script (N/A)
  • USD 23.05 - no version change, repo got renamed from USD to OpenUSD leading to a URI and package Hash change

    • Update CMake deps builder (4fe7626c49)
    • Add Package file to SVN (63439)
    • Add precompiled libraries (N/A)
      • Windows (N/A)
      • Linux (N/A)
      • macOS Intel (N/A)
      • macOS Arm (N/A)
    • Update install_linux_packages script (N/A)
  • tiff 4.5.1

    • Update CMake deps builder (#109133)
    • Add Package file to SVN (63418)
    • Add precompiled libraries
      • Windows (63434)
      • Linux(63426)
      • macOS Intel (63423)
      • macOS Arm (63423)
    • Update install_linux_packages script (N/A)
  • libxml2 2.10.4

    • Update CMake deps builder (#109133)
    • Add Package file to SVN (63418)
    • Add precompiled libraries
      • Windows (63434)
      • Linux(63426)
      • macOS Intel (63423)
      • macOS Arm (63423)
    • Update install_linux_packages script (N/A)
  • Python 3.10.12 (update again after 3.10.11)

    • Update CMake deps builder (#109133)
    • Add Package file to SVN (63418)
    • Add precompiled libraries
      • Windows (63434)
      • Linux(63426)
      • macOS Intel (63423)
      • macOS Arm (63423)
    • Update install_linux_packages script
  • OpenSSL 3.0.9

    • Update CMake deps builder (#109133)
    • Add Package file to SVN (63418)
    • Add precompiled libraries
      • Windows
      • Linux(63426)
      • macOS Intel (63423)
      • macOS Arm (63423)
    • Update install_linux_packages script (N/A)
Here is the list of all changes to our libraries for the Blender 4.0 release. there is no temp branch yet for the 4.0 lib work ### Summary - Windows: a number of deps that previously were build with Mingw64/GCC 3.x are now build with msys2/MSVC #105502 - LibXvid was dropped in favor of ffmpeg's build in support for this format #105502 ### Preliminary Libraries * nothing yet ### New Libraries * nothing yet ### Changed Libraries #### To Do #### WIP ## Done ### Done 23-09-26 - [x] OpenImageIO 2.4.15 - [x] Update CMake deps builder (#112529) - [x] Add Package file to SVN (63492) - [x] Add precompiled libraries - [x] Windows (63491) - [x] Linux (63489.) - [x] macOS Intel (63487) - [x] macOS Arm (63487) - [x] Update `install_linux_packages` script (N/A, keep using version of distro) - [x] OpenSSL 3.1.2 - [x] Update CMake deps builder (#112529) - [x] Add Package file to SVN (63492) - [x] Add precompiled libraries - [x] Windows (63491) - [x] Linux (63489) - [x] macOS Intel (63487) - [x] macOS Arm (63487) - [x] Update `install_linux_packages` script (N/A, keep using version of distro) - [x] Python 3.10.13 - [x] Update CMake deps builder (#112529) - [x] Add Package file to SVN (63492) - [x] Add precompiled libraries - [x] Windows (63491) - [x] Linux (63489) - [x] macOS Intel (63487) - [x] macOS Arm (63487) - [x] Update `install_linux_packages` script (N/A, keep using version of distro) - [x] Sndfile 1.2.2 - [x] Update CMake deps builder (#112529) - [x] Add Package file to SVN (63492) - [x] Add precompiled libraries - [x] Windows (63491) - [x] Linux (63489) - [x] macOS Intel (63487) - [x] macOS Arm (63487) - [x] Update `install_linux_packages` script (N/A, keep using version of distro) - [x] WebP 1.3.2 - [x] Update CMake deps builder (#112529) - [x] Add Package file to SVN (63492) - [x] Add precompiled libraries - [x] Windows (63491) - [x] Linux (63489) - [x] macOS Intel (63487) - [x] macOS Arm (63487) - [x] Update `install_linux_packages` script (N/A, keep using version of distro) - [x] SDL 2.28.2 - [x] Update CMake deps builder (1594f7fb956e7621d91ac8f6a795c2af3b418080) - [x] Add Package file to SVN (63492) - [x] Add precompiled libraries - [x] Windows (63491) - [x] Linux (63461) - [x] macOS Intel (63487) - [x] macOS Arm (63487) - [x] Update `install_linux_packages` script (N/A, keep using version of distro) - [x] OSL - rebuild for #110974 - [x] Update CMake deps builder (#110974) - [x] Add Package file to SVN (N/A) - [x] Add precompiled libraries - [x] Windows (63491) - [x] Linux (63461) - [x] macOS Intel (63455) - [x] macOS Arm (63455) - [x] Update `install_linux_packages` script (N/A) - [x] USD 23.05 - with patch to support OpenGL core profile - [x] Update CMake deps builder (#110474) - [x] Add Package file to SVN (N/A) - [x] Add precompiled libraries - [x] Windows (63491) - [x] Linux (63461) - [x] macOS Intel (63460) - [x] macOS Arm (63460) - [x] Update `install_linux_packages` script (N/A) ### Done - before we tracked dates - [x] embree 4.1.0 - no version change, there is an additional patch to resolve a crasher. - [x] Update CMake deps builder (ece704af4570) - [x] Add Package file to SVN (N/A patch change only) - [x] Add precompiled libraries - [x] Windows (63448) - [x] Linux (63461) - [x] macOS Intel (N/A, unaffected by patch) - [x] macOS Arm (N/A, unaffected by patch) - [x] Update `install_linux_packages` script (N/A, not building it) - [x] FFMpeg - no version change, rebuild without libxvid - [x] Update CMake deps builder (#105502) - [x] Add Package file to SVN (N/A) - [x] Add precompiled libraries - [x] Windows (63413) - [x] Linux(63426) - [x] macOS Intel (63460) - [x] macOS Arm (63460) - [x] Update `install_linux_packages` script (N/A) - [x] USD 23.05 - no version change, repo got renamed from USD to OpenUSD leading to a URI and package Hash change - [x] Update CMake deps builder (4fe7626c49) - [x] Add Package file to SVN (63439) - [x] Add precompiled libraries (N/A) - [x] Windows (N/A) - [x] Linux (N/A) - [x] macOS Intel (N/A) - [x] macOS Arm (N/A) - [x] Update `install_linux_packages` script (N/A) - [x] tiff 4.5.1 - [x] Update CMake deps builder (#109133) - [x] Add Package file to SVN (63418) - [x] Add precompiled libraries - [x] Windows (63434) - [x] Linux(63426) - [x] macOS Intel (63423) - [x] macOS Arm (63423) - [x] Update `install_linux_packages` script (N/A) - [x] libxml2 2.10.4 - [x] Update CMake deps builder (#109133) - [x] Add Package file to SVN (63418) - [x] Add precompiled libraries - [x] Windows (63434) - [x] Linux(63426) - [x] macOS Intel (63423) - [x] macOS Arm (63423) - [x] Update `install_linux_packages` script (N/A) - [x] Python 3.10.12 (update again after 3.10.11) - [x] Update CMake deps builder (#109133) - [x] Add Package file to SVN (63418) - [x] Add precompiled libraries - [x] Windows (63434) - [x] Linux(63426) - [x] macOS Intel (63423) - [x] macOS Arm (63423) - [x] Update `install_linux_packages` script - [x] OpenSSL 3.0.9 - [x] Update CMake deps builder (#109133) - [x] Add Package file to SVN (63418) - [x] Add precompiled libraries - [x] Windows - [x] Linux(63426) - [x] macOS Intel (63423) - [x] macOS Arm (63423) - [x] Update `install_linux_packages` script (N/A)
Ray molenkamp added the
Type
To Do
label 2023-06-22 15:45:25 +02:00
Bastien Montagne added this to the 4.0 milestone 2023-06-22 15:52:53 +02:00
Bastien Montagne added this to the Platforms, Builds Tests & Devices project 2023-06-22 15:53:04 +02:00
Bastien Montagne added the
Module
Platforms, Builds & Tests
label 2023-06-22 15:53:16 +02:00
Author
Member

@mont29 USD recently renamed them selves to OpenUSD leading to an URI/Hash change for the existing 23.05 release, unsure if that's relevant for the linux script, so i'm just pinging you so you're aware.

@mont29 USD recently renamed them selves to OpenUSD leading to an [URI/Hash change](https://github.com/PixarAnimationStudios/OpenUSD/issues/2511) for the existing 23.05 release, unsure if that's relevant for the linux script, so i'm just pinging you so you're aware.

@LazyDodo Ah thanks, for the heads up. Think the script remains unaffected by that (for now at least), since as far as I can tell neither Fedora nor Arch have updated their package names... Will have to keep an eye on that.

@LazyDodo Ah thanks, for the heads up. Think the script remains unaffected by that (for now at least), since as far as I can tell neither Fedora nor Arch have updated their package names... Will have to keep an eye on that.
Author
Member

We're bumping llvm to 15.0.x this will likely cause some clang-format changes, so landing this will need some coordination between the platform teams and core to have it all go over smoothly. Unsure how to go about this yet, @ideasman42 given you're on both teams and have a good handle on clang-format can you look into the ramifications of bumping to v15 (for clang format only, OSL being the only other consumer should be ok according to their documentation )

We're bumping llvm to 15.0.x this will likely cause some clang-format changes, so landing this will need some coordination between the platform teams and core to have it all go over smoothly. Unsure how to go about this yet, @ideasman42 given you're on both teams and have a good handle on clang-format can you look into the ramifications of bumping to v15 (for clang format only, OSL being the only other consumer _should_ be ok according to their documentation )

@LazyDodo bumping LLVM to 15.0.7 seems fine, is anything needed from my side?

  • LLVM 15.0.7 builds (needed to disable our local patch, looks to be macOS only).
  • Would the IGC_LLVM_VERSION be bumped too?
@LazyDodo bumping LLVM to 15.0.7 seems fine, is anything needed from my side? - LLVM 15.0.7 builds (needed to disable our local patch, looks to be macOS only). - Would the `IGC_LLVM_VERSION` be bumped too?
Author
Member

My main ask is for you to check out the formatting changes clang-format does with the V15 version as i'm not familiar enough with how this was setup, so i honestly don't know if it's wrong and needs a .clang-format change or it's doing a better job than v12 and everything is ok already.

My main ask is for you to check out the formatting changes clang-format does with the V15 version as i'm not familiar enough with how this was setup, so i honestly don't know if it's wrong and needs a .clang-format change or it's doing a better job than v12 and everything is ok already.

Ah, the formatting changes are good as far as I'm concerned.

Ah, the formatting changes are good as far as I'm concerned.
Author
Member

embree 4.1.0 rebuild with additional patch to resolve a crasher has been added, platform devs please rebuild main when convenient. (this is still in the pipeline for 3.6 as well, but please wait till the patch has landed there, i will add a work item in #104895 for that once the patch landed there)

embree 4.1.0 rebuild with additional patch to resolve a crasher has been added, platform devs please rebuild main when convenient. (this is still in the pipeline for 3.6 as well, but please wait till the patch has landed there, i will add a work item in #104895 for that once the patch landed there)

We will want to rebuild USD for #110474, but not yet since we may discover more issues to be fixed.

We will want to rebuild USD for #110474, but not yet since we may discover more issues to be fixed.
Author
Member

When it's ready, just add it to the starting post, I'll take that as it being an actionable work item.

Slightly related to this, historically we kept a list of changes/new/preliminary libs which when a dep needed a second rebuild in a cycle we'd just clear the checkmarks. I'm considering of moving that over to a todo / wip / done - style list.

  • todo : this work can be done and uploaded to SVN when time allows for it (this would contain all work items with unchecked checkboxes)
  • wip: lib update is being prepared not quite ready yet (this would be preliminary previously)
  • done: work has been completed. (items with all checks completed, with the date the whole block moved to done noted)

In order to

  • keep some history on LTS releases where the ticket be alive for 2+ years and odds of updating a single dep multiple times (cough python security releases cough) is relatively high, it be good to keep a historic list on when certain work items were completed.

  • It's nice to have an open work item list near the top of the ticket and a whole bunch of stuff you can ignore nicely separated into its own section

@brecht / @ideasman42 either of you have strong opinions on this?

When it's ready, just add it to the starting post, I'll take that as it being an actionable work item. Slightly related to this, historically we kept a list of changes/new/preliminary libs which when a dep needed a second rebuild in a cycle we'd just clear the checkmarks. I'm considering of moving that over to a `todo / wip / done` - style list. - todo : this work can be done and uploaded to SVN when time allows for it (this would contain all work items with unchecked checkboxes) - wip: lib update is being prepared not quite ready yet (this would be preliminary previously) - done: work has been completed. (items with all checks completed, with the date the whole block moved to done noted) In order to - keep some history on LTS releases where the ticket be alive for 2+ years and odds of updating a single dep multiple times (cough python security releases cough) is relatively high, it be good to keep a historic list on when certain work items were completed. - It's nice to have an open work item list near the top of the ticket and a whole bunch of stuff you can ignore nicely separated into its own section @brecht / @ideasman42 either of you have strong opinions on this?

It's fine with me to separate things like that.

It's fine with me to separate things like that.

This also seems fine to me.

This also seems fine to me.

Regarding the remaining library updates for 4.0, which I think we prefer to before Bcon3 (in less than 2 weeks).

LLVM 15 seems to be the only one that is listed but has no corresponding CMake changes yet. I propose to move that to Blender 4.1 unless there is a specific reason we should upgrade now?

Assuming that's the case, Linux seems to be up to date and for macOS I would only need to update SDL. For Windows there are more libraries to be updated, which I guess @LazyDodo will have time for still?

Regarding the remaining library updates for 4.0, which I think we prefer to before Bcon3 (in less than 2 weeks). LLVM 15 seems to be the only one that is listed but has no corresponding CMake changes yet. I propose to move that to Blender 4.1 unless there is a specific reason we should upgrade now? Assuming that's the case, Linux seems to be up to date and for macOS I would only need to update SDL. For Windows there are more libraries to be updated, which I guess @LazyDodo will have time for still?
Author
Member

llvm changes are in #110708 it was done mostly as a favor for the linaro people, i'm ok bumping to 4.1, especially since i saw a critical CVE fly by for zlib that may require a significant time allocation

llvm changes are in #110708 it was done mostly as a favor for the linaro people, i'm ok bumping to 4.1, especially since i saw a critical CVE fly by for zlib that may require a significant time allocation

Feel free to revert 607adbca47 if LLVM 15 is postponed (not critical, but better be consistent).

Feel free to revert 607adbca47 if LLVM 15 is postponed (not critical, but better be consistent).

I'd like to mention the following 3 updates for consideration:

  • libwebp: There's been a few CVEs recently here including a nasty one this week
  • OIIO 2.4.15 + 1 patch: Contains a few fixes for malformed files and would allow me to work toward closing 2 other blender bugs for 4.0. 2.4.16 releases on Oct 1st-ish if we don't want to apply that patch.
  • libHaru: Low priority. Would allow us to remove a small cmake workaround where we were forced to link it with libtiff for no good reason. But it does contain some build changes as their binary structure changed since our previous version.
I'd like to mention the following 3 updates for consideration: - libwebp: There's been a few CVEs recently here including a nasty one this week - OIIO 2.4.15 + 1 patch: Contains a few fixes for malformed files and would allow me to work toward closing 2 other blender bugs for 4.0. 2.4.16 releases on Oct 1st-ish if we don't want to apply that patch. - libHaru: Low priority. Would allow us to remove a small cmake workaround where we were forced to link it with libtiff for no good reason. But it does contain some build changes as their binary structure changed since our previous version.

@deadpin which OIIO patch are you referring to?

@deadpin which OIIO patch are you referring to?

@brecht Need to create a patch from this PR: https://github.com/OpenImageIO/oiio/pull/3984

@brecht Need to create a patch from this PR: https://github.com/OpenImageIO/oiio/pull/3984
Author
Member

handy shortcut, you can just add .diff to the end of any github pull

https://github.com/OpenImageIO/oiio/pull/3984.diff

(works for pulls on p.b.o as well)

handy shortcut, you can just add .diff to the end of any github pull https://github.com/OpenImageIO/oiio/pull/3984.diff (works for pulls on p.b.o as well)

@deadpin ok, I added WebP 3.1.2 and OIIO 2.4.15 + the patch in #112529 and #112528.

@deadpin ok, I added WebP 3.1.2 and OIIO 2.4.15 + the patch in #112529 and #112528.

We follow the VFX platform, and there OpenColorIO 2.3 is planned for next year.

We follow the VFX platform, and there OpenColorIO 2.3 is planned for next year.

If it's still possible, there was another bug that came in last week that requires a small patch to OIIO:
https://patch-diff.githubusercontent.com/raw/OpenImageIO/oiio/pull/3996.diff

Fixes a DDS load issue (https://github.com/OpenImageIO/oiio/issues/3993)

If it's still possible, there was another bug that came in last week that requires a small patch to OIIO: https://patch-diff.githubusercontent.com/raw/OpenImageIO/oiio/pull/3996.diff Fixes a DDS load issue (https://github.com/OpenImageIO/oiio/issues/3993)
Sign in to join this conversation.
No Label
Interest
Alembic
Interest
Animation & Rigging
Interest
Asset Browser
Interest
Asset Browser Project Overview
Interest
Audio
Interest
Automated Testing
Interest
Blender Asset Bundle
Interest
BlendFile
Interest
Collada
Interest
Compatibility
Interest
Compositing
Interest
Core
Interest
Cycles
Interest
Dependency Graph
Interest
Development Management
Interest
EEVEE
Interest
EEVEE & Viewport
Interest
Freestyle
Interest
Geometry Nodes
Interest
Grease Pencil
Interest
ID Management
Interest
Images & Movies
Interest
Import Export
Interest
Line Art
Interest
Masking
Interest
Metal
Interest
Modeling
Interest
Modifiers
Interest
Motion Tracking
Interest
Nodes & Physics
Interest
OpenGL
Interest
Overlay
Interest
Overrides
Interest
Performance
Interest
Physics
Interest
Pipeline, Assets & IO
Interest
Platforms, Builds & Tests
Interest
Python API
Interest
Render & Cycles
Interest
Render Pipeline
Interest
Sculpt, Paint & Texture
Interest
Text Editor
Interest
Translations
Interest
Triaging
Interest
Undo
Interest
USD
Interest
User Interface
Interest
UV Editing
Interest
VFX & Video
Interest
Video Sequencer
Interest
Virtual Reality
Interest
Vulkan
Interest
Wayland
Interest
Workbench
Interest: X11
Legacy
Blender 2.8 Project
Legacy
Milestone 1: Basic, Local Asset Browser
Legacy
OpenGL Error
Meta
Good First Issue
Meta
Papercut
Meta
Retrospective
Meta
Security
Module
Animation & Rigging
Module
Core
Module
Development Management
Module
EEVEE & Viewport
Module
Grease Pencil
Module
Modeling
Module
Nodes & Physics
Module
Pipeline, Assets & IO
Module
Platforms, Builds & Tests
Module
Python API
Module
Render & Cycles
Module
Sculpt, Paint & Texture
Module
Triaging
Module
User Interface
Module
VFX & Video
Platform
FreeBSD
Platform
Linux
Platform
macOS
Platform
Windows
Priority
High
Priority
Low
Priority
Normal
Priority
Unbreak Now!
Status
Archived
Status
Confirmed
Status
Duplicate
Status
Needs Info from Developers
Status
Needs Information from User
Status
Needs Triage
Status
Resolved
Type
Bug
Type
Design
Type
Known Issue
Type
Patch
Type
Report
Type
To Do
No Milestone
No Assignees
5 Participants
Notifications
Due Date
The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: blender/blender#109244
No description provided.