Library changes for Blender 4.1 #113157
Labels
No Label
Interest
Alembic
Interest
Animation & Rigging
Interest
Asset Browser
Interest
Asset Browser Project Overview
Interest
Audio
Interest
Automated Testing
Interest
Blender Asset Bundle
Interest
BlendFile
Interest
Collada
Interest
Compatibility
Interest
Compositing
Interest
Core
Interest
Cycles
Interest
Dependency Graph
Interest
Development Management
Interest
EEVEE
Interest
EEVEE & Viewport
Interest
Freestyle
Interest
Geometry Nodes
Interest
Grease Pencil
Interest
ID Management
Interest
Images & Movies
Interest
Import Export
Interest
Line Art
Interest
Masking
Interest
Metal
Interest
Modeling
Interest
Modifiers
Interest
Motion Tracking
Interest
Nodes & Physics
Interest
OpenGL
Interest
Overlay
Interest
Overrides
Interest
Performance
Interest
Physics
Interest
Pipeline, Assets & IO
Interest
Platforms, Builds & Tests
Interest
Python API
Interest
Render & Cycles
Interest
Render Pipeline
Interest
Sculpt, Paint & Texture
Interest
Text Editor
Interest
Translations
Interest
Triaging
Interest
Undo
Interest
USD
Interest
User Interface
Interest
UV Editing
Interest
VFX & Video
Interest
Video Sequencer
Interest
Virtual Reality
Interest
Vulkan
Interest
Wayland
Interest
Workbench
Interest: X11
Legacy
Blender 2.8 Project
Legacy
Milestone 1: Basic, Local Asset Browser
Legacy
OpenGL Error
Meta
Good First Issue
Meta
Papercut
Meta
Retrospective
Meta
Security
Module
Animation & Rigging
Module
Core
Module
Development Management
Module
EEVEE & Viewport
Module
Grease Pencil
Module
Modeling
Module
Nodes & Physics
Module
Pipeline, Assets & IO
Module
Platforms, Builds & Tests
Module
Python API
Module
Render & Cycles
Module
Sculpt, Paint & Texture
Module
Triaging
Module
User Interface
Module
VFX & Video
Platform
FreeBSD
Platform
Linux
Platform
macOS
Platform
Windows
Priority
High
Priority
Low
Priority
Normal
Priority
Unbreak Now!
Status
Archived
Status
Confirmed
Status
Duplicate
Status
Needs Info from Developers
Status
Needs Information from User
Status
Needs Triage
Status
Resolved
Type
Bug
Type
Design
Type
Known Issue
Type
Patch
Type
Report
Type
To Do
No Milestone
No project
No Assignees
7 Participants
Notifications
Due Date
No due date set.
Dependencies
No dependencies set.
Reference: blender/blender#113157
Loading…
Reference in New Issue
No description provided.
Delete Branch "%!s(<nil>)"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Here is the list of all changes to our libraries for the Blender 4.1 release.
temp branch:
tmp_libupdate_41Summary
Preliminary Libraries
New Libraries
Removed Libraries
Changed Libraries
To Do
WIP
Done yy-mm-dd
Done 24-03-17
OSL 1.13.7.0
install_linux_packagesscript (N/A)OpenImageDenoise 2.2.2
b32554e53binstall_linux_packagesscript (N/A)Superseded 2024-03-15
install_linux_packagesscript (N/A)Done
OpenImageDenoise 2.1.0
install_linux_packagesscript (90f83f71f2)wayland_protocols 1.32
7aa3d967ba)install_linux_packagesscript (N/A)mesa 23.3.0
install_linux_packagesscript (N/A)Python 3.11.6
install_linux_packagesscript (90f83f71f2)Numpy 1.24.3
install_linux_packagesscript (90f83f71f2)SQLite 3.42.0
install_linux_packagesscript (N/A)FFI 3.3.4
install_linux_packagesscript (N/A)MaterialX 1.38.8
install_linux_packagesscript (90f83f71f2)boost 1.82
install_linux_packagesscript (90f83f71f2)openexr 3.2.1
install_linux_packagesscript (90f83f71f2)libdeflate 1.18
install_linux_packagesscript (dfc00e8a18)opencolorio 2.3.0
install_linux_packagesscript (90f83f71f2)openvdb 11.0
install_linux_packagesscript (90f83f71f2)Vulkan Headers v1.3.270
install_linux_packagesscript (N/A)Vulkan Loader v1.3.270
install_linux_packagesscript (N/A)llvm 17.0.6
install_linux_packagesscript (ec5594ec7f)osl 1a7670600c8b08c2443a78d03c8c27e9a1149140 (this is what #110708 had, we can likely bump further than this)
install_linux_packagesscript (ec5594ec7f- not exact since not released yet)ispc 1.21.1
install_linux_packagesscript (N/A)ocloc patch
install_linux_packagesscript (N/A)libxml2 2.12.3
install_linux_packagesscript (N/A)OpenImageIO 2.5.6.0
install_linux_packagesscript (ec5594ec7f)USD 23.11
install_linux_packagesscript (ec5594ec7f)sse2neon cfaa59fc04fecb117c0a0f3fe9c82dece6f359ad
install_linux_packagesscript (N/A)opensubdiv 3.6.0
install_linux_packagesscript (90f83f71f2)openal 1.23.1
install_linux_packagesscript (N/A)openimagedenoise 2.2rc
install_linux_packagesscript (a48e825c1e)level zero 1.15.8
install_linux_packagesscript (a48e825c1e)opencolorio 2.3.2
deb4273565)install_linux_packagesscript (a48e825c1e)dpcpp patch for caching
install_linux_packagesscript (N/A)python 3.11.7
install_linux_packagesscript (N/A)vpx 1.14.0
install_linux_packagesscript (N/A)ffmpeg 6.1.1
install_linux_packagesscript (N/A)openssl 3.1.5
install_linux_packagesscript (N/A)sqlite 3.45.1
install_linux_packagesscript (N/A)Done 24-02-16
install_linux_packagesscript (N/A)CC @LazyDodo @ideasman42
I have no plans to start working on this immediately, but just so we have an issue to track things. The VFX platform upgrades should be straightforward, especially compared to last year.
Saw some people trying to build against a newer OCIO and having issues, there may be some reintegration work needed there.
started a working branch in
tmp_libupdate_41sofar only bumped the builder side for python. ffi and sqlite were updated to match pythons versions, materialX required an update to support building against py3.11. ffi's URI changed to github as the location we were using did not have the new release.openexr 3.2.1
OpenExr dropped zlib and gained a new dependency libdeflate 1.18
which it was determined to either download itself from git which is a
hard no for us, or use pkg_config to find it, which windows doesn't
use/have.
we were not the first ones to run into this, someone posted a
patch in an upstream ticket over at
https://github.com/AcademySoftwareFoundation/openexr/issues/1588
patch was used almost used verbatim with the sole change being
using the static version for the following reasons:
work harvesting and distributing the build artefacts on the blender
side of things.
I picked deflate 1.18 since it's what openexr defaults to 1.19 is
available though, unsure what we want to do here.
deps_builder: opencolorio 2.3.0
the patch for share/cmake/modules/Findpystring.cmake no longer applies
and could not be updated since none of the surrounding code appears
to be there any more, this was mac only, so @brecht may need to
take a look there.
The OpenColorIO patch can be eliminated completely.
For the OpenEXR deflate issue, I'll try to get it merged upstream, so that we can eliminate the patch in the future.
As part of this update I would also like to make OSL and FFmpeg dynamic, to resolve #113892. We could also upgrade to the latest OSL along with this, but it is not out yet though should be soon. I will need to do some work for API compatibility there.
Given we're likely gonna land all updates at once, any objections on landing #110708 in the temp branch? I'd rather have everything in single place.
No objections, the more of these updates we can batch together the easier for me.
A polite request for the following also in addition to these (they will help my ARM64 efforts):
d3d23ef3caf77239f463d305ffb87e68ed852ea7- sadly the SF version inversions.cmakeis way out of date now, and they haven't versioned for over a year on github, so it's currently a random commit hash00c6f241d4e79673392ef8dea5730e4037d728b3These (and a few of the ones you are already working on) are ones I use in my ARM64 port: https://gitlab.com/Linaro/windowsonarm/forks/blender/-/merge_requests/34/diffs#965e56bce6e484b139c6fd9e40899b4d75784c6f
FreeGLUT historically was only used for standalone cycles, i haven't checked up there in a while, i don't think it uses it anymore, we may be able to drop it all together, @brecht can you confirm this?
ISPC lists 1.21, 1.20 and 1.19 are working, we can probably just bump to 1.21 there, 1.21.1 is not listed as working, but has some oneapi fixes for linux, @xavierh you have any strong opinions on a preferred version here?
sse2neon no opinion, think this is a relatively trouble free dep
Vulkan , no opinion, @Jeroen-Bakker do you have any objections to bumping the SDK version?
I have no strong opinion for ISPC version, 1.21.1 is a good choice.
FreeGLUT can indeed be removed.
Bumping sse2neon should be fine too.
We can go to the latest vulkan 1.3 release. Will make it easier to add optional features in case we need them.
Excellent! thanks guys! i'll get those updates going
I bumped llvm+OSL to what we had in #110708 we can likely bump further than that, i think we're waiting on a new release of OSL there. llvm may need some ispc patches applied, i have not looked at this yet in details, figured i'd wait until the llvm version solidifies.
There will have to be some reintegration work for openvdb 11, cycles
ccl::ToNanoOpis throwing some template error while building, building withWITH_NANOVDB=Offworks fine though, so the work be likely minor in nature.I pushed changes to the
tmp_libupdate_41branch:Getting Python to use our own libraries for ssl, sqlite was tricky due to Python build system changes. It works for me on Ubuntu, hopefully works on Rocky as well.
We'll want to update OpenImageDenoise as well for CUDA, HIP and possibly Metal support. But this is not ready and would happen later in the release cycle.
After upgrading LLVM
make formatshows changes in hundreds of files. Previously we have been able to keep results nearly the same across a few versions. Ideally we can figure out a way to do the same here. Or we will need to take some care in landing the clang-format binaries for all platforms around the same time?Maybe @ideasman42 wants to take a look at this or remembers something about this? Most changes seem to be about placement of
{after if statements.We're on llvm15 currently since that's what the old patch had, given llvm bumps tend to be disruptive in the formatting department, do we want to bump to 16 or 17 so we have a little longer stable period?
I've added an ocloc update for Linux in the list, that would be good to do along with the other libraries. This does not affect Windows or macOS.
For LLVM and clang-format, I guess we might as well update to the latest?
Regarding clang-format, AFAICS the only practical option is to try to update LLVM for platforms at once - as I don't think there is a way to keep the old behavior.
Think my longest libs upload was shortly over 14 hours, landing "at the same time" will likely be impossible, I suspect the best we can do is land on the same day and tell people not to commit make format changes if the number of changes seem unreasonable.
I bumped llvm to 17.0..6 and osl and ipsc/oidn seem to build, but i have not yet had the time to do build/test blender against these libs
clang-format is statically linked and should not depend on anything else, so we could land it before or after the rest of the libs. I can even gather the binaries from all platforms and commit them myself in one go.
Builds on Linux & tests pass.
Committed an update to mesa which was needed as building with an old mesa was failing with the new LLVM.
macOS is ok too with the latest change.
I suggest the following plan:
tmp_libupdate_41, verify it builds ok with current libs on buildbot, and land the changes in mainclang-formatbinaryclang-formatbinaries to this issueclang-formatbinaries for all platforms and the correspondingmake formatchanges at the same moment@LazyDodo @ideasman42, what do you think?
That's pretty clever, sounds good!
Do you still want to turn ffmpeg shared on linux/mac? or do you want to carry that over to 4.2? (has always been shared on windows, no change there)
I forgot about ffmpeg. I think just making LLVM dynamic is enough to fix #113892, and it's not so important to do the same for ffmpeg.
It won't, I think we were a bit too enthusiastic requiring py 3.11 in the main CMakeLists.txt, this should probably be done "a little while"(tbd) after landing the lib update.
@LazyDodo @ideasman42 right, I reverted that Python bump to 3.11 commit now.
@brecht sounds good,
clang-formatbinary is self contained so this should work.I have hit a snag, I have a single failing test on a single architecture, the trouble maker is
this renders like this on my ivy-lake box, on my Skylake-U this appears to render correctly using the same set of blender binaries
my first instinct was to disable the AVX2 kernel for cycles on the skylake box with
CYCLES_CPU_NO_AVX2=1to see if it started failing there, but still a good render, validated it was using the sse4 kernelswell, i thought it was pretty clear at this point it HAS to be embree/ispc, as openvdb doesn't have any sse4/avx/avx2 specializations
disabled embree, logs mention .
.....still blocks.... to be super sure made a build with
WITH_CYCLES_EMBREE=Offand ......still blocks....got a copy of SDE
on the ivy box emulating skylake using sde renders correctly
on the skylake box emulating ivy using sde renders..... correctly... (and yes, validated it was indeed using the sse4 kernels)
I'm stumped...
Ivy Bridge has AVX but not AVX2.
CYCLES_CPU_NO_AVX2maybe doesn't do 100% of what it's supposed to do? I'm stumped it's happening with lib updates, as a next step I'd still recommend checking the openvdb upgrade in isolation?To be sure, is it really the new libraries that are causing the problem, or maybe it happens
main/ daily builds too and had not be discovered yet?I completely changed the kernel side NanoVDB sampling in
8ba474dand6cdb431a few days ago.ASAN on Linux found some undefined behavior, looking into it, in the hope it fixes this problem.
I started my build validations by validating
tmp_libupdate_41builds against main libs, this was done against either5a5561b3b5or something really close to it and that passed all tests.I'm still no closer to figuring out what's going on, but it appears the problem could be on the loading end of things, just opening up the test file with the UI gives a rather different looking result between the two archs
This was a false positive as far as I can tell. If the Blender viewport shows problem, it doesn't seem like a NanoVDB problem anyway, since that's not used there.
I looked through the OpenVDB changes between 10 and 11 but could not find anything particularly suspicious, although many lines of code changed.
Right now I don't have a better suggestion than bisecting the OpenVDB changes between v10 and v11.
More of a status update than anything note worthy to report
Ivy:
skylake:
Update: problem found!
https://github.com/AcademySoftwareFoundation/Imath/blob/main/src/Imath/half.h#L332
__lzcntis haswell+ , on ivy it executes without an invalid instruction exception, but performs a bitscanreverse instead sourcei'm out of town for the weekend, but in a quicktest borrowing some cycles code
solves the issue in my test code, for blender we'll need to patch both openvdb (they copy/pasted a chuck of imath into their own source tree) and imath in the following locations
https://github.com/AcademySoftwareFoundation/Imath/blob/main/src/Imath/half.h#L332
https://github.com/AcademySoftwareFoundation/openvdb/blob/master/openvdb/openvdb/math/Half.h#L345
if no-one picks this off this weekend, i'll get that patched up on monday.
Upstream fixes:
https://github.com/AcademySoftwareFoundation/Imath/pull/358
https://github.com/AcademySoftwareFoundation/openvdb/pull/1733
The upstream conversation reminds me, we still haven't bumped our flags to target sse42 (mostly since i didn't get word of this, the admin meeting notes where this got decided used the words "it was suggested" while someone attending the meeting a few weeks ago clarified this was actually decided) given this is not as easy as just "bump the flags in cmake" (as i very much would like to prevent a steady stream of "blender crashes at startup" tickets on the tracker) i'll make a design/discuss ticket next week and we'll target 4.2 for the actual bump?
Yes, I thought about that too. I don't think it's going to make a immediate big difference in performance to bump our flags to sse4.2 for external libraries, so not a big deal to postpone it. As far as I'm concerned we can even leave that for the end of next year, or whenever we happen to update many libraries and it doesn't add much extra work.
Updating Blender itself to sse4.2 could be done any time before that, but indeed a check in the Blender launchers on Windows and Linux would be good.
I'd like to request a bump of OIIO to 2.5.x. Ray provided a build for me to check and things seem fine wrt our tests and some cursory performance scenarios I tried.
This allows us to remove some of our patches, fixes an additional DDS bug, and provides a better base for some small tweaks and investigations I'd like to do early next year.
There is a bit of extra terminal spew that is solved with a tiny blender-side change that could wait until after the libs are committed as it doesn't impact the tests or performance.
Fine with me. I guess you already have the changes you can push to the branch?
Applied the upstream patches for both openexr and imath, also bumped libxml2 as the version we had was leaking exports into the blender binary. which just leaves oiio to be dealt with.
@deadpin I can either land the patch you send me last week, or you can make whatever changes you need in the 41 temp branch your self, either way works for me just let us know.
@LazyDodo The patch from last week is good. Can you land it since you're already setup?
not a problem,
on itdone!@brecht took a bit but can confirm that with the current state of
tmp_libupdate_41all tests are passing on windows, there are no more outstanding issues that i'm aware of, I think we can proceed with your plan.The branch has been merged. Precompiled libraries can now be committed for all platforms, and remember to leave out
clang-format.I will do the macOS libraries in the week of January 2-5.
Committing 4.1 Linux libraries to SVN:
The clang-format has been held back, this is the newly built version for 4.1:
https://download.blender.org/ftp/ideasman42/temp/clang-format.xz
here's mine
OIIO was including imath's
half.hwhen it didn't need to, which lead to build errors on lite builds, this has been fixed already upstream in OIIO PR 4062 i added the PR to our build and validated the issue to be solved. this does however mean the linux/windows libs that already have landed need a rebuild.Looks like sse2neon has versioned if you want to swap the random hash for that: https://github.com/DLTcollab/sse2neon/releases/tag/v1.7.0
I'll leave it up to @brecht as this doesn't affect linux+windows at this point, bumping it would require no rework there, if he has the libs done already just not uploaded, it's unlikely to be worth the trouble though.
Linux OIIO has been rebuilt.
macOS libs, packages and clang format have all landed.
For macOS Arm, I'm still investigating some failing tests with sse2neon and maybe opensubdiv. For this reason the sse2neon update has been temporarily reverted.
I'm not sure if it's completely new, but I'm noticing this message more after the Linux libs upgrade:
Apparently this is solved since OpenAL 1.22.0, while we are on 1.21.1. We could upgrade.
https://github.com/kcat/openal-soft/issues/554
best to upgrade, even if it's a harmless warn, people reporting on the tracker generally latch on to the first failure they see which in this case is a bit of a red herring
Confirmed the error happens on Linux & updating resolves the problem.
Submitted a PR that uses the latest version of OpenAL !116877.
@LazyDodo @ideasman42 I updated the to do list for opencolorio, openimagedenoise and level zero changes.
I will land the macOS libs shortly.
Also, I had to make a change to our pystring patch to get OpenColorIO to build. So if you are doing some partial update to an existing build folder, be sure to rebuild that as well or you might get a build error in OpenColorIO.
@LazyDodo @ideasman42 there's a patch for dpcpp now as well, hopefully not too much work to include that too.
Not sure when it will next version (a question for the intel guys maybe), but the next release of OIDN2 (commits currently in devel branch) has ARM64 support which would be extremely useful to me if we could get it included!
Also has a bunch of speedups for x64 CPU denoising, and has removed DNNL completely, so a bit more lighweight also AFAICT.
we've bumped to the 2.2RC a few days ago, which i think includes all of that?
You're absolutely right, apparently I managed to completely miss that - will rework my stuff for it
Jakub Marcowski referenced this issue2024-03-05 21:23:37 +01:00
Jakub Marcowski referenced this issue2024-03-05 21:24:08 +01:00