Library changes for Blender 3.5 #99618
Labels
No Label
Interest
Alembic
Interest
Animation & Rigging
Interest
Asset Browser
Interest
Asset Browser Project Overview
Interest
Audio
Interest
Automated Testing
Interest
Blender Asset Bundle
Interest
BlendFile
Interest
Collada
Interest
Compatibility
Interest
Compositing
Interest
Core
Interest
Cycles
Interest
Dependency Graph
Interest
Development Management
Interest
EEVEE
Interest
EEVEE & Viewport
Interest
Freestyle
Interest
Geometry Nodes
Interest
Grease Pencil
Interest
ID Management
Interest
Images & Movies
Interest
Import Export
Interest
Line Art
Interest
Masking
Interest
Metal
Interest
Modeling
Interest
Modifiers
Interest
Motion Tracking
Interest
Nodes & Physics
Interest
OpenGL
Interest
Overlay
Interest
Overrides
Interest
Performance
Interest
Physics
Interest
Pipeline, Assets & IO
Interest
Platforms, Builds & Tests
Interest
Python API
Interest
Render & Cycles
Interest
Render Pipeline
Interest
Sculpt, Paint & Texture
Interest
Text Editor
Interest
Translations
Interest
Triaging
Interest
Undo
Interest
USD
Interest
User Interface
Interest
UV Editing
Interest
VFX & Video
Interest
Video Sequencer
Interest
Virtual Reality
Interest
Vulkan
Interest
Wayland
Interest
Workbench
Interest: X11
Legacy
Blender 2.8 Project
Legacy
Milestone 1: Basic, Local Asset Browser
Legacy
OpenGL Error
Meta
Good First Issue
Meta
Papercut
Meta
Retrospective
Meta
Security
Module
Animation & Rigging
Module
Core
Module
Development Management
Module
EEVEE & Viewport
Module
Grease Pencil
Module
Modeling
Module
Nodes & Physics
Module
Pipeline, Assets & IO
Module
Platforms, Builds & Tests
Module
Python API
Module
Render & Cycles
Module
Sculpt, Paint & Texture
Module
Triaging
Module
User Interface
Module
VFX & Video
Platform
FreeBSD
Platform
Linux
Platform
macOS
Platform
Windows
Priority
High
Priority
Low
Priority
Normal
Priority
Unbreak Now!
Status
Archived
Status
Confirmed
Status
Duplicate
Status
Needs Info from Developers
Status
Needs Information from User
Status
Needs Triage
Status
Resolved
Type
Bug
Type
Design
Type
Known Issue
Type
Patch
Type
Report
Type
To Do
No Milestone
No project
No Assignees
13 Participants
Notifications
Due Date
No due date set.
Dependencies
No dependencies set.
Reference: blender/blender#99618
Loading…
Reference in New Issue
No description provided.
Delete Branch "%!s(<nil>)"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Here is the list of all changes to our libraries for the Blender 3.5 release.
Summary
Preliminary Libraries
New Libraries
ShaderC v2022.3
388bbc3290install_depsscript25e28f518cVulkan Loader v1.2.198
388bbc3290install_depsscript25e28f518cHarfbuzz 5.1.0 (support for bidirectional text, Requested by UI team)
388bbc3290install_depsscript25e28f518cFribidi v1.0.12 (support for bidirectional text, Requested by UI team)
388bbc3290install_depsscript25e28f518cmeson 0.63 pip module (build time dep required for Fribidi/Harfbuzz, standalone meson will not suffice on windows due to missing python)
388bbc3290UpdateN/A, we only use distro packages for fribidi/harfbuzz.install_depsscriptmatarialX 1.38.6
388bbc3290install_depsscriptb099e9924epybind11 (buildtime dep for ocio/oiio/materialx)
388bbc3290install_depsscripta318f3b8d6Updated Libraries
OpenImageIO 2.4.9.0 (updated from 2.4.6.0)
(388bbc3290){rBL63160}install_depsscript (#105958)9e332b113b,a318f3b8d6OpenVDB 10.0.0
388bbc3290install_depsscript9e332b113b,a318f3b8d6Boost 1.80.0
388bbc3290install_depsscript6347562fb,a318f3b8d6Python 3.10.9
3494ba3815install_depsscript6347562fb,a318f3b8d6Numpy 1.23.5
bcbd13201ainstall_depsscript6347562fbOpenSubDiv 3.5.0
388bbc3290install_depsscript9e332b113bOpenColorIO 2.2.0
388bbc3290install_depsscript9e332b113b,a318f3b8d6OSL 1.13-dev-1a7670600c8b08c2443a78d03c8c27e9a1149140
31ba4dd82einstall_depsscript4e027fdde6,9cef74f58bllvm 12.0.0 (no version change, but rebuild with NVPTX support for OSL/GPU support)
388bbc3290install_depsscript9cef74f58bYaml-cpp 0.7.0
388bbc3290UpdateN/A, using distro packagesinstall_depsscriptSQLite 3.39.4
bcbd13201aUpdateN/A, using distro packagesinstall_depsscriptUSD 22.11
388bbc3290install_depsscript4e027fdde6,a318f3b8d6SPNAV 1.1
396816faacUpdateN/A, using distro packagesinstall_depsscriptIGC 1.0.13064.7
install_depsscript (N/A)gmmlib 22.3.0
install_depsscript (N/A)ocloc 22.49.25018.21
install_depsscript (N/A)Changed status from 'Needs Triage' to: 'Confirmed'
Added subscribers: @LazyDodo, @brecht, @dr.sybren
I committed changes
tmp_libupdate_34branch. See commit logs for details. There were many more issues to solve that I expected, and still some remain.Notes:
Meson is probably being bundled in Blender site packages? I filtered it out for Linux/macOS. I modified the Python executable rpath so it can successfully do
from pxr import Usdandimport pyopenvdboutside of Blender. I'm not sure how Windows would handle this. I also had to workaround the static Python library getting embedded into the USD library leading to duplicate Python interpreter symbols that caused issues for this case.WITH_PYTHON_MODULEor separate build and install folders as on buildbot.WITH_PYTHON_INSTALLwhen building against system libraries.it is, there's really no neat way around that, i initially used standalone meson for libepoxy but standalone misses the python components and either harfbuzz or fridibi really wants to run some python, I don't have python in my path since deps will try to use and build/link against a version we don't want (cmake's findpython is ruthless, if there's a trace of python... it will find it...). Putting meson in at least the build version of python was needed , should probably be filtered out for shipping.
I have not tested importing from the python.exe we ship with blender, a quick test confirmed it to be failing, will add to my todo list
Static python is luckily not a thing on windows, so whatever bit you there won't be an issue on windows.
seems fair enough, some OCIO v2 support has been merged, but i'm not entirely sure what release that'll be in, and there's unmerged fixes on top of that to make mac happy, it just doesn't seem quite ready yet.
How important is the no-install scenario? does it work when you do install? At-least on windows install is required for any test involving blender.exe to pass
We probably at one point need to sit down with the python team and discuss
WITH_PYTHON_MODULEthere's no official support for it, yet every-time anyone tries to build it runs into trouble, the platform team still sinks in hours helping them since telling people to go pound sand is just not something we do. We as a group need to either step up and support and CI it or shoot it in the face, this limbo state is not working for me.I just thought of a workaround and committed it, so hopefully that holds up.
The issue is not really a no-install case, but that some executables that are run as part of the build process need these libraries, so they need to be available before the install step. Though probably I can avoid linking to these libraries and simplify things further.
Agreed.
This one is tricky, i can get it to work if i run
os.add_dll_directory(os.getcwd())(when run from the blender bin folder) beforeimport pyopenvdbas the path and current directory are no longer used by python.not entirely sure what we can do to make it work out of the box, maybe symlink the dlls into pythons dll folder will work but that feels hacky and will fail on FAT32 volumes....doesn't feel like the way to go
Maybe a patch like this (untested)?
It's not pretty but not sure what the better solution would be.
I did some further changes:
Testing the branch on the buildbot (so using the 3.3 libs), I noticed on Windows there's an error in the ffmpeg test:
https://builder.blender.org/admin/#/builders/57/builds/1465
I'm done with Linux/macOS unless new issues come up. I would like to land static libstdc++ and adding $ORIGIN/lib rpath as separate commits.
I'll get ffmpeg test sorted out, somewhat odd : there's 2 tests named
ffmpeg, we should probably rename one of them?I could not reproduce the issue locally, a bit on a hunch I stuck it into the blender_test executable hoping that would fix it, since that had no issues, and renamed it to ffmpeg_codecs while i was at it.
All platforms pass now with the 3.3 libs
https://builder.blender.org/admin/#/builders/57/builds/1467
That actually works (but... *1), while i was mucking about in site.py, I noticed some callback system in-place
if you place a
usercustomize.pyin thesite-packagesfolder with the contents of your patch it also works, bonus we don't have to hack around in the stock site.py*1 we internally hint USD to where it can find its metadata, while this fix lets you import
Usdwhen you try to use it it'll shit the bed and crash python when it can't find it's filesusing this as my usercustomize.py solves all issues and i can run HelloWorld.py from the python binary... but....yeah... that absolute path there.... i'm not entirely sure how to find the data folder from stock python... maybe we can generate this file from cmake which does know the major blender version?
i suspect Linux/Mac also need this.....
Indeed that needs to be solved for macOS and Linux too.
Finding the path is not so difficult, it can be done with
os.abspath(os.path.join(exe_dir, '..', '..', 'datafiles', 'usd')). However it's not ideal to load the USD library every time Python starts.It seems that by default USD looks for a
usdfolder next to the USD shared library (which is how it gets installed). So maybe it's easiest to just follow that layout and put the folder inlib/usdinstead of3.4/datafiles/usd.That was a choice by @dr.sybren i have honestly no opinion on it but he seemingly found it worth doing to the point of patching the USD library (no longer needed, but was when it landed) to get it done so i suspect he may have some strong feelings there.
i do have to note on windows it doesn't sit in
/libbut right next toblender.exesince rpath isn't a thing on windows, we may be able to get it in/libwith a second manifest like we did for the CRT, it'll require some fiddling, but i think it's doable.From D6287#149291, it appears the reason was that we were statically linking USD and you can't have a
usdfolder next to the Blender binary in a macOS app bundle. But now that we have a shared library in another folder, that's not a problem anymore. So I think it's fine to change.It would be nice to put all our dlls in lib folder on Windows but not sure it's worth sinking much time in if it's complicated. I think a
usdfolder at the top level would be acceptable.This issue was referenced by
a87d3edb98Added subscriber: @BrianSavery
@LazyDodo are you building the USD library with their "Shared Monolithic" method?
Windows Only update:
blender.sharedfolder (not married to the name, but it will have to be in theblender.[something]naming scheme. ) exceptucrtbase.dll(#88813) andBlendThumb.dll(could move but we'll have to change the registration code to account for it)blender.shared/usdso no more path hinting is requiredcustomsite.pywill be installed to site-packages to tell python about our shared library folder. The source file for this is currently inrelease\scripts\usercustomize.pywhich i'm not super thrilled about, but couldn't find a more appropriate place, open to suggestions hereAdded subscriber: @ray-1
Correct, you can find the exact build flags we will use over here
Removed subscriber: @ray-1
The main issue for #100569 is that we are building without OpenGL support currently. We could look into enabling that if Hydra support is going to be worked on soon.
Maybe
release/windows/python/usercustomize.py.Indeed it is. It would likely be added for 3.5. You mean you're not building the "USD Imaging" module which builds hydra and the GL Storm delegate I think?
It's building with USD imaging but
PXR_ENABLE_GL_SUPPORT=OFF.think with this last round of changes my plate is empty lib wise, I can do the work, however I'm not very familiar with USD's internals, for broader support would we also need to build with
PXR_ENABLE_VULKAN_SUPPORTand/orPXR_ENABLE_METAL_SUPPORTor will just GL cover us?PXR_ENABLE_GL_SUPPORTshould be enough for Hydra right now, but if it doesn't interfere with anything I'll enablePXR_ENABLE_METAL_SUPPORTas well since work on a Metal backend is ongoing.PXR_ENABLE_VULKAN_SUPPORTI don't think matters until we have a Vulkan backend in Blender, and that's a while off. It think it would also require Vulkan headers. I would not bother.looking at
usd.cmakeSeems like :
Yeah, I guess we also need OIO dynamic then to have texture in Hydra Storm. Also not happy about duplicating those image libs but agree dynamic OIIO + static image libs makes sense. I would try to make OpenEXR dynamic also since it seems to be the biggest, and I'd like to avoid having two OpenEXR thread pools. Maybe in the future we can replace most of our image reading code with OpenImageIO, but that's not going to happen overnight.
For Linux, I'll look into it.
Added subscriber: @deadpin
Historically oiio's support for packed textures is what has been preventing that, however oiio's ioproxy support should line up with most if not all the formats we care about (@deadpin has a list somewhere, edit: P2981) for oiio 2.4.x so that be an option for probably 3.5 or 3.6 depending on when that's coming out.
OIIO and OpenEXR are now dynamically linked on all platforms in the branch. I also enabled OpenGL support for USD, but did not test that on Windows.
On Linux the problem where USD links to libGL and libX11 remains, as well as only working with GLX. Ideally we'd solve that, I created an issue regarding that here:
https://github.com/PixarAnimationStudios/USD/issues/2011
Linking to those libraries is not really any worse than what Blender still does for official release builds, except if you're doing a Wayland only build with our precompiled libraries. Hydra not working on Wayland would be a bigger problem. Neither seems blocking for using the libs in this state for Blender 3.4 in my opinion.
Did a rebuild, needed a small fix for building with openvdb in USD, but beyond that building was all good, when running from blender all is still good, however i can't seem to import from the standalone python anymore, and i have to say, i'm not entirely sure why... i don't see it not finding a dll its looking for when spying with procmon, and when i manually force the usd lib into the python process all is well, so i'm not entirely sure where to start trouble shooting this....
Let me know when something is available (preferably on windows) we can test the hydra add-on against.
unclear why it's needed, but adding
to usercustomize.py seems to sidestep the issue
I'll contact you by email probably tomorrow with a download link for my current 3.4 lib folder, you can use that against the
tmp_libupdate_34branch asmasteris not ready for this set of libs.This issue was referenced by
1e1e9014cfAdded subscriber: @mont29
Regarding USD, there is problem with recent Linux distro: starting from glibc 2.34, some deprecated hooks have been removed, which are still used by USD 22.03: https://devtalk.blender.org/t/building-blender-on-linux-using-glibc-2-34-raises-linking-errors-from-the-usd-library/24185
The problem has been fixed in USD 22.08: https:*github.com/PixarAnimationStudios/USD/issues/1592, https:*github.com/PixarAnimationStudios/USD/commit/8b19842ad49d6c3eb439318fefea19cde3e87781
Also FYI I tried using USD 22.08, but it will require some changes in Blender code, some API had some incompatible changes it'd seem :(
This issue was referenced by
ded4604d71So
16af35054d(from @brecht patch P3180) solves the issue with USD 22.03 and glibc >= 2.34 for now.What's the plan for these 3.4 libs, I head something coming out of the admin meeting wanting a separate branch for the dynamic stuff? does that mean 3.4 will receive no updates and will stay on 3.3 level libs ?
Added subscriber: @Harley
I'd be okay without FriBiDi and Harbbuzz for a while. I've been been getting increasing resistance to small changes, so not optimistic about larger changes in the near term.
Yeah i'm not overly worried about those, if push comes to shove i'd be ok just cherry-picking those two out of the patch set and landing them in whatever form the 3.4 libs will take, the resistance is clearly not from the platform module here :)
There is no decision from the admins regarding what to do with dynamic libs, that's up to us platform maintainers. But I feel strongly that we should do the dynamic libs upgrade in 3.5 along with other VFX reference platform changes, to avoid potentially doing two disruptive releases for users and add-on developers. Putting some symbols into public namespaces before matching the VFX reference platform is also risky.
Maybe it turns out there are few to no problems. But I find it hard to predict, and there's few other developers who are even aware of the various obscure issues we've dealt with in the past: symbol conflicts with Python modules and drivers, initialization order issues, conflicts between memory allocators, C++ ABI issues, etc.
@LazyDodo, @dr.sybren, do you agree to postpone it to 3.5?
I have no objections to delaying to 3.5, that being said if we're going to do a temp SVN branch with the preliminary dynamic libs and update to VFX 2023 as libs become available i would like to do minimal or no changes for 3.4 just to keep it from becoming a time sink for platform maintainers
I think I mentioned this in the Asset meeting, but my 2 cents is to make the VFX alignment and the dynamic libraries at the same time. So maybe makes sense for 3.5.
Where is the resistence coming from? And what exactly are they resisting?
I also have no objection to postponing library updates to 3.5.
Library changes for Blender 3.4to Library changes for Blender 3.5Added subscriber: @hamza-el-barmaki
I'd probably go with something like
linux_glibc_228_x86_64I mean the glibc version is the most defining characteristic of set of libs, no? you could addCXX11_ABIbut i feel like once we switch it really doesn't matter anymore, as all libs will be cxx11I would strongly suggest that we stop using strings and
os.pathfor paths. The above can be written as:blender_dir = exe.resolve().parent.parent.parent.parent
os.add_dll_directory(blender_dir)
Fair point, though this stuff runs quite early and other functions in this file are using
os.path. So I'm not sure if using higher level modules is safe here, if that runs the risk of changing initialization order in some way that matters.The branch
tmp_libupdate_34was renamed totmp-vfx-platform-2023.For those working on USD/Hydra in branches, you can now try out building against the new libs, either locally or on the buildbot.
You would need to rebase on or merge with the
tmp-vfx-platform-2023branch for this to work.svn switchto https://svn.blender.org/svnroot/bf-blender/branches/vfx-platform-2023/lib/win64_vc15/This is for Windows only, which I guess the interested developers are using anyway.
Added subscriber: @SteffenD
Added subscriber: @zNight
Removed subscriber: @dr.sybren
This is still targeted for Blender 3.5? Any update for linux or Mac?
Yes, it's still targeting 3.5.
@LazyDodo did work to upgrade the libs to vfx platform 2023, and committed updated precompiled Windows libs. That can be used for testing as described in my previous comment.
I can commit the precompiled libs for macOS to the branch as well if you want to test on that platform?
For Linux we are a bit further away, we still need to set up Alma/Rocky Linux 8.
That task description still needs to be updated with the current library versions in the branch.
This issue was referenced by
ebff39d5bbThis issue was referenced by
70375c96d5This issue was referenced by
388bbc3290This issue was referenced by
20883841c7This issue was referenced by
6d27a2ff76This issue was referenced by None@63129
This issue was referenced by None@63130
This issue was referenced by
3494ba3815This issue was referenced by None@63132
This issue was referenced by None@63133
This issue was referenced by
3552f5d83cThis issue was referenced by None@63141
This issue was referenced by None@63142
there is an update about harfbuzz 6.0.0 see https://github.com/harfbuzz/harfbuzz/releases
Awesome. I hadn't noticed that.
Not a problem now though. We really won't release a version of Blender that uses Harfbuzz until 4.6 at the earliest. We are just getting a version in the repository now so that we can do some initial work, do testing, and to aid review. So we might do a version bump next release.
Added subscriber: @xavierh
@mont29 The change is subtle, it adds the
NVPTXtarget something like this would probably work (but I admit, didn't test)Thanks @LazyDodo, afaict it worked (although I do not really have the hardware/software setup here to actually test it ':) ).
Added subscriber: @howardt
I think we need to build the GMP library with --disable-assembly for the MacOS ARM architecture. See https://developer.blender.org/T103423
Added subscriber: @Caco-Oportot
This issue was referenced by
589cbbf0e3@howardt done.
Closing now that 3.5 was released.