Cycles oneAPI device #96840
Labels
No Label
Interest
Alembic
Interest
Animation & Rigging
Interest
Asset Browser
Interest
Asset Browser Project Overview
Interest
Audio
Interest
Automated Testing
Interest
Blender Asset Bundle
Interest
BlendFile
Interest
Collada
Interest
Compatibility
Interest
Compositing
Interest
Core
Interest
Cycles
Interest
Dependency Graph
Interest
Development Management
Interest
EEVEE
Interest
EEVEE & Viewport
Interest
Freestyle
Interest
Geometry Nodes
Interest
Grease Pencil
Interest
ID Management
Interest
Images & Movies
Interest
Import Export
Interest
Line Art
Interest
Masking
Interest
Metal
Interest
Modeling
Interest
Modifiers
Interest
Motion Tracking
Interest
Nodes & Physics
Interest
OpenGL
Interest
Overlay
Interest
Overrides
Interest
Performance
Interest
Physics
Interest
Pipeline, Assets & IO
Interest
Platforms, Builds & Tests
Interest
Python API
Interest
Render & Cycles
Interest
Render Pipeline
Interest
Sculpt, Paint & Texture
Interest
Text Editor
Interest
Translations
Interest
Triaging
Interest
Undo
Interest
USD
Interest
User Interface
Interest
UV Editing
Interest
VFX & Video
Interest
Video Sequencer
Interest
Virtual Reality
Interest
Vulkan
Interest
Wayland
Interest
Workbench
Interest: X11
Legacy
Blender 2.8 Project
Legacy
Milestone 1: Basic, Local Asset Browser
Legacy
OpenGL Error
Meta
Good First Issue
Meta
Papercut
Meta
Retrospective
Meta
Security
Module
Animation & Rigging
Module
Core
Module
Development Management
Module
EEVEE & Viewport
Module
Grease Pencil
Module
Modeling
Module
Nodes & Physics
Module
Pipeline, Assets & IO
Module
Platforms, Builds & Tests
Module
Python API
Module
Render & Cycles
Module
Sculpt, Paint & Texture
Module
Triaging
Module
User Interface
Module
VFX & Video
Platform
FreeBSD
Platform
Linux
Platform
macOS
Platform
Windows
Priority
High
Priority
Low
Priority
Normal
Priority
Unbreak Now!
Status
Archived
Status
Confirmed
Status
Duplicate
Status
Needs Info from Developers
Status
Needs Information from User
Status
Needs Triage
Status
Resolved
Type
Bug
Type
Design
Type
Known Issue
Type
Patch
Type
Report
Type
To Do
No Milestone
No project
No Assignees
10 Participants
Notifications
Due Date
No due date set.
Dependencies
No dependencies set.
Reference: blender/blender#96840
Loading…
Reference in New Issue
No description provided.
Delete Branch "%!s(<nil>)"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Tasks
For release in Blender 3.3:
For later:
Compiler Build Instructions
Additional build steps:
For Windows:
Or download prebuilded binaries from the release page
2) Download latest dGPU “Intel® Graphics Offline Compiler for OpenCL™ Code” from standalone components webpage: and extract it to the path of your choice.
3) Perform Blender configuration “make …” as usual
4) edit these CMake options (with SYCL_COMPILER_DIR=.\llvm\build\install from step 1) and rebuild
step 1 is optional now it's available from https://svn.blender.org/svnroot/bf-blender/trunk/lib/win64_vc15/
For Linux:
1/2/3/4 are optional since these now are available from https://svn.blender.org/svnroot/bf-blender/trunk/lib/linux_centos7_x86_64/
Added subscribers: @Sirgienko, @Stefan_Werner
The suggest patch to add this functionality and with detailed description tracked as D14480
Some preview about how this implementation is working: https://youtu.be/dyBbSJAW-Js
Added subscriber: @Alaska
Changed status from 'Needs Triage' to: 'Confirmed'
Added subscriber: @xavierh
Added subscriber: @brecht
Can you give an overview of which new files would be included in the Blender install and where they are located?
This should be automated as part of our precompiled libraries builder in
build_files/build_environment/CMakeLists.txt.Am I understanding correctly that the "oneAPI Base toolkit" contains the compiler, but the version it includes in Windows is too old, so we temporarily have to copy a newer versions over it? If so, I assume this base toolkit will be updated with the newer version in the future?
I don't think this is something we want to be doing, the whole Blender build should not run in a different environment just for oneAPI.
This I assume is also a temporary limitation? Since we can't ship Blender like this.
Similar comment to the Windows case.
I suppose this comes from running the full Blender build process inside the oneAPI environment, which we should not do.
This files will be new one:
Yes, it is possible, we have already have some code for this, but it is not in the patch yet (we have plan to add this during begin of Bcon2 phase, if it is possible).
Yes, exactly, this workaround is temporal - after few releases of "oneAPI Base toolkit" this won't be needed anymore.
It is possible to change CMake code in order to find oneAPI required files (like compiler) automatically from default environment.
True, we are planning to give Blender ability to build and ship binaries files for oneAPI like it have been done for other dependencies (oneAPI source code is in open-source).
We have plan this to check that all works fine and then suggest the required change during Bcon2 phase - and then Blender won't need any runtime installation (but will need to ship 4 additional
dlls with size ~5MB in total)
Similar to Windows, this is a temporal solution and will be improved later.
True - if Blender will search for oneAPI files without loading of environment, then this problem won't appear.
Updated build steps - no need to have oneAPI environment loaded anymore and problem with oneTBB on Linux also have gone.
Thanks, but we should not be requiring modifying
PATHorLD_LIBRARY_PATHeither. We don't require it for any other Blender features, and that sort of thing has a tendency to cause hard to track issues.It should be a CMake variable that you can pass, which is then used to locate the files, or if necessary added to the
PATHorLD_LIBRARY_PATHfor just the oneAPI kernel compilation commands.Ok, but note we are not going to enable oneAPI in Blender builds until this is resolved.
Added subscriber: @LazyDodo
Trying to follow along here, just to get a feel what it would take to get this to build, my initial feedback on the instructions for Windows:
Can you be more specific to what packages are required ? Installing 30GB worth of stuff, some of which will interfere with tools I use on a daily basis (vtune breaks xperf for instance) isn't great, if we could trim this down to just the bare essentials that are required for building, that be ideal.
I put some preliminary libs (windows only) in SVN, this step shouldn't be needed anymore, but it admit i haven't quite been able to get the oneAPI stuff building end to end, so haven't been able to confirm they are correct.
this one seems easy enough, i grabbed latest (as of now 101.1404) however having a minimum required version documented wouldn't be the worst idea here
The CI env is likely going to be virtualized, why is there a driver requirement?
Not ideal, but seems easy enough.
I'm weary here, what are these 4 dlls and where do they come from?
To clarify my expectations:
No host side dependency on SYCL, but using purely a dynamically loaded level zero driver library. Bundled binary or SPIR-V kernels that are not dlls, but more similar to CUDA or HIP binary kernel files that can be loaded and have kernels invoked without single source compilation mechanisms.
** A way to compile such binary kernels with a simple command like
/path/to/compiler source_file -o binary_file, without external cmake projects, visual studio xml files, special environments, etc.@brecht, can you please clarify, why this task is connected to BF Blender 2.90 project?
Seems it got automatically added when I moved this to Under Development, I'll remove that.
I got your points about the expectations, but can you please be more specific that you mean by "long term" (just to be on the same wave here)? I mean, do you mean that this should be adressed before Blender 3.3 or Blender 3.4 or "until next year" or even "until 2023"?
Well, VTune not needed for compilation, yes. You have a good point here - I will check, which exactly components is trully required and I will update builds steps accordinly.
The lowerest support version for compilation and execution will be 101.1661 if I am not mistaken about version number - it is not yet released, but will be available publicly on this week.
Well, it is actually needed for execution only, but, well, if the CI won't do an execution, then you don't need a driver on Windows at all. You will still need to get few packages related to GPU compiler and distributed along with driver distribution on Linux, but still there are no need to have really working driver and HW there for compilation either.
Well, you will need "sycl.dll", "ze_loader.dll", "pi_opencl.dll", "pi_level_zero.dll" shared libraries in order to able to load (and run) oneAPI rendering in Blender (overwise, a initialization code of oneAPI backend will found that these libraries are not available, and then will just safely disable oneAPI implementation). Right now this libraries is distributed with Intel® oneAPI Base toolkit or with Intel® oneAPI DPC++/C++ Compiler Runtime for Windows and it is also possible to get them by building oneAPI from source (I guess it will be easiest way for Blender, because in this case Blender can just distribute them like other 3rdparty Blender dependencies without any need to install something from end-user size).
I will try to make it happens and modify paths only for part of the build system, where oneAPI kernel compilation happens (because it is needed only there).
Also, it seems entire idea of "oneAPI environment" don't playing very well, so it possible, that oneAPI itself will address this issue in future - and then we will able to remove all related to this CMake code.
On Windows, ze_loader.dll comes with graphics drivers and is installed in System32, no need to distribute it or ask the user to install anything (beyond graphics drivers of course).
sycl.dll, pi_opencl.dll and pi_level_zero.dll can also be grabbed from "Intel DPC++/C++ Compiler for Windows " installation as redistributables (they're listed in licensing\credist.txt), we should also be able to recompile these from sources.
I don't have specific expectations about timelines here, the sooner the better but if it takes until next year so be it. After the initial version is in we can discuss in more detail about this, since I don't know exactly how deep some limitations are, or how exactly it will work for OSL or potential hardware-raytracing and how that will all fit together.
Added subscriber: @Sergey
Is it the same on Linux, or do we need to do something special in this platform?
According to the
credist.txtthese libraries are covered with "Intel End User License Agreement for Developer Tools" which does not seem to be compatible with GPL. If that's the case then we can not put those libraries to the Blender release.on linux, libze_loader.so comes from level-zero package, we advertise it in our gfx drivers installation guides but I don't know if it tends to be there in practice: https://dgpu-docs.intel.com/installation-guides/index.html
would using binaries you'd compile from https://github.com/intel/llvm be acceptable? compilation itself is quick, I've tried 2022-WW13 branch and it "works on my machine". If it's the solution you want to go with we can refine it to make it more reliable.
that generates these in build\bin
while that may generate the dll's, they are intrinsically linked to the dpcpp version they came out of, if they in the future add a parameter somewhere or change a type all hell will break lose if people download a new compiler from the intel website and use our 3 libs run against (also
sycl.dllneeds two other dllssvml_dispmd.dllandlibmmd.dll, so it's 5 really) the only way I could see this work is@xavierh, How reliable it is from a point of view of a newer DPC++/sycl for compilation? Those libraries seems to be quite low-level, easy to run out of sync from the compiler.
If we are to compile parts of the compiler, it might be better to compile the entire compiler, avoiding incompatibility surprises. But that's quite an escalation of scale of the project.
It might be easier and better for OpenSource in general if the redistreibutable libraries from Intel package will retain the code's license (which is Apache 2, I believe, in this case).
we should be able to remove need for svml_dispmd.dll and libmmd.dll by using their static versions.
The reliability is a question to me, as I said it's working on my machine, but yes there may be compatibility issues, that's something I need to get confirmations on.
Building the entire compiler is also an option but it'd be massive to ship it bundled, and if not shipping it, it will not solve potential compatibility issues with drivers runtimes.
I'll ask about credist.txt license change, I agree it'd be better.
Compatibility with drivers should be as fine as with proprietary version but from: https://github.com/intel/llvm/blob/sycl/README.md
Intel(R) oneAPI DPC++ Compiler is considered a downstream project and has a different license.
Is it really unacceptable in your view to have users download its runtime separately, if the other option right now is to use the less tested open source version - (but that also solves the Centos 7 compatibility issue until addressed downstream) ?
In both cases proprietary ocloc can be used: https:*intel.github.io/llvm-docs/GetStartedGuide.html#gpu + https:*www.intel.com/content/www/us/en/developer/articles/tool/oneapi-standalone-components.html
Downloading runtime libraries is against Blender's product vision. No runtime libraries/SDK installation should be needed in order to use Blender.
Drivers are a bit special in this regard, but those are typically pre-installed by an OS.
Is it just some special download, or we need to compile toolchain ourselves?
Noted!
The open source version is as I've documented a few comments above. It's https://github.com/intel/llvm and getting the DLLs is a short compilation (not the whole project needs to be built). Let us find the "best" commit to use.
I believe from licensing point of view dynamic or static linking is not that relevant. I do not see how static linking can change license of a library. It just makes it harder to see that library is used, but its (partial) content is still in the (derived) product.
The Blender code is GPL2+, and some components (like Cycles) are Apache 2, which renders the final Blender package to be GPL3+. We can not change this. Everything that is being distributed with Blender install must be compatible with GPL3+.
There is added complexity of this approach for developers. Ignoring that, I am not even sure this fully solves the packaging issue due to
svml_dispmdandlibmmd.if a a proprietary compiler is to be used for a device, it should not require putting any proprietary component into the package. Example of this is CUDA integration: a specific compiler needs to be installed on the builder machine, but no extra libraries are to be packaged with the software and everything comes from the driver.
If there is a way to use open-source compiler and libraries we should use those. Ideally, we would not need to re-compile the entire toolchain from scratch and the pre-compiled toolchain from oneAPI will be compatible with open source.
I think it is up to Intel to make it possible to use Intel toolkits usable by open source projects. Those hybrid solutions are always hard to implement in practice and verify from licensing perspective.
Moving forward please contact foundation@blender.org to further discuss licensing topics.
I'm looking into ways to avoid using svml_dispmd and libmmd at all since static linking is for sure not the way, I've just noticed they're covered by the same EULA as the other redistributables, sorry for proposing this.
pi_opencl.dll/libpi_opencl.so aren't needed anymore since 15c146091def58c6f1bd156611cda3efff982687
I've updated the overall build steps in the main task description.
Here is a more detailed script how to get the environment ready for compilation.
For JIT only intel/llvm compiler is needed. AoT needs our graphics compiler that can be downloaded for windows / built for Linux.
On Windows:
On Linux:
And on top of this for AoT support since current compile times are long:
On Windows:
On Linux:
current concerns:
sycl-nighlty/20220208linux usessycl-nightly/20220501pick a single version please all platforms should be on the same version, we don't want different bugs on different platforms.Note: dg2 not supported yet in 101.1960.then what's the point of it? Last time i tried to build on windows, i had to use an older arch to test, and i killed it after about 6 or 8 hours.. not doing that againthe rest looks easy enough to script.
Scripted sycl (and it's deps) for both windows and linux should be available in the deps builder of
cycles_oneapistill on my todo list:
all the buildscripts should be complete, they should build with no issues on both windows and centos (centos7 disclaimer, the stock flex is outdated, needs to be replaced with a build from source 2.6.4 or you'll have build errors), the blender windows build with dg2 enabled I made @xavierh was able to confirm as working, on linux all the puzzle pieces are in place but due to my centos container running out of ram i could not complete a blender build there.
What is not done yet is the harvesting, i'm not entirely sure how much of this we want to add to SVN, on windows you can get dpcpp down to < 200M so that shouldn't be an issue. On linux however... my install folders..
Given the place this branch is now at: @brecht / @Sergey feel free to step in and take over, I've done all I can.
I think you double counted igc in dpcpp. On my system it's closer to 300M.
libigc is indeed very large... gzip compression improves the situation a little bit but we can make it more reasonable by stripping debug symbols:
994M libigc.so.1.0.1
327M libigc.so.1.0.1.gz
85M libigc.so.1.0.1_stripped-debug-symbols
soo.. some good news, bad news.
In the good news column: on my ubuntu based WSL I managed to generate a kernel! no issues!
The bad news column is... more interesting..
Got some more ram for the centos container, looks like the problem was NOT memory, I hadn't looked too closely and just assumed it was ram since it was a severe ram constrained environment, turns out ocloc just sits on ~2.1GB for 35 minutes or so before crashing. crashlog in P2977 , actual peek with gdb in P2978
Then I tried building (dpcpp+igc+ocloc+kernel) on my ubuntu 20.04.1 based WSL setup, no issues there, then figured "maybe it's the container?" so I reinstalled centos7 as a full VM with plenty of ram... [half a day of compiling stuff later].... same issue as the container!! (crash after 35mins, 2.1GB tops)
Next idea: is it the build or the environment? Transfered dpcpp/igc/ocloc from centos7 to ubuntu 20.04.1 (needed some old ncurses lib before it ran, but nothing too crazy) kernel builds just fine (and yes, I've validated the right binaries were used)
so the build of dpcpp/igc/ocloc on centos is fine, but it won't have a happy time running on centos
next idea: maybe it's the memory allocator? lets swap it out! when preloading jemalloc centos 35->15 mins (but still crash) ubuntu 27 mins, no difference.
Next idea: kinda iffy it tops out at ~2GB memory usage, maybe a process limit? wrote a quick app that just allocated tons of ram and initalizes it with some random values, no issues going over 2GB
doing a debug build of igc now, hoping it'll be chattier in what is upsetting it, but open to other stuff you may think is useful trying.
alight, not what i was hoping for, given i have no desire to go debug ocloc but progress nonetheless
I did check as well and ran into the same stacktrace when building from centOS 7 end-to-end. More complete version:
I also confirm compiler binaries built from CentOS 7 and used on ubuntu show no such issue.
This is very strange and I'll check internally but it may take time. To continue progressing I recommend to just disable binaries generation for Linux from your CentOS 7 systems until we find a proper fix. JIT is working well, gets cached, and compilation is faster than on Windows (~10min).
Thanks for confirming my results, always good to know it's not just me :)
while building a reproducer for the graphics compiler team (issue is narrowed down to the intersection kernels), I ran into a great workaround which is to avoid using -ffast-math from CentOS7. From my testing on Windows, the impact of not using ffast-math is minimal (few %).
Can confirm, i can successfully build a kernel on centos now
@xavierh i see you quietly switched to sycl 20220529.
If you want to switch versions, that's fine, but rather than updating the build instructions above, please update(and test before committing) build_files/build_environment/cmake/versions.cmake this is what we'll be using to build the compilers. Be sure to keep an eye on the preferred versions of deps of dpcc they may also need to be changed when you change dpcpp version.
Or just mention that you want a version change and i'll do the work, (but this shouldn't ideally happen too often, this whole thing is kind of a time vampire)
you didn't give me the time to get loud about it :) current deps are good with this version of sycl as well, I'll just update the cmake file
Added subscriber: @manchakkay
This issue was referenced by
a02992f131Added subscriber: @Yuro
Added subscriber: @pawalls
@LazyDodo @Sergey, It make sense to update used SYCL compiler version, in order to get some fixes in JIT caching.
And I think, that this is worth to pick up this version: https://github.com/intel/llvm/releases/tag/sycl-nightly%2F20220812
I have checked this version and it have worked fine to me on Linux and Windows.
And also, some positive news I think - you can still build it on your own, but in the same time you can just grab prebuilt files for both Linux and Windows from this page.
@Sirgienko, Got it. Will see what I can do.
Unfortunately, the AoT on Windows is not addressed yet. Even after migration to a much faster hardware it took more than an hour to compile the AoT kernel (I can't say exact time since SSH session got closed after 1 hour of "inactivity").
On Linux on the same hardware compilation the AoT oneAPI takes 30min which is unideal, but manageable.
@Sirgienko, @xavierh Is there an update of Windows ocloc which brings its performance closer to Linux? :)
@Sergey, for Windows you can try to see if this version https://registrationcenter-download.intel.com/akdlm/irc_nas/18809/ocloc_win_dgpu_101.1743.zip works faster (this version from begin of the July).
But this version seems don't supporting some options, so I will provide better version later.
@Sirgienko Just gave it a whirl. It doesn't support the
--formatflag: https://builder.blender.org/admin/#/builders/30/builds/6404/steps/7/logs/stdio@Sergey, Yes, I know and I am on this topic (and we probably give your proper GPU compiler version under NDA).
Meanwhile, this timings not really expected - they should be twice lower, I would say. Are you using Virtual Machines for this? If so, then do you sure, that it is configured properly - Is Turbo Boost enabled, for exampled?
@Sergey, it maybe make sense to update description in the wiki (https://wiki.blender.org/wiki/Building_Blender/GPU_Binaries) due adding of AoT support (it is mentioned there as not added yet)?
Good point. I gave it a pass to put more recent information.