GPU rendering error: Illegal Address in CUDA queue copy_from_device #98348
Labels
No Label
Interest
Alembic
Interest
Animation & Rigging
Interest
Asset Browser
Interest
Asset Browser Project Overview
Interest
Audio
Interest
Automated Testing
Interest
Blender Asset Bundle
Interest
BlendFile
Interest
Collada
Interest
Compatibility
Interest
Compositing
Interest
Core
Interest
Cycles
Interest
Dependency Graph
Interest
Development Management
Interest
EEVEE
Interest
EEVEE & Viewport
Interest
Freestyle
Interest
Geometry Nodes
Interest
Grease Pencil
Interest
ID Management
Interest
Images & Movies
Interest
Import Export
Interest
Line Art
Interest
Masking
Interest
Metal
Interest
Modeling
Interest
Modifiers
Interest
Motion Tracking
Interest
Nodes & Physics
Interest
OpenGL
Interest
Overlay
Interest
Overrides
Interest
Performance
Interest
Physics
Interest
Pipeline, Assets & IO
Interest
Platforms, Builds & Tests
Interest
Python API
Interest
Render & Cycles
Interest
Render Pipeline
Interest
Sculpt, Paint & Texture
Interest
Text Editor
Interest
Translations
Interest
Triaging
Interest
Undo
Interest
USD
Interest
User Interface
Interest
UV Editing
Interest
VFX & Video
Interest
Video Sequencer
Interest
Virtual Reality
Interest
Vulkan
Interest
Wayland
Interest
Workbench
Interest: X11
Legacy
Blender 2.8 Project
Legacy
Milestone 1: Basic, Local Asset Browser
Legacy
OpenGL Error
Meta
Good First Issue
Meta
Papercut
Meta
Retrospective
Meta
Security
Module
Animation & Rigging
Module
Core
Module
Development Management
Module
EEVEE & Viewport
Module
Grease Pencil
Module
Modeling
Module
Nodes & Physics
Module
Pipeline, Assets & IO
Module
Platforms, Builds & Tests
Module
Python API
Module
Render & Cycles
Module
Sculpt, Paint & Texture
Module
Triaging
Module
User Interface
Module
VFX & Video
Platform
FreeBSD
Platform
Linux
Platform
macOS
Platform
Windows
Priority
High
Priority
Low
Priority
Normal
Priority
Unbreak Now!
Status
Archived
Status
Confirmed
Status
Duplicate
Status
Needs Info from Developers
Status
Needs Information from User
Status
Needs Triage
Status
Resolved
Type
Bug
Type
Design
Type
Known Issue
Type
Patch
Type
Report
Type
To Do
No Milestone
No project
No Assignees
10 Participants
Notifications
Due Date
No due date set.
Dependencies
No dependencies set.
Reference: blender/blender#98348
Loading…
Reference in New Issue
No description provided.
Delete Branch "%!s(<nil>)"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
System Information
Operating system: Windows-10-10.0.19044-SP0 64 Bits
Graphics card: NVIDIA GeForce RTX 3070 Ti/PCIe/SSE2 NVIDIA Corporation 4.5.0 NVIDIA 512.96
Blender Version
Broken: version: 3.1.2, branch: master, commit date: 2022-03-31 17:40, hash:
cc66d1020cWorked: (newest version of Blender that worked as expected)
Short description of error
I've been using blender for a few months now. And recently I got a PC with an Intel i5 12600k and a Gigabyte 3070 Ti (I do a lot of video work and gaming too). Everything works fine with the games(such as getting high fps and all) but when I use blender is when the problems arise. All the modelling and texturing works fine, but when I go into rendered view with my GPU Compute (and CUDA selected in my edit>preference>system tab with my graphics card checked) on, the rendering in the viewport starts but always ends but freezing and with an error message of Illegal Address in CUDA queue copy_from_device (integrator _shade surface integrator_sorted_paths_array prefix sum). This happens for complex projects with different textures, as well as for a simple project with the default cube only.
But when I switch to CPU with no other settings changed, the render completes itself in the viewport. And if I change it back to GPU compute, the screen turns completely grey and another error message appears - Failed to create CUDA context (Illegal address).
This is the same in the case of OptiX as well, when in CPU in the cycles menu, it works completely fine, but as soon as I switch to GPU compute, the screen turns grey with the same message - Failed to create CUDA context (Illegal address).
I've done quite a few things such as looked for the newest drivers for my Graphics card and updating them regularly, and I have opted for the Studio drivers over the Game drivers as well. I'm currently running on blender 3.1.
Exact steps for others to reproduce the error
This happens on the test blender files that we find on the internet as well, so I guess no steps are required to reach it in my system. It just happens as soon as render in the viewport with GPU.
Added subscriber: @vanshaeeee
#104367 was marked as duplicate of this issue
#104364 was marked as duplicate of this issue
#103368 was marked as duplicate of this issue
#101425 was marked as duplicate of this issue
#101159 was marked as duplicate of this issue
#100247 was marked as duplicate of this issue
#97633 was marked as duplicate of this issue
Added subscriber: @OmarEmaraDev
Changed status from 'Needs Triage' to: 'Needs User Info'
Are you overclocking your GPU or adjusting any of its options using things like Afterburner?
Please open Blender's installation directory and double click on the
blender_debug_gpu.cmd. This will start Blender in debug mode and create log files. Try to reproduce the error again. Once it crashes or you close Blender manually the Windows Explorer should open and show you up to two files, a debug log and the system information. Add them to your bug report by clicking on the upload button as shown in the screenshot below or via drag and drop. Please also upload the crash log located inC:\Users\[your username]\AppData\Local\Temp\[project name].crash.txt(or simply type%TEMP%into the path bar of the Windows Explorer).blender_system_info.txt
blender_debug_output.txt
I haven't overclocked the GPU at all or used any other apps on it. This problem has been there since I got the PC built from the seller directly. I have used the GeForce Experience app to update the studio drivers though. I even went ahead and made sure by reinstalling blender and the studio drivers, so they are both in their latest versions currently.
I tried some blender benchmark tests on it too. On the CPU, it takes 3 minutes for the BMW render test to work, but in the GPU its the same error. I did get a pop up notification from my windows quoting "Application blender.exe has been blocked from using graphics hardware". But i started getting this pop up form yesterday only, and never had this popup before, while the GPU error was still there a few weeks back.
I've attached the debugs report, but there isn't any crash report, as the program isn't crashing because of this error. The viewport freezes to a frame of the render, or turn grey (when OptiX is on) and when I switch back to CPU rendering, the program goes back to normal. Ive attached a few screenshots of the error page on the viewport as well well {F13103823}{F13103821}
Changed status from 'Needs User Info' to: 'Needs Triage'
Added subscriber: @mano-wii
I'm not very familiar with Cuda or Optix rendering, but could this hardware change be somehow confusing the Cycles kernels that are compiled for a specific GPU and cached?
If you delete these cache files this would force a recompile... Just a guess
Maybe you can find these cache files in one of these directories:
C:\Users\[your username]\AppData\OptixCacheC:\Users\[your username]\AppData\Roaming\Blender Foundation\Blender\3.3C:\Users\[your username]\AppData\Roaming\Blender Foundation\Blender\3.3\scripts\addons\cyclesChanged status from 'Needs Triage' to: 'Needs User Info'
I did try to find the cache directory with the folder paths that you mentioned above, but those paths don't exist in my C drive of the computer. The path ends at " C:\Users[your username]\AppData\Roaming\Blender Foundation\Blender\3.1" . And since I have the 3.1.2. version of blender installed I opened the 3.1 folder inside Blender Foundation\Blender , which led to the config folder and just has a bunch of text files in them. The screenshots have been posted below.
So I cleared the cache with Disk Cleanup with Windows, but no changes yet. I did run the same GPU Rendering in viewport and render mode a couple of times and got the same error in a few, but different errors as well. I've attached all the screenshots below.
Even with a simple default cube and a plane in the scene with no textures and the world lighting, the errors appear while using the GPU Compute, but not when using the CPU compute.
But I did get a pop up notification in one of the errors in which the notification stated " Application has been blocked form accessing Graphics Hardware - Application blender.exe has been blocked form accessing Graphics Hardware.
Maybe this helps find out what is causing the error?
Can you try the latest daily build? https://builder.blender.org/download/
Many bugs are fixed between the official releases.
I tried it all, but still no luck. I tried installing Blender 3.3.0 (Alpha), Blender 3.2.0 (Beta) , Blender 2.93.10 and even Blender 2.79, but while using cycles and GPU compute, I got the same error. The program did not crash even once in all of the versions, and worked fine and rendered without an issue when I switched to CPU rendering in cycles. I've{F13124658}
Changed status from 'Needs User Info' to: 'Needs Triage'
Just an update after yesterday. So I tried updating my Intel Drivers from scanning from their website, it automatically scanned and updated a few drivers here and there. But after doing so, I opened the BMW render test file in Blender 3.3 and it started rendering (In CUDA that is). But it only rendered up to a certain amount of samples (the BMW is 1225 samples per frame, and the GPU rendered it up to 161, sometimes 400 samples before showing the same illegal CUDA address error) I also started getting the pop up notification of "blender.exe is not authorized to us graphical hardware"
Here are a few images of how much it rendered in cycles before showing the error and the error notification as well.
No luck in OptiX though. Kept getting the error before any of the tiles or samples started rendering.
Added subscriber: @PratikPB2123
Hi, I would still suggest to check if GPU is running at stock speed or not
Reports complaining about similar CUDA errors was fixed when reporter disabled the overclocking.
#91729 (Cycles Illegal address on CUDA/OptiX GPU when overclocking GPU Core Clock)
#97924 (Cannot render with Cycles on any build: Illegal address in CUDA queue copy_from_device (integrator_intersect_closest))
Changed status from 'Needs Triage' to: 'Needs User Info'
Added subscribers: @Peter-Gerace, @ThomasDinges
I have a similar issue. I’m trying to make sense of it, but it seems to be so strange. It appears to happen randomly. In reality, I don’t think it’s random for me at all. Last time it happened to me, I got it on only frame 2 of my 460 frame 4K 60 FPS 4096 sample render. I restarted the render and it rendered over the next 16 hours perfectly fine, making me think it’s extremely unlikely to be any kind of hardware/GPU clock issue. My issue occurs in both the viewport and during actual renders. Both CUDA “illegal” errors require a restart of the program and then go back to working perfectly fine. I would say it happens with equal frequency in viewport and render.
Sorry for the late reply. The monsoons here have been giving a lot of power cuts, so couldn't work on my pc regularly.
I did try installing MSI Afterburner and it showed that my card was not overclocked. The Core clock and Memory Clock values were at 0. Even with this settings on, I was getting very random results in both the viewport and render window.(with the BMW test)
On the render window
Sometimes the frame rendered completely, all 1225 samples (only in CUDA), whereas sometimes the error pops up randomly at different samples (33, 161, 245,etc) and shows the error. This is in the render window. And when this error happens, sometimes the notification of "blender.exe is not authorized to use graphic hardware" pops up, but not on a regular basis. No luck on OptiX as it does not even render a single sample and the error appears almost instantly. The program does not crash though. It just shows the error and doesn't render. I can close the render window completely and still work on the modelling and everything, but when I click on the render button (F12), even in CUDA, the samples don't render at all, after the error has been shown previously and blender has not been restarted. The error appears instantly. So after restart blender and render (in CUDA) the samples start rendering to a random frame, and then stop again, (sometimes rendering the frame fully) at random sample rates. I did try under clocking the GPU, by a lot at different lower core and memory clocks. But the results are too random. Sometimes I get the full frame render while the GPU is underclocked or in normal settings, or sometimes it gets the error at random sample rates. Overclocking has pretty consistent results, the render does not even start and the error happens at the very first sample itself almost every time. But the error appears every time, along with the pop up notification.
But I did have some interesting results in the viewport.
When in rendered view in the viewport, while on normal clock settings of the GPU, the sample count reaches halfway (around 64 samples out of 128) and then freezes. I can move to other solid view mode and wok on the model , but the rendered view does not work in the viewport after the error has appeared once. But when I underclocked the GPU, the view poet render view worked completely file for a few minutes, rendering every sample to the fullest, and also rendering after used my middle mouse button to move around in 3D space. But after moving around like this for a few times, the program then froze, and had to restart it, if I wanted o see the viewport in rendered mode. This was the same case in OptiX as well. The pop up notification of "blender.exe is not authorized to use graphic hardware" did appear a few times when the error appeared.
All of these results are found in Blender 3.3 Alpha. I did try these exact things inn Blender 3.1.2, but the results aren't the same. I did not get any full renders at all, and the error appeared more quickly without reaching a high sample value.
Just an Update, I have been getting errors with he GPU on different software's such as Premiere Pro and wile using Adobe Media Encoder as well. When the GPU accelerated Rendering Option (CUDA) is selected, the export of videos hasn't happened fully even once and always gets an error mid way. But if I export with the CPU, (Open CL) the entire video is rendered from both premiere pro and media encoder. And I keep getting the same pop up notification "premiere.exe has been blocked to use graphic hardware". Maybe this helps figure out what the problem really is.
It sounds like your error might be very different from mine (my error report was merged into yours). I never receive any errors that say applications cannot use the graphics card, and my render crashes only occur in Blender.
It sounds to me like you might have a faulty graphics card if you are having issues in every program. Have you tried to reinstall the drivers after doing a full uninstall using Display Driver Uninstaller? Do you have Hardware Accelerated GPU Scheduling turned on or off? That kind of behavior you're experiencing across multiple programs leads me to believe the issue is somewhere within your own computer and not the fault of Blender itself. This is in contrast to mine where all other programs (and games) are stable 100% of the time and used daily without crashes over many months. The only program I have an issue with is Blender.
Have you tried searching "View Reliability History" in the Windows search bar and opening that tool? It might give you more info regarding GPU driver crashes.
Thanks for the solutions man! It did help for 50% of my problems, but some of it still persists.
Before doing any of the other things that you told me like the DDU and enabling Hardware GPU Scheduling, I had problems with most of the programs like Premiere Pro, Davinci Resolve, Adobe Media Encoder and Blender. I kept getting the pop up notification. But surprisingly enough, I never had problems with he GPU in games. I bought this PC around 2 months back, and have been playing Valorant on it every single day. And I'm averaging around 200-300 FPS inn the highest settings as well. And while seeing the performance tab in Task Manager, it clearly shows that the GPU is in usage most of the time at around 80%. So the problem hasn't been there in games. But I always got black screens and render errors while enabling GPU in After Effects and premiere pro, and crashes in Davinci resolve and the illegal CUDA errors in Blender.
But after enabling Hardware Accelerated GPU Scheduling(which wasn't turned on until you told me) programs such as Davinci resolve work like butter. NO lags and no errors and I can been get 4k videos rendered really quickly. But i still get mixed errors in Premiere pro and Media Encoder. 50% percent of the times, the rendering with CUDA through Premiere and Encoder happen smoothly, with no pop ups or errors. But in the other 50% I get errors, black screen in premiere project view window (which fixes itself without rebooting after I switch to open cl) when turning on CUDA. The error still happens in bender and blender had way more inconsistent results with the image being rendered fully sometimes, but most of the times the error appeared. And most of the times, there is the pop of notification of graphical hardware being blocked for the software in use. But the programs that I use never crash at all. They just have an error when Cuda or the GPU is in use, and work fine when I turn it back to CPU rendering. Such inconsistent results have made it very hard to pin point what the actual problem is.
And after I ran the Display Driver Uninstaller, and it did give me hope at the beginning. While in cycles (Blender 3.2 and 3.3) i could render in the viewport with GPU and all the samples were rendered with no errors at all. I could even change the view in the viewport and all the samples would render. But after doing this for a few times, the error still occurred. (Illegal address in CUDA queue) And the pop up notification of application blocked from using graphics hardware still pop up every time I get an error in blender or when I use software's like Premiere and Encoder, but not with Davinci resolve anymore. The frames sometimes render completely in blender, but it is still a rarety , but no program crashes so far, and work fine when I switch to CPU , even after the error.
Changed status from 'Needs User Info' to: 'Needs Triage'
GPU rendering errorto GPU rendering error: Illegal Address in CUDA queue copy_from_deviceI'm glad some of my advice helped you. I'm not sure if you've solved your problem, but I can share what I did to solve my issue and to get flawless renders that can run for days without a single crash. So basically Blender was causing GPU instability at the default clock speeds. I installed MSI Afterburner and made a separate profile for my Blender renders. For gaming and all other software, I actually applied a nice overclock to 2025 MHz GPU core and 7400 MHz to memory on my 2080 Ti. This runs at 1025 mV. Then I have a sperate profile I enable during my Blender renders. This profile is set to 1675 MHz GPU and 7000 MHz memory at 1040 mV (more voltage combined with lower clocks for more stability). Hopefully this helps others with similar issues.
Added subscriber: @SWOOP1R
Added subscriber: @Mizino
Added subscriber: @drimkat4er
Added subscriber: @H.Diedrichs
This comment was removed by @H.Diedrichs
I'm having a similar illegal address issue but only when using ScatterVolume or AbsorptionVolume.
cycles 3.4.0 c++, cycles as library in another app.
Occurs in PathTraceWorkGPU::enqueue_path_iteration() at
integrator_next_shadow_path_index_ looks fine to me. I checked address to make sure they were 4 byte aligned. I have NVidia driver as of 12/1/22
This comment was removed by @H.Diedrichs
Added subscribers: @EXUS, @iss
I have resolved my issue with volume shaders crashing optix and cuda. We were creating a shader then a graph then calling shader->set_graph(graph) without filling in the graph with nodes. Then we filled in the graph and called tag_update. set_graph sets a bunch of flags to turn on features by examining the nodes in the graph such as if it finds a volume node it enables volumes. Wonder if its a good idea to add those checks in tag_update as well. In any case, filling in the graph first before calling set_graph fixed my issue.
Added subscriber: @nn950815
My cpu dont have overclocking , i tried to set the cycles render devices to none is will work well , but using cuda and optix will cause same result . I try in 3.3.2 with same setting can work well , in 3.4 and 3.4.1 will cause the issue don't know how to fix it .
Added subscriber: @BB12
Hi, similar issue has been fixed recently, see: #105522
Please check whether it's also fixed for you (in 3.5): https://builder.blender.org/download/daily/
Hi, I probably managed to solve the problem by updating the VBIOS version of my graphics card (in my case a 3060 from EVGA) to a newer version. Hopefully this tip will help someone else as well. I did a few tests in the latest experimental version (3.5.0 release candidate) where the error would normally have occurred, but this was no longer the case.
My graphics card was never overclocked or underclocked.
Thanks @EXUS . I'll keep this report open until a majority of subscribers claims this as fixed.
Same issue on blender 3.5 , i try to render in cycles but shows this error this time.
Not found in cuModuleGetFunction(&kernel.function, cuModule, function_name.c_str()) (C:\Users\blender\git\blender-v350\blender.git\intern\cycles\device\cuda\kernel.cpp:26)
@nn950815 , looks different than this report. Could you open a new report with all necessary info?
Any updates on this?
No activity for more than a week. As per the tracker policy we assume the issue is gone and can be closed.
Thanks again for the report. If the problem persists please open a new report with the required information.