CUDA error on RTX 4090 if using Volumetrics in scene #114035

Closed
opened 2023-10-22 17:24:39 +02:00 by ShizzyZzZzZz · 13 comments

System Information
Operating system: Windows 11
Graphics card: NVIDIA GeForce RTX 4090/PCIe/SSE2 NVIDIA Corporation 4.5.0 NVIDIA 537.42

Blender Version
Broken: version: 3.6.4, branch: blender-v3.6-release, commit date: 2023-09-25 13:24, hash: 21bfc5e7fe3f This happens on all versions of blender, on all previous ones after 3.5 version
Worked: (newest version of Blender that worked as expected)

Short description of error
When adding object like sphere or box in to the scene with transparent shader node and volume scatter node most of the times after you doing pre-redner of vieport it gives me an error[first screenshot]

Illegal address in CUDA queue copy_from_device (integrator_shade_surface integrator_sorted_paths_array integrator_compact_shadow_states prefix_sum)

after that rendered section freezes and only pressing "Z" or switching in solid mode gives you ability to navigate, and working in general without any problems, but you can not use render option, because when you trying to start another pre-redner it gives you another error message [second screenshot]

Failed to create CUDA context (Illegal address)

to fix that you need to restart blender completely, reloading project not helping, only complete restart.

This happens only when using volumetrics, and the bigger object the bigger chance to get that error. But when you just start finish render it goes well, i got this error on finish render only 2 times, like from 100~ so it's more like accident, but in viewport it's 95% chance to get this error at start of the processing pre-render of the viewport, sometimes it happens after 40-100 samples, sometimes after 300+ sometimes it is not happens at all, but most of the time it happens around 40-100 samples, unfortunately i can not share any of my scenes, but i can say it's mostly people with complex pbr skin shaders, multyscattering levels and etc, memory usage in acceptable limits so this is not the case, because this also happens when i just rendering single human model on hdri background, where gpu memory usage around 5-8gb, it's happens only on 4090, i tested it on my laptop with 1060 and everything works fine.

Exact steps for others to reproduce the error

  • Using global volume scatter on world or adding object with transparent and volume scatter nodes

UPT 31.10.2023:

I found 1 pattern in which the error occurs more often. When you selecting limited region for render and scrolling in to it with pre-render on in viewport, you can get this error more often when zooming too close to that region[screenshot 3] Also using DOF with Empty sphere as target for focus, just in case.

**System Information** Operating system: Windows 11 Graphics card: NVIDIA GeForce RTX 4090/PCIe/SSE2 NVIDIA Corporation 4.5.0 NVIDIA 537.42 **Blender Version** Broken: version: 3.6.4, branch: blender-v3.6-release, commit date: 2023-09-25 13:24, hash: `21bfc5e7fe3f` This happens on all versions of blender, on all previous ones after 3.5 version Worked: (newest version of Blender that worked as expected) **Short description of error** When adding object like sphere or box in to the scene with transparent shader node and volume scatter node most of the times after you doing pre-redner of vieport it gives me an error[**first screenshot**] ``` Illegal address in CUDA queue copy_from_device (integrator_shade_surface integrator_sorted_paths_array integrator_compact_shadow_states prefix_sum) ``` after that rendered section freezes and only pressing "Z" or switching in solid mode gives you ability to navigate, and working in general without any problems, but you can not use render option, because when you trying to start another pre-redner it gives you another error message [**second screenshot**] ``` Failed to create CUDA context (Illegal address) ``` to fix that you need to restart blender completely, reloading project not helping, only complete restart. This happens only when using volumetrics, and the bigger object the bigger chance to get that error. But when you just start finish render it goes well, i got this error on finish render only 2 times, like from 100~ so it's more like accident, but in viewport it's 95% chance to get this error at start of the processing pre-render of the viewport, sometimes it happens after 40-100 samples, sometimes after 300+ sometimes it is not happens at all, but most of the time it happens around 40-100 samples, unfortunately i can not share any of my scenes, but i can say it's mostly people with complex pbr skin shaders, multyscattering levels and etc, memory usage in acceptable limits so this is not the case, because this also happens when i just rendering single human model on hdri background, where gpu memory usage around 5-8gb, it's happens only on 4090, i tested it on my laptop with 1060 and everything works fine. **Exact steps for others to reproduce the error** - Using global volume scatter on world or adding object with transparent and volume scatter nodes UPT 31.10.2023: I found 1 pattern in which the error occurs more often. When you selecting limited region for render and scrolling in to it with pre-render on in viewport, you can get this error more often when zooming too close to that region[screenshot 3] Also using DOF with Empty sphere as target for focus, just in case.
ShizzyZzZzZz added the
Status
Needs Triage
Priority
Normal
Type
Report
labels 2023-10-22 17:24:40 +02:00

I'm not able to reproduce using a 4090 here. Can you provide a small .blend file that can be used or a better set of steps to follow?

image

I'm not able to reproduce using a 4090 here. Can you provide a small .blend file that can be used or a better set of steps to follow? ![image](/attachments/bbb929cf-64e6-4be3-a502-df3ee5d0c11a)
1.7 MiB
Jesse Yurkovich added
Status
Needs Information from User
and removed
Status
Needs Triage
labels 2023-10-23 04:00:23 +02:00
Author

As i said in post i can not provide any scenes with my models, and as i said this problem happens when scene have complex models and shaders on them with multy-layered scattering for pbr skin shading, also alot of glassy refraction and etc, the more complex scene the higher chance of this errors, in simple scenes everything wortks fine. In that scene(nsfw) from screenshots i used limited box with volumetric and noise texture for volume vibrance as sauna's steam, also there is water shader with refrection, and fluid shader which uses and refraction and volume scatter shading on it. There is 3 human models with pbr skin shaders with additional stuff like polygonal hairs with alot textures with alpha-channels, so scene is complicated. and my light paths setup is kinda expensive for resources, like alot for transparency and also for volume, in older versions i used 1 for volume and 12 for transparency, but in that case volume is very poor on details and etc, so this can be also the case... i don't know)

As i said in post i can not provide any scenes with my models, and as i said this problem happens when scene have complex models and shaders on them with multy-layered scattering for pbr skin shading, also alot of glassy refraction and etc, the more complex scene the higher chance of this errors, in simple scenes everything wortks fine. In that scene(nsfw) from screenshots i used limited box with volumetric and noise texture for volume vibrance as sauna's steam, also there is water shader with refrection, and fluid shader which uses and refraction and volume scatter shading on it. There is 3 human models with pbr skin shaders with additional stuff like polygonal hairs with alot textures with alpha-channels, so scene is complicated. and my light paths setup is kinda expensive for resources, like alot for transparency and also for volume, in older versions i used 1 for volume and 12 for transparency, but in that case volume is very poor on details and etc, so this can be also the case... i don't know)
Jesse Yurkovich added
Status
Needs Triage
and removed
Status
Needs Information from User
labels 2023-10-23 11:21:50 +02:00

Thanks for the report, @ShizzyZzZzZz. Usually, before forwarding it to the developers, the report is first confirmed by a triage team (necessary due to the large flow of daily bug reports).

While we understand that you are unable to share your production files, it would be helpful if you could try to reproduce the issue in a simplified setup and provide an example .blend file that triggers the issue. This will allow us to investigate the issue more effectively.

To better understand the problem, could you please clarify the following:

  • Does the error occur consistently when using any specific settings or configurations with the transparent shader and volume scatter nodes? Are there any specific values or parameters that trigger the error more frequently?

  • Have you tried adjusting the CUDA settings in Blender? For example, changing the tile size or enabling/disabling GPU features like OptiX denoising. If so, did it have any effect on the issue?

Based on the information provided, here are a few possible causes and solutions for the error you're encountering:

  1. GPU memory limitations: The error could be related to the GPU running out of memory when rendering scenes with volumetrics. Try reducing the complexity of the scene, such as lowering the number of scattering levels or simplifying the PBR skin shaders, to see if it helps alleviate the issue.

  2. CUDA compatibility: There might be a compatibility issue between Blender and the CUDA version used by the RTX 4090. Make sure you're using the latest version of Blender and CUDA, and check if there are any known issues or updates related to CUDA support for your graphics card.

  3. Software bug: It's possible that the error is caused by a bug in Blender. Since you mentioned that the issue started occurring after a specific version (3.5), it would be helpful to know if you encountered the problem in any earlier versions. If you have access to older versions of Blender, please try rendering the same scene to see if the error persists.

Also, based on the error messages, this issue may be related, or even a duplicate of one of these others:
#110175: CUDA misaligned address error when rendering the image
#110657: Blender Cycles "Launch failed in CUDA queue copy from device (integrator_intersect_shadow)"
#112843: Error when editing displacement values: Illegal address in CUDA queue copy_from_device
#113212: Illegal adrees in CUDA queue copy_from_device
#113709: Blender Crashing Mid-Render
#112880: Blender lockup/stops responding on Pop!_OS 22.04
#113521: Cycles renderer only loads 1 texture when set to GPU compute
#105623: Mesh to Volume crash on small geometry
#111639: "Illegal address in CUDA queue" error occurs when I use both Principled BSDF and Adaptive Subdivision

Thanks for the report, @ShizzyZzZzZz. Usually, before forwarding it to the developers, the report is first confirmed by a triage team (necessary due to the large flow of daily bug reports). While we understand that you are unable to share your production files, it would be helpful if you could try to reproduce the issue in a simplified setup and provide an example .blend file that triggers the issue. This will allow us to investigate the issue more effectively. To better understand the problem, could you please clarify the following: - Does the error occur consistently when using any specific settings or configurations with the transparent shader and volume scatter nodes? Are there any specific values or parameters that trigger the error more frequently? - Have you tried adjusting the CUDA settings in Blender? For example, changing the tile size or enabling/disabling GPU features like OptiX denoising. If so, did it have any effect on the issue? Based on the information provided, here are a few possible causes and solutions for the error you're encountering: 1. GPU memory limitations: The error could be related to the GPU running out of memory when rendering scenes with volumetrics. Try reducing the complexity of the scene, such as lowering the number of scattering levels or simplifying the PBR skin shaders, to see if it helps alleviate the issue. 2. CUDA compatibility: There might be a compatibility issue between Blender and the CUDA version used by the RTX 4090. Make sure you're using the latest version of Blender and CUDA, and check if there are any known issues or updates related to CUDA support for your graphics card. 3. Software bug: It's possible that the error is caused by a bug in Blender. Since you mentioned that the issue started occurring after a specific version (3.5), it would be helpful to know if you encountered the problem in any earlier versions. If you have access to older versions of Blender, please try rendering the same scene to see if the error persists. Also, based on the error messages, this issue may be related, or even a duplicate of one of these others: #110175: CUDA misaligned address error when rendering the image #110657: Blender Cycles "Launch failed in CUDA queue copy from device (integrator_intersect_shadow)" #112843: Error when editing displacement values: Illegal address in CUDA queue copy_from_device #113212: Illegal adrees in CUDA queue copy_from_device #113709: Blender Crashing Mid-Render #112880: Blender lockup/stops responding on Pop!_OS 22.04 #113521: Cycles renderer only loads 1 texture when set to GPU compute #105623: Mesh to Volume crash on small geometry #111639: "Illegal address in CUDA queue" error occurs when I use both Principled BSDF and Adaptive Subdivision
Germano Cavalcante added
Status
Needs Information from User
and removed
Status
Needs Triage
labels 2023-10-23 21:13:48 +02:00
Author

As ia said it happens ONLY when using volumetrics in scene, it doesn't matter if it's global volumetrics or applied to object for limited volumetrics. Usual setup of my working scene is:

  1. Hdri map, in viewport, i using from 4K to 8K resolution for good performance in viewport and 10k-16k for final render.
  2. Enviroment objects such as rooms or just furniture, nothing special, using PBR materials from Poligon or PBR assets created by me in Substance Painter, not the case.
  3. Main models - People, as described before with comlex material of which i can not share too, because it's paid thing which i don't want to share with anyone. I only can show some math nodes for SSS radius and for Specular IOR.
    In general multySSS shader based on 3 principled shaders[Multiscatter GGX, Random Walk(Fixed Radius)]
  4. This is not VRAM problem, as i showed on screenshot earlier im not reaching limits of VRAM, it just hepeens more often when scene is more complex and have a lot of stuff, but sometimes it happens even with simple scene, like simple pinup with one person, where average VRAM usage = 7-11gb in general.
  5. In vieport i do not using any denoising features, my "System" setup is right now "OptiX" i will try to switch to just "CUDA" and test it today with my scenes. Will make an update if something will change in better or in worst way.

The problem is happens always accidentaly, so i can not track exact reason why it happend or what triggered it, sometimes it happens just at start of prerender in vieport, sometimes it happens when im trying to scale up by mouse wheel the frame from camera view, sometimes it happens when im trying to move something in scene like lights while prerender is active to find better lighting, but as i said i can not see dependency between my actions, so it's super annoying because i don't know what to do to avoid that problem.

I have an assumption that this may be a problem associated with a large number of calculations of shadows and volumetrics in polygonal hairstyles with a large number of textures with alpha channels, since the limit of calculations is 64 for transparency, then perhaps at some depth an error occurs related to volumetrics, which the blender also tries calculate along with the transparency of the textures and the shadow cast by them. I also use "translucent" to have it on outline of the hair, screenshot of hair material is attached.

As ia said it happens ONLY when using volumetrics in scene, it doesn't matter if it's global volumetrics or applied to object for limited volumetrics. Usual setup of my working scene is: 1. Hdri map, in viewport, i using from 4K to 8K resolution for good performance in viewport and 10k-16k for final render. 2. Enviroment objects such as rooms or just furniture, nothing special, using PBR materials from Poligon or PBR assets created by me in Substance Painter, not the case. 3. Main models - People, as described before with comlex material of which i can not share too, because it's paid thing which i don't want to share with anyone. I only can show some math nodes for SSS radius and for Specular IOR. In general multySSS shader based on 3 principled shaders[Multiscatter GGX, Random Walk(Fixed Radius)] 4. This is not VRAM problem, as i showed on screenshot earlier im not reaching limits of VRAM, it just hepeens more often when scene is more complex and have a lot of stuff, but sometimes it happens even with simple scene, like simple pinup with one person, where average VRAM usage = 7-11gb in general. 5. In vieport i do not using any denoising features, my "System" setup is right now "OptiX" i will try to switch to just "CUDA" and test it today with my scenes. Will make an update if something will change in better or in worst way. The problem is happens always accidentaly, so i can not track exact reason why it happend or what triggered it, sometimes it happens just at start of prerender in vieport, sometimes it happens when im trying to scale up by mouse wheel the frame from camera view, sometimes it happens when im trying to move something in scene like lights while prerender is active to find better lighting, but as i said i can not see dependency between my actions, so it's super annoying because i don't know what to do to avoid that problem. I have an assumption that this may be a problem associated with a large number of calculations of shadows and volumetrics in polygonal hairstyles with a large number of textures with alpha channels, since the limit of calculations is 64 for transparency, then perhaps at some depth an error occurs related to volumetrics, which the blender also tries calculate along with the transparency of the textures and the shadow cast by them. I also use "translucent" to have it on outline of the hair, screenshot of hair material is attached.

Hmm, the steps are still too complex and too time consuming for us to track down :\

And, even though not directly related to volume, I believe this is probably a duplicate of #112843: Error when editing displacement values: Illegal address in CUDA queue copy_from_device

I will leave the report open to try to confirm at another time...

Hmm, the steps are still too complex and too time consuming for us to track down :\ And, even though not directly related to volume, I believe this is probably a duplicate of #112843: Error when editing displacement values: Illegal address in CUDA queue copy_from_device I will leave the report open to try to confirm at another time...
Author

This is not about displace at all, do you read it carefully? IT IS ONLY HAPPENS WHEN VOLUME SCATTER IN THE SCENE, ONLY IN THAT CASE it crashed, it's not about displace, not about anything else. Without volumetric everything works fine, i repeated it million times already.

This is not about displace at all, do you read it carefully? IT IS ONLY HAPPENS WHEN VOLUME SCATTER IN THE SCENE, ONLY IN THAT CASE it crashed, it's not about displace, not about anything else. Without volumetric everything works fine, i repeated it million times already.

"even though not directly related to volume" I was referring to #112843

#112843 is also not directly related to displacement, but in that file the displacement seems to trigger a bug with the same error message.

"even though not directly related to volume" I was referring to #112843 #112843 is also not directly related to displacement, but in that file the displacement seems to trigger a bug with the same error message.

I have the same problem in blender 3.6.

Illegal address in CUDA queue copy_from_device (integrator_shade_surface integrator_sorted_paths_array integrator_compact_shadow_states prefix_sum)

Interestingly, blender 4.0 beta doesn't seem to have this issue. Problematic scene renders there without error.

I have the same problem in blender 3.6. _Illegal address in CUDA queue copy_from_device (integrator_shade_surface integrator_sorted_paths_array integrator_compact_shadow_states prefix_sum)_ Interestingly, blender 4.0 beta doesn't seem to have this issue. Problematic scene renders there without error.
Member

#110657 occurring with volumetrics. Though I'm not able to confirm it locally.
@ShizzyZzZzZz , would be helpful if you can check #110657
And a blend file would help us for further investigation.

#110657 occurring with volumetrics. Though I'm not able to confirm it locally. @ShizzyZzZzZz , would be helpful if you can check #110657 And a blend file would help us for further investigation.
Pratik Borhade added
Status
Needs Information from User
and removed
Status
Needs Triage
labels 2023-11-13 11:05:55 +01:00
Author

I described everyting above(aslo little addition to original post with UPT tag in the end), if any of developers need visual representation/demonstration i can show them my few scenes in discord voice channel if they don't mind NSFW stuff to see(becasue i am NSFW artist yup), I got one where is even no chance to render the scene because error happens everytime and in viewport(randomly) and when im trying to start render.(99% constantly) So maybe they will be able to see patterns in it, and as i said a lot of times already i can not share any of my projects becasue of privacy(no chance of it).

Btw here screenshot of console when error happens, idk if this could help to localize it

I described everyting above(aslo little addition to original post with UPT tag in the end), if any of developers need visual representation/demonstration i can show them my few scenes in discord voice channel if they don't mind NSFW stuff to see(becasue i am NSFW artist yup), I got one where is even no chance to render the scene because error happens everytime and in viewport(randomly) and when im trying to start render.(99% constantly) So maybe they will be able to see patterns in it, and as i said a lot of times already i can not share any of my projects becasue of privacy(no chance of it). Btw here screenshot of console when error happens, idk if this could help to localize it
Member

Sharing the affected file publicly will allow multiple devs to have a look at it for investigation.
Maybe create another sample scene which you can share publicly and where problem is more visible.
Not much we can do without being able to replicate the bug locally (hard to identify the buggy code).

Similar error has been triggered before due to various factors.

Sharing the affected file publicly will allow multiple devs to have a look at it for investigation. Maybe create another sample scene which you can share publicly and where problem is more visible. Not much we can do without being able to replicate the bug locally (hard to identify the buggy code). Similar error has been triggered before due to various factors.
Author

And it will also give other people to get freely my models and use them to gain profits so no, as i said thousands times, i can not. and i will not not, even for big money.
I tried to make different scene with complex elements but it was okay, without errors, because my models was build for years and have too many small things to repeat to get that level of "complexity" to get that error, so i have no time for it to waste a week creating assets just to reproduce error, becuase i can not say even if i will get error or not, so i can not afford to waste so much time without understanding the final result, and when this result will be received, it may happen the next day or it may not happen within a month, it's too random. But as i said i can show it in discord, so if no one wants to spend 30 minutes - 1 hour on this to check it in discord stream and see and try changing different scene settings, then I won’t spend a day on it. I have my work to do.

And it will also give other people to get freely my models and use them to gain profits so no, as i said thousands times, i can not. and i will not not, even for big money. I tried to make different scene with complex elements but it was okay, without errors, because my models was build for years and have too many small things to repeat to get that level of "complexity" to get that error, so i have no time for it to waste a week creating assets just to reproduce error, becuase i can not say even if i will get error or not, so i can not afford to waste so much time without understanding the final result, and when this result will be received, it may happen the next day or it may not happen within a month, it's too random. But as i said i can show it in discord, so if no one wants to spend 30 minutes - 1 hour on this to check it in discord stream and see and try changing different scene settings, then I won’t spend a day on it. I have my work to do.
Member

I understand that you can not share the file publicly and recreating same situation in new file is quite time consuming.
In such situation I would close the report due to insufficient resources for replicating and fixing the bug.
Thanks for offering the discord streaming but it won't help much in identifying the responsible code. For debugging the code, we require a .blend file to reproduce the problem locally or exact steps to create the situation from scratch (which may not be possible in this case since the issue is happening in complex scene).

tagging @lichtwerk in case I've missed something.

I understand that you can not share the file publicly and recreating same situation in new file is quite time consuming. In such situation I would close the report due to insufficient resources for replicating and fixing the bug. Thanks for offering the discord streaming but it won't help much in identifying the responsible code. For debugging the code, we require a .blend file to reproduce the problem locally or exact steps to create the situation from scratch (which may not be possible in this case since the issue is happening in complex scene). tagging @lichtwerk in case I've missed something.
Blender Bot added
Status
Archived
and removed
Status
Needs Information from User
labels 2023-11-14 11:12:17 +01:00
Sign in to join this conversation.
No Label
Interest
Alembic
Interest
Animation & Rigging
Interest
Asset Browser
Interest
Asset Browser Project Overview
Interest
Audio
Interest
Automated Testing
Interest
Blender Asset Bundle
Interest
BlendFile
Interest
Collada
Interest
Compatibility
Interest
Compositing
Interest
Core
Interest
Cycles
Interest
Dependency Graph
Interest
Development Management
Interest
EEVEE
Interest
EEVEE & Viewport
Interest
Freestyle
Interest
Geometry Nodes
Interest
Grease Pencil
Interest
ID Management
Interest
Images & Movies
Interest
Import Export
Interest
Line Art
Interest
Masking
Interest
Metal
Interest
Modeling
Interest
Modifiers
Interest
Motion Tracking
Interest
Nodes & Physics
Interest
OpenGL
Interest
Overlay
Interest
Overrides
Interest
Performance
Interest
Physics
Interest
Pipeline, Assets & IO
Interest
Platforms, Builds & Tests
Interest
Python API
Interest
Render & Cycles
Interest
Render Pipeline
Interest
Sculpt, Paint & Texture
Interest
Text Editor
Interest
Translations
Interest
Triaging
Interest
Undo
Interest
USD
Interest
User Interface
Interest
UV Editing
Interest
VFX & Video
Interest
Video Sequencer
Interest
Virtual Reality
Interest
Vulkan
Interest
Wayland
Interest
Workbench
Interest: X11
Legacy
Blender 2.8 Project
Legacy
Milestone 1: Basic, Local Asset Browser
Legacy
OpenGL Error
Meta
Good First Issue
Meta
Papercut
Meta
Retrospective
Meta
Security
Module
Animation & Rigging
Module
Core
Module
Development Management
Module
EEVEE & Viewport
Module
Grease Pencil
Module
Modeling
Module
Nodes & Physics
Module
Pipeline, Assets & IO
Module
Platforms, Builds & Tests
Module
Python API
Module
Render & Cycles
Module
Sculpt, Paint & Texture
Module
Triaging
Module
User Interface
Module
VFX & Video
Platform
FreeBSD
Platform
Linux
Platform
macOS
Platform
Windows
Priority
High
Priority
Low
Priority
Normal
Priority
Unbreak Now!
Status
Archived
Status
Confirmed
Status
Duplicate
Status
Needs Info from Developers
Status
Needs Information from User
Status
Needs Triage
Status
Resolved
Type
Bug
Type
Design
Type
Known Issue
Type
Patch
Type
Report
Type
To Do
No Milestone
No project
No Assignees
5 Participants
Notifications
Due Date
The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: blender/blender#114035
No description provided.