Blender Cycles "Launch failed in CUDA queue copy from device (integrator_intersect_shadow)" #110657

New Issue

gareth-hirst · 2023-07-31T17:28:15+02:00

gareth-hirst commented

2023-07-31 17:28:15 +02:00

System Information
Operating system: Windows 10 - 22H2
Graphics card: 3090ti 31.0.15.3667

Blender Version
Broken: (3.5, 3.5.1, 3.6.1)
Worked: None

Blender Cycles "Launch failed in CUDA queue copy from device (integrator_intersect_shadow)"

Exact steps for others to reproduce the error
I have a volumetric setup in the scene. To show up as haze on the horizon. Error only occurs when Volumetric is active.

Ensure the WorldSphere is enabled for render and just render animation for 1279.
It will do 1 - 10 frames before throws the error attached.

File is too big for the attacher, please get blend file from google drive link.

https://drive.google.com/file/d/15WXfA5MA_zIuBbMDTG4QrukyM-g69LNi/view?usp=drive_link

Simplified: https://drive.google.com/file/d/1OUnxpgy-9X2YV1EMS-YvjMth4CdaXegz/view?usp=drive_link

**System Information** Operating system: Windows 10 - 22H2 Graphics card: 3090ti 31.0.15.3667 **Blender Version** Broken: (3.5, 3.5.1, 3.6.1) Worked: None Blender Cycles "Launch failed in CUDA queue copy from device (integrator_intersect_shadow)" **Exact steps for others to reproduce the error** I have a volumetric setup in the scene. To show up as haze on the horizon. Error only occurs when Volumetric is active. Ensure the WorldSphere is enabled for render and just render animation for 1279. It will do 1 - 10 frames before throws the error attached. File is too big for the attacher, please get blend file from google drive link. https://drive.google.com/file/d/15WXfA5MA_zIuBbMDTG4QrukyM-g69LNi/view?usp=drive_link Simplified: https://drive.google.com/file/d/1OUnxpgy-9X2YV1EMS-YvjMth4CdaXegz/view?usp=drive_link

image.png

50 KiB

gareth-hirst added the

labels 2023-07-31 17:28:16 +02:00

Pratik Borhade commented

2023-08-01 04:46:02 +02:00

Hi, drive link asks for the access. Could you make it public?

Pratik Borhade added

Status

Needs Information from User

and removed

Status

Needs Triage

labels 2023-08-01 04:46:12 +02:00

gareth-hirst commented

2023-08-01 05:00:10 +02:00

Done, sorry. Thought i had done this previous.

Pratik Borhade commented

2023-08-01 05:08:35 +02:00

Thanks. File is quite large though (~1.8 GB) :/
Could you simplify it further? (remove objects, shaders, image textures, etc.)
Investigation will be difficult in large files

Thanks. File is quite large though (~1.8 GB) :/ Could you simplify it further? (remove objects, shaders, image textures, etc.) Investigation will be difficult in large files

gareth-hirst commented

2023-08-01 05:17:29 +02:00

Ill have to try later,
The Haze Settings are part of an addon. Apperently this has been an issue before.
True Sky support team has said:
Blender just need to address the error message which is that their ray calculations for volumes in the world is broken again. The sphere just allows the volume to be seen correctly in the render when it's not there the volume just clamps to blow the 0 on z axis and looks like a flat line

Ill have to try later, The Haze Settings are part of an addon. Apperently this has been an issue before. True Sky support team has said: Blender just need to address the error message which is that their ray calculations for volumes in the world is broken again. The sphere just allows the volume to be seen correctly in the render when it's not there the volume just clamps to blow the 0 on z axis and looks like a flat line

gareth-hirst commented

2023-08-02 07:48:13 +02:00

@PratikPB2123 I have tried to simplify the scene more. Got it down to 800mb.

https://drive.google.com/file/d/1OUnxpgy-9X2YV1EMS-YvjMth4CdaXegz/view?usp=drive_link

@PratikPB2123 I have tried to simplify the scene more. Got it down to 800mb. https://drive.google.com/file/d/1OUnxpgy-9X2YV1EMS-YvjMth4CdaXegz/view?usp=drive_link

gareth-hirst commented

2023-08-09 05:49:05 +02:00

Hello, Is there any update on this?

Pratik Borhade commented

2023-08-09 05:50:40 +02:00

Thanks, I missed the earlier notification 😅
Will check

Thanks, I missed the earlier notification 😅 Will check

Pratik Borhade added

Status

Needs Triage

and removed

Status

Needs Information from User

labels 2023-08-09 05:50:49 +02:00

Pratik Borhade commented

2023-08-14 12:15:05 +02:00

I rendered first few frames from 1359 but got no error so far.
Does it happen between particular frame range in simplified file?

I rendered first few frames from 1359 but got no error so far. Does it happen between particular frame range in simplified file?

DorianB commented

2023-08-17 00:05:16 +02:00

I had the exact same issue, cycles seems to dismiss Rendered view and throw this error message "Launch exceeded timeout in CUDA queue copy_from_device (integrator_intersect_shadow)". (Specs: WIN11, RTX4070 Gameready driver 536.99)

@gareth-hirst A driver reinstall fixed the issue for me (it seems).
https://blender.stackexchange.com/questions/299204/how-to-fix-launch-exceeded-timeout-in-cuda-queue-copy-from-device-integrator-i/299205#299205

I had the exact same issue, cycles seems to dismiss Rendered view and throw this error message `"Launch exceeded timeout in CUDA queue copy_from_device (integrator_intersect_shadow)`". (Specs: WIN11, RTX4070 Gameready driver 536.99) @gareth-hirst A driver reinstall fixed the issue for me (it seems). https://blender.stackexchange.com/questions/299204/how-to-fix-launch-exceeded-timeout-in-cuda-queue-copy-from-device-integrator-i/299205#299205

gareth-hirst commented

2023-08-17 10:51:13 +02:00

@PratikPB2123 For me it is still happening, it happens randomly after a few frames. @dodododorian, i have updated to the latest 3090ti driver -- 536.67

gareth-hirst commented

2023-08-17 10:57:42 +02:00

There was a new GPU update on 8/8. I have updated and will test again.

DorianB commented

2023-08-17 11:32:54 +02:00

@gareth-hirst nevermind i still have the issue
can you check in your windows event viewer if you have similar type of GPU warning/error message? Maybe this error message is linked with blender CUDA compilation, i did a full repair of my system and clean driver install, it still persists, it only happens with blender, i start wondering if it's a blender cycles issue.

.

@gareth-hirst nevermind i still have the issue can you check in your windows event viewer if you have similar type of GPU warning/error message? Maybe this error message is linked with blender CUDA compilation, i did a full repair of my system and clean driver install, it still persists, it only happens with blender, i start wondering if it's a blender cycles issue. ![image](/attachments/070484f8-3973-4fcc-b5ac-6f088e1e4f66).

image.png

85 KiB

gareth-hirst commented

2023-08-17 11:56:44 +02:00

Yes, getting the same errors

Yes, getting the same errors ![image](/attachments/e1cd7547-db73-496e-9bc7-4aa7e8cce020)

image.png

60 KiB

gareth-hirst commented

2023-08-17 11:58:29 +02:00

I believe its a problem inside cycles. The Addon Support advised that it is an issue:
Blender just need to address the error message which is that their ray calculations for volumes in the world is broken again. The sphere just allows the volume to be seen correctly in the render when it's not there the volume just clamps to blow the 0 on z axis and looks like a flat line.
I have also tried on 3.6.2 beta use of blender - issue persists.

I believe its a problem inside cycles. The Addon Support advised that it is an issue: Blender just need to address the error message which is that their ray calculations for volumes in the world is broken again. The sphere just allows the volume to be seen correctly in the render when it's not there the volume just clamps to blow the 0 on z axis and looks like a flat line. I have also tried on 3.6.2 beta use of blender - issue persists.

Pratik Borhade commented

2023-08-17 12:14:24 +02:00

Thanks for the info. nvlddmkm is part of NVIDIA driver.
Previously similar error occurred due to overclocking GPU. Does this happen at stock speed or default GPU settings.
Another workaround you could try is to increase tdr delay.

Thanks for the info. nvlddmkm is part of NVIDIA driver. Previously similar error occurred due to overclocking GPU. Does this happen at stock speed or default GPU settings. Another workaround you could try is to increase tdr delay.

gareth-hirst commented

2023-08-17 12:15:46 +02:00

I have not clocked my GPU/CPU. It is left default.

👍 1

Pratik Borhade commented

2023-08-17 12:16:45 +02:00

Not sure if this is an issue within Cycles render or system/hardware side problem.

@brecht , can you check?

Not sure if this is an issue within Cycles render or system/hardware side problem. @brecht , can you check?

DorianB commented

2023-08-17 12:17:51 +02:00

@gareth-hirst why are you so sure the issue is linked with volumetric? i don't have any volumetric in my scene, yet i have the same errors as you have. Do you want to test if the error occurs in my (heavy) scene? -Bests

@PratikPB2123
I did added a 'TdrDelay' and 'TdrDdiDelay' of 30seconds, still have the issues! isn't that enough? No overclocking, didn't tweak GPU's.
To be clear, blender cycles is the only GPU heavy app where my GPU froze like this, and give me these error messages (CUDA queue copy from device (integrator_intersect_shadow) and these warnings/errors in windows EventViewer)

@gareth-hirst why are you so sure the issue is linked with volumetric? i don't have any volumetric in my scene, yet i have the same errors as you have. Do you want to test if the error occurs in my (heavy) scene? -Bests @PratikPB2123 I did added a 'TdrDelay' and 'TdrDdiDelay' of 30seconds, still have the issues! isn't that enough? No overclocking, didn't tweak GPU's. To be clear, blender cycles is the only GPU heavy app where my GPU froze like this, and give me these error messages (CUDA queue copy from device (integrator_intersect_shadow) and these warnings/errors in windows EventViewer)

gareth-hirst commented

2023-08-17 12:18:13 +02:00

Is this two issues then? Launch failed in CUDA queue copy from device (integrator_intersect_shadow) and Driver issues?

gareth-hirst commented

2023-08-17 12:22:34 +02:00

@dodododorian I only get the issue with Volumetric on. I have rendered all the frames without issue if I turn off Volumetric on the WorldSphere.

DorianB commented

2023-08-17 12:23:56 +02:00

@dodododorian I only get the issue with Volumetric on. I can render the whole scene with out issue if I turn off Volumetric.

I couldn't replicate your error, on your scene, it's working just fine in rendered view! such a strange issue we are having
The errors messages i'm getting in blender console are similar to @gareth-hirst

Launch failed in CUDA queue copy from device (integrator_intersect_shadow)

Failed to create CUDA context (Launch exceeded timeout)

Refer to the Cycles GPU rendering documentation for possible solutions:
https://docs.blender.org/manual/en/latest/render/cycles/gpu_rendering.html

Invalid value in cuCtxPushCurrent(device->cuContext) (C:\Users\blender\git\blender-v360\blender.git\intern\cycles\device\cuda\util.cpp:13)
Invalid context in cuStreamCreate(&cuda_stream_, CU_STREAM_NON_BLOCKING) (C:\Users\blender\git\blender-v360\blender.git\intern\cycles\device\cuda\queue.cpp:20)
Invalid context in cuCtxPopCurrent(NULL) (C:\Users\blender\git\blender-v360\blender.git\intern\cycles\device\cuda\util.cpp:18)
Invalid value in cuCtxPushCurrent(device->cuContext) (C:\Users\blender\git\blender-v360\blender.git\intern\cycles\device\cuda\util.cpp:13)
Invalid context in cuCtxPopCurrent(NULL) (C:\Users\blender\git\blender-v360\blender.git\intern\cycles\device\cuda\util.cpp:18)
Invalid value in cuCtxPushCurrent(device->cuContext) (C:\Users\blender\git\blender-v360\blender.git\intern\cycles\device\cuda\util.cpp:13)
Invalid context in cuCtxPopCurrent(NULL) (C:\Users\blender\git\blender-v360\blender.git\intern\cycles\device\cuda\util.cpp:18)
Invalid value in cuCtxDestroy_v2(cuContext) (C:\Users\blender\git\blender-v360\blender.git\intern\cycles\device\cuda\device_impl.cpp:126)

Launch failed in CUDA queue copy from device (integrator_intersect_shadow)

Launch exceeded timeout in CUDA queue copy_from_device (integrator_intersect_shadow integrator_queued_shadow_paths_array)

Refer to the Cycles GPU rendering documentation for possible solutions:
https://docs.blender.org/manual/en/latest/render/cycles/gpu_rendering.html

Launch exceeded timeout in cuStreamCreate(&cuda_stream_, CU_STREAM_NON_BLOCKING) (C:\Users\blender\git\blender-v360\blender.git\intern\cycles\device\cuda\queue.cpp:20)
Launch exceeded timeout in CUDA queue copy_from_device ()
Launch exceeded timeout in CUDA queue copy_to_device ()

Launch exceeded timeout in cuMemFree_v2((CUdeviceptr)device_pointer) (C:\Users\blender\git\blender-v360\blender.git\intern\cycles\device\cuda\device_impl.cpp:521)
Launch exceeded timeout in cuMemFree_v2((CUdeviceptr)device_pointer) (C:\Users\blender\git\blender-v360\blender.git\intern\cycles\device\cuda\device_impl.cpp:521)
... repeating..

> @dodododorian I only get the issue with Volumetric on. I can render the whole scene with out issue if I turn off Volumetric. I couldn't replicate your error, on your scene, it's working just fine in rendered view! such a strange issue we are having The errors messages i'm getting in blender console are similar to @gareth-hirst `Launch failed in CUDA queue copy from device (integrator_intersect_shadow)` ``` Failed to create CUDA context (Launch exceeded timeout) Refer to the Cycles GPU rendering documentation for possible solutions: https://docs.blender.org/manual/en/latest/render/cycles/gpu_rendering.html Invalid value in cuCtxPushCurrent(device->cuContext) (C:\Users\blender\git\blender-v360\blender.git\intern\cycles\device\cuda\util.cpp:13) Invalid context in cuStreamCreate(&cuda_stream_, CU_STREAM_NON_BLOCKING) (C:\Users\blender\git\blender-v360\blender.git\intern\cycles\device\cuda\queue.cpp:20) Invalid context in cuCtxPopCurrent(NULL) (C:\Users\blender\git\blender-v360\blender.git\intern\cycles\device\cuda\util.cpp:18) Invalid value in cuCtxPushCurrent(device->cuContext) (C:\Users\blender\git\blender-v360\blender.git\intern\cycles\device\cuda\util.cpp:13) Invalid context in cuCtxPopCurrent(NULL) (C:\Users\blender\git\blender-v360\blender.git\intern\cycles\device\cuda\util.cpp:18) Invalid value in cuCtxPushCurrent(device->cuContext) (C:\Users\blender\git\blender-v360\blender.git\intern\cycles\device\cuda\util.cpp:13) Invalid context in cuCtxPopCurrent(NULL) (C:\Users\blender\git\blender-v360\blender.git\intern\cycles\device\cuda\util.cpp:18) Invalid value in cuCtxDestroy_v2(cuContext) (C:\Users\blender\git\blender-v360\blender.git\intern\cycles\device\cuda\device_impl.cpp:126) ``` `Launch failed in CUDA queue copy from device (integrator_intersect_shadow)` ``` Launch exceeded timeout in CUDA queue copy_from_device (integrator_intersect_shadow integrator_queued_shadow_paths_array) Refer to the Cycles GPU rendering documentation for possible solutions: https://docs.blender.org/manual/en/latest/render/cycles/gpu_rendering.html Launch exceeded timeout in cuStreamCreate(&cuda_stream_, CU_STREAM_NON_BLOCKING) (C:\Users\blender\git\blender-v360\blender.git\intern\cycles\device\cuda\queue.cpp:20) Launch exceeded timeout in CUDA queue copy_from_device () Launch exceeded timeout in CUDA queue copy_to_device () Launch exceeded timeout in cuMemFree_v2((CUdeviceptr)device_pointer) (C:\Users\blender\git\blender-v360\blender.git\intern\cycles\device\cuda\device_impl.cpp:521) Launch exceeded timeout in cuMemFree_v2((CUdeviceptr)device_pointer) (C:\Users\blender\git\blender-v360\blender.git\intern\cycles\device\cuda\device_impl.cpp:521) ... repeating.. ```

DorianB commented

2023-08-23 01:37:51 +02:00

Any on what could be the source on these errors? (asking for all the potential internet visitors that are typing this error message in their web search engine)

gareth-hirst commented

2023-08-29 01:50:24 +02:00

@PratikPB2123 Any update on this? It is now happening with my other projects. Project that don't have volumes involved. Just Landscape projects.

Pratik Borhade commented

2023-08-29 05:37:00 +02:00

Unfortunately I'm not sure about the possible culprit here. Neither I'm able to reproduce this locally.
Maybe Cycles developers have an idea. I'll ask around for help.

Unfortunately I'm not sure about the possible culprit here. Neither I'm able to reproduce this locally. Maybe Cycles developers have an idea. I'll ask around for help.

Maarten-Nauta commented

2023-09-14 03:17:34 +02:00

Hi getting the same issue. Huge landscape scene for a commercial archviz project and all of a sudden this error strikes. Recovery files also getting the same error. Completely unable to render out the whole project now. And at a loss of what I can do.

On a recovery file that doesn't have the textures, but it does seem to render, the GPU usage seems totally messed up as its rendering!

Hi getting the same issue. Huge landscape scene for a commercial archviz project and all of a sudden this error strikes. Recovery files also getting the same error. Completely unable to render out the whole project now. And at a loss of what I can do. ![image](/attachments/f839017e-55b8-4afd-9b92-67eea86531b6) On a recovery file that doesn't have the textures, but it does seem to render, the GPU usage seems totally messed up as its rendering! ![image](/attachments/8e261d42-a023-4b55-9299-e199f0490d92)

image.png

1.9 MiB

image.png

68 KiB

Maarten-Nauta commented

2023-09-14 04:06:50 +02:00

this is the log with the error.

HillsBuildLAPTOPPORT_AUTUMN_SAFE_-CloseGroundLeft_0140.1.0.11315801.txt

4.5 MiB

Dorian commented

2023-10-06 12:18:47 +02:00

Still getting the issue, what's the meaning of the error message we are having? I bet it's related to a bad nvidia driver perhaps

Alex-202 commented

2023-10-15 21:00:00 +02:00

I have the same problem.
I can't work for 5 days... (((
I have not found a solution to this problem. I will continue searching.

I have the same problem. I can't work for 5 days... ((( I have not found a solution to this problem. I will continue searching.

😕 1

Juan-Bilotta commented

2023-10-22 01:59:11 +02:00

Hi everyone! I have tested three different RTX 3090 GPUs with this error:

"Illegal address in CUDA queue copy_from_device (integrator_shade_surface integrator_sorted_paths_array prefix_sum)"

It occurs randomly in various projects. I've tried everything:

-Drivers (all types and versions)
-Windows 7, 8, and 11, including a clean Windows format
-All Blender versions, from 3.0 to 3.6
-Three different MSI RTX 3090 GPUs, but my RTX 3070 Ti never failed in the same projects.
-I've tested it on all my projects, and it's more likely to fail in the heavier ones. But the 3070ti won´t fail!
-I've replaced my motherboard
-I've upgraded my power supply from 850W to 1050W.

Any insights? What else can we try? Linux??
How can we contact competent Nvidia developers?

Hi everyone! I have tested three different RTX 3090 GPUs with this error: "Illegal address in CUDA queue copy_from_device (integrator_shade_surface integrator_sorted_paths_array prefix_sum)" It occurs randomly in various projects. I've tried everything: -Drivers (all types and versions) -Windows 7, 8, and 11, including a clean Windows format -All Blender versions, from 3.0 to 3.6 -Three different MSI RTX 3090 GPUs, but my RTX 3070 Ti never failed in the same projects. -I've tested it on all my projects, and it's more likely to fail in the heavier ones. But the 3070ti won´t fail! -I've replaced my motherboard -I've upgraded my power supply from 850W to 1050W. Any insights? What else can we try? Linux?? How can we contact competent Nvidia developers?

Germano Cavalcante referenced this issue

2023-10-23 21:13:36 +02:00

CUDA error on RTX 4090 if using Volumetrics in scene #114035

Pratik Borhade referenced this issue

2023-11-13 11:05:43 +01:00

CUDA error on RTX 4090 if using Volumetrics in scene #114035

Alaska commented

2023-12-31 23:32:04 +01:00

I just stumbled across this report and just wanted to take a quick look at it.

Does anyone have a scene that can reliably reproduce the Illegal address in CUDA queue copy_from_device issue? If so, I would like to know this information about it:

Can you share the file with us?
What are the exact steps to reproduce the issue?
Does the issue occur every time, or is it more random?
What Blender version are you using? Is this Blender version the one provided by the Blender foundation? Or is it sourced from somewhere else?
Do you have any third-party addons installed?
- Does turning off your third-party addons fix the issue? (The easiest way to test this is to select from the top of Blender File -> Defaults -> Load Factory Settings then selecting File -> Open Recent -> THE FILE WITH ISSUES. Then enabling your GPU in the preferences and testing the scene)
What GPU hardware do you have? Specifically:
- What GPU do you have?
- What GPU driver version are you using?
- What operating system are you on?
- Are you rendering the scene with CUDA or OptiX?
- Are you using CPU + GPU or just GPU?
- Are you using multiple GPUs or just one?

Just for reference, I tried rendering the simplified scene from @gareth-hirst on a RTX 4090 with driver 537.42 on Windows 11 in Blender 3.6.1 with OptiX and got no issues.

I just stumbled across this report and just wanted to take a quick look at it. Does anyone have a scene that can reliably reproduce the `Illegal address in CUDA queue copy_from_device` issue? If so, I would like to know this information about it: 1. Can you share the file with us? 2. What are the exact steps to reproduce the issue? 3. Does the issue occur every time, or is it more random? 4. What Blender version are you using? Is this Blender version the one provided by the Blender foundation? Or is it sourced from somewhere else? 5. Do you have any third-party addons installed? - Does turning off your third-party addons fix the issue? (The easiest way to test this is to select from the top of Blender `File -> Defaults -> Load Factory Settings` then selecting `File -> Open Recent -> THE FILE WITH ISSUES`. Then enabling your GPU in the preferences and testing the scene) 6. What GPU hardware do you have? Specifically: - What GPU do you have? - What GPU driver version are you using? - What operating system are you on? - Are you rendering the scene with CUDA or OptiX? - Are you using CPU + GPU or just GPU? - Are you using multiple GPUs or just one? Just for reference, I tried rendering the simplified scene from @gareth-hirst on a RTX 4090 with driver 537.42 on Windows 11 in Blender 3.6.1 with OptiX and got no issues.

Pratik Borhade added

Status

Needs Information from User

and removed

Status

Needs Triage

labels 2024-01-01 06:52:34 +01:00

Pratik Borhade commented

2024-03-01 05:40:46 +01:00

Hi, any updates?