Cuda Copy_from_device error in final render #105442

New Issue

niels · 2023-03-04T20:10:23+01:00

niels commented

2023-03-04 20:10:23 +01:00

System Information
Operating system: Windows-10-10.0.19045-SP0 64 Bits
Graphics card: NVIDIA GeForce GTX 1080/PCIe/SSE2 NVIDIA Corporation 4.5.0 NVIDIA 528.49

Blender Version
Broken: version: 3.6.0 Alpha, branch: main, commit date: 2023-03-01 01:34, hash: ea97bb1641b9
Worked: 3.4.1 Stable

Short description of error
I have 5 levels of multires modifier on a female body and when trying to render i get this error "Illegal address in CUDA queue copy_from_device (integrator_intersect_closest)"
Blender build 3.5.0 Beta has the same error while 3.4.1 Stable works fine
i'm on the latest studio drivers from nvidia

Exact steps for others to reproduce the error

Download file attached below
In .blend file, select CUDA from preferences > systems tab
Start rndering an image
https://drive.google.com/file/d/1qtbG4ajGajCJN-NDrQW39oW1_xzyMRdS/view?usp=sharing

**System Information** Operating system: Windows-10-10.0.19045-SP0 64 Bits Graphics card: NVIDIA GeForce GTX 1080/PCIe/SSE2 NVIDIA Corporation 4.5.0 NVIDIA 528.49 **Blender Version** Broken: version: 3.6.0 Alpha, branch: main, commit date: 2023-03-01 01:34, hash: `ea97bb1641b9` Worked: 3.4.1 Stable **Short description of error** I have 5 levels of multires modifier on a female body and when trying to render i get this error "Illegal address in CUDA queue copy_from_device (integrator_intersect_closest)" Blender build 3.5.0 Beta has the same error while 3.4.1 Stable works fine i'm on the latest studio drivers from nvidia **Exact steps for others to reproduce the error** - Download file attached below - In .blend file, select CUDA from `preferences > systems` tab - Start rndering an image https://drive.google.com/file/d/1qtbG4ajGajCJN-NDrQW39oW1_xzyMRdS/view?usp=sharing

niels added the

labels 2023-03-04 20:10:23 +01:00

Pratik Borhade commented

2023-03-06 12:20:12 +01:00

Hi, thanks for the report. Could you share full logs generated with the error?: https://docs.blender.org/manual/en/dev/troubleshooting/crash.html#windows
Issue could be same as reported in: #98348

Hi, thanks for the report. Could you share full logs generated with the error?: https://docs.blender.org/manual/en/dev/troubleshooting/crash.html#windows Issue could be same as reported in: #98348

Pratik Borhade added

Status

Needs Information from User

and removed

Status

Needs Triage

labels 2023-03-06 12:20:22 +01:00

niels commented

2023-03-06 17:16:52 +01:00

this is what i get from the system console:

Illegal address in CUDA queue copy_from_device (integrator_intersect_closest)

Refer to the Cycles GPU rendering documentation for possible solutions:
https://docs.blender.org/manual/en/latest/render/cycles/gpu_rendering.html

Illegal address in cuMemFree_v2((CUdeviceptr)device_pointer) (C:\Users\blender\git\blender-vdev\blender.git\intern\cycles\device\cuda\device_impl.cpp:520)

This line is repeated abouut 50 times

Error: Illegal address in CUDA queue copy_from_device (integrator_intersect_closest)

this is what i get from the system console: Illegal address in CUDA queue copy_from_device (integrator_intersect_closest) Refer to the Cycles GPU rendering documentation for possible solutions: https://docs.blender.org/manual/en/latest/render/cycles/gpu_rendering.html Illegal address in cuMemFree_v2((CUdeviceptr)device_pointer) (C:\Users\blender\git\blender-vdev\blender.git\intern\cycles\device\cuda\device_impl.cpp:520) This line is repeated abouut 50 times Error: Illegal address in CUDA queue copy_from_device (integrator_intersect_closest)

Pratik Borhade commented

2023-03-07 03:30:25 +01:00

Hi, please attach complete log files:
Open Blender's installation directory and double click on the blender_debug_gpu.cmd. This will start Blender in debug mode and create log files. Try to reproduce the error again. Once it crashes or you close Blender manually the Windows Explorer should open and show you up to two files, a debug log and the system information. Add them to your bug report by clicking on the upload button as shown in the screenshot below or via drag and drop. Please also upload the crash log located in C:\Users\[your username]\AppData\Local\Temp\[project name].crash.txt (or simply type %TEMP% into the path bar of the Windows Explorer).

Did you overclock the GPU? Is this started to happen recently? Have you made any changes on system side (bios, drivers, OS updates)?

Hi, please attach complete log files: Open Blender's installation directory and double click on the `blender_debug_gpu.cmd`. This will start Blender in debug mode and create log files. Try to reproduce the error again. Once it crashes or you close Blender manually the Windows Explorer should open and show you up to two files, a debug log and the system information. Add them to your bug report by clicking on the upload button as shown in the screenshot below or via drag and drop. Please also upload the crash log located in `C:\Users\[your username]\AppData\Local\Temp\[project name].crash.txt` (or simply type `%TEMP%` into the path bar of the Windows Explorer).![2019_12_04_upload_icon_developer_blender_org.png](https://archive.blender.org/developer/F8190038/2019_12_04_upload_icon_developer_blender_org.png) - - - Did you overclock the GPU? Is this started to happen recently? Have you made any changes on system side (bios, drivers, OS updates)?

niels commented

2023-03-07 16:30:24 +01:00

Hi,
i did not overclock the gpu or anything, this has been an issuue ever since i updated to the alpha version of 3.5.0, i did not change anything in the system like bios or drivers, 3.4.1 stable works just fine for me.

Hi, i did not overclock the gpu or anything, this has been an issuue ever since i updated to the alpha version of 3.5.0, i did not change anything in the system like bios or drivers, 3.4.1 stable works just fine for me.

blender_debug_output.txt

108 KiB

blender_system_info.txt

24 KiB

Alfredo Rodrigues commented

2023-03-07 17:34:54 +01:00

I'm also having the same problem in Blender 3.5. It's working in Blender 3.4.
I don't know if it helps.

I'm also having the same problem in Blender 3.5. It's working in Blender 3.4. I don't know if it helps.

blender_system_info.txt

23 KiB

blender_debug_output.txt

73 KiB

Pratik Borhade added

Status

Needs Triage

and removed

Status

Needs Information from User

labels 2023-03-08 02:27:42 +01:00

Pratik Borhade referenced this issue

2023-03-10 04:06:44 +01:00

Illegal Address in Cuda #105522

Pratik Borhade commented

2023-03-10 04:38:28 +01:00

Hi, I'm able to confirm the error in 3.5
In prior versions, I caught up with out of memory issue.
Will raise the priority since this looks like a regression from comments.

Hi, I'm able to confirm the error in 3.5 In prior versions, I caught up with out of memory issue. Will raise the priority since this looks like a regression from comments.

Pratik Borhade added

and removed

labels 2023-03-10 04:38:58 +01:00

Pratik Borhade changed title from ~~Cuda Copy_from_device bug~~ to Cuda Copy_from_device error in final render

2023-03-10 04:39:17 +01:00

Brecht Van Lommel commented

2023-03-10 18:54:06 +01:00

I'm unable to get this file to render on 3.3, 3.4 or 3.5 as it runs out of memory in all of them. Maybe I can reproduce it when I simplify the file.

However I did notice this file has Open Shading Language enabled, which is supported for OptiX since 3.5. So you could try disabling that and see if the error still happens.

I'm unable to get this file to render on 3.3, 3.4 or 3.5 as it runs out of memory in all of them. Maybe I can reproduce it when I simplify the file. However I did notice this file has Open Shading Language enabled, which is supported for OptiX since 3.5. So you could try disabling that and see if the error still happens.

niels commented

2023-03-10 19:42:08 +01:00

Disabling the OSL didn't do anything for me since it was turned on but no OSL shaders are active, rendering purely on cpu works for me even in 3.5, if i change to Optix i get "Failed to create CUDA context (Illegal address)",
When i chhange render devices to both GPU and cpu i also get "Failed to create CUDA context (Illegal address)",
i prefer to use CUDA instead of optix since in most my projects CUDA does Out of core rnedering and optix doesnt atleast on my system

Disabling the OSL didn't do anything for me since it was turned on but no OSL shaders are active, rendering purely on cpu works for me even in 3.5, if i change to Optix i get "Failed to create CUDA context (Illegal address)", When i chhange render devices to both GPU and cpu i also get "Failed to create CUDA context (Illegal address)", i prefer to use CUDA instead of optix since in most my projects CUDA does Out of core rnedering and optix doesnt atleast on my system

Pratik Borhade commented

2023-03-11 02:42:53 +01:00

@brecht , try replicating with CUDA.

Brecht Van Lommel added this to the 3.5 milestone 2023-03-14 16:43:44 +01:00

Brecht Van Lommel commented

2023-03-15 19:34:11 +01:00

I can sometimes get it to run out of memory with CUDA in 3.5, but not consistently. I noticed that it uses a few % more memory than 3.4 which may just be pushing it over the limit. But I'm not sure if that's really the problem.

I'll look further to figure out what's really going on. This could be a bug in host memory fallback, in which case 6dcfb6d could have caused it.

There are also recent memory usage optimizations in 3.6 that help significantly (6c03339), and I have not been able to reproduce with the very latest version there.

I can sometimes get it to run out of memory with CUDA in 3.5, but not consistently. I noticed that it uses a few % more memory than 3.4 which may just be pushing it over the limit. But I'm not sure if that's really the problem. I'll look further to figure out what's really going on. This could be a bug in host memory fallback, in which case 6dcfb6d could have caused it. There are also recent memory usage optimizations in 3.6 that help significantly (6c03339), and I have not been able to reproduce with the very latest version there.

niels commented

2023-03-15 23:29:54 +01:00

I don't know if it is usefull info but at the second my GPU runs out of VRAM and needs to go out of core it instantly gives me this error, might have to do with that ?

Brecht Van Lommel referenced this issue from a commit

2023-03-17 21:53:30 +01:00

Fix #105442: Cycles CUDA and HIP host memory fallback not working

Brecht Van Lommel closed this issue

2023-03-17 21:56:00 +01:00

Blender Bot added

and removed

labels 2023-03-17 21:56:02 +01:00

Shady Hamza commented

2023-03-18 06:34:36 +01:00

it still give me that error in 3.6 but not in 3.5

Brecht Van Lommel commented

2023-03-18 08:22:18 +01:00

@Shady.Hamza what's the full name of the 3.6 build (with build hash included) that you tested? Are you testing the .blend file from the description in this bug report, or something else?

Shady Hamza commented

2023-03-18 08:58:07 +01:00

I'm using the latest
blender-3.6.0-alpha+main.3eff28a158ca-windows.amd64-release

I submitted a report #105522 (comment) using different file, it's too large to upload here, and i can't share most of it unfortunately, i was getting many types of errors in both cuda and optix including this one , all screenshots in report, the file wasn't rendering in both 3.5 and 3.6, only in 3.4.1,now 3.5 renders it but not 3.6, let me know if i can provide something that can help, thank you

I'm using the latest blender-3.6.0-alpha+main.3eff28a158ca-windows.amd64-release I submitted a report https://projects.blender.org/blender/blender/issues/105522#issuecomment-903441 using different file, it's too large to upload here, and i can't share most of it unfortunately, i was getting many types of errors in both cuda and optix including this one , all screenshots in report, the file wasn't rendering in both 3.5 and 3.6, only in 3.4.1,now 3.5 renders it but not 3.6, let me know if i can provide something that can help, thank you

Pratik Borhade referenced this issue

2023-03-18 14:07:14 +01:00

CUDA Crash #105845

Brecht Van Lommel commented

2023-03-20 12:56:44 +01:00

Let's continue discussion in #105522 then.

Pratik Borhade referenced this issue

2023-06-07 08:15:44 +02:00

Illegal address in CUDA queue copy_from_device #108202

Sign in to join this conversation.