Cycles/Cuda crash when live rendering an object with an animated array modifier #48856

New Issue

Ralf Hölzemer · 2016-07-14T20:11:50+02:00

Ralf Hölzemer commented

2016-07-14 20:11:50 +02:00

System Information
system-info.txt

Tested on Debian Stretch and at work on multiple Windows 10 machines.

Blender Version
Broken: 39dee8a
Worked: unknown

Short description of error
When a scene with an object with an animated array modifier is live rendered in the viewport, and the current frame is changed, blender crashes on win10 and gives a timeout error on Linux.

The crash on Windows 10 does happen with the builder.blender.org build from Wed, Jul 13, which is labeled as blender-2.77-eb04908-win64.
Unfortunately, I can only debug at home on Linux where I get a timeout instead of a crash with the following log.

Read new prefs: /home/ralf/.config/blender/2.77/config/userpref.blend
Cannot connect to server socket err = No such file or directory
Cannot connect to server request channel
jack server is not running or cannot be started
JackShmReadWritePtr::~JackShmReadWritePtr - Init not done for 4294967295, skipping unlock
JackShmReadWritePtr::~JackShmReadWritePtr - Init not done for 4294967295, skipping unlock
AL lib: (WW) ALCjackBackendFactory_init: jack_client_open() failed, 0x11
AL lib: (WW) alc_initconfig: Failed to initialize backend "jack"
GL_CONTEXT_PROFILE_MASK = 0 (unknown profile)
found bundled python: /home/ralf/Source/blender/build/bin/2.77/python
read blend: /home/ralf/crash.blend
CUDA error: Launch exceeded timeout in cuCtxSynchronize()

Refer to the Cycles GPU rendering documentation for possible solutions:
http://www.blender.org/manual/render/cycles/gpu_rendering.html

Saved session recovery to '/tmp/quit.blend'
CUDA error: Launch exceeded timeout in cuMemFree(cuda_device_ptr(mem.device_pointer))
CUDA error: Launch exceeded timeout in cuMemFree(cuda_device_ptr(mem.device_pointer))
CUDA error: Launch exceeded timeout in cuGraphicsUnregisterResource(pmem.cuPBOresource)
CUDA error: Launch exceeded timeout in cuMemFree(cuda_device_ptr(mem.device_pointer))
CUDA error: Launch exceeded timeout in cuMemFree(cuda_device_ptr(mem.device_pointer))
CUDA error: Launch exceeded timeout in cuMemFree(cuda_device_ptr(mem.device_pointer))
CUDA error: Launch exceeded timeout in cuMemFree(cuda_device_ptr(mem.device_pointer))
CUDA error: Launch exceeded timeout in cuMemFree(cuda_device_ptr(mem.device_pointer))
CUDA error: Launch exceeded timeout in cuMemFree(cuda_device_ptr(mem.device_pointer))
CUDA error: Launch exceeded timeout in cuMemFree(cuda_device_ptr(mem.device_pointer))
CUDA error: Launch exceeded timeout in cuMemFree(cuda_device_ptr(mem.device_pointer))
CUDA error: Launch exceeded timeout in cuMemFree(cuda_device_ptr(mem.device_pointer))
CUDA error: Launch exceeded timeout in cuMemFree(cuda_device_ptr(mem.device_pointer))
CUDA error: Launch exceeded timeout in cuMemFree(cuda_device_ptr(mem.device_pointer))
CUDA error: Launch exceeded timeout in cuMemFree(cuda_device_ptr(mem.device_pointer))
CUDA error: Launch exceeded timeout in cuMemFree(cuda_device_ptr(mem.device_pointer))
CUDA error: Launch exceeded timeout in cuMemFree(cuda_device_ptr(mem.device_pointer))
CUDA error: Launch exceeded timeout in cuMemFree(cuda_device_ptr(mem.device_pointer))
CUDA error: Launch exceeded timeout in cuMemFree(cuda_device_ptr(mem.device_pointer))
CUDA error: Launch exceeded timeout in cuMemFree(cuda_device_ptr(mem.device_pointer))
CUDA error: Launch exceeded timeout in cuMemFree(cuda_device_ptr(mem.device_pointer))
CUDA error: Launch exceeded timeout in cuMemFree(cuda_device_ptr(mem.device_pointer))
CUDA error: Launch exceeded timeout in cuMemFree(cuda_device_ptr(mem.device_pointer))
CUDA error: Launch exceeded timeout in cuMemFree(cuda_device_ptr(mem.device_pointer))

Exact steps for others to reproduce the error

open crash.blend
set the viewport mode to rendered

scrub the timeline

**System Information** [system-info.txt](https://archive.blender.org/developer/F322839/system-info.txt) Tested on Debian Stretch and at work on multiple Windows 10 machines. **Blender Version** Broken: 39dee8a Worked: unknown **Short description of error** When a scene with an object with an animated array modifier is live rendered in the viewport, and the current frame is changed, blender crashes on win10 and gives a timeout error on Linux. The crash on Windows 10 does happen with the builder.blender.org build from Wed, Jul 13, which is labeled as blender-2.77-eb04908-win64. Unfortunately, I can only debug at home on Linux where I get a timeout instead of a crash with the following log. ``` Read new prefs: /home/ralf/.config/blender/2.77/config/userpref.blend Cannot connect to server socket err = No such file or directory Cannot connect to server request channel jack server is not running or cannot be started JackShmReadWritePtr::~JackShmReadWritePtr - Init not done for 4294967295, skipping unlock JackShmReadWritePtr::~JackShmReadWritePtr - Init not done for 4294967295, skipping unlock AL lib: (WW) ALCjackBackendFactory_init: jack_client_open() failed, 0x11 AL lib: (WW) alc_initconfig: Failed to initialize backend "jack" GL_CONTEXT_PROFILE_MASK = 0 (unknown profile) found bundled python: /home/ralf/Source/blender/build/bin/2.77/python read blend: /home/ralf/crash.blend CUDA error: Launch exceeded timeout in cuCtxSynchronize() Refer to the Cycles GPU rendering documentation for possible solutions: http://www.blender.org/manual/render/cycles/gpu_rendering.html Saved session recovery to '/tmp/quit.blend' CUDA error: Launch exceeded timeout in cuMemFree(cuda_device_ptr(mem.device_pointer)) CUDA error: Launch exceeded timeout in cuMemFree(cuda_device_ptr(mem.device_pointer)) CUDA error: Launch exceeded timeout in cuGraphicsUnregisterResource(pmem.cuPBOresource) CUDA error: Launch exceeded timeout in cuMemFree(cuda_device_ptr(mem.device_pointer)) CUDA error: Launch exceeded timeout in cuMemFree(cuda_device_ptr(mem.device_pointer)) CUDA error: Launch exceeded timeout in cuMemFree(cuda_device_ptr(mem.device_pointer)) CUDA error: Launch exceeded timeout in cuMemFree(cuda_device_ptr(mem.device_pointer)) CUDA error: Launch exceeded timeout in cuMemFree(cuda_device_ptr(mem.device_pointer)) CUDA error: Launch exceeded timeout in cuMemFree(cuda_device_ptr(mem.device_pointer)) CUDA error: Launch exceeded timeout in cuMemFree(cuda_device_ptr(mem.device_pointer)) CUDA error: Launch exceeded timeout in cuMemFree(cuda_device_ptr(mem.device_pointer)) CUDA error: Launch exceeded timeout in cuMemFree(cuda_device_ptr(mem.device_pointer)) CUDA error: Launch exceeded timeout in cuMemFree(cuda_device_ptr(mem.device_pointer)) CUDA error: Launch exceeded timeout in cuMemFree(cuda_device_ptr(mem.device_pointer)) CUDA error: Launch exceeded timeout in cuMemFree(cuda_device_ptr(mem.device_pointer)) CUDA error: Launch exceeded timeout in cuMemFree(cuda_device_ptr(mem.device_pointer)) CUDA error: Launch exceeded timeout in cuMemFree(cuda_device_ptr(mem.device_pointer)) CUDA error: Launch exceeded timeout in cuMemFree(cuda_device_ptr(mem.device_pointer)) CUDA error: Launch exceeded timeout in cuMemFree(cuda_device_ptr(mem.device_pointer)) CUDA error: Launch exceeded timeout in cuMemFree(cuda_device_ptr(mem.device_pointer)) CUDA error: Launch exceeded timeout in cuMemFree(cuda_device_ptr(mem.device_pointer)) CUDA error: Launch exceeded timeout in cuMemFree(cuda_device_ptr(mem.device_pointer)) CUDA error: Launch exceeded timeout in cuMemFree(cuda_device_ptr(mem.device_pointer)) CUDA error: Launch exceeded timeout in cuMemFree(cuda_device_ptr(mem.device_pointer)) ``` **Exact steps for others to reproduce the error** - open [crash.blend](https://archive.blender.org/developer/F322853/crash.blend) - set the viewport mode to rendered # scrub the timeline

Ralf Hölzemer commented

2016-07-14 20:11:50 +02:00

Changed status to: 'Open'

Ralf Hölzemer commented

2016-07-14 20:11:50 +02:00

Added subscriber: @cheleb

Sergey Sharybin commented

2016-07-15 09:30:07 +02:00

Added subscriber: @Sergey

Sergey Sharybin commented

2016-07-15 09:30:07 +02:00

Timeout is an intrinsic behavior of CUDA, unfortunately. Internally driver limits amount of time you can spend in the compute kernel if you're computing on the display device (video card which has monitor connected to it). This timeout is nothing we can avoid from our side. You can try simplifying settings to make tiles rendering faster, but that's rather a workaround.

But all that being said, Blender should not crash -- as in it should only show error message and stop working, but allow to keep working further. Please make sure you're using latest drivers form nvidia.com site (don't rely on Windows installing drivers for you). And are the steps reproducing crash in Windows same as you noted above?

Timeout is an intrinsic behavior of CUDA, unfortunately. Internally driver limits amount of time you can spend in the compute kernel if you're computing on the display device (video card which has monitor connected to it). This timeout is nothing we can avoid from our side. You can try simplifying settings to make tiles rendering faster, but that's rather a workaround. But all that being said, Blender should not crash -- as in it should only show error message and stop working, but allow to keep working further. Please make sure you're using latest drivers form nvidia.com site (don't rely on Windows installing drivers for you). And are the steps reproducing crash in Windows same as you noted above?

step-ani-motion commented

2016-07-15 14:46:36 +02:00

Added subscriber: @STEP-ANI-MOTION

step-ani-motion commented

2016-07-15 14:46:36 +02:00

Hi Sergey,

this is me again posting from our d.b.o account at work. Thanks for the quick reply. Yeah, I know about the timeout thing and I deployed more useful tdr settings via GPO at our Studio for the render clients that do have a display connected to a rendering card, which works really well so far.

Our workstations however all have a dedicated card just for the display and additional cards to render via Cuda. I should have mentioned that in the initial post of course - sorry about that.

Yes, the reproduction steps above are also valid for windows. I just did update my workstation to the latest driver from NVidia (we always use the vendor drivers, not the windows update ones) and got the latest b.b.o build (blender-2.77-71bbf0e-win64), but the scene still crashed.

Workstation info:
system-info.txt
console_log.txt

Here's some screenies from the NVidia driver panel and blender.

Hi Sergey, this is me again posting from our d.b.o account at work. Thanks for the quick reply. Yeah, I know about the timeout thing and I deployed more useful tdr settings via GPO at our Studio for the render clients that do have a display connected to a rendering card, which works really well so far. Our workstations however all have a dedicated card just for the display and additional cards to render via Cuda. I should have mentioned that in the initial post of course - sorry about that. Yes, the reproduction steps above are also valid for windows. I just did update my workstation to the latest driver from NVidia (we always use the vendor drivers, not the windows update ones) and got the latest b.b.o build (blender-2.77-71bbf0e-win64), but the scene still crashed. **Workstation info:** [system-info.txt](https://archive.blender.org/developer/F323179/system-info.txt) [console_log.txt](https://archive.blender.org/developer/F323181/console_log.txt) Here's some screenies from the NVidia driver panel and blender. ![driver_version.png](https://archive.blender.org/developer/F323173/driver_version.png) ![cards_overview.png](https://archive.blender.org/developer/F323171/cards_overview.png) ![cuda_settings_nvidia.png](https://archive.blender.org/developer/F323175/cuda_settings_nvidia.png) ![cuda_settings_blender.png](https://archive.blender.org/developer/F323177/cuda_settings_blender.png)

Sergey Sharybin self-assigned this 2016-07-15 15:34:21 +02:00

Sergey Sharybin commented

2016-07-15 15:34:21 +02:00

Very interesting. For some reason there's a timelimit associated with your 680, which makes it considered a Display device. This shouldn't really happen.

But anyway, Blender should not crash.

Very interesting. For some reason there's a timelimit associated with your 680, which makes it considered a Display device. This shouldn't really happen. But anyway, Blender should not crash.

Sergey Sharybin commented

2016-07-15 18:33:18 +02:00

I've committed fix which potentially affects on this at ac061de.

Hopefully buildbot will deliver new builds soon, so you can test those. Let me know if after that fix the issue still happens.

If it does not help, please check if same issue exists in 2.77a.

I've committed fix which potentially affects on this at ac061de. Hopefully buildbot will deliver new builds soon, so you can test those. Let me know if after that fix the issue still happens. If it does not help, please check if same issue exists in 2.77a.

Ralf Hölzemer commented

2016-07-15 18:52:56 +02:00

Thanks for the quick fix!

ac061de doesn't timeout on my Linux machine at home anymore, which is promising already. I will test the windows build on my machine at work as soon as they are available on b.b.o and report back.

Thanks for the quick fix! ac061de doesn't timeout on my Linux machine at home anymore, which is promising already. I will test the windows build on my machine at work as soon as they are available on b.b.o and report back.

Ralf Hölzemer commented

2016-07-15 21:16:14 +02:00

Allright! Just did some tests on some win10 machines at work with the fresh build and the timeouts/crashes gone. Thanks again!

Sergey Sharybin commented

2016-07-15 21:34:01 +02:00

Changed status from 'Open' to: 'Resolved'

Sergey Sharybin closed this issue

2016-07-15 21:34:01 +02:00

Sergey Sharybin commented

2016-07-15 21:34:01 +02:00

Nice! Considering the report as resolved then!

Sign in to join this conversation.

No Label

Download

What's New

Blender Studio

Manual

Developers Blog

Documentation

Benchmark

Blender Conference

Development Fund

One-time Donations

Cycles/Cuda crash when live rendering an object with an animated array modifier #48856

scrub the timeline