Cycles baking: "CUDA error: Launch exceeded timeout in ..." #40370

New Issue

Wojciech Matyjewicz · 2014-05-26T21:54:43+02:00

2014-05-26 21:54:43 +02:00

System Information and Blender Version

GPU: NVIDIA GTX 670
The error occurs both on Linux and Windows
- Linux: Blender build b33d83bf51, Debian jessie x64, nvidia-glx 331.67-2, nvidia-cuda-toolkit 5.5.22-3 (6.0 not yet available in jessie)
- Windows: Blender 2.71, Windows 8 64-bit, NVIDIA driver version 337.88 (the most recent ones)

Short description of error
Baking COMBINED pass with Cycles on a GPU does not work for higher number of samples. It fails with a series of errors "CUDA error: Launch exceeded timeout in ...". See log.txt.

Exact steps for others to reproduce the error

Open cuda-error.blend. It should have number of samples set to 32.
Click 'Bake'. No error should occur.
Change the number of samples to 64.
Click 'Bake'. It should fail with errors described above.

**System Information and Blender Version** - GPU: NVIDIA GTX 670 - The error occurs both on Linux and Windows - Linux: Blender build b33d83bf51, Debian jessie x64, nvidia-glx 331.67-2, nvidia-cuda-toolkit 5.5.22-3 (6.0 not yet available in jessie) - Windows: Blender 2.71, Windows 8 64-bit, NVIDIA driver version 337.88 (the most recent ones) **Short description of error** Baking COMBINED pass with Cycles on a GPU does not work for higher number of samples. It fails with a series of errors "CUDA error: Launch exceeded timeout in ...". See [log.txt](https://archive.blender.org/developer/F91242/log.txt). **Exact steps for others to reproduce the error** - Open [cuda-error.blend](https://archive.blender.org/developer/F91243/cuda-error.blend). It should have number of samples set to 32. - Click 'Bake'. No error should occur. - Change the number of samples to 64. - Click 'Bake'. It should fail with errors described above.

Wojciech Matyjewicz commented

2014-05-26 21:54:43 +02:00

Changed status to: 'Open'

Wojciech Matyjewicz commented

2014-05-26 21:54:43 +02:00

Added subscriber: @wmatyjewicz

Dalai Felinto commented

2014-05-26 22:00:45 +02:00

Added subscribers: @brecht, @dfelinto

Dalai Felinto commented

2014-05-26 22:00:45 +02:00

@brecht could you confirm this? no CUDA here

Brecht Van Lommel commented

2014-05-26 22:09:40 +02:00

Yes, I can confirm it. For graphics cards used for display the operating system puts a limit on how long a kernel can run, and if that is exceeded it is cancelled.

The only way to solve this is to split up the baking code so it only computes one sample each time the kernel is called, the same way regular rendering only does one sample at a time. That's quite a bit of work though, and it's a bit late in the release cycle. It may be best to disable CUDA baking for the release.

Yes, I can confirm it. For graphics cards used for display the operating system puts a limit on how long a kernel can run, and if that is exceeded it is cancelled. The only way to solve this is to split up the baking code so it only computes one sample each time the kernel is called, the same way regular rendering only does one sample at a time. That's quite a bit of work though, and it's a bit late in the release cycle. It may be best to disable CUDA baking for the release.

Martijn Berger commented

2014-05-27 07:50:06 +02:00

Added subscriber: @MartijnBerger

Martijn Berger commented

2014-05-27 07:50:06 +02:00

Is it possible to bake using multiple kernel invocations? As the timeout is per kernel invocation this could provide a way around this limitation.

Brecht Van Lommel commented

2014-05-27 10:49:39 +02:00

Yes, that's what should be done. It's already split now by the number of pixels, but if you have many samples a single pixel can also take long, so it also needs to be split by the number of samples.

Wojciech Matyjewicz commented

2014-05-27 18:31:09 +02:00

I can confirm that as a workaround one can configure BIOS and OS not to use the CUDA device as the display device. At least for Linux. I have not managed to configure Windows not to treat the NVIDIA GPU as a potential display device (even though no display was connected to it).

I can confirm that as a workaround one can configure BIOS and OS not to use the CUDA device as the display device. At least for Linux. I have not managed to configure Windows not to treat the NVIDIA GPU as a *potential* display device (even though no display was connected to it).

blender-admin commented

2014-06-06 15:39:04 +02:00

This issue was referenced by e4e58d4612

This issue was referenced by e4e58d46128dc7fe4fb9b881d73b38173f00f5c3

Brecht Van Lommel commented

2014-06-06 15:39:36 +02:00

Changed status from 'Open' to: 'Resolved'

Brecht Van Lommel closed this issue

2014-06-06 15:39:36 +02:00

Brecht Van Lommel commented

2014-06-06 15:39:36 +02:00

Closed by commit e4e58d4612.

Closed by commit e4e58d4612.

istavang commented

2014-08-23 18:59:53 +02:00

(Migrated from localhost:3001)

Added subscriber: @istavang

istavang commented

2014-08-23 18:59:53 +02:00

(Migrated from localhost:3001)

is this resolved ? I still get cuda error on gtx670 driver ver 340 52
win 8.1 64b

erroe msg: not found cuModule getglobal cumem and cubytes cumodulename when set to rendered mode

is this resolved ? I still get cuda error on gtx670 driver ver 340 52 win 8.1 64b erroe msg: not found cuModule getglobal cumem and cubytes cumodulename when set to rendered mode

Dalai Felinto commented

2014-08-26 02:56:54 +02:00

@istavang this report is related to baking. If you get an error when setting to rendered mode then it's not a baking bug. Please create a new but report for that.

Sign in to join this conversation.

No Label

Download

What's New

Blender Studio

Manual

Developers Blog

Documentation

Benchmark

Blender Conference

Development Fund

One-time Donations

Cycles baking: "CUDA error: Launch exceeded timeout in ..." #40370