Regression: GPU memory accumulation in Cycles render #95837

New Issue

Semih Iyikalender · 2022-02-17T15:23:57+01:00

Semih Iyikalender commented

2022-02-17 15:23:57 +01:00

System Information
Operating system: Linux-5.13.0-28-generic-x86_64-with-glibc2.31 64 Bits
Graphics card: NVIDIA GeForce GTX 1660 SUPER/PCIe/SSE2 NVIDIA Corporation 4.5.0 NVIDIA 495.29.05

Blender Version
Broken: version: 3.1.0 Beta, branch: master, commit date: 2022-02-16 18:49, hash: 05697470ab 3.1.0 Beta, branch: master, commit date: 2022-02-13 02:13, hash: 517afcc858b0
Worked: blender-3.0.0-linux-x64

This broke between bad 922ae55a16 & good 37d6ae16be (20cb2c72a5 possibly?)

Short description of error
There is a GPU memory accumulation in Cycles render.
1024 samples with denoiser on. Also, extra compositing on frames results "out of GPU memory" error in later frames.
I watched with watch nvidia-smi command, each frame GPU memory increases. In the process section blender starts around 200Mb memory usage, after rendering 200 frames becomes 3Gb.
And quitting the render window doesn't decrease the memory usage, stays in the last state.
Tested with 128, 8, 1 samples, with and without denoiser and disabled compositing, there is still a memory accumulation that occurs in each scenario.

Exact steps for others to reproduce the error
Just render it.
render_bug_tests.blend

note: the following error gets spit to the console:

Invalid graphics context in cuGraphicsUnregisterResource(cu_graphicsresource) (intern/cycles/device/cuda/graphics_interop.cpp:36)

Other related reports with the same error message include

#95733 (Cycles Cuda/Optix error message with multi GPU devices. (Invalid graphics context in cuGraphicsUnregisterResource))
#95651 (GPU error (Invalid graphics context in cuGraphicsUnregisterResource))
#95631 (VRAM is not being freed when rendering (Invalid graphics context in cuGraphicsUnregisterResource))
#89747 (Cycles Render - Textures Disappear then Crashes the Render)
#95763 (Cycles freeze with hologram shader)

**System Information** Operating system: Linux-5.13.0-28-generic-x86_64-with-glibc2.31 64 Bits Graphics card: NVIDIA GeForce GTX 1660 SUPER/PCIe/SSE2 NVIDIA Corporation 4.5.0 NVIDIA 495.29.05 **Blender Version** Broken: version: 3.1.0 Beta, branch: master, commit date: 2022-02-16 18:49, hash: `05697470ab` 3.1.0 Beta, branch: master, commit date: 2022-02-13 02:13, hash: `517afcc858b0` Worked: blender-3.0.0-linux-x64 This broke between bad 922ae55a16 & good 37d6ae16be (20cb2c72a5 possibly?) **Short description of error** There is a GPU memory accumulation in Cycles render. 1024 samples with denoiser on. Also, extra compositing on frames results "out of GPU memory" error in later frames. I watched with `watch nvidia-smi` command, each frame GPU memory increases. In the process section blender starts around 200Mb memory usage, after rendering 200 frames becomes 3Gb. And quitting the render window doesn't decrease the memory usage, stays in the last state. Tested with 128, 8, 1 samples, with and without denoiser and disabled compositing, there is still a memory accumulation that occurs in each scenario. **Exact steps for others to reproduce the error** Just render it. [render_bug_tests.blend](https://archive.blender.org/developer/F12871282/render_bug_tests.blend) note: the following error gets spit to the console: ``` Invalid graphics context in cuGraphicsUnregisterResource(cu_graphicsresource) (intern/cycles/device/cuda/graphics_interop.cpp:36) ``` Other related reports with the same error message include - #95733 (Cycles Cuda/Optix error message with multi GPU devices. (Invalid graphics context in cuGraphicsUnregisterResource)) - #95651 (GPU error (Invalid graphics context in cuGraphicsUnregisterResource)) - #95631 (VRAM is not being freed when rendering (Invalid graphics context in cuGraphicsUnregisterResource)) - #89747 (Cycles Render - Textures Disappear then Crashes the Render) - #95763 (Cycles freeze with hologram shader)

Semih Iyikalender commented

2022-02-17 15:23:57 +01:00

Added subscriber: @merkwur

tempdevnova commented

2022-02-17 15:55:27 +01:00

Added subscriber: @tempdevnova

tempdevnova commented

2022-02-17 15:55:27 +01:00

Yep can definetly confirm this with my own blender files on my GTX 1080.

Can't really figure out the cause howver... I always assumed it was because of persistent data option but you didn't even have it on.

The same issue also exists within the viewport, where after rendering an image and switching the viewport to rendered mode it would report "out of host and shared memory" and cause blender to crash.

Yep can definetly confirm this with my own blender files on my GTX 1080. Can't really figure out the cause howver... I always assumed it was because of persistent data option but you didn't even have it on. The same issue also exists within the viewport, where after rendering an image and switching the viewport to rendered mode it would report "out of host and shared memory" and cause blender to crash.

Semih Iyikalender commented

2022-02-17 16:25:18 +01:00

I haven't experienced any issue in the viewport which includes 700 objects, 500k trinagles, two armature, and mb-lab data.
I can say that I have a relatively smooth experience with navigating around in the viewport-render view with all the objects and textures included, and occasionally rendering a single image.
Hence, I can't relate to your issue in this. That might be some other issue like; CUDA version or driver, etc?
But definitely, there is some issue with the rendering of multiple frames.

I haven't experienced any issue in the viewport which includes 700 objects, 500k trinagles, two armature, and mb-lab data. I can say that I have a relatively smooth experience with navigating around in the viewport-render view with all the objects and textures included, and occasionally rendering a single image. Hence, I can't relate to your issue in this. That might be some other issue like; CUDA version or driver, etc? But definitely, there is some issue with the rendering of multiple frames.

Philipp Oeser commented

2022-02-18 09:16:24 +01:00

Added subscriber: @lichtwerk