CUDA animation render with Motion blur. #33465

%%%System: Quad Core AMD 64, 6 GB RAM, GeForce GTX 560 Ti w 1024 MB DDR5 PCIe 2.0

Ubuntu 12.04 LTS &/or Windows 7

current version 2.64.9 r52837 - however it was behaving the same previous versions. (I also tested with the latest daily build)

My blender file is made of 170810 verts / 143624 faces / 16 objects / 0 lamps

I rendered animation on both Ubuntu and Windows 7 with Cycles. NVIDIA CUDA(NVIDIA-Linux-x86_64-295.49 - Latest Windows driver) & motion blur enabled. After four or so frames rendered; it either renders ten more blank frames before crashing, or it crashes completely giving only the following information on the command line:

CUDA error: Out of memory in cuArrayCreate(&handle, &desc)
Bus error (core dumped)

This may also contribute to the issue:

I am rendering one object in the scene with an image sequence texture of two jpeg images set to auto refresh with each frame.

I assume that this is a bug because it works without issue while rendering with the CPU & GPU compatible shaders.%%%

Changed status to: 'Open'

%%%What's your Graphics Card, in windows you can monitor the memory load with a widget.

Probably not happening with CPU because it accesses your main RAM (supposedly bigger than your gpu's). Try tile rendering on GPU with Save buffers enabled on :)%%%

%%%My Graphics Card is - GeForce GTX 560 Ti w 1024 MB DDR5 PCIe 2.0

I have tile rendering : tile size X & Y = 256. "Save Buffers" is on. I also have "Static BVH" with "Use Spatial Splits" and "Cache BVH" enabled.%%%

%%%We really need a way to redo issues to fix them, that means an example .blend file that crashes in this case. Further it helps if you can try to narrow down where the issue is. You say auto refresh may have an influence, that's possible to verify then by disabling it and testing if it still gives the same issue afterwards.%%%

%%%I'll give it a try. I did not share the blend because of the 5meg limit. My file is 8 megs. But I'll see what I can do.%%%

%%%I'll try to get the file size to 5meg in order to post it. I removed the image sequence entirely and I got the same result. It rendered 3 frames and then 8 blank frames and crashed.%%%

%%%https://dl.dropbox.com/u/82843316/nSEG-animatedVBM.blend

Above is the link to the file: I was unable to reduce the size.%%%

%%%can confirm this, it think the problem is motion blur here , you can repeat this with these steps...

open default scene
set renderer to cycles
set viewport to "rendered" view
set compute device to cuda
set samples to 100 for preview and 250 for render
enable motion blur set motion blur to 1.5
set tile size to x :256 y :256
enable auto key and select default cube
press g and create key in 0 frame then set time line to new frame like 100
move cube in space and create animation sequence
set time line to frame 50 and press render f12 5-6 times
now press escape and you will get error message on the viewport top info bar

" CUDA Error : Out of memory in cuLaunchGrid (cuPathTrace,xblocks,yblocks) "

I will try to upload a videoshowing the exact steps and outcome...

%%%

%%%I couldn't redo either of these issues on Ubuntu 12.10, with a GTX 460, driver version 304.43. Tried both the official Blender 2.65 release and own build.

For the issue with the rendered cube, this scene of course takes almost no memory. The bigger scene upload takes about 170 MB of memory here, and "nvidia-smi -q -d MEMORY" show memory usage doesn't increase here as more frames are rendered. Maybe it is leaking render buffer memory somehow, but I'm not sure where.

Please test the official 2.65 release from blender.org if you didn't already
If you test rendering the exact same scene with motion blur on and off, does it make a difference, i.e. is it this particular option that makes it fail?
Does turning on/off the image sequence auto refresh make any difference?
Are you using multi GPU rendering or do you have just one GPU?
%%%

%%%Hi Brecht, I understand your question wasnt directed at me but I will answer the ones i can.
Nvidia Driver version 306.97
Specs:

Home computer:
  Amd phenom II 1100T
  Windows 7 64
  Nvidia 580 GTX

16 Gb Ram

Work Computer:

 AMd Phenom II 1100T
Windows 7 64
Nvidia 570 GTX

16 Gb Ram

Its the offical release of 2.65
Yes , cube with motion blur on GPU gives error. Cube with motion blur off doesnt.
Question is for an image sequence texture i think? The cube doesnt have any materials or textures or UVs :)
Single GPU on both computers

other tests i did are ...

cpu cycles rendering with 1280x720 res 250 samples and 256x256 tiles gives no errors.Render takes 50 secs on average.35 secs without motion blur.

after the initial error message changing viewport settings or render settings (sample,exposure,tile size motion blur size , motion blur on/off,clamp size) doesnt revert the cuda state you have restart blender.

here is the dropbox link for the test i did at work showing render without motion blur not giving error and render with motion blur giving error.

https://www.dropbox.com/s/wyp3hpqhskf85vq/bug_report_from_office_265_cyclesmotionblur.avi

%%%

%%%I just downloaded the 2.65 release this morning and I'm no longer experiencing the issue in Ubuntu 12.04. I have not tried windows 7 again.%%%

%%%Good to hear that issue is fixed.

So probably we are dealing with two separate issues then, and there still one remaining. Maybe there's some sort of issue when rendering with the viewport open, I'll do some testing on Windows later.%%%

%%%I read something like that a few days ago. If I find it I pass it on.%%%

%%%I've tried this now on Windows 7 with a GTX 460 and Blender 2.65, following the steps given by Yigit, but I couldn't get it to crash. Maybe it's the different graphics card, 580 has shader model 2.0 while 460 has 2.1. Sometimes I can redo such issues by running the 2.0 kernel on my card, but that also didn't make it crash.

There is obviously a bug here as shown in the video, but I don't know how to start fixing it without a way to redo. I looked over the code and don't see any particular reason motion blur should cause such errors.%%%

%%%? ISSUE RESOLVED ?

Cause : Corrupt default scene / ?? VBO's causing timeout on graphics card (its a guess no proof) ??

Hi Brecht;this bug never created a hard crash, or any crash for that matter, it drains the memory of the gpu,and the gpu stops responding.So renderin animation sequences causes problems, which is the main purpose for motion blur in the first place :).Anyway; I did some tests last night and heres what i manage to find.

I downloaded 2.65.3 (r53103) from builder.blender.org , the bug is still there .. .:(
I used "save buffers" option in performance tab , although it allows 7-8 frames to render it gives a half rendered tile and then a black render and when i

  switch to viewport it reads a different error "Cuda Error : Unknown Error"

I used a "Cache BVH" option in performance tab, again it allows more frames to be render 7-8 but gives a black render and switching to viewport you get

   the old message " CUDA Error : Out of memory in cuLaunchGrid (cuPathTrace,xblocks,yblocks) "

** ?ınteresting event ? **

I build up a new scene with a game asset of mine, using :

  cycles (gpu render / 250 samples / 1920x1080 resoulution / 256x256 tiles)
  armature(33 deform bones,around 27-28 control /helper bones)
  subsurf modifier (4 levels on1800 vertex mesh) 
  displace modifier (32 bit exr map) 
  animated camera

memory used : 150.6 M

as long the motion blur (1.5 factor) is active i get the same bug and black renders but when i enabled both "save buffers" and "cache bvh" i managed to render 42 frames of animation with motion blur.No errors no black renders.

Then I opened blender and loaded factory defaults. Then I couldnt recreate the bug.The cube motion blur with factory default works fine.The character animation i did renders fine.This lead me to believe that my default scene is somehow corrupt. The only difference i thnik will make a difference in the preferences/system tab is that i enable CUDA and VBO's by default. Changing VBO setting in the corrupted default scene doesnt made any difference i still get the black renders and Cuda error message. Anyway, this seems a problem from my side and the bug is non existent at the moment. I will upload the corrupt motion blur cube scene and the character animation test rendered by forcing save buffers and cache bvh in case you would like to investigate more.
%%%

%%%? ISSUE RESOLVED ? Cause : Corrupt default scene / ?? VBO's causing timeout on graphics card (its a guess no proof) ?? Hi Brecht;this bug never created a hard crash, or any crash for that matter, it drains the memory of the gpu,and the gpu stops responding.So renderin animation sequences causes problems, which is the main purpose for motion blur in the first place :).Anyway; I did some tests last night and heres what i manage to find. 1) I downloaded 2.65.3 (r53103) from builder.blender.org , the bug is still there .. .:( 2) I used "save buffers" option in performance tab , although it allows 7-8 frames to render it gives a half rendered tile and then a black render and when i ``` switch to viewport it reads a different error "Cuda Error : Unknown Error" ``` 3) I used a "Cache BVH" option in performance tab, again it allows more frames to be render 7-8 but gives a black render and switching to viewport you get ``` the old message " CUDA Error : Out of memory in cuLaunchGrid (cuPathTrace,xblocks,yblocks) " ``` ** ?ınteresting event ? ** 4) I build up a new scene with a game asset of mine, using : ``` cycles (gpu render / 250 samples / 1920x1080 resoulution / 256x256 tiles) armature(33 deform bones,around 27-28 control /helper bones) subsurf modifier (4 levels on1800 vertex mesh) displace modifier (32 bit exr map) animated camera ``` memory used : 150.6 M as long the motion blur (1.5 factor) is active i get the same bug and black renders but when i enabled both "save buffers" and "cache bvh" i managed to render 42 frames of animation with motion blur.No errors no black renders. Then I opened blender and loaded factory defaults. Then I couldnt recreate the bug.The cube motion blur with factory default works fine.The character animation i did renders fine.This lead me to believe that my default scene is somehow corrupt. The only difference i thnik will make a difference in the preferences/system tab is that i enable CUDA and VBO's by default. Changing VBO setting in the corrupted default scene doesnt made any difference i still get the black renders and Cuda error message. Anyway, this seems a problem from my side and the bug is non existent at the moment. I will upload the corrupt motion blur cube scene and the character animation test rendered by forcing save buffers and cache bvh in case you would like to investigate more. %%%

%%%Yigit Savtur: Can you please test if the issue still happens in latest svn? (builder.download.org). Thanks. %%%

%%%Hi Thomas the issue is resolved, i tested the corrupt startup scene and a factory default and cant produce the bug (with or with out VBO's) :) thanks.
Tested with Blender 2.65.9 r54002%%%

Changed status from 'Open' to: 'Resolved'

Download

What's New

Blender Studio

Manual

Developers Blog

Documentation

Benchmark

Blender Conference

Development Fund

One-time Donations

CUDA animation render with Motion blur. #33465