OIDN on GPU conflicts with Persistent data in some cases #123145
Labels
No Label
Interest
Alembic
Interest
Animation & Rigging
Interest
Asset System
Interest
Audio
Interest
Automated Testing
Interest
Blender Asset Bundle
Interest
BlendFile
Interest
Collada
Interest
Compatibility
Interest
Compositing
Interest
Core
Interest
Cycles
Interest
Dependency Graph
Interest
Development Management
Interest
EEVEE
Interest
Freestyle
Interest
Geometry Nodes
Interest
Grease Pencil
Interest
ID Management
Interest
Images & Movies
Interest
Import Export
Interest
Line Art
Interest
Masking
Interest
Metal
Interest
Modeling
Interest
Modifiers
Interest
Motion Tracking
Interest
Nodes & Physics
Interest
OpenGL
Interest
Overlay
Interest
Overrides
Interest
Performance
Interest
Physics
Interest
Pipeline, Assets & IO
Interest
Platforms, Builds & Tests
Interest
Python API
Interest
Render & Cycles
Interest
Render Pipeline
Interest
Sculpt, Paint & Texture
Interest
Text Editor
Interest
Translations
Interest
Triaging
Interest
Undo
Interest
USD
Interest
User Interface
Interest
UV Editing
Interest
VFX & Video
Interest
Video Sequencer
Interest
Viewport & EEVEE
Interest
Virtual Reality
Interest
Vulkan
Interest
Wayland
Interest
Workbench
Interest: X11
Legacy
Asset Browser Project
Legacy
Blender 2.8 Project
Legacy
Milestone 1: Basic, Local Asset Browser
Legacy
OpenGL Error
Meta
Good First Issue
Meta
Papercut
Meta
Retrospective
Meta
Security
Module
Animation & Rigging
Module
Core
Module
Development Management
Module
Grease Pencil
Module
Modeling
Module
Nodes & Physics
Module
Pipeline, Assets & IO
Module
Platforms, Builds & Tests
Module
Python API
Module
Render & Cycles
Module
Sculpt, Paint & Texture
Module
Triaging
Module
User Interface
Module
VFX & Video
Module
Viewport & EEVEE
Platform
FreeBSD
Platform
Linux
Platform
macOS
Platform
Windows
Severity
High
Severity
Low
Severity
Normal
Severity
Unbreak Now!
Status
Archived
Status
Confirmed
Status
Duplicate
Status
Needs Info from Developers
Status
Needs Information from User
Status
Needs Triage
Status
Resolved
Type
Bug
Type
Design
Type
Known Issue
Type
Patch
Type
Report
Type
To Do
No Milestone
No project
No Assignees
3 Participants
Notifications
Due Date
No due date set.
Dependencies
No dependencies set.
Reference: blender/blender#123145
Loading…
Reference in New Issue
Block a user
No description provided.
Delete Branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
System Information
Operating system: Windows-10-10.0.19045-SP0 64 Bits
Graphics card: NVIDIA GeForce RTX 3080/PCIe/SSE2 NVIDIA Corporation 4.6.0 NVIDIA 555.85
Blender Version
Broken: version: 4.1.1, branch: blender-v4.1-release, commit date: 2024-04-15 15:11, hash:
e1743a0317bc
Worked: before OIDN gone on GPU
Short description of error
I discovered a strange behavior in certain cases: when your scene is quite heavy and eats all VRAM AND you enable Persistent data option in Render Settings / Performance tab AND OIDN on GPU is enabled, in half cases render crashes on random of first few frames. The only solution is to fallback to use OIDN on CPU.
I cannot upload such troublesome file because of NDA and file's size (about 7 GiBs).
A have attached the crash log and console output for this scene.
Hi, thanks for the report. I suspect the system ran out of memory when persistent data is enabled (AFAICS ~8 gb memory is used already ). How much RAM/VRAM do you have? Can you monitor memory usage during the render?
Good time of day!
My guess is that OIDN data unpredictably replaces some crucial data of render itself (part of persistent data), because of lack of vram. But interesting part is that it happens absolutely random.
In my case I have several full-CG shots of full feture movie, they all represent the same scene shot by different angles and optics. In some cases shot renders with OIDN on gpu flawlessly, and in other cases, sometimes even simpler in quantity of data, crashes randomly.
Right now one of the shots being rendered (I set OIDN to CPU for reliability reasons) and there is screenshot of RAM and VRAM usage. For one frame render cycle.
Thanks. I see VRAM is almost tanked up. So crash is likely due to the "out of memory" case.
High memory usage due to persistent data is expected. But I'm in doubt about few things:
@Alaska hi, any idea here? Otherwise will have to poke cycles devs.
If it could help there is some additional info I forgot to tell: my renders set up to use OptiX, not CUDA. RAM reservation for CUDA is set by default in driver settings. I'm not sure what does it means: is it on or off? Neithertheless it shouldn't have to do anything with optix render, isn't it?
The scene itself composed of 50-80 millions of triangles (most of it is instanced) and uses few dosens of 8k textures (some are 8-bit, most are 24-bit and three or four are floating point ones).
The error seems to be occuring during a
memcopy
on the CPU during the setup of the camera motionblur for the next frame of rendering.Based on the screenshots provided, there's plenty of RAM left for the CPU. So I'm not sure why this is occuring, or why OIDN on the GPU impacts things. Unless I'm mis-understanding how memory is distributed accross devices (which is likely).
@efimpetelin Have you overclocked/udervolted your CPU or your RAM (Including settings like XMP and XPO in your BIOS)? If so, can you try disabling it and see if it helps? I'm thinking there's a possibility of some sort of hardware issue impacting things here, but I could be wrong.
Can you also share what CPU you have? 13th and 14th generation Intel CPUs have been having some stability issues in some programs leading to random crashes and errors.
It might be best to try and reproduce the issue first, then try contact a Cycles developers for input.
I ran some quick tests and couldn't reproduce this issue, but I could be doing something wrong.
@Alaska Hello! No, I haven't overclocked/undervolted any of my gear parts. And never had any cpu-ram problems on it. But I'm using XMP profile (never thought it could be treated as overclocking though). There is bunch of specs screenshots in attachments.
Oops, I occasionaly closed this ticked, sorry!
Does disabling XMP resolve the issue?
Intel considers XMP a overclock. And in my personal experience, enabling XMP can make a computer produce unexpected results related to memory if the speeds in the profile are too high, or the BIOS doesn't apply it properly.
@Alaska It's really strange: I tried to record video of rendering crash with OBS, with XMP and without it. And it doesn't crash today! )) I will try to report you later, when the same crash will occur.
I have encountered this error again, and there is video, explaining all things.
After all I disabled XMP in BIOS, and there weren't any crashes under any circumstances.
My apologizes, perhaps it's some kind of hardware/software related things.
P.S.
Nope. It crashed without XMP (and firefox )) right after 1003 frame.
I tried recrating a scenario that could cause the issue based on the information you've shared. So that's a scene that:
I also tried opening up Firefox (since the issue seemed more common with Firefox open for you) and watched a 4k YouTube video while the render was running.
I could not reproduce the issue. But there's still a possibilty that I'm doing something wrong.
@efimpetelin Can you run some extra tests for us?
System Information
Operating system: Windows-10-10.0.22631-SP0 64 Bits
Graphics card: NVIDIA GeForce RTX 4090/PCIe/SSE2 NVIDIA Corporation 4.6.0 NVIDIA 555.99
Blender version: 4.1.1, branch: blender-v4.1-release, commit date: 2024-04-15 15:11, hash:
e1743a0317bc
Hello!
I'm very busy these days so can't hassle with all this experiments for now. Neithertheless I will try as soon as some free time will appear.
The only thing that wasn't time-consuming was to test this scene under Ubuntu. And right after rendering was done, on the denoising phase Blender just says "Out of memory". It doesn't crash, just stops with this error.
My guess is still the same: peristent data, kept on GPU, prevents OIDN to allocate enough of VRAM to initialise proprely.