Heavy EXR images use too much memory when loading them (e.g. multi-threaded thumbnail generation causes swapping) #70584

Closed
opened 2019-10-06 14:28:08 +02:00 by Jason · 24 comments

System Information
Operating system: Windows-10-10.0.18362 64 Bits
Graphics card: GeForce GTX 1050 Ti/PCIe/SSE2 NVIDIA Corporation 4.5.0 NVIDIA 431.86

Blender Version
Broken: version: 2.81 (sub 13), branch: master, commit date: 2019-10-04 22:33, hash: ab519b91b2
Worked: (optional)

Short description of error

Folder full of .EXR Environment maps that are 20 to 70 mb each
The thumbnail generation of these freezes my PC for 10 - 20 min as it generates them.
Windows doesn't freeze when generating them though.

in Blender 2.79 it crashes.

I'm not sure if this can be helped.

Exact steps for others to reproduce the error

Open Blender 2.81 or 2.80

Shader editor > World > Environment Texture > Open > Browse to a folder containing around 10 .exr images of 20 - 70 mb each > Switch to Thumbnail view.

*PC slows down and eventually freezes for 10 - 20 min as it generates the thumbnails.

**System Information** Operating system: Windows-10-10.0.18362 64 Bits Graphics card: GeForce GTX 1050 Ti/PCIe/SSE2 NVIDIA Corporation 4.5.0 NVIDIA 431.86 **Blender Version** Broken: version: 2.81 (sub 13), branch: master, commit date: 2019-10-04 22:33, hash: `ab519b91b2` Worked: (optional) **Short description of error** Folder full of .EXR Environment maps that are 20 to 70 mb each The thumbnail generation of these freezes my PC for 10 - 20 min as it generates them. Windows doesn't freeze when generating them though. in Blender 2.79 it crashes. I'm not sure if this can be helped. **Exact steps for others to reproduce the error** Open Blender 2.81 or 2.80 Shader editor > World > Environment Texture > Open > Browse to a folder containing around 10 .exr images of 20 - 70 mb each > Switch to Thumbnail view. *PC slows down and eventually freezes for 10 - 20 min as it generates the thumbnails.

#85182 was marked as duplicate of this issue

#85182 was marked as duplicate of this issue
Author

Added subscriber: @Reoxur

Added subscriber: @Reoxur
Author

In 2.81 I loaded Factory Defaults, opened the folder and Blender crashed.

In 2.81 I loaded Factory Defaults, opened the folder and Blender crashed.
Author

I've experimented and found that:

Introducing 2-3 .exr's at a time to Blenders file browser loads them fine without crashing
However, trying to load 15 of them at once in a Folder either freezes the PC or crashes blender

I think the Solutionwould be to limit .exr thumbnail generation to 2-3 at a time and not all at once.

I've experimented and found that: Introducing 2-3 .exr's at a time to Blenders file browser loads them fine without crashing However, trying to load 15 of them at once in a Folder either freezes the PC or crashes blender I think the **Solution**would be to limit .exr thumbnail generation to 2-3 at a time and not all at once.

Added subscriber: @mont29

Added subscriber: @mont29

Hrmmmm… A modern computer should be able to read more than two or three EXRs at a time, especially reasonably "small" ones… How many threads do you have on your CPU? and how many RAM?

Hrmmmm… A modern computer should be able to read more than two or three EXRs at a time, especially reasonably "small" ones… How many threads do you have on your CPU? and how many RAM?
Author

I have a Ryzen 5 1600 6 cores and 12 threads
And 16 gig Ram DDR4

I have a Ryzen 5 1600 6 cores and 12 threads And 16 gig Ram DDR4

Well, that would be 12 (or 13) EXRs opened at the same time… I’d have a hard time believing that could eat all of your 16 GB of ram…

What happens if you try to open 12 of those EXRs simultaneously in viewers e.g.? How many RAM does that takes?

Well, that would be 12 (or 13) EXRs opened at the same time… I’d have a hard time believing that could eat all of your 16 GB of ram… What happens if you try to open 12 of those EXRs simultaneously in viewers e.g.? How many RAM does that takes?
Author

Unfortunately I'm not sure how to get rid of the generated thumbnails
I can't test them in any Blender versions anymore.

I've tried to Disk Cleanup the Thumbnails but it did not affect the .EXR's in Blenders File Browser.

Unfortunately I'm not sure how to get rid of the generated thumbnails I can't test them in any Blender versions anymore. I've tried to Disk Cleanup the Thumbnails but it did not affect the .EXR's in Blenders File Browser.

I was asking to open those EXRs in another application, like an image viewer, 12 at once. To check whether your computer is able to handle that many EXR reading or not (Blender does nothing more than that, but with 12 threads it might do 12 at once).

I was asking to open those EXRs in another application, like an image viewer, 12 at once. To check whether your computer is able to handle that many EXR reading or not (Blender does nothing more than that, but with 12 threads it might do 12 at once).
Author

explorer_2019-10-13_12-36-58.png

gimp-2.10_2019-10-13_12-35-40.png

Opened 12 of them in Gimp, PC did slow down a bit, Task manager froze for a while
Then it was fine after

Not sure but maybe it's my Hardrive that cant keep up? its just a normal HDD from 3 years ago or so.

All I know is that in any Blender version when I tried to load the thumbnails of entire folder of massive EXR's my PC froze for a long time every time, until I started testing it with only 3 - 5 of them at a time and it then worked

So I'm just not sure if it's my Hardware or not. These specific EXR's are pretty large and its the first time that happened.
Taskmgr_2019-10-13_12-36-12.png

Taskmgr_2019-10-13_12-36-03.png

Taskmgr_2019-10-13_12-35-57.png

Taskmgr_2019-10-13_12-36-26.png

![explorer_2019-10-13_12-36-58.png](https://archive.blender.org/developer/F7812939/explorer_2019-10-13_12-36-58.png) ![gimp-2.10_2019-10-13_12-35-40.png](https://archive.blender.org/developer/F7812942/gimp-2.10_2019-10-13_12-35-40.png) Opened 12 of them in Gimp, PC did slow down a bit, Task manager froze for a while Then it was fine after Not sure but maybe it's my Hardrive that cant keep up? its just a normal HDD from 3 years ago or so. All I know is that in any Blender version when I tried to load the thumbnails of entire folder of massive EXR's my PC froze for a long time every time, until I started testing it with only 3 - 5 of them at a time and it then worked So I'm just not sure if it's my Hardware or not. These specific EXR's are pretty large and its the first time that happened. ![Taskmgr_2019-10-13_12-36-12.png](https://archive.blender.org/developer/F7812945/Taskmgr_2019-10-13_12-36-12.png) ![Taskmgr_2019-10-13_12-36-03.png](https://archive.blender.org/developer/F7812946/Taskmgr_2019-10-13_12-36-03.png) ![Taskmgr_2019-10-13_12-35-57.png](https://archive.blender.org/developer/F7812947/Taskmgr_2019-10-13_12-35-57.png) ![Taskmgr_2019-10-13_12-36-26.png](https://archive.blender.org/developer/F7812948/Taskmgr_2019-10-13_12-36-26.png)

Added subscriber: @iss

Added subscriber: @iss

Changed status from 'Needs Developer To Reproduce' to: 'Confirmed'

Changed status from 'Needs Developer To Reproduce' to: 'Confirmed'

I have only 8 thread CPU, and tried to open folder with 20M EXR's (8K render of cube, rgba+zbuf zip)

Mem usage went from 3GB to 13GB

Though each image should have only ~500M size in memory

I have only 8 thread CPU, and tried to open folder with 20M EXR's (8K render of cube, rgba+zbuf zip) Mem usage went from 3GB to 13GB Though each image should have only ~500M size in memory

5 components float32 times 33M pixels, that’s more like over 600MB of memory I think? Times 8, you already reach 5GB. If we have two buffers (one in OIIO library, one in our own IMBuf thingy), there you get your 10GB. And with twelve threads, would be 8GB/16GB…

Also annoying, would consider this more like a known limitation than really a bug, code is working as expected I think here, just those heavy images are very memory-greedy. Though improvement is of course possible, one way (limiting amount of threads reading files beyond a certain size?) or the other (fixing the read code to only use one buffer at a time would only push back the issue though, not actually fixing it).

5 components float32 times 33M pixels, that’s more like over 600MB of memory I think? Times 8, you already reach 5GB. If we have two buffers (one in OIIO library, one in our own IMBuf thingy), there you get your 10GB. And with twelve threads, would be 8GB/16GB… Also annoying, would consider this more like a known limitation than really a bug, code is working as expected I think here, just those heavy images are very memory-greedy. Though improvement is of course possible, one way (limiting amount of threads reading files beyond a certain size?) or the other (fixing the read code to only use one buffer at a time would only push back the issue though, not actually fixing it).
Bastien Montagne changed title from .exr Thumbnail generation causes computer to Freeze for 10 - 30 min to Heavy EXR images use too much memory when loading them (e.g. multi-threaded thumbnail generation causes swapping) 2020-01-14 14:40:10 +01:00

Added subscriber: @Arken

Added subscriber: @Arken

Added subscriber: @rjg

Added subscriber: @rjg

Seems like a design issue with the thumbnail generation code. Loading the images greedily, when the bottleneck is memory, doesn't seem like a good idea. In fact doing the entire processing sequentially in a thread would probably be a better trade-off between performance and reliability, since this can currently very easily crash Blender with OOM.

I also don't see a memory consumption that matches the napkin math in the comments above. On a processor with 12 logical cores, two buffers and .exr files with 8192 x 4096 px, this would be 12 * 2 * 8192 * 4096 * 4 * 32 bit = 12.88 GB. My system with 32 GB runs out of memory when attempting to show the thumbnails in a directory with 224 HDRI files.

Seems like a design issue with the thumbnail generation code. Loading the images greedily, when the bottleneck is memory, doesn't seem like a good idea. In fact doing the entire processing sequentially in a thread would probably be a better trade-off between performance and reliability, since this can currently very easily crash Blender with OOM. I also don't see a memory consumption that matches the napkin math in the comments above. On a processor with 12 logical cores, two buffers and .exr files with 8192 x 4096 px, this would be 12 * 2 * 8192 * 4096 * 4 * 32 bit = 12.88 GB. My system with 32 GB runs out of memory when attempting to show the thumbnails in a directory with 224 HDRI files.

Added subscriber: @Sworly

Added subscriber: @Sworly

Screenshot 2022-02-09 at 13.56.57.png I'm not seeing thumbnails for 32bit EXRs, only 16bit. (ZIP, RLE Compression made no difference) (dimensions made no difference). Is that expected behaviour?

Blender 3.0.1 on Macos 10.15.7, iMac Pro 64Gb ram, 8 core Xeon, Vega 56

![Screenshot 2022-02-09 at 13.56.57.png](https://archive.blender.org/developer/F12857329/Screenshot_2022-02-09_at_13.56.57.png) I'm not seeing thumbnails for 32bit EXRs, only 16bit. (ZIP, RLE Compression made no difference) (dimensions made no difference). Is that expected behaviour? Blender 3.0.1 on Macos 10.15.7, iMac Pro 64Gb ram, 8 core Xeon, Vega 56

Added subscriber: @Shady.Hamza

Added subscriber: @Shady.Hamza
Member

Added subscriber: @Harley

Added subscriber: @Harley
Member

Changed status from 'Confirmed' to: 'Resolved'

Changed status from 'Confirmed' to: 'Resolved'
Harley Acheson self-assigned this 2022-05-20 03:19:45 +02:00
Member

Resolved with commit f600a2aa6d as that will thumbnail all EXRs no matter what type or size and will do so with far less RAM.

Resolved with commit f600a2aa6d as that will thumbnail all EXRs no matter what type or size and will do so with far less RAM.
Sign in to join this conversation.
No Label
Interest
Alembic
Interest
Animation & Rigging
Interest
Asset Browser
Interest
Asset Browser Project Overview
Interest
Audio
Interest
Automated Testing
Interest
Blender Asset Bundle
Interest
BlendFile
Interest
Collada
Interest
Compatibility
Interest
Compositing
Interest
Core
Interest
Cycles
Interest
Dependency Graph
Interest
Development Management
Interest
EEVEE
Interest
EEVEE & Viewport
Interest
Freestyle
Interest
Geometry Nodes
Interest
Grease Pencil
Interest
ID Management
Interest
Images & Movies
Interest
Import Export
Interest
Line Art
Interest
Masking
Interest
Metal
Interest
Modeling
Interest
Modifiers
Interest
Motion Tracking
Interest
Nodes & Physics
Interest
OpenGL
Interest
Overlay
Interest
Overrides
Interest
Performance
Interest
Physics
Interest
Pipeline, Assets & IO
Interest
Platforms, Builds & Tests
Interest
Python API
Interest
Render & Cycles
Interest
Render Pipeline
Interest
Sculpt, Paint & Texture
Interest
Text Editor
Interest
Translations
Interest
Triaging
Interest
Undo
Interest
USD
Interest
User Interface
Interest
UV Editing
Interest
VFX & Video
Interest
Video Sequencer
Interest
Virtual Reality
Interest
Vulkan
Interest
Wayland
Interest
Workbench
Interest: X11
Legacy
Blender 2.8 Project
Legacy
Milestone 1: Basic, Local Asset Browser
Legacy
OpenGL Error
Meta
Good First Issue
Meta
Papercut
Meta
Retrospective
Meta
Security
Module
Animation & Rigging
Module
Core
Module
Development Management
Module
EEVEE & Viewport
Module
Grease Pencil
Module
Modeling
Module
Nodes & Physics
Module
Pipeline, Assets & IO
Module
Platforms, Builds & Tests
Module
Python API
Module
Render & Cycles
Module
Sculpt, Paint & Texture
Module
Triaging
Module
User Interface
Module
VFX & Video
Platform
FreeBSD
Platform
Linux
Platform
macOS
Platform
Windows
Priority
High
Priority
Low
Priority
Normal
Priority
Unbreak Now!
Status
Archived
Status
Confirmed
Status
Duplicate
Status
Needs Info from Developers
Status
Needs Information from User
Status
Needs Triage
Status
Resolved
Type
Bug
Type
Design
Type
Known Issue
Type
Patch
Type
Report
Type
To Do
No Milestone
No project
No Assignees
8 Participants
Notifications
Due Date
The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: blender/blender#70584
No description provided.