Heavy EXR images use too much memory when loading them (e.g. multi-threaded thumbnail generation causes swapping) #70584

System Information
Operating system: Windows-10-10.0.18362 64 Bits
Graphics card: GeForce GTX 1050 Ti/PCIe/SSE2 NVIDIA Corporation 4.5.0 NVIDIA 431.86

Blender Version
Broken: version: 2.81 (sub 13), branch: master, commit date: 2019-10-04 22:33, hash: ab519b91b2
Worked: (optional)

Short description of error

Folder full of .EXR Environment maps that are 20 to 70 mb each
The thumbnail generation of these freezes my PC for 10 - 20 min as it generates them.
Windows doesn't freeze when generating them though.

in Blender 2.79 it crashes.

I'm not sure if this can be helped.

Exact steps for others to reproduce the error

Open Blender 2.81 or 2.80

Shader editor > World > Environment Texture > Open > Browse to a folder containing around 10 .exr images of 20 - 70 mb each > Switch to Thumbnail view.

*PC slows down and eventually freezes for 10 - 20 min as it generates the thumbnails.

#85182 was marked as duplicate of this issue

Added subscriber: @Reoxur

In 2.81 I loaded Factory Defaults, opened the folder and Blender crashed.

I've experimented and found that:

Introducing 2-3 .exr's at a time to Blenders file browser loads them fine without crashing
However, trying to load 15 of them at once in a Folder either freezes the PC or crashes blender

I think the Solutionwould be to limit .exr thumbnail generation to 2-3 at a time and not all at once.

Added subscriber: @mont29

Hrmmmm… A modern computer should be able to read more than two or three EXRs at a time, especially reasonably "small" ones… How many threads do you have on your CPU? and how many RAM?

I have a Ryzen 5 1600 6 cores and 12 threads
And 16 gig Ram DDR4

Well, that would be 12 (or 13) EXRs opened at the same time… I’d have a hard time believing that could eat all of your 16 GB of ram…

What happens if you try to open 12 of those EXRs simultaneously in viewers e.g.? How many RAM does that takes?

Unfortunately I'm not sure how to get rid of the generated thumbnails
I can't test them in any Blender versions anymore.

I've tried to Disk Cleanup the Thumbnails but it did not affect the .EXR's in Blenders File Browser.

I was asking to open those EXRs in another application, like an image viewer, 12 at once. To check whether your computer is able to handle that many EXR reading or not (Blender does nothing more than that, but with 12 threads it might do 12 at once).

Opened 12 of them in Gimp, PC did slow down a bit, Task manager froze for a while
Then it was fine after

Not sure but maybe it's my Hardrive that cant keep up? its just a normal HDD from 3 years ago or so.

All I know is that in any Blender version when I tried to load the thumbnails of entire folder of massive EXR's my PC froze for a long time every time, until I started testing it with only 3 - 5 of them at a time and it then worked

So I'm just not sure if it's my Hardware or not. These specific EXR's are pretty large and its the first time that happened.

Added subscriber: @iss

Changed status from 'Needs Developer To Reproduce' to: 'Confirmed'

I have only 8 thread CPU, and tried to open folder with 20M EXR's (8K render of cube, rgba+zbuf zip)

Mem usage went from 3GB to 13GB

Though each image should have only ~500M size in memory

5 components float32 times 33M pixels, that’s more like over 600MB of memory I think? Times 8, you already reach 5GB. If we have two buffers (one in OIIO library, one in our own IMBuf thingy), there you get your 10GB. And with twelve threads, would be 8GB/16GB…

Also annoying, would consider this more like a known limitation than really a bug, code is working as expected I think here, just those heavy images are very memory-greedy. Though improvement is of course possible, one way (limiting amount of threads reading files beyond a certain size?) or the other (fixing the read code to only use one buffer at a time would only push back the issue though, not actually fixing it).

Added subscriber: @Arken

Added subscriber: @rjg

Seems like a design issue with the thumbnail generation code. Loading the images greedily, when the bottleneck is memory, doesn't seem like a good idea. In fact doing the entire processing sequentially in a thread would probably be a better trade-off between performance and reliability, since this can currently very easily crash Blender with OOM.

I also don't see a memory consumption that matches the napkin math in the comments above. On a processor with 12 logical cores, two buffers and .exr files with 8192 x 4096 px, this would be 12 * 2 * 8192 * 4096 * 4 * 32 bit = 12.88 GB. My system with 32 GB runs out of memory when attempting to show the thumbnails in a directory with 224 HDRI files.

Added subscriber: @Sworly

I'm not seeing thumbnails for 32bit EXRs, only 16bit. (ZIP, RLE Compression made no difference) (dimensions made no difference). Is that expected behaviour?

Blender 3.0.1 on Macos 10.15.7, iMac Pro 64Gb ram, 8 core Xeon, Vega 56

Added subscriber: @Shady.Hamza

Added subscriber: @Harley

Changed status from 'Confirmed' to: 'Resolved'

Resolved with commit f600a2aa6d as that will thumbnail all EXRs no matter what type or size and will do so with far less RAM.

Download

What's New

Blender Studio

Manual

Developers Blog

Documentation

Benchmark

Blender Conference

Development Fund

One-time Donations

Heavy EXR images use too much memory when loading them (e.g. multi-threaded thumbnail generation causes swapping) #70584