Lookdev HDRIs are never unloaded, a sort of memory leak (+ other HDRI issues) #68867

New Issue

no way · 2019-08-20T13:38:06+02:00

no way commented

2019-08-20 13:38:06 +02:00

System Information
Operating system: Windows-7-6.1.7601-SP1 64 Bits
Graphics card: GeForce GTX 560 Ti/PCIe/SSE2 NVIDIA Corporation 4.5.0 NVIDIA 388.59

Blender Version Broken: version: 2.81 sub 2 , branch: master, commit date: 2019-08-16 18:51, hash: e8ffe25eb1

Short description of error
Lookdev never unloads the HDRIs given to it. This results in runaway memory usage, and eventually an OOM crash.

Exact steps for others to reproduce the error
Install a large number of lookdev HDRIs. (50 or more)
Observe memory usage. Several gigabytes will be taken up.
Restart Blender.
Observe memory usage. Blender will use a minimal amount of memory at this point, as expected.
Open the lookdev pop-up, and try to change the HDRI. (Alternatively, open the HDRI manager.)
Observe memory usage as Blender loads in the "thumbnails".
Pick a HDRI and close the pop-up. Memory will still be taken up by all the other ones, no matter how long we work on the scene.

Comment
A user's HDRI collection can be potentially large, and the current import and load behavior is very inefficient. Copying the images to the settings folder can have additional bad consequences. (small partition, backup software trying to pick up the HDRIs, waste of disk space when several programs are using the same collection)
Consider importing the images as path+thumbnail only.

**System Information** Operating system: Windows-7-6.1.7601-SP1 64 Bits Graphics card: GeForce GTX 560 Ti/PCIe/SSE2 NVIDIA Corporation 4.5.0 NVIDIA 388.59 **Blender Version** Broken: version: 2.81 sub 2 , branch: master, commit date: 2019-08-16 18:51, hash: e8ffe25eb1 **Short description of error** Lookdev never unloads the HDRIs given to it. This results in runaway memory usage, and eventually an OOM crash. **Exact steps for others to reproduce the error** Install a large number of lookdev HDRIs. (50 or more) Observe memory usage. Several gigabytes will be taken up. Restart Blender. Observe memory usage. Blender will use a minimal amount of memory at this point, as expected. Open the lookdev pop-up, and try to change the HDRI. (Alternatively, open the HDRI manager.) Observe memory usage as Blender loads in the "thumbnails". Pick a HDRI and close the pop-up. Memory will still be taken up by all the other ones, no matter how long we work on the scene. **Comment** A user's HDRI collection can be potentially large, and the current import and load behavior is very inefficient. Copying the images to the settings folder can have additional bad consequences. (small partition, backup software trying to pick up the HDRIs, waste of disk space when several programs are using the same collection) Consider importing the images as path+thumbnail only.

no way commented

2019-08-20 13:38:06 +02:00

Added subscriber: @stop_using_recaptcha

Philipp Oeser commented

2020-01-02 14:32:20 +01:00

Added subscribers: @fclem, @Jeroen-Bakker, @lichtwerk

Philipp Oeser commented

2020-01-02 14:32:20 +01:00

Changed status from 'Needs Triage' to: 'Archived'

Philipp Oeser closed this issue

2020-01-02 14:32:20 +01:00

Philipp Oeser self-assigned this 2020-01-02 14:32:20 +01:00

Philipp Oeser commented

2020-01-02 14:32:20 +01:00

Not 100% sure, but there might be ways to free up some data [a Studio Lights' equirect_radiance_buffer is kept forever (alongside other stuff probably) as you mentioned -- for all studio lights].

This happens as soon as we have a call to BKE_studiolight_ensure_flag(sl, STUDIOLIGHT_EXTERNAL_IMAGE_LOADED) -- which includes generating the previews...
So this would mean a different design would need to take effect, where only the previews are kept, unneeded buffers freed and only when actually switching to another studiolight, calculations would happen.

This would be a nice improvement, however this is not strictly a bug (since I dont see this being intended to free unneeded data in ints current state in the code...).

I'll CC @Jeroen-Bakker & @fclem here to consider adding this as a TODO to #64282 (EEVEE & Viewport Module), but will close this for now [unless these two have objections?], we can always reopen later...

So thx for the report, if this doesnt get added to the roadmap immediately, please consider using other channels for user feedback and feature requests: https://wiki.blender.org/wiki/Communication/Contact#User_Feedback_and_Requests
For more information on why this isn't considered a bug, visit: https://wiki.blender.org/wiki/Reference/Not_a_bug

Not 100% sure, but there might be ways to free up some data [a `Studio Lights`' `equirect_radiance_buffer` is kept forever (alongside other stuff probably) as you mentioned -- for all studio lights]. This happens as soon as we have a call to `BKE_studiolight_ensure_flag(sl, STUDIOLIGHT_EXTERNAL_IMAGE_LOADED)` -- which includes generating the previews... So this would mean a different design would need to take effect, where only the previews are kept, unneeded buffers freed and only when actually switching to another studiolight, calculations would happen. This would be a nice improvement, however this is not strictly a bug (since I dont see this being intended to free unneeded data in ints current state in the code...). I'll CC @Jeroen-Bakker & @fclem here to consider adding this as a TODO to #64282 (EEVEE & Viewport Module), but will close this for now [unless these two have objections?], we can always reopen later... So thx for the report, if this doesnt get added to the roadmap immediately, please consider using other channels for user feedback and feature requests: https://wiki.blender.org/wiki/Communication/Contact#User_Feedback_and_Requests For more information on why this isn't considered a bug, visit: https://wiki.blender.org/wiki/Reference/Not_a_bug

Jason commented

2023-09-26 17:14:05 +02:00

This issue still exists in 3.6.3 and below

Merely looking at HDRI thumbnails causes memory to balloon to 10 GB until you close blender.
This wasn't an issue long ago when we only had like 5 HDRI

I have a folder of 34 which = to 1.4 GB, so I'm not sure why Blender insists on 10 GB memory

This issue still exists in 3.6.3 and below Merely looking at HDRI thumbnails causes memory to balloon to 10 GB until you close blender. This wasn't an issue long ago when we only had like 5 HDRI I have a folder of 34 which = to 1.4 GB, so I'm not sure why Blender insists on 10 GB memory

Clipboarder.2023.09.26-008.png

356 KiB

Clipboarder.2023.09.26-009.png

327 KiB