Asset System & UI: Improve memory management for assets and previews #111614

New Issue

Julian Eisel · 2023-08-28T15:30:10+02:00

Julian Eisel commented

2023-08-28 15:30:10 +02:00

Loading asset libraries can lead to significant memory use, since all contained assets need to be represented in memory. This may be thousands of assets. Further, the UI may request previews for assets, where each preview has a notable memory footprint (around 70kB for each preview at the default 256x256 image size).

As it stands, Blender is not mindful about this at all. For example simply calling the "Add" menu in geometry nodes editors loads all asset libraries into memory (!), including all previews (!) and keeps all this data in memory until another .blend is read (!). This is bad and wasteful design that users have to pay for.

Status Quo

Memory and asset lifetime is managed rather differently in the asset browser and the rest of the UI (asset shelves, add menus, etc.).

Current Asset Browser Memory Management:

The situation for the asset browser is not too bad (because it's simpler).

Good: Only the currently shown asset library is loaded into memory.
Good: Previews are lazy loaded into a cache based on the assets in view.
Good: Previews are removed if the cache is filled with a resonably small amount of previews.
Bad: Storage is per asset browser instance, so when multiple are open and showing the same asset libraries, storage is duplicated (especially wasteful for the "All" asset library).
Bad: The preview cache is cleared and reloaded often, causing flickering on common interactions (e.g. undo/redo).
Bad: The preview cache fails miserably with large asset libraries and causes continuous freezes, making the asset browser unusable with large asset libraries.

Rest of the UI:

(Asset views, the node add menu, the node search menu, asset-view template, etc.)

Challenge: Usually use the "All" asset library, meaning all assets are loaded when showing such UIs.
Good: Memory is shared between these UIs (static storage), so libraries are only loaded and stored once. (Asset browser storage is still separate).
Bad: No lazy loading of previews, all previews are loaded into memory
Bad: Only freed when closing the file.
Bad: Previews are also loaded when none are displayed (e.g. node "Add" menu)

How to solve it?

Good solution characteristics

A good solution should give us the following:

Only hold asset previews in storaged that are in view of the user
Don't keep asset libraries in storage that the user isn't interacting with (at least not regularly, see next point)
Provide fast repetetive access to asset libraries (e.g. assets shouldn't be reloaded every time the "Add" menu is called in the geometry nodes editor)
Support shared storage. All UIs giving access to assets should be able to access the same static asset library and preview image storage.

(Further improvements are possible still, see below)

Lazy Loading Previews

Previews should always be lazy loaded based on which assets the user sees, and only if the previews are actually displayed (e.g. not for the node "Add" menu). #109234 does most of the necessary work for this.

Garbage Collector?

A garbage collector could be implemented to remove unused asset libraries and asset previews from storage. Some kind of user-counting will be needed to detect the unused state, even when multiple parts of the UI display the same asset library (and thus may share storage). By being time based rather than user-count based only, the garbage collector still gives enough time for repetitive access to asset libraries. For example the asset library may still be in storage for multiple calls to the node "Add" menu, but be freed from storage if it hasn't been called for at least 5 minutes.

Note that this doesn't need to be asset system specific. There are other UI previews that we load from disk that could make use of this. So this could be a more general UI garbage collector.

Possible Further Improvements

Partial asset library loading?

Users rarely ever look at all assets of a library. They typically look at assets of a certain type (e.g. in the node "Add" menu), a specific catalog, or the view simply only has space for a subset of the available assets. So this is another case where we can avoid keeping data in storage, e.g. by lazy loading assets or deleting them once they are recognized as irrelevant.

Keeping track of which assets are loaded and which have to be made available can be finicky. Plus we may run into issues when asset library contents were updated in-between partial reads.

Lazy load extended metadata

In most cases, all the user sees from an asset is the name, and potentially the preview. Keeping all the asset metadata in storage is a bit of a waste. We could lazy load this as well, although this can become finicky too, since some metadata needs to be accessed faster or more frequently (e.g. basic type information, tags for search, etc.) than other.

Compressed preview buffers

Preview buffers could be compressed to reduce the memory footprint of each, potentially leading to faster load times too.

Loading asset libraries can lead to significant memory use, since all contained assets need to be represented in memory. This may be thousands of assets. Further, the UI may request previews for assets, where each preview has a notable memory footprint (around 70kB for each preview at the default 256x256 image size). As it stands, Blender is not mindful about this at all. For example simply calling the "Add" menu in geometry nodes editors loads all asset libraries into memory (!), including all previews (!) and keeps all this data in memory until another .blend is read (!). This is bad and wasteful design that users have to pay for. ## Status Quo Memory and asset lifetime is managed rather differently in the asset browser and the rest of the UI (asset shelves, add menus, etc.). **Current Asset Browser Memory Management:** The situation for the asset browser is not too bad (because it's simpler). - **Good**: Only the currently shown asset library is loaded into memory. - **Good**: Previews are lazy loaded into a cache based on the assets in view. - **Good**: Previews are removed if the cache is filled with a resonably small amount of previews. - **Bad**: Storage is per asset browser instance, so when multiple are open and showing the same asset libraries, storage is duplicated (especially wasteful for the "All" asset library). - **Bad**: The preview cache is cleared and reloaded often, causing flickering on common interactions (e.g. undo/redo). - **Bad**: The preview cache fails miserably with large asset libraries and causes continuous freezes, making the asset browser unusable with large asset libraries. **Rest of the UI:** (Asset views, the node add menu, the node search menu, asset-view template, etc.) - Challenge: Usually use the "All" asset library, meaning all assets are loaded when showing such UIs. - **Good**: Memory is shared between these UIs (static storage), so libraries are only loaded and stored once. (Asset browser storage is still separate). - **Bad**: No lazy loading of previews, all previews are loaded into memory - **Bad**: Only freed when closing the file. - **Bad**: Previews are also loaded when none are displayed (e.g. node "Add" menu) --- ## How to solve it? ### Good solution characteristics A good solution should give us the following: - Only hold asset previews in storaged that are in view of the user - Don't keep asset libraries in storage that the user isn't interacting with (at least not regularly, see next point) - Provide fast repetetive access to asset libraries (e.g. assets shouldn't be reloaded every time the "Add" menu is called in the geometry nodes editor) - Support shared storage. All UIs giving access to assets should be able to access the same static asset library and preview image storage. (Further improvements are possible still, see below) ### Lazy Loading Previews Previews should always be lazy loaded based on which assets the user sees, and only if the previews are actually displayed (e.g. not for the node "Add" menu). #109234 does most of the necessary work for this. ### Garbage Collector? A garbage collector could be implemented to remove unused asset libraries and asset previews from storage. Some kind of user-counting will be needed to detect the unused state, even when multiple parts of the UI display the same asset library (and thus may share storage). By being time based rather than user-count based only, the garbage collector still gives enough time for repetitive access to asset libraries. For example the asset library may still be in storage for multiple calls to the node "Add" menu, but be freed from storage if it hasn't been called for at least 5 minutes. Note that this doesn't need to be asset system specific. There are other UI previews that we load from disk that could make use of this. So this could be a more general UI garbage collector. --- ## Possible Further Improvements ## Partial asset library loading? Users rarely ever look at all assets of a library. They typically look at assets of a certain type (e.g. in the node "Add" menu), a specific catalog, or the view simply only has space for a subset of the available assets. So this is another case where we can avoid keeping data in storage, e.g. by lazy loading assets or deleting them once they are recognized as irrelevant. Keeping track of which assets are loaded and which have to be made available can be finicky. Plus we may run into issues when asset library contents were updated in-between partial reads. ## Lazy load extended metadata In most cases, all the user sees from an asset is the name, and potentially the preview. Keeping all the asset metadata in storage is a bit of a waste. We could lazy load this as well, although this can become finicky too, since some metadata needs to be accessed faster or more frequently (e.g. basic type information, tags for search, etc.) than other. ## Compressed preview buffers Preview buffers could be compressed to reduce the memory footprint of each, potentially leading to faster load times too.

Julian Eisel added the

Download

What's New

Blender Studio

Manual

Developers Blog

Documentation

Benchmark

Blender Conference

Development Fund

One-time Donations

Asset System & UI: Improve memory management for assets and previews #111614