AssetBrowser: Indexing Asset Library #91406

New Issue

Jeroen Bakker · 2021-09-14T15:41:43+02:00

Jeroen Bakker commented

2021-09-14 15:41:43 +02:00

Goal

make it possible to read assets + meta data that is used by the asset browser really fast.

Technical Requirements

Index should be stored on local file system. Asset libraries can be stored on a shared and having the index local would speed up. (.cache folder)
Index should be a file per asset file.
Index is refreshed when the asset file has changed or when triggered via an refresh operator.
Index item contains library name + asset name + meta?
Data stored in the Index should readable (stored in a textual format Expected JSON).
Library full path Version number of the index format.
Per Asset inside the asset file:* IDName (including object type prefix)
Catelog ID(s). Meta data.
*** icon hash? if it cannot be extracted from the data already present.
Index will be refreshes with file updates when the asset library is activated.

Out of scope
We consider storing icons as part of the index, but it has not that many benefits and needs a lot of space (70k per icon).

Areas in the code
AssetList::fetch triggers filelist_readjob_start.
file list type = FILE_LOADLIB for reading asset libraries stored.
FileListInternEntry is the data structure that we should be able to rebuild.

NOTE: When moving the indexing too close to the filelist_readjob_list_lib the indexing makes less sense performance wise. When moving it to close to AssetList there is a chance of code duplication and that indexes are rebuilt too many times (not per asset file, but per change in the asset library would trigger regenerating any index). The performance gain would be that we don't need to seek in the blend file...

Decisions

In assetlist valid indexes are read. An index is valid when the timestamp of the index matches the timestamp of the blend file.
From valid indexes the FileListInternItems are constructed. Invalid indexes are deleted.
When there are still files left that have no index, they will be read using FILE_LOADLIB, but only files that don't have an index are read.
After this the missing indexes are constructed from the read files.
When a file has no assets, an empty index is generated so we can still detect on the outside. We could even skip the reading of the index based on the size of the index.
When the asset library is renamed it should still use the index files.
In the cache folder it should be clear by which asset library the index belongs to. /Blender/Assets/<assetlib_hash>_<assetlib_readable_name/<folder>/<assetfile>.index.json for example.
Cache files are deleted when asset library is removed.
Cache files are relocated when asset libraries are renamed.
Cache files are remove when asset file is deleted
The index should be able to construct a BLODataBlockInfo struct.
The source for indexing should be located in assetlib code. Use callback(struct) for clean code.

Changes

Edit editors/space_file/... add index callback functions/struct. Perhaps even in the job?
Add: editors/assets/intern/asset_index.cc

**Goal** * make it possible to read assets + meta data that is used by the asset browser really fast. **Technical Requirements** * Index should be stored on local file system. Asset libraries can be stored on a shared and having the index local would speed up. (`.cache` folder) * Index should be a file per asset file. * Index is refreshed when the asset file has changed or when triggered via an refresh operator. * Index item contains library name + asset name + meta? * Data stored in the Index should readable (stored in a textual format Expected `JSON`). **Library full path** Version number of the index format. **Per Asset inside the asset file:*** IDName (including object type prefix) ***Catelog ID(s).*** Meta data. *** icon hash? if it cannot be extracted from the data already present. * Index will be refreshes with file updates when the asset library is activated. **Out of scope** We consider storing icons as part of the index, but it has not that many benefits and needs a lot of space (70k per icon). **Areas in the code** `AssetList::fetch` triggers `filelist_readjob_start`. `file list type = FILE_LOADLIB` for reading asset libraries stored. `FileListInternEntry` is the data structure that we should be able to rebuild. NOTE: When moving the indexing too close to the `filelist_readjob_list_lib` the indexing makes less sense performance wise. When moving it to close to `AssetList` there is a chance of code duplication and that indexes are rebuilt too many times (not per asset file, but per change in the asset library would trigger regenerating any index). The performance gain would be that we don't need to `seek` in the blend file... # Decisions * In assetlist valid indexes are read. An index is valid when the timestamp of the index matches the timestamp of the blend file. * From valid indexes the FileListInternItems are constructed. Invalid indexes are deleted. * When there are still files left that have no index, they will be read using `FILE_LOADLIB`, but only files that don't have an index are read. * After this the missing indexes are constructed from the read files. * When a file has no assets, an empty index is generated so we can still detect on the outside. We could even skip the reading of the index based on the size of the index. * When the asset library is renamed it should still use the index files. * In the cache folder it should be clear by which asset library the index belongs to. `/Blender/Assets/<assetlib_hash>_<assetlib_readable_name/<folder>/<assetfile>.index.json` for example. * Cache files are deleted when asset library is removed. * Cache files are relocated when asset libraries are renamed. * Cache files are remove when asset file is deleted * The index should be able to construct a `BLODataBlockInfo` struct. * The source for indexing should be located in assetlib code. Use callback(struct) for clean code. # Changes * Edit `editors/space_file/...` add index callback functions/struct. Perhaps even in the job? * Add: `editors/assets/intern/asset_index.cc`

Jeroen Bakker self-assigned this 2021-09-14 15:41:43 +02:00

Jeroen Bakker commented

2021-09-14 15:41:43 +02:00

Changed status from 'Needs Triage' to: 'Confirmed'

Jeroen Bakker commented

2021-09-14 15:41:43 +02:00

Added subscribers: @Jeroen-Bakker, @JulianEisel

Julian Eisel commented

2021-09-14 16:00:42 +02:00

what will happen when the library is updated at the same time it is displayed in a different blender instance. I would expect that this is detected and the index would be refreshed

Current design of the Asset Browser is: While showing any library that is not the Current File one, there always has to be an explicit refresh of the Asset Browser to get changes in the library to show up. I'd keep the same design here, we read the asset library data once and that's what the user gets to work with, until a reload is triggered.

> what will happen when the library is updated at the same time it is displayed in a different blender instance. I would expect that this is detected and the index would be refreshed Current design of the Asset Browser is: While showing any library that is not the *Current File* one, there always has to be an explicit refresh of the Asset Browser to get changes in the library to show up. I'd keep the same design here, we read the asset library data once and that's what the user gets to work with, until a reload is triggered.

Julian Eisel commented

2021-09-18 17:44:16 +02:00

Index is refreshed when the asset file has changed or when triggered via an refresh operator.

I'm not sure what this means. I wouldn't expect the Asset Browser to refresh assets when an asset file is changed (detecting that isn't trivial). Or are you just talking about updating the index when a change is found while reading the library? Plus an extra operator to entirely reload the asset library index?

Can an asset have multiple catelog IDs?

No, in the current design assets can be in only one catalog.

> Index is refreshed when the asset file has changed or when triggered via an refresh operator. I'm not sure what this means. I wouldn't expect the Asset Browser to refresh assets when an asset file is changed (detecting that isn't trivial). Or are you just talking about updating the index when a change is found *while reading the library*? Plus an extra operator to entirely reload the asset library index? > Can an asset have multiple catelog IDs? No, in the current design assets can be in only one catalog.

blender-admin commented

2021-09-24 08:39:58 +02:00

This issue was referenced by fc7beac8d6

This issue was referenced by fc7beac8d6f45d9eca344ec4ae8879c2e73f0731

blender-admin commented

2021-10-12 08:42:08 +02:00

This issue was referenced by 70fd6a313e

This issue was referenced by 70fd6a313e70ead89f94090d1de3d90032f778d5

blender-admin commented

2021-11-24 08:20:18 +01:00

This issue was referenced by dbeab82a89

This issue was referenced by dbeab82a89c4979b7e9e62e740938546394eca3e

RedMser commented

2021-12-25 15:52:14 +01:00

Added subscriber: @RedMser

slowburn commented

2021-12-28 22:18:40 +01:00

Added subscriber: @slowburn

Maciej Morgaś commented

2022-01-13 09:50:27 +01:00

Added subscriber: @MaciejMorgas

Jeroen Bakker removed their assignment 2022-01-19 17:54:12 +01:00

Jeroen Bakker commented

2022-01-19 17:54:12 +01:00

Core stuff is in master. The rest don't have a high prio. And can be taken over by community/module.

Julian Eisel commented

2022-02-01 17:31:30 +01:00

Changed status from 'Confirmed' to: 'Resolved'

Julian Eisel closed this issue

2022-02-01 17:31:30 +01:00

Julian Eisel self-assigned this 2022-02-01 17:31:30 +01:00

Julian Eisel commented

2022-02-01 17:31:30 +01:00

I'm going to close these parent tasks since there are just two smaller tasks remaining which can stand on their own, the parent tasks don't add much value (but may be confusing).

slowburn commented

2022-02-08 14:31:26 +01:00

Does this mean that indexing should be working for end user already? Because I'm not seeing any improvement compared to earlier builds.

Deanna Greyoak commented

2022-03-25 17:53:14 +01:00

Added subscriber: @Deanna

Sign in to join this conversation.

No Label

Download

What's New

Blender Studio

Manual

Developers Blog

Documentation

Benchmark

Blender Conference

Development Fund

One-time Donations

AssetBrowser: Indexing Asset Library #91406

Decisions

Changes