Refactor: Retrieve node operator asset with property #110018

Merged

Hans Goudey merged 11 commits from HooglyBoogly/blender:node-group-operator-path-prop into main

2023-08-01 14:55:07 +02:00

Hans Goudey commented

2023-07-12 18:28:01 +02:00

Member

In order to more reliably pass the asset to the operator, use operator
properties making up the asset path instead of using a context pointer.
This is required to support the quick favorites menu, shortcuts, and
to draw operator inputs in the redo panel. In all of those situations
the original context isn't available. This also feels safer, since we
rely less on storing pointers to data with a less-defined lifetime.

In order to more reliably pass the asset to the operator, use operator properties making up the asset path instead of using a context pointer. This is required to support the quick favorites menu, shortcuts, and to draw operator inputs in the redo panel. In all of those situations the original context isn't available. This also feels safer, since we rely less on storing pointers to data with a less-defined lifetime.

Hans Goudey added 1 commit 2023-07-12 18:28:09 +02:00

0ac84d8fbd Refactor: Retrieve node operator asset with property

In order to more reliably pass the asset to the operator, use a string
property with the full asset path instead of using a context pointer.
This is required to support the quick favorites menu, shortcuts, and
to draw operator inputs in the redo panel. In all of those situations
the original context isn't available. This also feels safer, since we
rely less on storing pointers to data with a less-defined lifetime.

Hans Goudey requested review from Julian Eisel 2023-07-12 18:28:17 +02:00

Hans Goudey added this to the Nodes & Physics project 2023-07-12 18:28:24 +02:00

Hans Goudey added the

Nodes & Physics

labels 2023-07-12 18:28:39 +02:00

Julian Eisel commented

2023-07-13 15:01:17 +02:00

Member

Loading all asset libraries (possibly 1000s of assets) just to get a single asset doesn't sound like a good idea. For the node add menu and such we accept that, but only because we keep requiring quick access to them (this should be improved still though, so that unused assets are freed or garbage collected somehow). It also supports asynchronous loading, which isn't handled properly by the operator.

The operator could expect the library in question to be loaded already (static AssetLibrary::foreach_loaded() can be used then). Or we let the operator know about the asset library as well and let the asset system load the library in a smarter way. For example by temporarily loading the asset library, or making it possible to load a single asset representation from a library.

Loading all asset libraries (possibly 1000s of assets) just to get a single asset doesn't sound like a good idea. For the node add menu and such we accept that, but only because we keep requiring quick access to them (this should be improved still though, so that unused assets are freed or garbage collected somehow). It also supports asynchronous loading, which isn't handled properly by the operator. The operator could expect the library in question to be loaded already (`static AssetLibrary::foreach_loaded()` can be used then). Or we let the operator know about the asset library as well and let the asset system load the library in a smarter way. For example by temporarily loading the asset library, or making it possible to load a single asset representation from a library.

Hans Goudey commented

2023-07-13 15:28:44 +02:00

Author

Member

Loading all asset libraries (possibly 1000s of assets) just to get a single asset doesn't sound like a good idea

I think all asset libraries will already be loaded already anyway (by build_catalog_tree), in order for the operator to be displayed in the menu in the first place. So I don't think this changes anything. Am I missing something?

>Loading all asset libraries (possibly 1000s of assets) just to get a single asset doesn't sound like a good idea I think all asset libraries will already be loaded already anyway (by `build_catalog_tree`), in order for the operator to be displayed in the menu in the first place. So I don't think this changes anything. Am I missing something?

Julian Eisel commented

2023-07-13 15:48:37 +02:00

Member

I think all asset libraries will already be loaded already anyway (by build_catalog_tree), in order for the operator to be displayed in the menu in the first place. So I don't think this changes anything. Am I missing something?

Where you call the operator the all asset library was already read, or at least is in the process of reading, yes. But this may not be the case when the operator is called from the quick favorites, for example. And there it will likely break still, since the operator invokes async loading of the library, but doesn't wait until the assets are actually loaded. If the operator does any loading (and I guess it has to if we want this to work from the quick favorites), it needs to wait until the asset is found or the loading completed.

> I think all asset libraries will already be loaded already anyway (by `build_catalog_tree`), in order for the operator to be displayed in the menu in the first place. So I don't think this changes anything. Am I missing something? Where you call the operator the all asset library was already read, or at least is in the process of reading, yes. But this may not be the case when the operator is called from the quick favorites, for example. And there it will likely break still, since the operator invokes async loading of the library, but doesn't wait until the assets are actually loaded. If the operator does any loading (and I guess it has to if we want this to work from the quick favorites), it needs to wait until the asset is found or the loading completed.

Julian Eisel requested changes 2023-07-13 16:24:46 +02:00

Julian Eisel left a comment

Member

Setting to changes requested because of the given reasons.

Setting to changes requested because of the given reasons.

Hans Goudey commented

2023-07-13 16:32:17 +02:00

Author

Member

Okay, that makes sense, thanks. In menus, crucially this asset will already be loaded if it's displayed. But yeah, if you call the operator through quick favorites or shortcuts right when you enter the mode before the asset is loaded, the call would fail.

I guess you're suggesting the operator call wait until the asset is loaded? That sounds reasonable to me!

As for accomplishing that, I'm not quite sure. Theoretically, to execute the operator, the asset isn't even necessary since we already have the node group's full path. But there is currently no way to "ensure imported" without the asset, because that behavior is tied to the AssetRepresentation class. So there are two options I think:

Separate the asset import behavior (including "append and reuse" etc) from the asset itself
- This probably requires storing more of the asset behavior like import method as operator properties
- We wouldn't be able to use AssetRepresentation::get_local_id(), but that hasn't been foolproof in my testing anyway, and we have BLO_LIBLINK_APPEND_LOCAL_ID_REUSE for that anyway.
Add a way to make loading the operator's asset blocking, but otherwise don't change the code.

I think I prefer option 1. How about you?

Okay, that makes sense, thanks. In menus, crucially this asset will already be loaded if it's displayed. But yeah, if you call the operator through quick favorites or shortcuts right when you enter the mode before the asset is loaded, the call would fail. I guess you're suggesting the operator call wait until the asset is loaded? That sounds reasonable to me! As for accomplishing that, I'm not quite sure. Theoretically, to execute the operator, the asset isn't even necessary since we already have the node group's full path. But there is currently no way to "ensure imported" without the asset, because that behavior is tied to the `AssetRepresentation` class. So there are two options I think: 1. Separate the asset import behavior (including "append and reuse" etc) from the asset itself - This probably requires storing more of the asset behavior like import method as operator properties - We wouldn't be able to use `AssetRepresentation::get_local_id()`, but that hasn't been foolproof in my testing anyway, and we have `BLO_LIBLINK_APPEND_LOCAL_ID_REUSE` for that anyway. 2. Add a way to make loading the operator's asset blocking, but otherwise don't change the code. I think I prefer option 1. How about you?

Julian Eisel commented

2023-07-13 17:23:18 +02:00

Member

I guess you're suggesting the operator call wait until the asset is loaded? That sounds reasonable to me!

Yes, although we could also load some temporary representation of the imported asset and replace that with the real one once done.

As for accomplishing that, I'm not quite sure. Theoretically, to execute the operator, the asset isn't even necessary since we already have the node group's full path. But there is currently no way to "ensure imported" without the asset, because that behavior is tied to the AssetRepresentation class. So there are two options I think:

Separate the asset import behavior (including "append and reuse" etc) from the asset itself

This probably requires storing more of the asset behavior like import method as operator properties

We wouldn't be able to use AssetRepresentation::get_local_id(), but that hasn't been foolproof in my testing anyway, and we have BLO_LIBLINK_APPEND_LOCAL_ID_REUSE for that anyway.

Add a way to make loading the operator's asset blocking, but otherwise don't change the code.

I think we need a design where you can request any asset to be imported through the asset system. The asset system design foresees non-ID assets and assets that are not loaded from disk, for example that need to be downloaded from a web server. So simple file path based link/append is too limiting.

So I guess (2) will be needed either way, since importing an asset may be an expensive operation that should be done asynchronously. And I think that means we still require an AssetRepresentation. How exactly we get that needs to be defined still, I see a few ways. I also thought earlier that we could assume the library is already loaded, but yeah, that won't work as mentioned.

> I guess you're suggesting the operator call wait until the asset is loaded? That sounds reasonable to me! Yes, although we could also load some temporary representation of the imported asset and replace that with the real one once done. > As for accomplishing that, I'm not quite sure. Theoretically, to execute the operator, the asset isn't even necessary since we already have the node group's full path. But there is currently no way to "ensure imported" without the asset, because that behavior is tied to the `AssetRepresentation` class. So there are two options I think: > 1. Separate the asset import behavior (including "append and reuse" etc) from the asset itself > - This probably requires storing more of the asset behavior like import method as operator properties > - We wouldn't be able to use `AssetRepresentation::get_local_id()`, but that hasn't been foolproof in my testing anyway, and we have `BLO_LIBLINK_APPEND_LOCAL_ID_REUSE` for that anyway. > 2. Add a way to make loading the operator's asset blocking, but otherwise don't change the code. I think we need a design where you can request any asset to be imported through the asset system. The asset system design foresees non-ID assets and assets that are not loaded from disk, for example that need to be downloaded from a web server. So simple file path based link/append is too limiting. So I guess (2) will be needed either way, since importing an asset may be an expensive operation that should be done asynchronously. And I think that means we still require an `AssetRepresentation`. How exactly we get that needs to be defined still, I see a few ways. I also thought earlier that we could assume the library is already loaded, but yeah, that won't work as mentioned.

Hans Goudey commented

2023-07-13 17:43:35 +02:00

Author

Member

Everything you say makes sense, we shouldn't make it too hard to add support for operator node groups that are stored in the cloud sometime in the future.

I'd appreciate some more specific advice about how this should move forward though; I'm not sure who you imagine working on that more generalized "import asset based on non-asset-representation-path in a blocking way" feature, or where it fits into the existing asset code.

Everything you say makes sense, we shouldn't make it too hard to add support for operator node groups that are stored in the cloud sometime in the future. I'd appreciate some more specific advice about how this should move forward though; I'm not sure who you imagine working on that more generalized "import asset based on non-asset-representation-path in a blocking way" feature, or where it fits into the existing asset code.

~~Hans Goudey referenced this pull request 2023-07-13 19:50:36 +02:00~~

Geometry Nodes: Draw node operator inputs in redo panel #109975

~~Hans Goudey referenced this pull request 2023-07-13 19:52:05 +02:00~~

Node Tools Overview #101778

Hans Goudey added 2 commits 2023-07-25 21:40:41 +02:00

9761531bf7 Merge branch 'main' into node-group-operator-path-prop

c70745bfc1 Add different warning when asset loading is unfinished

Hans Goudey commented

2023-07-25 21:49:02 +02:00

Author

Member

Discussed this in the node module meeting-- I added a different warning for when the asset wasn't found and asset libraries are still loading: Asset loading is unfinished. The operator just returns early in that case. In the end users a bit more flexibility this way anyway.

Discussed this in the node module meeting-- I added a different warning for when the asset wasn't found and asset libraries are still loading: `Asset loading is unfinished`. The operator just returns early in that case. In the end users a bit more flexibility this way anyway.

Hans Goudey requested review from Julian Eisel 2023-07-25 21:49:07 +02:00

Julian Eisel commented

2023-07-28 18:05:32 +02:00

Member

The absolute asset path may be okay, but it can still break as an asset library gets moved around, or in future project libraries, etc. It also won't work once we support assets that are not on disk (online asset libraries).

I think for now the best thing to is to use AssetWeakReference, which should be pretty simple:

The menu creates one using asset.make_weak_reference()
The operator can take its components as individual properties (asset library type eAssetLibraryType, asset library identifier and relative asset identifier).
The operator reconstructs the AssetWeakReference and gets the full path using AssetLibraryService::resolve_asset_weak_reference_to_full_path() (which probably has to be wrapped into a static AssetLibrary::resolve_asset_weak_reference_to_full_path()).
I think local IDs need special care since the resolving doesn't support these. So if the library type is ASSET_LIBRARY_LOCAL, just using the relative asset identifier and comparing that against asset->get_identifier().full_path() should work.

Longer term I hope we can simply replace this by a URI, but for now this is the closest and most future proof thing we have.

The absolute asset path may be okay, but it can still break as an asset library gets moved around, or in future project libraries, etc. It also won't work once we support assets that are not on disk (online asset libraries). I think for now the best thing to is to use `AssetWeakReference`, which should be pretty simple: - The menu creates one using `asset.make_weak_reference()` - The operator can take its components as individual properties (asset library type `eAssetLibraryType`, asset library identifier and relative asset identifier). - The operator reconstructs the `AssetWeakReference` and gets the full path using `AssetLibraryService::resolve_asset_weak_reference_to_full_path()` (which probably has to be wrapped into a `static AssetLibrary::resolve_asset_weak_reference_to_full_path()`). - I think local IDs need special care since the resolving doesn't support these. So if the library type is `ASSET_LIBRARY_LOCAL`, just using the relative asset identifier and comparing that against `asset->get_identifier().full_path()` should work. Longer term I hope we can simply replace this by a URI, but for now this is the closest and most future proof thing we have.

Hans Goudey added 8 commits 2023-07-31 18:14:09 +02:00

ec1ee38b89 Merge branch 'main' into node-group-operator-path-prop

11394026c4 Fix build error

b74852df21 Progress

9c92624234 Merge branch 'main' into node-group-operator-path-prop

ad067e1582 Merge branch 'main' into node-group-operator-path-prop

4607f14842 Progress

107f81ef95 Merge branch 'main' into node-group-operator-path-prop

2348afbd95 Split full path into three separate operator properties

Hans Goudey commented

2023-07-31 18:14:23 +02:00

Author

Member

Thanks for the clear description, that was very helpful. I've implemented what you described now.

Thanks for the clear description, that was very helpful. I've implemented what you described now.

Julian Eisel approved these changes 2023-08-01 11:52:33 +02:00

Julian Eisel left a comment

Member

Alright, LGTM now!

Alright, LGTM now!

Hans Goudey referenced this issue from a commit

2023-08-01 14:55:07 +02:00

Refactor: Retrieve node operator asset with property

Hans Goudey merged commit 65591b2c8c into main

2023-08-01 14:55:07 +02:00

Hans Goudey deleted branch node-group-operator-path-prop

2023-08-01 14:55:10 +02:00

Sign in to join this conversation.

No reviewers

No Label

Animation & Rigging

Asset Browser Project Overview

Automated Testing

Blender Asset Bundle

Dependency Graph

Development Management

EEVEE & Viewport

Images & Movies

Motion Tracking

Nodes & Physics

Pipeline, Assets & IO

Platforms, Builds & Tests

Render & Cycles

Render Pipeline

Sculpt, Paint & Texture

Video Sequencer

Virtual Reality

Blender 2.8 Project

Milestone 1: Basic, Local Asset Browser

Good First Issue

Animation & Rigging

Development Management

EEVEE & Viewport

Nodes & Physics

Pipeline, Assets & IO

Platforms, Builds & Tests

Render & Cycles

Sculpt, Paint & Texture

Needs Info from Developers

Needs Information from User

No Milestone

No project

No Assignees

2 Participants

Notifications

Due Date

The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: blender/blender#110018

No description provided.