AssetBrowser: Add IDProperties to Asset Indexer. #92306

New Issue

Jeroen Bakker · 2021-10-18T08:30:19+02:00

Jeroen Bakker commented

2021-10-18 08:30:19 +02:00

Object/collection asset workflow would need the bounding box for snapping.
The bounding box is stored inside an ID property. Currently ID properties aren't
stored in the asset index.

This task adds storing/reading ID properties as part of the asset index.

Approach

Custom Properties is a general data structure and the conversion between blender::io::serialize::Value should be accessible to other areas of blender as well.
There are two places where this could be added (BLI_serialize.h or BKE_idprops.h) I prefer it to be a part of BKE_idprops; reasoning is that BLI_serialize will be kept abstract and to the point.

NOTE: IDP_ID will not be supported. When passed these custom properties will be ignored. A warning will be logged.

Introduce blenkernel/intern/idprop_serialize.cc containing the data transformation methods.
Add test cases in blenkernel/intern/idprop_serialize_test.cc for data transformation between custom properties and blender::io::serialize::Value.
- We should add test cases for each specific custom property type.
- We could add a test case that randomly generates custom properties , transform them to serialize and back. Compare with input data.
Add CPP-only section to blenkernel/BKE_idprops.h.
Call the conversion methods from asset_indexer.cc.

Data Mapping

For data mapping we store the internal structure of IDProperty to the indexer (including meta-data) to be able to deserialize it back.

[
  {
    "name":  ..,
    "value": ..,
    "type": ..,
    /* `subtype` and `length` are only available for IDP_ARRAYs. */
    "subtype": ..,
    "length": ..,
  },
]

DNA	Serialize type	Note
IDProperty.name	StringValue
IDProperty.type	StringValue	"IDP_STRING", "IDP_INT", "IDP_FLOAT", "IDP_ARRAY", "IDP_GROUP", "IDP_DOUBLE"
IDProperty.subtype	StringValue	"IDP_INT", "IDP_FLOAT", "IDP_GROUP", "IDP_DOUBLE"
IDProperty.value	StringValue	When type is IDP_STRING
IDProperty.value	IntValue	When type is IDP_INT
IDProperty.value	DoubleValue	When type is IDP_FLOAT/IDP_DOUBLE
IDProperty.value	ArrayValue	When type is IDP_GROUP. Recursively uses the same structure as described in this section.
IDProperty.value	ArrayValue	When type is IDP_ARRAY. Each element holds a single element as described in this section.
IDProperty.length	IntValue

NOTE: IDP_ID and IDP_IDARRAY arent' supported. The entry will not be added.

Example

[
  {
    "name": "MyIntValue,
    "type": "IDP_INT",
    "value": 6,
  },
  {
    "name": "myComplexArray",
    "type": "IDP_ARRAY",
    "subtype": "IDP_GROUP",
    "length": 1,
    "value": [
        [
          {
            "name": ..
            ....
          }
        ]
    ]
  }
]

Alternatives

Add conversion functions inside asset_indexer; makes generic code part of a specific solution.
Add conversion functions inside BLI_serialize; would add data transformation responsibilities inside a unit that is currently only responsible for formatting.
Use direct mapping between IDP properties and Values; leads to missing information and edge cases (empty primitive arrays) that could not be deserialized.

Object/collection asset workflow would need the bounding box for snapping. The bounding box is stored inside an ID property. Currently ID properties aren't stored in the asset index. This task adds storing/reading ID properties as part of the asset index. ## Approach Custom Properties is a general data structure and the conversion between `blender::io::serialize::Value` should be accessible to other areas of blender as well. There are two places where this could be added (BLI_serialize.h or BKE_idprops.h) I prefer it to be a part of BKE_idprops; reasoning is that `BLI_serialize` will be kept abstract and to the point. NOTE: IDP_ID will not be supported. When passed these custom properties will be ignored. A warning will be logged. - Introduce `blenkernel/intern/idprop_serialize.cc` containing the data transformation methods. - Add test cases in `blenkernel/intern/idprop_serialize_test.cc` for data transformation between custom properties and `blender::io::serialize::Value`. - We should add test cases for each specific custom property type. - We could add a test case that randomly generates custom properties , transform them to serialize and back. Compare with input data. - Add CPP-only section to `blenkernel/BKE_idprops.h`. - Call the conversion methods from `asset_indexer.cc`. ## Data Mapping For data mapping we store the internal structure of IDProperty to the indexer (including meta-data) to be able to deserialize it back. ``` [ { "name": .., "value": .., "type": .., /* `subtype` and `length` are only available for IDP_ARRAYs. */ "subtype": .., "length": .., }, ] ``` | **DNA** | **Serialize type** | **Note** | | -- | -- | -- | | IDProperty.name | StringValue| | | IDProperty.type | StringValue| "IDP_STRING", "IDP_INT", "IDP_FLOAT", "IDP_ARRAY", "IDP_GROUP", "IDP_DOUBLE"| | IDProperty.subtype | StringValue| "IDP_INT", "IDP_FLOAT", "IDP_GROUP", "IDP_DOUBLE" | | IDProperty.value | StringValue | When type is IDP_STRING | | IDProperty.value | IntValue | When type is IDP_INT | | IDProperty.value | DoubleValue | When type is IDP_FLOAT/IDP_DOUBLE | | IDProperty.value | ArrayValue | When type is IDP_GROUP. Recursively uses the same structure as described in this section. | | IDProperty.value | ArrayValue | When type is IDP_ARRAY. Each element holds a single element as described in this section. | | IDProperty.length | IntValue| | NOTE: IDP_ID and IDP_IDARRAY arent' supported. The entry will not be added. Example ``` [ { "name": "MyIntValue, "type": "IDP_INT", "value": 6, }, { "name": "myComplexArray", "type": "IDP_ARRAY", "subtype": "IDP_GROUP", "length": 1, "value": [ [ { "name": .. .... } ] ] } ] ``` ## Alternatives - Add conversion functions inside `asset_indexer`; makes generic code part of a specific solution. - Add conversion functions inside `BLI_serialize`; would add data transformation responsibilities inside a unit that is currently only responsible for formatting. - Use direct mapping between IDP properties and Values; leads to missing information and edge cases (empty primitive arrays) that could not be deserialized.

Jeroen Bakker commented

2021-10-18 08:30:19 +02:00

Added subscriber: @Jeroen-Bakker

Jeroen Bakker self-assigned this 2021-10-19 09:04:07 +02:00

Julian Eisel commented

2021-10-19 14:16:34 +02:00

Added subscriber: @JulianEisel

Julian Eisel commented

2021-10-19 14:16:34 +02:00

Generally makes sense to me. I don't expect this to be that difficult, although I'm unsure about the UI data properties store, the design for that has changed recently.

IDP_UI_DATA_TYPE_ID will not be supported. When passed these custom properties will be ignored. A warning will be logged.

I guess you mean IDP_ID here?

Introduce blenkernel/intern/idprops.cc containing the data transformation methods.

Maybe call it idprops_serialize.cc? Just comment clearly that this doesn't include .blend file (de)serialization.

Generally makes sense to me. I don't expect this to be that difficult, although I'm unsure about the UI data properties store, the design for that has changed recently. > IDP_UI_DATA_TYPE_ID will not be supported. When passed these custom properties will be ignored. A warning will be logged. I guess you mean `IDP_ID` here? > Introduce blenkernel/intern/idprops.cc containing the data transformation methods. Maybe call it `idprops_serialize.cc`? Just comment clearly that this doesn't include .blend file (de)serialization.

Julian Eisel commented

2021-10-19 14:47:24 +02:00

I checked with Hans (who implemented the current UI data design of ID properties). There shouldn't be a need to store the UI data (min/max values, tooltip) in the index. This data will be attached to the property at runtime if it is registered in .py with these information bits. So I think we only need to write the name and value for each property?

Jeroen Bakker commented

2021-10-19 15:27:59 +02:00

In #92306#1238141, @JulianEisel wrote:
So I think we only need to write the name and value for each property?

That is my expectation also. We will see during implementation if it holds :-)

> In #92306#1238141, @JulianEisel wrote: > So I think we only need to write the name and value for each property? That is my expectation also. We will see during implementation if it holds :-)

Jeroen Bakker commented

2021-11-23 08:48:44 +01:00

Changed status from 'Needs Triage' to: 'Confirmed'

RedMser commented

2021-12-25 15:56:56 +01:00

Added subscriber: @RedMser

blender-admin commented

2022-01-18 11:12:02 +01:00

This issue was referenced by 36068487d0

This issue was referenced by 36068487d076bfd8320d9d6b05b69a6a8966c78b

Jeroen Bakker commented

2022-01-19 09:53:15 +01:00

Changed status from 'Confirmed' to: 'Resolved'

Jeroen Bakker closed this issue

2022-01-19 09:53:15 +01:00

Thomas Dinges added this to the 3.1 milestone 2023-02-08 15:53:05 +01:00

Sign in to join this conversation.

No Label

Download

What's New

Blender Studio

Manual

Developers Blog

Documentation

Benchmark

Blender Conference

Development Fund

One-time Donations

AssetBrowser: Add IDProperties to Asset Indexer. #92306

Approach

Data Mapping

Alternatives