USD: Import and export custom properties #118938

Merged
Jesse Yurkovich merged 15 commits from CharlesWardlaw/blender:feature/custom_properties into main 2024-04-23 19:27:52 +02:00

Adding support for converting between Blender custom properties and
USD user-defined custom attributes. Custom attributes on Xforms, many
data types, and materials are all supported for round-tripping.

Please see the USD attributes documentation for more information on custom attributes.

Properties are exported with a userProperties: namespace for simple
filtering in external apps. This namespace is stripped on import,
but other namespace are allowed to persist.

  • An "Import Attributes" parameter has been added with options "None" (do not import
    attributes), "User" (import attributes in the 'userProperties' namespace only), "All custom"
    (import all USD custom attributes, the default).

  • An "Export Custom Properties" export option has been added.

  • The property conversion code handles float, double, string and bool types, as well as
    tuples of size 2, 3 and 4. Note that USD quaternions and arrays of arbitrary length are
    not yet supported.

  • There is currently no attempt to set the Blender property subtype based on the USD
    type "role" (e.g., specifying Color or XYZ vector subtypes). This can be addressed in
    future work.

  • In addition to exporting custom properties, the original Blender object and data names
    are now saved as USD custom string attributes "userProperties:blenderName:object" and
    "userProperties:blenderName:data", respectively, on the corresponding USD prims. This
    feature is enabled with the "Author Blender Name" export option.

  • If a Blender custom string property is named "displayName", it's handled in a special way on
    export in that its value is used to set the USD prim's "displayName" metadata.

Testing

The attached custom_attr.usda file contains custom attributes of type asset, string, bool, int, double, float and half and tuples of various sizes, types and roles (e.g., colors, vectors and texture coordinates) which should successfully import as Blender custom properties of subtype "Plain Data".

Adding support for converting between Blender custom properties and USD user-defined custom attributes. Custom attributes on Xforms, many data types, and materials are all supported for round-tripping. Please see the [USD attributes documentation](https://openusd.org/release/api/class_usd_attribute.html#details) for more information on custom attributes. Properties are exported with a userProperties: namespace for simple filtering in external apps. This namespace is stripped on import, but other namespace are allowed to persist. - An "Import Attributes" parameter has been added with options "None" (do not import attributes), "User" (import attributes in the 'userProperties' namespace only), "All custom" (import all USD custom attributes, the default). - An "Export Custom Properties" export option has been added. - The property conversion code handles float, double, string and bool types, as well as tuples of size 2, 3 and 4. Note that USD quaternions and arrays of arbitrary length are not yet supported. - There is currently no attempt to set the Blender property subtype based on the USD type "role" (e.g., specifying Color or XYZ vector subtypes). This can be addressed in future work. - In addition to exporting custom properties, the original Blender object and data names are now saved as USD custom string attributes "userProperties:blenderName:object" and "userProperties:blenderName:data", respectively, on the corresponding USD prims. This feature is enabled with the "Author Blender Name" export option. - If a Blender custom string property is named "displayName", it's handled in a special way on export in that its value is used to set the USD prim's "displayName" metadata. ### Testing The attached `custom_attr.usda` file contains custom attributes of type `asset`, `string`, `bool`, `int`, `double`, `float` and `half` and tuples of various sizes, types and roles (e.g., colors, vectors and texture coordinates) which should successfully import as Blender custom properties of subtype "Plain Data".
Charles Wardlaw added the
Interest
USD
label 2024-02-29 23:22:57 +01:00
Charles Wardlaw self-assigned this 2024-02-29 23:22:57 +01:00
Charles Wardlaw added this to the Pipeline, Assets & IO project 2024-02-29 23:23:11 +01:00
Charles Wardlaw force-pushed feature/custom_properties from 181ad1abcc to 3dacbcd3e5 2024-02-29 23:35:14 +01:00 Compare
Charles Wardlaw requested review from Michael Kowalski 2024-02-29 23:35:34 +01:00
Charles Wardlaw requested review from Jesse Yurkovich 2024-02-29 23:35:42 +01:00
Michael Kowalski added 2 commits 2024-04-13 01:50:47 +02:00
be905f1342 Merge branch 'main' into feature/custom_properties
Note that this commit won't compile due to unresolved merge conflicts.
Michael Kowalski changed title from USD: Added the ability to import and export custom properties. Xforms, many data types, and materials are all supported for round-tripping. to USD: Import and export custom properties 2024-04-15 21:52:14 +02:00

Per discussion with @CharlesWardlaw, I'm extending the current implementation to handle a wider range of USD attribute types. I will mark the pull request as WIP until I've completed these changes.

Per discussion with @CharlesWardlaw, I'm extending the current implementation to handle a wider range of USD attribute types. I will mark the pull request as WIP until I've completed these changes.
Michael Kowalski changed title from USD: Import and export custom properties to WIP: USD: Import and export custom properties 2024-04-16 17:46:50 +02:00
Michael Kowalski added 2 commits 2024-04-17 01:30:56 +02:00
698b47a3e4 USD: Support additional custom attribute types
Now handling bool, asset, half and arbitrary tuple types
on import and bool type on export.

Added support for bool, asset, half and arbitrary tuple types on import and bool type on export.

Added support for bool, asset, half and arbitrary tuple types on import and bool type on export.
Michael Kowalski added 4 commits 2024-04-18 01:25:42 +02:00
Michael Kowalski reviewed 2024-04-18 23:29:59 +02:00
@ -241,2 +241,4 @@
object_ = BKE_object_add_only_object(bmain, OB_MESH, name_.c_str());
object_->data = mesh;
set_props(&object_->id, prim_, use_parent_xform());

I noticed that this call to set_props() in create_object() here and in the other reader implementations can cause a bug, in that if use_parent_xform() evaluates to false then the prim properties are duplicated on both the object and its data. E.g., if you load the custom_attr.usda you can see the duplicated properties.

I think the solution is to combine the functionality currently in USDPrimReader::read_object_data() and USDPrimReaader::set_props() into a single function and call it from USDXformReader::read_object_data().

I will commit this fix shortly.

I noticed that this call to `set_props()` in `create_object()` here and in the other reader implementations can cause a bug, in that if `use_parent_xform()` evaluates to false then the prim properties are duplicated on both the object and its data. E.g., if you load the `custom_attr.usda` you can see the duplicated properties. I think the solution is to combine the functionality currently in `USDPrimReader::read_object_data()` and `USDPrimReaader::set_props()` into a single function and call it from ` USDXformReader::read_object_data()`. I will commit this fix shortly.
makowalski marked this conversation as resolved
Michael Kowalski reviewed 2024-04-18 23:34:49 +02:00
@ -164,0 +167,4 @@
void set_props(ID *id,
const pxr::UsdPrim &prim,
const bool use_parent = false,
std::optional<double> motionSampleTime = std::nullopt);

IMHO, using std::optional here and elsewhere might not be necessary as it can be easily replaced with parameter pxr::UsdTimeCode motionSampleTime = pxr::UsdTimeCode::Default(). I'll make this change, but am certainly open to discussing this and we can change it back if necessary.

IMHO, using `std::optional` here and elsewhere might not be necessary as it can be easily replaced with parameter `pxr::UsdTimeCode motionSampleTime = pxr::UsdTimeCode::Default()`. I'll make this change, but am certainly open to discussing this and we can change it back if necessary.
makowalski marked this conversation as resolved
Michael Kowalski added 1 commit 2024-04-19 00:02:30 +02:00
buildbot/vexp-code-patch-lint Build done. Details
buildbot/vexp-code-patch-darwin-x86_64 Build done. Details
buildbot/vexp-code-patch-darwin-arm64 Build done. Details
buildbot/vexp-code-patch-linux-x86_64 Build done. Details
buildbot/vexp-code-patch-windows-amd64 Build done. Details
buildbot/vexp-code-patch-coordinator Build done. Details
02f71094ed
USD: fix duplicated custom properties on import
Merged the functionaliy in USDPrimReader::read_object_data() into
USDPrimReaader::set_props(), which is now called from
USDXformReader::read_object_data().  The new logic avoids duplicating
prim properties on the object when USDXformReader::use_parent_xform()
returns false.
Michael Kowalski changed title from WIP: USD: Import and export custom properties to USD: Import and export custom properties 2024-04-19 00:07:23 +02:00

@deadpin I just removed the WIP tag, as I believe this PR is now ready for review.

@deadpin I just removed the WIP tag, as I believe this PR is now ready for review.
Michael Kowalski reviewed 2024-04-19 00:43:36 +02:00
Michael Kowalski left a comment
Member

@CharlesWardlaw I'm done with the revisions I had in mind and have a couple of questions/comments. I welcome your feedback. Thanks!

@CharlesWardlaw I'm done with the revisions I had in mind and have a couple of questions/comments. I welcome your feedback. Thanks!
@ -214,1 +236,4 @@
relative_paths,
add_properties_namespace,
export_custom_properties,
author_blender_name,

I don't believe 'add_properties_namespace' and 'author_blender_name' are actually exposed in the UI. I wonder if they should be. For example, I can imagine cases where automatically adding the 'userProperties' namespace might not be desirable, so maybe this behavior should be optional. I'm open to other opinions about this, of course.

I don't believe 'add_properties_namespace' and 'author_blender_name' are actually exposed in the UI. I wonder if they should be. For example, I can imagine cases where automatically adding the 'userProperties' namespace might not be desirable, so maybe this behavior should be optional. I'm open to other opinions about this, of course.

Actually, I thought that all custom DCC properties should be namespaced under userProperties - so in that sense we should just always do it? Are there really cases they shouldn't be?
https://openusd.org/release/glossary.html?highlight=userproperties#user-properties

Looks like author_blender_name is exposed in the UI though.

Actually, I thought that all custom DCC properties should be namespaced under `userProperties` - so in that sense we should just always do it? Are there really cases they shouldn't be? https://openusd.org/release/glossary.html?highlight=userproperties#user-properties Looks like `author_blender_name` is exposed in the UI though.

Ah, author_blender_name is there, indeed. Not sure how my text search missed it. Thanks for the correction.

You're right that adding the userProperties namespace is standard, but I do recall seeing some cases where it wasn't used. Also, we allow importing custom attributes without this namespace.

Regardless, I don't think this is a big issue, so probably leaving the current behavior as is would be fine. In that case, we should maybe remove the add_properties_namespace option from the code altogether. I can go either way on this.

Ah, `author_blender_name` is there, indeed. Not sure how my text search missed it. Thanks for the correction. You're right that adding the `userProperties` namespace is standard, but I do recall seeing some cases where it wasn't used. Also, we allow importing custom attributes without this namespace. Regardless, I don't think this is a big issue, so probably leaving the current behavior as is would be fine. In that case, we should maybe remove the `add_properties_namespace` option from the code altogether. I can go either way on this.

I'll resolve this conversation as I'm fine with always adding the namespace.

It would be cleaner if we remove the add_properties_namespace option if it's always true, but I don't feel strongly about this.

I'll resolve this conversation as I'm fine with always adding the namespace. It would be cleaner if we remove the `add_properties_namespace` option if it's always true, but I don't feel strongly about this.
makowalski marked this conversation as resolved
@ -833,0 +890,4 @@
"attr_import_mode",
rna_enum_usd_attr_import_mode_items,
USD_ATTR_IMPORT_ALL,
"Import Attributes",

I wonder whether we should rename this "Import Custom Attributes" to make it very clear that this only applies to custom attributes and not primvars. Likewise, maybe the related enums should have "custom" in the name (e.g., rename USD_ATTR_IMPORT_NONE to USD_CUSTOM_ATTR_IMPORT_NONE). But am open to feedback on this, as maybe this is already clear.

I wonder whether we should rename this "Import Custom Attributes" to make it very clear that this only applies to custom attributes and not primvars. Likewise, maybe the related enums should have "custom" in the name (e.g., rename USD_ATTR_IMPORT_NONE to USD_CUSTOM_ATTR_IMPORT_NONE). But am open to feedback on this, as maybe this is already clear.

Maybe "Import Custom Properties" since "attributes" can mean something else in Blender terminology?

Maybe "Import Custom Properties" since "attributes" can mean something else in Blender terminology?

Yes, I think "import Custom Properties" is good.

Yes, I think "import Custom Properties" is good.

I'm fine with either "Import Custom Properties" or "Import Custom Attributes". I don't feel strongly about either option.

I'm fine with either "Import Custom Properties" or "Import Custom Attributes". I don't feel strongly about either option.

Let's go with "Properties" in general since that is the term used in Blender for these and since they'll end up as "userProperties" inside the USD file itself.

Let's go with "Properties" in general since that is the term used in Blender for these and since they'll end up as "userProperties" inside the USD file itself.
deadpin marked this conversation as resolved

@blender-bot build

@blender-bot build
Michael Kowalski added 2 commits 2024-04-19 02:54:06 +02:00
buildbot/vexp-code-patch-lint Build done. Details
buildbot/vexp-code-patch-linux-x86_64 Build done. Details
buildbot/vexp-code-patch-darwin-x86_64 Build done. Details
buildbot/vexp-code-patch-darwin-arm64 Build done. Details
buildbot/vexp-code-patch-windows-amd64 Build done. Details
buildbot/vexp-code-patch-coordinator Build done. Details
7478cc5128
USD: fix linux/mac builds

@blender-bot build

@blender-bot build
Jesse Yurkovich requested changes 2024-04-19 03:42:50 +02:00
Dismissed
Jesse Yurkovich left a comment
Member

Seems to work pretty well! I'll run through a bit more round-trip tests tonight. I think the biggest piece of feedback is about the displayName processing and the boilerplate checks inside the writers.

Seems to work pretty well! I'll run through a bit more round-trip tests tonight. I think the biggest piece of feedback is about the `displayName` processing and the boilerplate checks inside the writers.
@ -9,0 +10,4 @@
#include "usd.hh"
#include "WM_types.hh"

Seems like using "DNA_object_types.h" instead of "WM_types.hh" would be more apropriate.

Seems like using "DNA_object_types.h" instead of "WM_types.hh" would be more apropriate.
CharlesWardlaw marked this conversation as resolved
@ -9,0 +15,4 @@
#include <pxr/usd/sdf/path.h>
#include <pxr/usd/usd/prim.h>
#include <optional>

Unused

Unused <optional>
CharlesWardlaw marked this conversation as resolved
@ -16,6 +16,7 @@
#include <pxr/usd/sdf/path.h>
#include <pxr/usd/usd/prim.h>
#include <optional>

Can get rid of this one now too.

Can get rid of this one now too.
CharlesWardlaw marked this conversation as resolved
@ -0,0 +8,4 @@
#include <iostream>
#include "BLI_utildefines.h"

Both iostream and utildefines are unused.

Both iostream and utildefines are unused.
CharlesWardlaw marked this conversation as resolved
@ -0,0 +181,4 @@
if (attr_names.size() > 2 && is_user_prop && attr_names[1] == "blenderName") {
/* Skip the deprecated userProperties:blenderName namespace attribs. */
continue;

How can they be deprecated already when we just added them? This seems to prevent the newly added author_blender_name option from working I think.

What are we hoping to achieve with the blenderName:object and blenderName:data properties to begin with?

How can they be deprecated already when we just added them? This seems to prevent the newly added `author_blender_name` option from working I think. What are we hoping to achieve with the `blenderName:object` and `blenderName:data` properties to begin with?

That comment is not accurate, as the properties are not deprecated, as you say. I think this was just an oversight.

An immediate use for these properties could be in the USD export hook, to help map USD prims back to original Blender object names. We might also at some point use these properties to restore original Blender object names on import, though we are not doing so now. Also, such names could be useful in a pipeline, if it matters what the original object names were.

I'm not sure it makes sense to create custom properties with such names on import in Blender, since that doesn't seem useful in and of itself, but I could be wrong.

That comment is not accurate, as the properties are not deprecated, as you say. I think this was just an oversight. An immediate use for these properties could be in the USD export hook, to help map USD prims back to original Blender object names. We might also at some point use these properties to restore original Blender object names on import, though we are not doing so now. Also, such names could be useful in a pipeline, if it matters what the original object names were. I'm not sure it makes sense to create custom properties with such names on import in Blender, since that doesn't seem useful in and of itself, but I could be wrong.
Author
Member

I think there are arguments in both directions. On import, Blender might mangle names and by having the original name metadata be round-trippable, we at least preserve the ability for external tools to track changes despite scene names being different.

I've removed the comment. I believe it was referring to something internal to the original implementation that we changed while testing.

I think there are arguments in both directions. On import, Blender might mangle names and by having the original name metadata be round-trippable, we at least preserve the ability for external tools to track changes despite scene names being different. I've removed the comment. I believe it was referring to something internal to the original implementation that we changed while testing.

Oh, I see. We're writing those names out but not explicitly doing anything internally to use them when read back in (at least today). I misunderstood and thought that we'd re-assign the actual blender object and data names ourselves during import based on these properties.

Oh, I see. We're writing those names out but not explicitly doing anything internally to use them when read back in (at least today). I misunderstood and thought that we'd re-assign the actual blender object and data names ourselves during import based on these properties.
deadpin marked this conversation as resolved
@ -42,8 +43,109 @@ static std::string get_mesh_active_uvlayer_name(const Object *ob)
return name ? name : "";
}
template<typename VECT>

Nit: I'd at least use USDT here, instead of VECT, as we use that term elsewhere and it helps to know that we should be feeding a USD type in here.

Nit: I'd at least use `USDT` here, instead of VECT, as we use that term elsewhere and it helps to know that we should be feeding a USD type in here.
CharlesWardlaw marked this conversation as resolved
@ -168,0 +297,4 @@
}
if (id.properties) {
write_user_properties(prim, (IDProperty *)id.properties, timecode);

Nit: The cast to IDProperty * is redundant

Nit: The cast to IDProperty * is redundant
CharlesWardlaw marked this conversation as resolved
@ -168,0 +315,4 @@
const StringRef displayName_identifier = "displayName";
IDProperty *prop;

Can declare prop inside the for loop

Can declare `prop` inside the for loop
CharlesWardlaw marked this conversation as resolved
@ -70,0 +73,4 @@
void write_user_properties(pxr::UsdPrim &prim,
IDProperty *properties,
pxr::UsdTimeCode = pxr::UsdTimeCode::Default());

Both of the new methods can be const and can take in prim as a const reference.

Both of the new methods can be `const` and can take in `prim` as a const reference.
CharlesWardlaw marked this conversation as resolved
@ -100,0 +100,4 @@
if (usd_export_context_.export_params.export_custom_properties && camera) {
auto prim = usd_camera.GetPrim();
write_id_properties(prim, camera->id, timecode);

The write_id_properties function already checks the export params for other criteria, can the check for export_custom_properties be moved inside there so all these callers don't have to have boilerplate checks? Similarly, checking camera isn't useful here because we would have died already if camera is null. The same applies for the other writers I think.

The `write_id_properties` function already checks the export params for other criteria, can the check for `export_custom_properties` be moved inside there so all these callers don't have to have boilerplate checks? Similarly, checking `camera` isn't useful here because we would have died already if camera is null. The same applies for the other writers I think.
Author
Member

I addressed the call sites-- this made the code a bit more concise. Thanks for the suggestion.

I addressed the call sites-- this made the code a bit more concise. Thanks for the suggestion.
deadpin marked this conversation as resolved
Charles Wardlaw added 1 commit 2024-04-19 18:38:23 +02:00
Charles Wardlaw added 1 commit 2024-04-19 22:32:22 +02:00
Author
Member

"namespace" not "deadspace" 🤣

"namespace" not "deadspace" 🤣
Michael Kowalski approved these changes 2024-04-20 00:27:41 +02:00
Michael Kowalski left a comment
Member

I'm approving this pull request now, as I'll be out all of next week.

There are a couple of minor questions in io_usd.cc:

  • It would be cleaner if we remove the add_properties_namespace option if it's always true.
  • I think it would probably be clearer if the Import Attributes option is renamed Import Custom Attributes or Import Custom Properties.

However, I don't feel strongly as to how these questions are resolved, so I'll leave that up to @deadpin and @CharlesWardlaw to decide on whether to follow up on these points.

I'm approving this pull request now, as I'll be out all of next week. There are a couple of minor questions in `io_usd.cc`: - It would be cleaner if we remove the `add_properties_namespace` option if it's always true. - I think it would probably be clearer if the `Import Attributes` option is renamed `Import Custom Attributes` or `Import Custom Properties`. However, I don't feel strongly as to how these questions are resolved, so I'll leave that up to @deadpin and @CharlesWardlaw to decide on whether to follow up on these points.
Jesse Yurkovich reviewed 2024-04-22 21:30:51 +02:00
@ -168,0 +282,4 @@
return;
}
if (!id.properties) {

This conditional should be removed. It's checked below and it prevents data_name from being written e.g. for the default cube.

This conditional should be removed. It's checked below and it prevents data_name from being written e.g. for the default cube.
CharlesWardlaw marked this conversation as resolved
Charles Wardlaw added 1 commit 2024-04-22 22:47:12 +02:00
buildbot/vexp-code-patch-lint Build done. Details
buildbot/vexp-code-patch-linux-x86_64 Build done. Details
buildbot/vexp-code-patch-darwin-x86_64 Build done. Details
buildbot/vexp-code-patch-windows-amd64 Build done. Details
buildbot/vexp-code-patch-darwin-arm64 Build done. Details
buildbot/vexp-code-patch-coordinator Build done. Details
0792862c73
A few final renamings, and removed add_properties_namespace option.
Jesse Yurkovich approved these changes 2024-04-23 00:25:15 +02:00
Jesse Yurkovich left a comment
Member

Looks all good from what I can see.

Looks all good from what I can see.

@blender-bot build

@blender-bot build
Jesse Yurkovich merged commit 60bc34b494 into main 2024-04-23 19:27:52 +02:00
Sign in to join this conversation.
No Label
Interest
Alembic
Interest
Animation & Rigging
Interest
Asset Browser
Interest
Asset Browser Project Overview
Interest
Audio
Interest
Automated Testing
Interest
Blender Asset Bundle
Interest
BlendFile
Interest
Collada
Interest
Compatibility
Interest
Compositing
Interest
Core
Interest
Cycles
Interest
Dependency Graph
Interest
Development Management
Interest
EEVEE
Interest
EEVEE & Viewport
Interest
Freestyle
Interest
Geometry Nodes
Interest
Grease Pencil
Interest
ID Management
Interest
Images & Movies
Interest
Import Export
Interest
Line Art
Interest
Masking
Interest
Metal
Interest
Modeling
Interest
Modifiers
Interest
Motion Tracking
Interest
Nodes & Physics
Interest
OpenGL
Interest
Overlay
Interest
Overrides
Interest
Performance
Interest
Physics
Interest
Pipeline, Assets & IO
Interest
Platforms, Builds & Tests
Interest
Python API
Interest
Render & Cycles
Interest
Render Pipeline
Interest
Sculpt, Paint & Texture
Interest
Text Editor
Interest
Translations
Interest
Triaging
Interest
Undo
Interest
USD
Interest
User Interface
Interest
UV Editing
Interest
VFX & Video
Interest
Video Sequencer
Interest
Virtual Reality
Interest
Vulkan
Interest
Wayland
Interest
Workbench
Interest: X11
Legacy
Blender 2.8 Project
Legacy
Milestone 1: Basic, Local Asset Browser
Legacy
OpenGL Error
Meta
Good First Issue
Meta
Papercut
Meta
Retrospective
Meta
Security
Module
Animation & Rigging
Module
Core
Module
Development Management
Module
EEVEE & Viewport
Module
Grease Pencil
Module
Modeling
Module
Nodes & Physics
Module
Pipeline, Assets & IO
Module
Platforms, Builds & Tests
Module
Python API
Module
Render & Cycles
Module
Sculpt, Paint & Texture
Module
Triaging
Module
User Interface
Module
VFX & Video
Platform
FreeBSD
Platform
Linux
Platform
macOS
Platform
Windows
Priority
High
Priority
Low
Priority
Normal
Priority
Unbreak Now!
Status
Archived
Status
Confirmed
Status
Duplicate
Status
Needs Info from Developers
Status
Needs Information from User
Status
Needs Triage
Status
Resolved
Type
Bug
Type
Design
Type
Known Issue
Type
Patch
Type
Report
Type
To Do
No Milestone
No Assignees
3 Participants
Notifications
Due Date
The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: blender/blender#118938
No description provided.