USD: Support Python hooks and chasers to extend IO functionality #108825
This task is meant to initiate a discussion of design strategies for supporting Python hooks and callbacks (also known as
chasers) for extending USD import and export functionality. Although the current discussion will focus on USD IO in particular, the intent is to design an approach that can be generalized to support other formats as well.
As context, some DCCs allow extending the application's USD IO functionality through custom plugins. For example,
Solaris in Houdini supports authoring custom material shader translators as Python plugins . Maya’s USD plugin supports the concept of
chasers for import and export.
There have been previous discussions in the
pipeline-assets-io-module chat for supporting Python customization of Blender’s USD IO, as described here. In addition, a limited prototype was implemented in the
universal-scene-description branch for converting USD materials through Python callbacks. This prototype is currently being refactored and will be described in a separate design task in the near future.
Such hooks can be very useful for adding missing or experimental USD conversion functionality and for handling custom schemas and attributes. The set of supported hooks might include the following:
- On Import/Export. Called immediately after import and before export finalizes, respectively.
- On Prim Import/Export. Customize per-prim conversion.
- Material Conversion. Handle converting custom material types. On export, multiple hooks could be used to support different shading contexts for the same material.
Arguments for such callbacks would typically include a pointer to the USD stage and a mapping between USD prims and Blender objects.
The implementation strategy for this feature in Blender requires further discussion. As a starting point, it may be useful to describe a simple and direct approach for invoking such callbacks in user-specified modules:
- Users implement Python add-ons that define the required chaser functions.
- The chaser module names are provided as a parameter to the USD IO operators.
- The USD IO code calls the chaser functions in the given modules using the
boost::pythonbindings provided by the USD libraries.
An advantage of this strategy is that it is straightforward to implement and requires no changes to Blender code outside the USD IO module. However, providing module names as a parameter requires an extra step in the workflow and is error prone.
An improved design would be to allow registering new types of classes for implementing USD IO hooks, using the RNA framework. This has the advantages of allowing automatic registration of hooks and potentially better integration with Blender's UI.
I've provided a simple example implementations of a new
bpy.types.USDHook class type which defines the callbacks as member functions and which can be sub classed and registered in add-ons. A prototype implementation of this strategy is in pull request #108823.
I welcome feedback and further discussion of these ideas.
Thanks for posting this Michael. Having a chaser like functionality would be a huge benefit for many pipelines.
Just to give a better sense of the use case, each project at a studio may have different export needs. For example, custom schema data to enable features in their game engine. Currently this requires either of two approaches:
Either there's a fork of Blender for every project with the changes added in. This has a lot of overhead in keeping Blender versions in sync across projects, managing diffs and requires someone who's adept at C++ on each project.
Or you have a new operator that wraps the default exporter. This is easier than the forked version, however every single parameter needs to be forwarded up resulting in duplicated work, and there's overhead in that approach too.
Additionally, for non-studio use, this allows other add-ons to extend use cases for users. For example, if an add-on has bespoke data within Blender that it wants to persist in the export/import of USD. It's beholden to the same issues as above.
Chasers would solve both issues, since each project and add-on can maintain their own list of chasers that let them tailor IO in Blender to their needs. It would give the users a lot more flexibility without encumbering the Blender project itself for adding support for everything directly.
I agree with your description, and prognosis that it makes most sense to start with the USD module first. As you said, future iterations could let chasers identify which IO operators they support, but USD is a great place to start with it, and keeps things simple and contained for the time being.
My one change would be (and I think it was implied:
On Import/Export. Called immediately after import and before export, respectively.
I would say "before export finalizes" instead, because I think we'd want it to run after the brunt of the export code has run, but before its actually saved the file to disk. That way chasers can modify the contents as needed.
I think this change would be very welcome by the community, especially those in studios. Personally, its been a huge accelerator for pipeline workflows in other DCCs.
Thanks, Dhruv! Those are great examples of use-cases, and I completely agree with your points. Good catch regarding the description of the "on-export" chaser. I made the change.
I believe this has been addressed by the merge of !108823, closing.
Thanks @mont29! I just wanted to note that #108823 currently implements the export hooks only, but the same design and framework can be used to support import hooks as well, so I agree it's appropriate to close the current task. We can implement import hooks in a separate pull request in the future.
No due date set.
No dependencies set.
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?