Collections for Import/Export #68933

New Issue

Problem Statement

Currently importing and exporting files like glTF, FBX, Alembic or USD is done through operators. This is fine for a one time export, but when re-importing or re-exporting the same asset many times it becomes cumbersome. We need to store relations to such files and settings permanently, so re-import and re-export is quick, and may even be done automatically on changes.

Example Use Cases

Creating assets for a game, where the same asset is tweaked and exported many times.
Production pipeline loading animated characters from Alembic file and using Blender for rendering.
Production pipeline step for an animator, import USD layer, animate character and export a modified USD layer.
Realtime collaboration through USD / Omniverse.

Design

Collection Settings

The basic idea is that Collection datablock would get settings for import and export.

For export, a collection could have one or more associated exporters. Blender would natively support USD and Alembic exporters. However this could be an API that Python add-ons can plug into, to also support file formats like FBX and glTF. The settings would be defined by the exporter, and could be exactly the same as existing export operator properties. Once export settings are set up, re-exporting one (or all) collections would be a single click, much like rendering.

For import, the simplest case would be similar to export. A collection would have one associated importer and settings. Re-importing would be done with a single click as well. This would delete the objects in the collection and add new objects, however any changes to these objects would be lost.

Libraries and Overrides

For a production pipeline, we could treat USD or Alembic files like libraries and linked datablocks. They would be loaded on .blend file load, be non-editable by default and then support overrides and make local just like linked datablocks.

To support such a system, the existing Library datablock would be used. It would contain the filepath and other importer settings. Objects and other datablocks coming from USD or Alembic collections would be associated with a library much like linked datablocks.

That system would also replace the current Alembic importer system. That system adds local objects to the Blender scene, and then dynamically loads in geometry through modifiers. In that system only geometry can change dynamically, while we want for example object or material properties to update on changes also. If the override system is used, the choice of which properties to modify locally is up to the user rather than being a predefined subset. Implementing #69046 would no longer be necessary.

USD / Hydra

USD Layers and Blender Collections have similar purpose. Evolving Blender Collections to match USD Layers more closely would help integrate well with USD.

When using Hydra or another renderer that can directly load USD, there is no need to convert to Blender data structures. In such cases the importer may create only a bounding box or other stand-in object for Blender. This feature would be useful also when not using Hydra, to only partially load a scene for performance reasons.

Nodes

For additional flexibility, a USD importer or exporter may want to do some processing on the data. The design is compatible with an importer or exporter having a node graph to do such processing. Also without importers/exporters, it may be useful to be able to have a collection that can be generated by some node graph based on the contents of other collections.

To keep the project scope under control, it seems unlikely that the initial implementation would have support for nodes.

Realtime Collaboration

A system like Omniverse for realtime collaboration could use some of the same infrastructure. Collections could have settings associating them with e.g. a URI instead of a file path.

Such a system might support an advanced production pipeline use case where there is a USD layer coming in, overrides are applied and the result is then exported as another USD layer. However the initial or main target would probably be cases where datablocks are simply edited without Blender overrides. Rather the datablocks would be local and editable for the duration of the session, and any changes would be synchronized with the server.

### Problem Statement Currently importing and exporting files like glTF, FBX, Alembic or USD is done through operators. This is fine for a one time export, but when re-importing or re-exporting the same asset many times it becomes cumbersome. We need to store relations to such files and settings permanently, so re-import and re-export is quick, and may even be done automatically on changes. ### Example Use Cases * Creating assets for a game, where the same asset is tweaked and exported many times. * Production pipeline loading animated characters from Alembic file and using Blender for rendering. * Production pipeline step for an animator, import USD layer, animate character and export a modified USD layer. * Realtime collaboration through USD / Omniverse. ### Design **Collection Settings** The basic idea is that Collection datablock would get settings for import and export. For export, a collection could have one or more associated exporters. Blender would natively support USD and Alembic exporters. However this could be an API that Python add-ons can plug into, to also support file formats like FBX and glTF. The settings would be defined by the exporter, and could be exactly the same as existing export operator properties. Once export settings are set up, re-exporting one (or all) collections would be a single click, much like rendering. For import, the simplest case would be similar to export. A collection would have one associated importer and settings. Re-importing would be done with a single click as well. This would delete the objects in the collection and add new objects, however any changes to these objects would be lost. **Libraries and Overrides** For a production pipeline, we could treat USD or Alembic files like libraries and linked datablocks. They would be loaded on .blend file load, be non-editable by default and then support overrides and make local just like linked datablocks. To support such a system, the existing Library datablock would be used. It would contain the filepath and other importer settings. Objects and other datablocks coming from USD or Alembic collections would be associated with a library much like linked datablocks. That system would also replace the current Alembic importer system. That system adds local objects to the Blender scene, and then dynamically loads in geometry through modifiers. In that system only geometry can change dynamically, while we want for example object or material properties to update on changes also. If the override system is used, the choice of which properties to modify locally is up to the user rather than being a predefined subset. Implementing #69046 would no longer be necessary. **USD / Hydra** USD Layers and Blender Collections have similar purpose. Evolving Blender Collections to match USD Layers more closely would help integrate well with USD. When using Hydra or another renderer that can directly load USD, there is no need to convert to Blender data structures. In such cases the importer may create only a bounding box or other stand-in object for Blender. This feature would be useful also when not using Hydra, to only partially load a scene for performance reasons. **Nodes** For additional flexibility, a USD importer or exporter may want to do some processing on the data. The design is compatible with an importer or exporter having a node graph to do such processing. Also without importers/exporters, it may be useful to be able to have a collection that can be generated by some node graph based on the contents of other collections. To keep the project scope under control, it seems unlikely that the initial implementation would have support for nodes. **Realtime Collaboration** A system like Omniverse for realtime collaboration could use some of the same infrastructure. Collections could have settings associating them with e.g. a URI instead of a file path. Such a system might support an advanced production pipeline use case where there is a USD layer coming in, overrides are applied and the result is then exported as another USD layer. However the initial or main target would probably be cases where datablocks are simply edited without Blender overrides. Rather the datablocks would be local and editable for the duration of the session, and any changes would be synchronized with the server.

The other way around, for importing it would similarly be a welcome improvement if you could link an FBX, OBJ, Alembic etc into Blender and once the associated file is updated, you would be able to update the contents of the file in Blender. Many popular rendering programs are also able to memorise material overrides (i.e. they propagate the 'old' material assignments upon updating the geometry, so the materials stay the same). I hope this will also be considered at a later stage of development.

Download

What's New

Blender Studio

Manual

Developers Blog

Documentation

Benchmark

Blender Conference

Development Fund

One-time Donations

Collections for Import/Export #68933

Problem Statement

Example Use Cases

Design