Add Existing Import/Export Functionality to Collection Properties #100367

New Issue

Sonny Campbell · 2022-08-12T10:04:18+02:00

Sonny Campbell commented

2022-08-12 10:04:18 +02:00

Problem Statement
Currently importing and exporting files like glTF, FBX, Alembic or USD is done through operators. This is fine for a one time export, but when re-importing or re-exporting the same asset many times it becomes cumbersome. We need to store relations to such files and settings permanently, so re-import and re-export is quick, and may even be done automatically on changes.

Collection Settings
The basic idea is that Collection datablock would get settings for import and export.

For export, a collection could have one or more associated exporters. Blender would natively support USD and Alembic exporters. However this could be an API that Python add-ons can plug into, to also support file formats like FBX and glTF. The settings would be defined by the exporter, and could be exactly the same as existing export operator properties. Once export settings are set up, re-exporting one (or all) collections would be a single click, much like rendering.

For import, the simplest case would be similar to export. A collection would have one associated importer and settings. Re-importing would be done with a single click as well. This would delete the objects in the collection and add new objects, however any changes to these objects would be lost.

Goal
The first step will be to wrap the existing import and export logic using Collection Properties.

Requirements

A user should be able to select the file for import/export
A user should be able to set the required import/export parameters
A user should be able to import all data from a supported file type as children of the selected Collection.
A user should be able to re-import all data from the selected file. This would delete the objects in the collection and add new objects, however any changes to these objects would be lost.
- This would probably require a pop up to ask the user if they are sure the want to re-import and overwrite the collection children
A user should be able to export all objects that are a child of the Collection back to the selected file.
A collection could have one or more associated exporters.
- This was mentioned in the main design task, is it a realistic/requested use case?

Design Goals

It should be generic enough to be usable for any importer/exporter
Start thinking about how the Collection data block should be updated to support the longer term goals of using the collection properties import/export as a sort of live library to the file contents.

An initial thought for a UI design could be something like below (all the relevant import/export parameters have not been added to the list). Ideally we should be able to re-use the existing operator an UI logic as much as possible.

Implementation Notes

Import for USD already has an option to import all objects to the active collection. Not much needs to be changed there.
I believe export for USD only currently supports options for All, All Selected, All Visible. Would have to be updated to support exporting all children of the active collection.
(I am unfamiliar with the other import/exporters)

Questions

If there is work that needs to be done for each individual file type to support this workflow, who is responsible for that implementation? Does it fall to the owner of the importer, or should they all be tackled together before moving on to the next phase of this work?

Technical Design

To support import, the Collection data-block would need to be updated to store the import filepath and the import parameters. If we want to keep this as generic as possible, what would be the best way to store these parameters? Obviously each format will have a different set of parameters, and we want to keep it generic enough to be easy to add other formats in the future. Currently the import parameters are contained entirely in their respective modules (USDImportParams used only in ../io/usd and .../io/editors/usd), how should the parameters be exposed at a lower level?

To support a single file export, the same technical requirements and questions apply regarding best way to make generic. To support multiple file export, Collection will need to store a list of export parameter blocks.

**Problem Statement** Currently importing and exporting files like glTF, FBX, Alembic or USD is done through operators. This is fine for a one time export, but when re-importing or re-exporting the same asset many times it becomes cumbersome. We need to store relations to such files and settings permanently, so re-import and re-export is quick, and may even be done automatically on changes. **Collection Settings** The basic idea is that Collection datablock would get settings for import and export. For export, a collection could have one or more associated exporters. Blender would natively support USD and Alembic exporters. However this could be an API that Python add-ons can plug into, to also support file formats like FBX and glTF. The settings would be defined by the exporter, and could be exactly the same as existing export operator properties. Once export settings are set up, re-exporting one (or all) collections would be a single click, much like rendering. For import, the simplest case would be similar to export. A collection would have one associated importer and settings. Re-importing would be done with a single click as well. This would delete the objects in the collection and add new objects, however any changes to these objects would be lost. --- **Goal** The first step will be to wrap the existing import and export logic using Collection Properties. **Requirements** - A user should be able to select the file for import/export - A user should be able to set the required import/export parameters - A user should be able to import all data from a supported file type as children of the selected Collection. - A user should be able to re-import all data from the selected file. This would delete the objects in the collection and add new objects, however any changes to these objects would be lost. - This would probably require a pop up to ask the user if they are sure the want to re-import and overwrite the collection children - A user should be able to export all objects that are a child of the Collection back to the selected file. - A collection could have one or more associated exporters. - This was mentioned in the main design task, is it a realistic/requested use case? **Design Goals** - It should be generic enough to be usable for any importer/exporter - Start thinking about how the Collection data block should be updated to support the longer term goals of using the collection properties import/export as a sort of live library to the file contents. An initial thought for a UI design could be something like below (all the relevant import/export parameters have not been added to the list). Ideally we should be able to re-use the existing operator an UI logic as much as possible. ![image.png](https://archive.blender.org/developer/F13364811/image.png) **Implementation Notes** - Import for USD already has an option to import all objects to the active collection. Not much needs to be changed there. - I believe export for USD only currently supports options for All, All Selected, All Visible. Would have to be updated to support exporting all children of the active collection. - (I am unfamiliar with the other import/exporters) **Questions** - If there is work that needs to be done for each individual file type to support this workflow, who is responsible for that implementation? Does it fall to the owner of the importer, or should they all be tackled together before moving on to the next phase of this work? **Technical Design** To support import, the Collection data-block would need to be updated to store the import filepath and the import parameters. If we want to keep this as generic as possible, what would be the best way to store these parameters? Obviously each format will have a different set of parameters, and we want to keep it generic enough to be easy to add other formats in the future. Currently the import parameters are contained entirely in their respective modules (`USDImportParams` used only in `../io/usd` and `.../io/editors/usd`), how should the parameters be exposed at a lower level? To support a single file export, the same technical requirements and questions apply regarding best way to make generic. To support multiple file export, Collection will need to store a list of export parameter blocks.

Sonny Campbell commented

2022-08-12 10:04:18 +02:00

Added subscribers: @SonnyCampbell_Unity, @makowalski

Thomas Marcos commented

2022-08-12 21:07:28 +02:00

Added subscriber: @thomasmcs

Satish Goda commented

2022-08-13 01:55:07 +02:00

Added subscriber: @satishgoda1

Zhen Dai commented

2022-08-15 16:27:24 +02:00

Added subscriber: @Zhen-Dai

Philipp Oeser commented

2022-08-15 18:50:27 +02:00

Added subscriber: @lichtwerk

Philipp Oeser commented

2022-08-19 10:06:30 +02:00

Changed status from 'Needs Triage' to: 'Needs Developer To Reproduce'

Philipp Oeser commented

2022-08-19 10:06:30 +02:00

Not sure if this has been discussed in the module(s)? Will reflect in the task status (to get it out of the triaging queue)

Bastien Montagne commented

2022-08-19 10:57:08 +02:00

Added subscriber: @mont29

Bastien Montagne commented

2022-08-19 10:57:08 +02:00

Changed status from 'Needs Developer To Reproduce' to: 'Confirmed'

Bastien Montagne commented

2022-08-19 10:57:08 +02:00

Yes this has been, still need to actually read on this though...

Brian Savery (AMD) commented

2022-08-22 21:53:09 +02:00

Added subscriber: @BrianSavery

Brian Savery (AMD) commented

2022-08-22 21:53:09 +02:00

Hi @SonnyCampbell_Unity we have a similar proposal https://developer.blender.org/T100569 here. The main difference is we propose being able to load a USD via "Reference" in which you could override the location, material, etc of linked objects. You would just scan the USD for the prims, and load their geometry at render time. Otherwise I think we came to similar ideas on using collections for this.

Sonny Campbell commented

2022-08-23 09:00:02 +02:00

Hey @BrianSavery - thanks a million for creating that design task! That is esentially the plan for this work too. In the design discussion for this work we laid out a couple of steps for this work is to be integrated.

The first task is adding the existing import/export functionality to collections in a generic way, so that you can import from one file type and possibly export to any file type. There were design decisions that needed to be made and this design task was created for that discussion.

The step after that would be to add the functionality for importing via "reference", but again needs to be approach in a generic way to apply to a format like Alembic too. It looks like you have solved it the way we were planning with the library data, which is great because you'll be able to point out and potential headaches and pitfalls with it. I haven't dug into that code much yet, so if you have code already doing that it will be a massive headstart when we get to implementing it.

After that the plan was to think about USD-specific features and we would design and integrate them in a sensible way (like with the prim_path in your screenshots).

The guidance from the Blender team has been that it needs to be introduced in as generic a way as possible at first so that the design isn't tied to any specific format.

Would be great if you could join the regular pipelines meetings so we could plan this out together. Ideally we'd be able to spend most of our effort helping to convert the work you've already done into a design that the Blender Foundation is happy to integrate, rather than starting from scratch, but the team have a lot of design and implementation requirements that need to be thought out and solved first.

Hey @BrianSavery - thanks a million for creating that design task! That is esentially the plan for this work too. In the design discussion for this work we laid out a couple of steps for this work is to be integrated. The first task is adding the existing import/export functionality to collections in a generic way, so that you can import from one file type and possibly export to any file type. There were design decisions that needed to be made and this design task was created for that discussion. The step after that would be to add the functionality for importing via "reference", but again needs to be approach in a generic way to apply to a format like Alembic too. It looks like you have solved it the way we were planning with the library data, which is great because you'll be able to point out and potential headaches and pitfalls with it. I haven't dug into that code much yet, so if you have code already doing that it will be a massive headstart when we get to implementing it. After that the plan was to think about USD-specific features and we would design and integrate them in a sensible way (like with the prim_path in your screenshots). The guidance from the Blender team has been that it needs to be introduced in as generic a way as possible at first so that the design isn't tied to any specific format. --- Would be great if you could join the regular pipelines meetings so we could plan this out together. Ideally we'd be able to spend most of our effort helping to convert the work you've already done into a design that the Blender Foundation is happy to integrate, rather than starting from scratch, but the team have a lot of design and implementation requirements that need to be thought out and solved first.

Tristan commented

2022-08-29 09:08:07 +02:00

Added subscriber: @Plyro

Bastien Montagne commented

2022-08-31 10:58:56 +02:00

Sorry for the delay, finally had time off from overrides...

Design is fine, reflects what we discussed during the meetings.

Regarding raised questions:

Having several exporters in a collection: I do not see any technical issue with it, implementing it should be trivial. Whether this is a real need for users is another question... Should be trivial to add later anyway if needed, so we can definitely go without it for a start.
Once the specs are clear for IO code for how to hook into the 'Collection I/O' system, think it should be up for each maintainer of our various formats to decide whether they want to support it, and add this support to their code. Would expect required changes to be very minimal and easy to do though.

Technical Design

I would recommend to keep our Operator system here, with some minor tweaks. that way, Collection can simply store a set of IDProperties matching the relevant operator, in a similar way as what is done for e.g. the 'redo' panel in window manager code (see wmOperatorType.last_properties usages, WM_operator_last_properties_store function, etc.).

Then we can have a specific registering of those operators to declare themselves as supporting the Collection I/O feature, also requiring a minimal subset of common properties (would think about e.g. at least a numeric field to store the collection session_uuid, maybe a boolean one to tell the operator it is being called from Collection I/O code, etc.).

*Sorry for the delay, finally had time off from overrides...* ----------------- Design is fine, reflects what we discussed during the meetings. Regarding raised questions: * Having several exporters in a collection: I do not see any technical issue with it, implementing it should be trivial. Whether this is a real need for users is another question... Should be trivial to add later anyway if needed, so we can definitely go without it for a start. * Once the specs are clear for IO code for how to hook into the 'Collection I/O' system, think it should be up for each maintainer of our various formats to decide whether they want to support it, and add this support to their code. Would expect required changes to be very minimal and easy to do though. ### Technical Design I would recommend to keep our Operator system here, with some minor tweaks. that way, Collection can simply store a set of IDProperties matching the relevant operator, in a similar way as what is done for e.g. the 'redo' panel in window manager code (see `wmOperatorType.last_properties` usages, `WM_operator_last_properties_store` function, etc.). Then we can have a specific registering of those operators to declare themselves as supporting the Collection I/O feature, also requiring a minimal subset of common properties (would think about e.g. at least a numeric field to store the collection `session_uuid`, maybe a boolean one to tell the operator it is being called from Collection I/O code, etc.).

Alexey Adamitsky commented

2022-08-31 14:25:46 +02:00

Added subscriber: @AlexeyAdamitsky

RedMser commented

2022-08-31 15:45:14 +02:00

Added subscriber: @RedMser

Georgiy Markelov commented

2022-09-04 17:14:09 +02:00

Added subscriber: @DagerD

Sonny Campbell self-assigned this 2022-09-07 16:33:43 +02:00

Hadrien Brissaud commented

2022-09-07 19:32:31 +02:00

Added subscriber: @hadrien

Charles Wardlaw commented

2022-09-23 18:26:34 +02:00

Added subscriber: @CharlesWardlaw

Charles Wardlaw commented

2022-09-23 18:26:34 +02:00

Hey folks,

I've been working with Michael and Sonny on a demo prototype and I think we're at a good point. Please find the video attached, understanding that the code is just an attempt to explore the problem. Implementation details are certainly open for discussion.

collections_03.mp4

The short version:

Starting from the operators, I do get_rna_type(), iterate their properties, and create mirror storage versions on the Collection type automatically.
I display the storage types, filtering by import or export and on the file type.
I wrap the import / export operators with a set just for Collection File IO. These fill the operators with the proper values from the stored properties on the active Collection, then call the operator EXEC_DEFAULT to skip the file dialog.

I'm curious what people think of this workflow example.

Hey folks, I've been working with Michael and Sonny on a demo prototype and I think we're at a good point. Please find the video attached, understanding that the code is just an attempt to explore the problem. Implementation details are certainly open for discussion. [collections_03.mp4](https://archive.blender.org/developer/F13560839/collections_03.mp4) The short version: - Starting from the operators, I do get_rna_type(), iterate their properties, and create mirror storage versions on the Collection type automatically. - I display the storage types, filtering by import or export and on the file type. - I wrap the import / export operators with a set just for Collection File IO. These fill the operators with the proper values from the stored properties on the active Collection, then call the operator EXEC_DEFAULT to skip the file dialog. I'm curious what people think of this workflow example.

Scurest commented

2022-09-24 06:16:19 +02:00

Added subscriber: @scurest

Scurest commented

2022-09-24 06:16:19 +02:00

How will "versioning" of the operator parameters work? eg. what should happen if a property changes type or name?

Kevin Collins commented

2022-09-24 06:32:24 +02:00

Added subscriber: @codeloadgame

Sonny Campbell commented

2022-09-27 17:18:09 +02:00

Hey all, as @CharlesWardlaw mentioned we've been working on some prototypes for this functionality, and I've been working on a prototype to implement this functionality through native code.

2022-09-27 15-47-31.mp4

Some of the questions I have:

As I mentioned in the video, I don't know how to map the collection->import_properties to the op->ptr in the wm_usd_import_exec callback of the existing USD importer logic. If that can be easily solved, and the same solution works for setting the operator properties of python importers, it's pretty trivial to extend this to any importer.
Also as mentioned in the video, having to set the import_export_type enum on each operator type feels a bit hacky, but it might be the correct way to maintain that dropdown across each of the operator types.
I wasn't sure how to set the order of the panel in native code, so the new panel always appeared at the top of the list. Is there an ordering property somewhere that I need to set in native code?
I also wasn't sure how to default the panel to closed in native code.
Another issue will be that while the properties get laid out correctly in the UI panel, we would ideally like to be able to group related properties together rather than just displaying a long list. It would mean writing some custom drawing logic based on the importer type, but shouldn't be a lot of extra work. See this example from the existing USD importer:

If these things are fairly easy to resolve, it's overall a pretty small amount of code to implement. Appreciate any feedback.

Hey all, as @CharlesWardlaw mentioned we've been working on some prototypes for this functionality, and I've been working on a prototype to implement this functionality through native code. [2022-09-27 15-47-31.mp4](https://archive.blender.org/developer/F13581851/2022-09-27_15-47-31.mp4) Some of the questions I have: - As I mentioned in the video, I don't know how to map the `collection->import_properties` to the `op->ptr` in the `wm_usd_import_exec` callback of the existing USD importer logic. If that can be easily solved, and the same solution works for setting the operator properties of python importers, it's pretty trivial to extend this to any importer. - Also as mentioned in the video, having to set the `import_export_type` enum on each operator type feels a bit hacky, but it might be the correct way to maintain that dropdown across each of the operator types. - I wasn't sure how to set the order of the panel in native code, so the new panel always appeared at the top of the list. Is there an ordering property somewhere that I need to set in native code? - I also wasn't sure how to default the panel to closed in native code. - Another issue will be that while the properties get laid out correctly in the UI panel, we would ideally like to be able to group related properties together rather than just displaying a long list. It would mean writing some custom drawing logic based on the importer type, but shouldn't be a lot of extra work. See this example from the existing USD importer: ![image.png](https://archive.blender.org/developer/F13581925/image.png) If these things are fairly easy to resolve, it's overall a pretty small amount of code to implement. Appreciate any feedback.

Sonny Campbell commented

2022-09-30 14:51:25 +02:00

Here's a follow-up to the preview video that resolves the first issue I was having where I was unable to map the collection->import_properties to the op->ptr in the wm_usd_import_exec callback.

It also resolves the second point of having to set the import_export_type enum on each operator type by adding it to the wrapper_ot instead.

The other three points are fairly minor UI things that I'm sure can be resolved, but this at least functionally achieves what we're trying to do in a completely generic way for all importers.

Again any feedback is really appreciated.

2022-09-30 13-38-38.mp4

Here's a follow-up to the preview video that resolves the first issue I was having where I was unable to map the `collection->import_properties` to the `op->ptr` in the `wm_usd_import_exec` callback. It also resolves the second point of having to set the `import_export_type` enum on each operator type by adding it to the `wrapper_ot` instead. The other three points are fairly minor UI things that I'm sure can be resolved, but this at least functionally achieves what we're trying to do in a completely generic way for all importers. Again any feedback is really appreciated. [2022-09-30 13-38-38.mp4](https://archive.blender.org/developer/F13597867/2022-09-30_13-38-38.mp4)

Brian Savery (AMD) commented

2022-10-02 21:31:50 +02:00

So how does this work if you have two collections that import the same object (maybe with different material or color for example). Is two objects created or is there some override being calculated?

Still thinking ahead to us being able to import objects by reference. Otherwise looks ok.

So how does this work if you have two collections that import the same object (maybe with different material or color for example). Is two objects created or is there some override being calculated? Still thinking ahead to us being able to import objects by reference. Otherwise looks ok.

Sonny Campbell commented

2022-10-03 09:20:15 +02:00

For the minute there is no reference kept to anything outside the Blender scene. This change just exposes the existing importer/exporter logic as an option for the collection, so if you import something multiple times it will create that many duplicates in the scene.

As you have mentioned, the longer term goal will be to get to a point of importing things by reference. This is setting up some of the groudwork to allow us to do that. We want to be sure Blender is happy with the implementation details at each step.

For the minute there is no reference kept to anything outside the Blender scene. This change just exposes the existing importer/exporter logic as an option for the collection, so if you import something multiple times it will create that many duplicates in the scene. As you have mentioned, the longer term goal will be to get to a point of importing things by reference. This is setting up some of the groudwork to allow us to do that. We want to be sure Blender is happy with the implementation details at each step.

Sonny Campbell commented

2022-10-06 22:52:14 +02:00

@mont29 here is the related prototype code https://developer.blender.org/D16176

Julien Duroure commented

2022-10-21 21:23:28 +02:00

Added subscriber: @JulienDuroure

Brecht Van Lommel commented

2022-12-01 17:43:43 +01:00

Added subscriber: @brecht

Brecht Van Lommel commented

2022-12-01 17:43:43 +01:00

Regarding how to store properties or reuse operators, I think this is the direction we should go in: #68935 (Unified object and geometry I/O handling)

An initial implementation can be internal and not exposed in the Python API, while we figure out the right interface.

Regarding how to store properties or reuse operators, I think this is the direction we should go in: #68935 (Unified object and geometry I/O handling) An initial implementation can be internal and not exposed in the Python API, while we figure out the right interface.

Bastien Montagne added this to the Pipeline, Assets & IO project 2023-02-09 15:39:30 +01:00

Bastien Montagne modified the project from Pipeline, Assets & IO to Core

2023-02-09 15:42:54 +01:00

Bastien Montagne removed this from the Core project 2023-02-09 18:20:30 +01:00

Philipp Oeser removed the

Download

What's New

Blender Studio

Manual

Developers Blog

Documentation

Benchmark

Blender Conference

Development Fund

One-time Donations

Add Existing Import/Export Functionality to Collection Properties #100367

Technical Design