Asset Deduplication with "Append & Reuse" #115660

New Issue

Dalai Felinto · 2023-12-01T12:19:35+01:00

Dalai Felinto commented

2023-12-01 12:19:35 +01:00

Asset Deduplication with "Append & Reuse"

This topic was discussed as part of the Geomerty Nodes workshop in November 2023. Although it popped up in the context of nodes, at the core of it are assets.

The introduction of "Append (Reuse Data)" for assets, made production files clutter with duplicated versions of the same assets (e.g., essential node groups). That happens due to linking different files which have appended/reused the same assets. That leads to the following problems:

Namespace pollution.
Changes not being propagated on them.

On the flip side, it made local changes easy.

To address these problems we want Blender to deduplicate identical data-blocks:

It is up to the implementation to decide how to determine whether a data-block is identical. If non-relevant changes (e.g., node position) can't be told apart from relevant changes (values) we err on the side of considering them different.
This should be handle automatically by Blender, which internally will need to find a way to tell different versions of the same asset apart (internal versionioning, hash, ...).

The proposal is to change the existing "Append (Reuse Data)" so it brings a data-block into the file in a new way:

It should work similarly to linking, where users have to make the data-block local before changing its properties.
It preserves a reference to the original data-block (and its version/hash/...) so it give users the option to update an asset.
If the relation with the original data-block is severed (e.g., when using an asset from outside the essential in a renderfarm) the file is still fully valid (it contains a full copy of the original data-block).

Pain points:

The logic to determine whether the datablocks are identical may be expensive. In fact it may not be worth pursuiting this due to potential problems this may bring. It can be done as a separate project too, while a simpler .blend timestamp check could be used meanwhile.
This is too close to the concept of variants and versions for IDs, at least for the backend implementation. It makes sense to tackle them together.
This should be budgeted as a project within the Blender planning.
How it is called, and how this relates with linking needs to be clear and well communicated.
To differentiate this even more from regular Append, we could rename it to "Append & Reuse" on the user interface.

Patch: !125389

# Asset Deduplication with "Append & Reuse" _This topic was discussed as part of the [Geomerty Nodes workshop](https://code.blender.org/2023/11/geometry-nodes-workshop-november-2023/) in November 2023. Although it popped up in the context of nodes, at the core of it are assets._ The introduction of "Append (Reuse Data)" for assets, made production files clutter with duplicated versions of the same assets (e.g., essential node groups). That happens due to linking different files which have appended/reused the same assets. That leads to the following problems: * Namespace pollution. * Changes not being propagated on them. On the flip side, it made local changes easy. ---- To address these problems we want Blender to **deduplicate identical data-blocks**: * It is up to the implementation to decide how to determine whether a data-block is identical. If non-relevant changes (e.g., node position) can't be told apart from relevant changes (values) we err on the side of considering them different. * This should be handle automatically by Blender, which internally will need to find a way to tell different versions of the same asset apart (internal versionioning, hash, ...). The proposal is to change the existing "Append (Reuse Data)" so it brings a data-block into the file in a new way: * It should work similarly to linking, where users have to make the data-block local before changing its properties. * It preserves a reference to the original data-block (and its version/hash/...) so it give users the option to update an asset. * If the relation with the original data-block is severed (e.g., when using an asset from outside the essential in a renderfarm) the file is still fully valid (it contains a full copy of the original data-block). ## Pain points: * The logic to determine whether the datablocks are identical may be expensive. In fact it may not be worth pursuiting this due to potential problems this may bring. It can be done as a separate project too, while a simpler .blend timestamp check could be used meanwhile. * This is too close to the concept of variants and versions for IDs, at least for the backend implementation. It makes sense to tackle them together. * This should be budgeted as a project within the Blender planning. * How it is called, and how this relates with linking needs to be clear and well communicated. * To differentiate this even more from regular Append, we could rename it to "Append & Reuse" on the user interface. --- Patch: !125389

❤️ 3 🎉 1

Dalai Felinto added the

Type

Design

label 2023-12-01 12:19:35 +01:00

Dalai Felinto added this to the Pipeline, Assets & IO project 2023-12-01 12:19:37 +01:00

Dalai Felinto changed title from ~~Asset Deduplication and Append & Re-use~~ to Asset Deduplication with "Append & Reuse"

2023-12-01 12:21:50 +01:00

Jon Edis commented

2024-01-16 07:40:48 +01:00

Hi Dalai, thank you for addressing this issue! I can't seem to find much information on this online.
My projects are constantly getting cluttered with essentially duplicate materials and node groups when I copy over models and materials from other blender files.

I would love the ability to reuse materials and node data blocks when appending files. Maybe if an appended material has the same name the user could be asked if they want to reuse the data block. Maybe there is an optional flag that could be on each material and node? I understand that it is too expensive of a calculation to evaluate and compare every material that is imported.

Hi Dalai, thank you for addressing this issue! I can't seem to find much information on this online. My projects are constantly getting cluttered with essentially duplicate materials and node groups when I copy over models and materials from other blender files. I would love the ability to reuse materials and node data blocks when appending files. Maybe if an appended material has the same name the user could be asked if they want to reuse the data block. Maybe there is an optional flag that could be on each material and node? I understand that it is too expensive of a calculation to evaluate and compare every material that is imported.

Dalai Felinto commented

2024-01-18 10:26:59 +01:00

Hi @Jonathan-Edis although this is not the use case we are focusing on here, it may benefit from it.

At first the idea is to tackle data-blocks which are appended using the asset system. This option could/should be exposed for regular append as well though, which would contemplate your case.

Hi @Jonathan-Edis although this is not the use case we are focusing on here, it may benefit from it. At first the idea is to tackle data-blocks which are appended using the asset system. This option could/should be exposed for regular append as well though, which would contemplate your case.

Hans Goudey referenced this issue

2024-06-12 17:31:23 +02:00

Geometry Nodes - Modifiers | Bug / Usability Issue (Destruction of Presets possible) #123082

Dalai Felinto referenced this issue

2024-07-25 16:42:10 +02:00

Out of the Box: Online Assets #125429

Duarte Farrajota Ramos commented

2024-08-06 16:33:13 +02:00

Maybe we don't have to try to be too smart about it. Automating will over complicate, and we always risk doing something the user doesn't mean to.

I'd say Blender could just show potential duplicate candidates, and leave the actual merging explicitly up to the user.
The outliner already has a mode for orphaned data, maybe add a new one for duplicates. Show potential candidates for merging.

Maybe show asset datablocks that say share the same base name with a numbering suffix, nodegroups/trees with similar input and output socket names, or keep track of same origin file (there may be privacy concerns with this). Maybe we can show multiple criteria in different categories and gradually improve our way to better detection methods.

Maybe we don't have to try to be too smart about it. Automating will over complicate, and we always risk doing something the user doesn't mean to. I'd say Blender could just show potential duplicate candidates, and leave the actual merging explicitly up to the user. The outliner already has a mode for orphaned data, maybe add a new one for duplicates. Show potential candidates for merging. Maybe show asset datablocks that say share the same base name with a numbering suffix, nodegroups/trees with similar input and output socket names, or keep track of same origin file (there may be privacy concerns with this). Maybe we can show multiple criteria in different categories and gradually improve our way to better detection methods.

Sign in to join this conversation.

No Label

Download

What's New

Blender Studio

Manual

Developers Blog

Documentation

Benchmark

Blender Conference

Development Fund

One-time Donations

Asset Deduplication with "Append & Reuse" #115660

Asset Deduplication with "Append & Reuse"

Pain points: