Ensure name collisions between liboverrides and their reference data does not breaks liboverrides. #110421

New Issue

Bastien Montagne · 2023-07-24T17:14:27+02:00

Bastien Montagne commented

2023-07-24 17:14:27 +02:00

The Problem

Pets production at the Blender studio as revealed an issue with library overrides, which is fairly difficult to trigger in normal usages cases, but can easily appear when production files get messy: name collision between linked reference IDs and override ones with a collection of IDs.

In practice, this is mainly affecting collections' list of objects.

At some point, given specific set of events (having several liboverrides of the same data, having several objects named the 'same' in the library file - door, door.00, door.002, etc. - and having several resyncs due to such objects being added and removed, you can end up in a situation where the override of object door.004 is named door.002, and the override of object door.002 is named door.001.

This can lead to breakage in liboverride applying code, as at some point during the process the liboverride collection can have two objects named door.002, the liboverride one, and the linked one. This breaks reference of objects of a collection by their names.

Two commits have already been made to alleviate the issue (a05419f18b and e11da03e7a), but they only make such problem much, much less likely to happen, they cannot guarantee it will never happen again.

Possible Solutions

The only way to solve this issue is to ensure that in liboverride, references to IDs within ID RNAcollections are unique and can never collide with any other ID. There can be two ways to achieve this:

Replace the .001 etc. counter by some sort of alpha-numeric UID in liboverride names.
Also store a reference to the library ID within liboverride operations for RNA collection items.
Also store the ID pointer itself within liboverride operations for RNA collection items.

The first solution essentially introduces some concept of UUID to data-blocks, but in a poor way, potentially still brittle, and in any case fairly 'ugly' since these alphanumeric codes would be exposed everywhere in the UI.

The second solution, suggested @brecht and also discussed with @Sergey, was tried out first. From an initial analysis, it implied:

Adding a reference and locale Library pointer to the IDOverrideLibraryPropertyOperation. Would be null, except for RNA collection items referencing linked IDs.
- Note that do_version code may be a bit tricky here.
These pointers will be initialized during the liboverride/RNA diffing process, when liboverride data is created or updated.
RNA collection items accessor API will need to be updated to allow search by name and libpointer for its ID items.
- This could also be linked to existing py_rna code that allows to lookup an ID in a RNA collection of IDs by its name and library path.

However, it was actually not working for linked liboverrides, as libraries are always local IDs and never linked, this data is lost when linking the liboverride into another file. This could be worked around by storing the library as a file path oinstead of a pointer to the ID, however this would make the code significantly more complicated.

So instead, a weak reference to the ID pointer itself will be used (similar to how linked objects are referenced by local viewlayer's base data e.g.). This solution is very similar to using the Library pointer, with the key difference that ID pointers also work 'by definition' when linking liboverrides. Note that:

Weak reference can be used here, since it can be assumed that a linked ID referenced in that context is also used as reference of its liboverride (otherwise, something is very wrong, and getting rid of that linked reference is likely a good thing then).
For the same reason, there is no need for ID refcounting on these pointers.
While it could look like this is creating a very strong bound between a specific ID and the RNA collection item, in fact nothing changes here, since in case the ID name changes in the library, the linked reference to it are lost, so the strong bind between the ID name and the ID itself already exist de-facto in linked data.

Implemented as !110773.

# The Problem Pets production at the Blender studio as revealed an issue with library overrides, which is fairly difficult to trigger in normal usages cases, but can easily appear when production files get messy: name collision between linked reference IDs and override ones with a collection of IDs. In practice, this is mainly affecting collections' list of objects. At some point, given specific set of events (having several liboverrides of the same data, having several objects named the 'same' in the library file - `door`, `door.00`, `door.002`, etc. - and having several resyncs due to such objects being added and removed, you can end up in a situation where the override of object `door.004` is named `door.002`, and the override of object `door.002` is named `door.001`. This can lead to breakage in liboverride applying code, as at some point during the process the liboverride collection can have two objects named `door.002`, the liboverride one, and the linked one. This breaks reference of objects of a collection by their names. Two commits have already been made to alleviate the issue (a05419f18b and e11da03e7a), but they only make such problem much, much less likely to happen, they cannot guarantee it will never happen again. # Possible Solutions The only way to solve this issue is to ensure that in liboverride, references to IDs within ID RNAcollections are unique and can never collide with any other ID. There can be two ways to achieve this: 1. Replace the `.001` etc. counter by some sort of alpha-numeric UID in liboverride names. 2. Also store a reference to the library ID within liboverride operations for RNA collection items. 3. Also store the ID pointer itself within liboverride operations for RNA collection items. The first solution essentially introduces some concept of UUID to data-blocks, but in a poor way, potentially still brittle, and in any case fairly 'ugly' since these alphanumeric codes would be exposed everywhere in the UI. The second solution, suggested @brecht and also discussed with @Sergey, was tried out first. From an initial analysis, it implied: * Adding a `reference` and `locale` `Library` pointer to the `IDOverrideLibraryPropertyOperation`. Would be null, except for RNA collection items referencing linked IDs. * *Note that do_version code may be a bit tricky here.* * These pointers will be initialized during the liboverride/RNA diffing process, when liboverride data is created or updated. * RNA collection items accessor API will need to be updated to allow search by name and libpointer for its ID items. * This could also be linked to existing py_rna code that allows to lookup an ID in a RNA collection of IDs by its name and library path. However, it was actually not working for linked liboverrides, as libraries are always local IDs and never linked, this data is lost when linking the liboverride into another file. This could be worked around by storing the library as a file path oinstead of a pointer to the ID, however this would make the code significantly more complicated. So instead, a weak reference to the ID pointer itself will be used (similar to how linked objects are referenced by local viewlayer's base data e.g.). This solution is very similar to using the Library pointer, with the key difference that ID pointers also work 'by definition' when linking liboverrides. Note that: * Weak reference can be used here, since it can be assumed that a linked ID referenced in that context is also used as reference of its liboverride (otherwise, something is very wrong, and getting rid of that linked reference is likely a good thing then). * For the same reason, there is no need for ID refcounting on these pointers. * While it could look like this is creating a very strong bound between a specific ID and the RNA collection item, in fact nothing changes here, since in case the ID name changes in the library, the linked reference to it are lost, so the strong bind between the ID name and the ID itself already exist de-facto in linked data. Implemented as !110773.

Bastien Montagne added the

Type

Design

Download

What's New

Blender Studio

Manual

Developers Blog

Documentation

Benchmark

Blender Conference

Development Fund

One-time Donations

Ensure name collisions between liboverrides and their reference data does not breaks liboverrides. #110421

The Problem

Possible Solutions