How to handle preservations of linked IDs when they are effectively unused. #106321

New Issue

Bastien Montagne · 2023-03-30T17:55:27+02:00

Bastien Montagne commented

2023-03-30 17:55:27 +02:00

133dde41bb refactored how linked data 'directly linked' versus 'indirectly linked' status is handled in code, such that only linked IDs effectively used in current blend file are tagged as 'directly linked'.

This broke some existing use-case, where previous broken behavior was 'abused' to keep linked IDs in current .blend files even if they did not have actual real local usages. E.g. Texts data-blocks, or node groups... See #103867, #105687, #105786, and 'fixes' commits a60f651502 and 3d49d738b7.

A better solution than these temp hacks is needed. Here are three different proposals.

1. Keep track of IDs explicitly linked by users.

The idea here is to add a new flag to IDs, which would only be set for linked IDs that were actually selected by the user (as opposed to these implicitly linked as their dependencies).

These explicitly linked IDs would then always be kept in current blend file, unless users explicitly delete them (through regular ID deletion or Purge operation e.g.).

The 'Fake User' flag would then be fully ignored on linked data.

This solution would also help in case 'fake user' gets removed in the future? See #61209.

2. Make 'Fake User' Editable on Linked Data.

'Fake User' would be editable on linked data. Linked IDs with this flag set would always be considered as directly linked.

When linking a new ID, the value of the 'Fake User' flag in the library file would be ignored. Instead, the Fake User flag would be set or cleared based on the ID type (in a similar way as some ID types currently get a fake user by default on creation, e.g. texts)
This would imply adding a new IDType flag to identify such ID types, which would be a good side-cleanup thing.

3. Fully ignore 'Fake User' status of linked data.

This option would systematically clear that flag when reading linked data, and only keep linked data that is effectively used in the current .blend file.

User would then have to explicitly reference linked data they want to keep around, e.g. via custom property pointers.

This task is closely related to #99262.

133dde41bb refactored how linked data 'directly linked' versus 'indirectly linked' status is handled in code, such that only linked IDs effectively used in current blend file are tagged as 'directly linked'. This broke some existing use-case, where previous broken behavior was 'abused' to keep linked IDs in current .blend files even if they did not have actual real local usages. E.g. Texts data-blocks, or node groups... See #103867, #105687, #105786, and 'fixes' commits a60f651502 and 3d49d738b7. A better solution than these temp hacks is needed. Here are three different proposals. ### 1. Keep track of IDs explicitly linked by users. The idea here is to add a new flag to IDs, which would only be set for linked IDs that were actually selected by the user (as opposed to these implicitly linked as their dependencies). These explicitly linked IDs would then always be kept in current blend file, unless users explicitly delete them (through regular ID deletion or Purge operation e.g.). The 'Fake User' flag would then be fully ignored on linked data. This solution would also help in case 'fake user' gets removed in the future? See #61209. ### 2. Make 'Fake User' Editable on Linked Data. 'Fake User' would be editable on linked data. Linked IDs with this flag set would always be considered as directly linked. When linking a new ID, the value of the 'Fake User' flag in the library file would be ignored. Instead, the Fake User flag would be set or cleared based on the ID type (in a similar way as some ID types currently get a fake user by default on creation, e.g. texts) *This would imply adding a new IDType flag to identify such ID types, which would be a good side-cleanup thing.* ### 3. Fully ignore 'Fake User' status of linked data. This option would systematically clear that flag when reading linked data, and only keep linked data that is effectively used in the current .blend file. User would then have to explicitly reference linked data they want to keep around, e.g. via custom property pointers. ------------------ This task is closely related to #99262.

👍 1

Bastien Montagne added the

labels 2023-03-30 17:56:00 +02:00

Bastien Montagne added this to the 3.6 LTS milestone 2023-03-30 17:56:12 +02:00

Bastien Montagne referenced this issue

2023-03-30 17:56:59 +02:00

How to handle linked data with 'fake user' set #105786

Iliya Katushenock commented

2023-03-30 18:09:11 +02:00

Subscribed

*Subscribed*

Bastien Montagne commented

2023-04-12 11:41:59 +02:00

@brecht @dfelinto you guys have any opinions regarding proposed solutions, or other ideas? Think it would be good to properly address that issue for 3.6 LTS.

Bastien Montagne commented

2023-04-13 10:38:24 +02:00

@SimonThommes @eyecandy @Mets You guys may also have some ideas here?

Brecht Van Lommel commented

2023-04-14 13:51:25 +02:00

I think we should have a discussion to figure out where we want to go with fake users and reference counting in general, and what we can change in 4.0, before committing to a solution here. Since it might change the conclusion.

(1) and (2) in a way are not that different. If "explicitly linked" is user editable it's like a local "fake user"? And the "fake user" from the original .blend is not relevant when linked, so in the UI that can be replaced with this.

For both #103867 and #105687 I think the proper solution is not to rely on this though. If a rig needs a script, we should have some way of attaching that script to the rig instead so it comes along and gets removed automatically as needed. For an add-on using a node group, it should also be linked through a custom property I think.

I think we should have a discussion to figure out where we want to go with fake users and reference counting in general, and what we can change in 4.0, before committing to a solution here. Since it might change the conclusion. (1) and (2) in a way are not that different. If "explicitly linked" is user editable it's like a local "fake user"? And the "fake user" from the original .blend is not relevant when linked, so in the UI that can be replaced with this. For both #103867 and #105687 I think the proper solution is not to rely on this though. If a rig needs a script, we should have some way of attaching that script to the rig instead so it comes along and gets removed automatically as needed. For an add-on using a node group, it should also be linked through a custom property I think.

Iliya Katushenock referenced this issue

2023-05-23 22:06:44 +02:00

Node assets bring along unwanted data #108208

Demeter Dzadik referenced this issue from studio/blender-studio-pipeline

2023-07-10 11:01:35 +02:00

Blender Crawl: During Purge Remove Fake User for Linked Data-Blocks #105

Demeter Dzadik commented

2023-07-10 11:09:12 +02:00

Quick 2 cents:

use_fake_user should only be considered on local IDs. The flag of linked IDs should be ignored, otherwise linked IDs end up filling up shot files, and can only be properly cleaned up with Python scripting.
If user wants to keep a linked ID even when it has no users, either simply don't allow it (sorry) or take the ugly design path and add a separate flag for that specific behaviour (use_fake_user_linked).

Of course a longer term solution would probably involve a rework of Blender's garbage collection or so, but that's probably a bigger project.

Quick 2 cents: - use_fake_user should only be considered on local IDs. The flag of linked IDs should be ignored, otherwise linked IDs end up filling up shot files, and can only be properly cleaned up with [Python scripting](https://projects.blender.org/studio/blender-studio-pipeline/pulls/105). - If user wants to keep a linked ID even when it has no users, either simply don't allow it (sorry) or take the ugly design path and add a separate flag for that specific behaviour (use_fake_user_linked). Of course a longer term solution would probably involve a rework of Blender's garbage collection or so, but that's probably a bigger project.

Nick Alberelli referenced this issue from a commit

2023-07-12 17:45:20 +02:00

Blender Crawl: During Purge Remove Fake User for Linked Data-Blocks (#105)

Bastien Montagne modified the milestone from 3.6 LTS to 4.0

2023-07-18 11:16:15 +02:00

Brecht Van Lommel commented

2023-07-18 13:28:10 +02:00

I think (3) makes sense, but there is a problem that I think we need to solve along with it.

As far as I know, there is no way to attach a (auto-run) Text datablock to a rig without using Python scripting. In the Custom Properties panel you can not create datablock pointer properties. You can do it by defining a PointerProperty in Python, but of course to do that you need to be able to run a script first.

Can we make this possible, so that we can recommend that as the solution to keep around arbitrary datablocks from other datablocks going forward?

I think (3) makes sense, but there is a problem that I think we need to solve along with it. As far as I know, there is no way to attach a (auto-run) Text datablock to a rig without using Python scripting. In the Custom Properties panel you can not create datablock pointer properties. You can do it by defining a `PointerProperty` in Python, but of course to do that you need to be able to run a script first. Can we make this possible, so that we can recommend that as the solution to keep around arbitrary datablocks from other datablocks going forward?

Demeter Dzadik commented

2023-07-19 15:22:56 +02:00

You can do it by defining a PointerProperty in Python, but of course to do that you need to be able to run a script first.

Slight correction: You don't need to define a PointerProperty, IDProperties support pointers, so you can just do this:
my_rig.data['script'] = bpy.data.texts['rig_script.py']
But it is correct that there is no UI way to do this. (Add-ons can provide a UI to do it, like Rigify does, and to do that, you indeed need a PointerProperty(type=Text))

I agree that implementing UI into bpy.ops.wm.properties_edit that allows the aforementioned line of code should be possible, and it is a nice way to address the use case where users would want linked data to stick around.

Only problem is that Operators can't get IDs passed to them, but if you pass ID type, name, and libpath, you're good to go.

I recently implemented this in an add-on, may it inspire: ID Management Pie

> You can do it by defining a `PointerProperty` in Python, but of course to do that you need to be able to run a script first. Slight correction: You don't need to define a PointerProperty, IDProperties support pointers, so you can just do this: `my_rig.data['script'] = bpy.data.texts['rig_script.py']` But it is correct that there is no UI way to do this. (_Add-ons can provide a UI to do it, like Rigify does, and to do that, you indeed need a `PointerProperty(type=Text)`_) I agree that implementing UI into `bpy.ops.wm.properties_edit` that allows the aforementioned line of code should be possible, and it is a nice way to address the use case where users would want linked data to stick around. Only problem is that Operators can't get IDs passed to them, but if you pass ID type, name, and libpath, you're good to go. I recently implemented this in an add-on, may it inspire: [ID Management Pie](https://projects.blender.org/studio/blender-studio-pipeline/pulls/127#issuecomment-982252) ![image](/attachments/d8a5ed93-0fbf-45c4-8b62-2e112b23541c)

image.png

38 KiB

Bastien Montagne commented

2023-07-25 12:36:33 +02:00

Adding proper UI support for custom properties to ID pointers seems doable, here is a quickly slapped prototype: !110458.

Bastien Montagne referenced this issue

2023-08-11 12:39:02 +02:00

Always clear 'fake user' flag from linked data. #111042

Bastien Montagne commented

2023-08-11 12:40:31 +02:00

And !111042 implements solution #3 above (always fully ignoring 'fake user' status of linked data - fairly trivial in fact).

Bastien Montagne referenced this issue from a commit