Dynamic Overrides Use Case - Giant object-based 'rig' with a few key controlers. #96092

New Issue

Ad Morris · 2022-03-01T05:02:49+01:00

Ad Morris commented

2022-03-01 05:02:49 +01:00

Dynamic Overrides are a good candidate to support extreme relationships nightmare involving thousands of objects, like the one in Machine.zip.

Library Overrides cannot realistically handle such a massive amount of IDs/relationships, but Dynamic Overrides could only override the relevant properties of the few key controller objects while keeping everything fully linked, in a similar way to what proxies used to do.

Original report:

System Information
Operating system: Windows-10-10.0.19041-SP0 64 Bits
Graphics card: NVIDIA GeForce RTX 3060/PCIe/SSE2 NVIDIA Corporation 4.5.0 NVIDIA 511.65

Blender Version
Broken: version: 3.0.1, branch: master, commit date: 2022-01-25 17:19, hash: dc2d180181
Worked: 2.9

Short description of error
It's now impossible to use a single control object/bone from a linked rig without creating an override for the entire hierarchy or parent collection. This is completely un-workable for professionals working with rigged assets made up of many hundreds or thousands of parts.

Exact steps for others to reproduce the error
I can't share my scene file as it contains sensitive intellectual property belonging to a client. The issue will be evident for any remotely complex rig however.

In the project I'm currently working on, I have a top level collection of bone controllers that have no parents. Previously I could create a proxy for each controller and use those to animate the linked asset whilst retaining the full functionality of the rig. Now that create proxy has been removed, I can't just override a single controller. Instead I'm forced to unnecessarily create overrides for several thousand objects! An operation which for this rig takes in excess of 30 minutes to run (if at all)! Then after saving the file (once the operation is complete), Blender outright crashes on each attempt to re-open.

So although the new override system works in theory. There's no parity between the two systems yet in terms of the ability to override a single object. And previous functionality has been lost without a viable workaround available. Which in my estimation constitutes a bug.

With this in mind, would you please kindly add the create proxy option back again, until the overrides system is fully functional? I'll personally now have to roll back to 2.9 until this issue is resolved.

Thanks

Dynamic Overrides are a good candidate to support extreme relationships nightmare involving thousands of objects, like the one in [Machine.zip](https://archive.blender.org/developer/F12894482/Machine.zip). Library Overrides cannot realistically handle such a massive amount of IDs/relationships, but Dynamic Overrides could only override the relevant properties of the few key controller objects while keeping everything fully linked, in a similar way to what proxies used to do. ------------------------------------------ Original report: **System Information** Operating system: Windows-10-10.0.19041-SP0 64 Bits Graphics card: NVIDIA GeForce RTX 3060/PCIe/SSE2 NVIDIA Corporation 4.5.0 NVIDIA 511.65 **Blender Version** Broken: version: 3.0.1, branch: master, commit date: 2022-01-25 17:19, hash: `dc2d180181` Worked: 2.9 **Short description of error** It's now impossible to use a single control object/bone from a linked rig without creating an override for the entire hierarchy or parent collection. This is completely un-workable for professionals working with rigged assets made up of many hundreds or thousands of parts. **Exact steps for others to reproduce the error** I can't share my scene file as it contains sensitive intellectual property belonging to a client. The issue will be evident for any remotely complex rig however. In the project I'm currently working on, I have a top level collection of bone controllers that have no parents. Previously I could create a proxy for each controller and use those to animate the linked asset whilst retaining the full functionality of the rig. Now that create proxy has been removed, I can't just override a single controller. Instead I'm forced to unnecessarily create overrides for several thousand objects! An operation which for this rig takes in excess of 30 minutes to run (if at all)! Then after saving the file (once the operation is complete), Blender outright crashes on each attempt to re-open. So although the new override system works in theory. There's no parity between the two systems yet in terms of the ability to override a single object. And previous functionality has been lost without a viable workaround available. Which in my estimation constitutes a bug. With this in mind, would you please kindly add the create proxy option back again, until the overrides system is fully functional? I'll personally now have to roll back to 2.9 until this issue is resolved. Thanks

Ad Morris commented

2022-03-01 05:02:49 +01:00

Added subscriber: @AdamMorris

Aaron Carlisle commented

2022-03-01 05:17:03 +01:00

Added subscriber: @Blendify

Aaron Carlisle commented

2022-03-01 05:17:03 +01:00

Changed status from 'Needs Triage' to: 'Resolved'

Aaron Carlisle closed this issue

2022-03-01 05:17:03 +01:00

Aaron Carlisle self-assigned this 2022-03-01 05:17:03 +01:00

Aaron Carlisle commented

2022-03-01 05:17:03 +01:00

Proxies have been removed in 3.0 in favor of overrides,

See https://wiki.blender.org/wiki/Reference/Release_Notes/3.0/Core#Proxy_Removal

Proxies have been removed in 3.0 in favor of overrides, See https://wiki.blender.org/wiki/Reference/Release_Notes/3.0/Core#Proxy_Removal

Aaron Carlisle commented

2022-03-01 05:23:08 +01:00

Changed status from 'Resolved' to: 'Needs Triage'

Aaron Carlisle reopened this issue

2022-03-01 05:23:08 +01:00

Aaron Carlisle commented

2022-03-01 05:23:09 +01:00

Apologizes I read this at face value and did not see

Then after saving the file (once the operation is complete), Blender outright crashes on each attempt to re-open.

Can you test the upcoming 3.1 release which should bring some performance improvements and we will see if it fixes the crash.

Apologizes I read this at face value and did not see > Then after saving the file (once the operation is complete), Blender outright crashes on each attempt to re-open. Can you test the upcoming 3.1 release which should bring some performance improvements and we will see if it fixes the crash.

Aaron Carlisle commented

2022-03-01 05:24:05 +01:00

Added subscriber: @mont29

Aaron Carlisle commented

2022-03-01 05:24:05 +01:00

Also CCing @mont29 to comment on the parity issue.

Aaron Carlisle removed their assignment 2022-03-01 05:25:17 +01:00

Bastien Montagne commented

2022-03-01 11:26:04 +01:00

Changed status from 'Needs Triage' to: 'Needs User Info'

Bastien Montagne commented

2022-03-01 11:26:04 +01:00

Added subscriber: @Mets

Bastien Montagne commented

2022-03-01 11:26:04 +01:00

By default creating overrides creates an override for all objects, however it is possible to override IDs individually, user (or py script) is then responsible to handle relationships between them though...

I think your way of rigging though does not match the 'canonical' way of doing so in Blender - where typically controllers are bones part of the armature, using 'shape' objects to get some form of 'UI' for the animators. But think a rigger might know better here, maybe @Mets can shine in?

In any case, I would indeed recommend trying out at least upcoming 3.1, and maybe even current alpha 3.2, there has been huge improvements in speed (and reliability) regarding overrides in those.

Also, if you cannot provide us with a reproducible case, we cannot investigate that report, so it will have to be closed...

Regarding proxies, they won't be brought back, they had a giant amount of issues creeping all over the codebase, and indeed they are not on feature-parity with overrides - the latter allow much, much more than what was possible with proxies. But indeed liboverrides are not a drop-in replacement of proxies, they may imply some new limitations and some workflows may need to be adapted for them.

By default creating overrides creates an override for all objects, however it is possible to override IDs individually, user (or py script) is then responsible to handle relationships between them though... I think your way of rigging though does not match the 'canonical' way of doing so in Blender - where typically controllers are bones part of the armature, using 'shape' objects to get some form of 'UI' for the animators. But think a rigger might know better here, maybe @Mets can shine in? In any case, I would indeed recommend trying out at least upcoming 3.1, and maybe even current alpha 3.2, there has been huge improvements in speed (and reliability) regarding overrides in those. Also, if you cannot provide us with a reproducible case, we cannot investigate that report, so it will have to be closed... Regarding proxies, they won't be brought back, they had a giant amount of issues creeping all over the codebase, and indeed they are not on feature-parity with overrides - the latter allow much, much more than what was possible with proxies. But indeed liboverrides are not a drop-in replacement of proxies, they may imply some new limitations and some workflows may need to be adapted for them.

Ad Morris commented

2022-03-01 17:44:13 +01:00

OK Allow me to try out the 3.1 and 3.2 builds and get back to you.

As far as following the canonical/conventional method for rigging in Blender. That is the method I use. The controller bones are indeed bones, with custom shapes assigned to them to give the animator a UI. All objects in the rig (which is a mechanical rig i.e. no skins or weights) are constrained to those controller bones using child of constraints etc. However I think in this context that might be a moot point. Especially given that there are no direct hierarchies between object and controller i.e. constraints are used as opposed to hierarchical parenting. Which again makes the need to override the entire linked collection seem very inefficient. With respect however, I don't understand how the system is implemented on the back end.

I will do some test with the new builds and report back.

OK Allow me to try out the 3.1 and 3.2 builds and get back to you. As far as following the canonical/conventional method for rigging in Blender. That is the method I use. The controller bones are indeed bones, with custom shapes assigned to them to give the animator a UI. All objects in the rig (which is a mechanical rig i.e. no skins or weights) are constrained to those controller bones using child of constraints etc. However I think in this context that might be a moot point. Especially given that there are no direct hierarchies between object and controller i.e. constraints are used as opposed to hierarchical parenting. Which again makes the need to override the entire linked collection seem very inefficient. With respect however, I don't understand how the system is implemented on the back end. I will do some test with the new builds and report back.

Ad Morris commented

2022-03-01 19:29:48 +01:00

Right so attempting this with Blender 3.1, there is now an option for Make Library Override Single. This is available exclusively for bones and not empties, so using bone controllers as mentioned above is the assumed preferred workflow.

Running Make Library Override Single on one of the controllers from the Outliner's context menu has no visible effect. Shift + Clicking the object data block (to create a local override) in the object properties panel unexpectedly creates a duplicate of it. As opposed to the override behaviour happening in-place. This duplicate can be then transformed (similar to the result of running the create proxy operation). However, the rig is not effected by the transforms of the new duplicate controller at all. Presumably as the constraints are still pointing at the original controller. Which remains locked and in the position read in from the linked file.

Attempting to run the Make Library Override Hierarchy operation on the full collection unfortunately causes Blender to hang once again. Currently for 15 minutes and counting at the time of writing this.

I will run the same test with 3.2 and follow up shortly.

Right so attempting this with **Blender 3.1**, there is now an option for Make Library Override Single. This is available exclusively for bones and not empties, so using bone controllers as mentioned above is the assumed preferred workflow. Running Make Library Override Single on one of the controllers from the Outliner's context menu has no visible effect. Shift + Clicking the object data block (to create a local override) in the object properties panel unexpectedly creates a duplicate of it. As opposed to the override behaviour happening in-place. This duplicate can be then transformed (similar to the result of running the create proxy operation). However, the rig is not effected by the transforms of the new duplicate controller at all. Presumably as the constraints are still pointing at the original controller. Which remains locked and in the position read in from the linked file. Attempting to run the Make Library Override Hierarchy operation on the full collection unfortunately causes Blender to hang once again. Currently for 15 minutes and counting at the time of writing this. I will run the same test with 3.2 and follow up shortly.

Ad Morris commented

2022-03-01 22:39:04 +01:00

After repeating the same process, Blender 3.2 had exactly the same behaviour as above.

Both versions eventually completed the Make Library Override Hierarchy operation, after leaving them running for nearly an hour. Unlike with 3.0.1 though, the result of this was a complete duplicate of the entire collection with overrides applied (again not an override "in-place") for both 3.1 and 3.2.

After saving new versions from each respective build. Opening the 3.1 file again caused Blender to hang for 2 minutes 53 seconds before eventually loading it. After which the override collection and controllers were fully functional. Opening the 3.2 file again caused Blender to hang for 2 minutes 7 seconds before loading. Again with the override collection and controllers being fully functional thereafter.

With 2.93 on the other hand, with proxies created for each controller. The saved file loads in 7 seconds!

Based on the above. Surely this needs more serious consideration and work, before taking away the proxy system from users?

After repeating the same process, **Blender 3.2** had exactly the same behaviour as above. Both versions eventually completed the Make Library Override Hierarchy operation, after leaving them running for nearly an hour. Unlike with 3.0.1 though, the result of this was a complete duplicate of the entire collection with overrides applied (again not an override "in-place") for both 3.1 and 3.2. After saving new versions from each respective build. Opening the 3.1 file again caused Blender to hang for **2 minutes 53 seconds** before eventually loading it. After which the override collection and controllers were fully functional. Opening the 3.2 file again caused Blender to hang for **2 minutes 7 seconds** before loading. Again with the override collection and controllers being fully functional thereafter. With 2.93 on the other hand, with proxies created for each controller. The saved file loads in **7 seconds!** Based on the above. Surely this needs more serious consideration and work, before taking away the proxy system from users?

Aaron Carlisle commented

2022-03-01 23:31:38 +01:00

Is there a way that you could create a blend-file that shows the issue by redacting any proprietary assets for example by replacing them with other objects?

This way it can be used by developers to profile what it taking so much time and cant point to where developers might want to focus time to speed up the process.

Is there a way that you could create a blend-file that shows the issue by redacting any proprietary assets for example by replacing them with other objects? This way it can be used by developers to profile what it taking so much time and cant point to where developers might want to focus time to speed up the process.

Ad Morris commented

2022-03-02 16:07:36 +01:00

OK could you please send me a private email to adam@renderkick.co.uk so that I can send you a download link?

Bastien Montagne commented

2022-03-02 16:27:33 +01:00

@AdamMorris Blender being an open project, we keep communication about it and its issues also open and public, please attach here such a file, indeed you can taint it as much as needed to remove any sensitive information. Or even generate a dummy example for this purpose.

From your comments though, I suspect override hierarchy ends up dealing with many thousands of data-blocks in a single override hierarchy, for some reasons... If that is the case, it might be worth reconsidering your rigging pipeline, or we may have to tweak override creation process to deal better with those specific cases... But quite impossible to go further without a good example file to investigate.

@AdamMorris Blender being an open project, we keep communication about it and its issues also open and public, please attach here such a file, indeed you can taint it as much as needed to remove any sensitive information. Or even generate a dummy example for this purpose. From your comments though, I suspect override hierarchy ends up dealing with many thousands of data-blocks in a single override hierarchy, for some reasons... If that is the case, it might be worth reconsidering your rigging pipeline, or we may have to tweak override creation process to deal better with those specific cases... But quite impossible to go further without a good example file to investigate.

Ad Morris commented

2022-03-02 17:21:09 +01:00

OK understood. Here's a version of the file with zero verts for all objects. It has the same object tree as the original so should hopefully enable you to run some tests.Machine.zip

OK understood. Here's a version of the file with zero verts for all objects. It has the same object tree as the original so should hopefully enable you to run some tests.[Machine.zip](https://archive.blender.org/developer/F12894482/Machine.zip)

Bastien Montagne commented

2022-03-03 09:58:54 +01:00

OK, so I am not an expert in rigging (@Mets please feel free to share your expertise on this), but am fairly sure this Machine collection is not the expected way of rigging something in Blender. From what I can see, you are essentially not using armatures at all, and are instead using parenting (and some constraints) to 'rig' almost 14 000 objects together! Most of those objects should be replaced by actual bones from a single armature.

Some stats:

13847 Objects (len(D.collections['Machine'].all_objects)).
- 6 Armatures (len([ob for ob in D.collections['Machine'].all_objects if ob.type == 'ARMATURE']))
  - 6 Bones in total (sum([len(ob.pose.bones) for ob in D.collections['Machine'].all_objects if ob.type == 'ARMATURE'])).
- 6774 Empties (len([ob for ob in D.collections['Machine'].all_objects if ob.type == 'EMPTY'])).
- 11049 Parented Objects (len([ob for ob in D.collections['Machine'].all_objects if ob.parent != None])).
- 577 Constraints (sum([len(ob.constraints) for ob in D.collections['Machine'].all_objects])).
- 38656 Relationships from all those objects to other data-blocks (sum(len(v) for v in D.user_map(subset=D.collections['Machine'].all_objects).values())).

In any case, this situation is definitively not supported by library overrides, this is simply impossible. Most likely, such situation would work fairly well with future dynamic overrides, although there would likely be some other limitations to take into account.

OK, so I am not an expert in rigging (@Mets please feel free to share your expertise on this), but am fairly sure this `Machine` collection is not the expected way of rigging something in Blender. From what I can see, you are essentially not using armatures at all, and are instead using parenting (and some constraints) to 'rig' almost 14 000 objects together! Most of those objects should be replaced by actual bones from a single armature. Some stats: * 13847 Objects (`len(D.collections['Machine'].all_objects)`). * 6 Armatures (`len([ob for ob in D.collections['Machine'].all_objects if ob.type == 'ARMATURE'])`) * 6 Bones in total (`sum([len(ob.pose.bones) for ob in D.collections['Machine'].all_objects if ob.type == 'ARMATURE'])`). * 6774 Empties (`len([ob for ob in D.collections['Machine'].all_objects if ob.type == 'EMPTY'])`). * 11049 Parented Objects (`len([ob for ob in D.collections['Machine'].all_objects if ob.parent != None])`). * 577 Constraints (`sum([len(ob.constraints) for ob in D.collections['Machine'].all_objects])`). * 38656 Relationships from all those objects to other data-blocks (`sum(len(v) for v in D.user_map(subset=D.collections['Machine'].all_objects).values())`). In any case, this situation is definitively not supported by library overrides, this is simply impossible. Most likely, such situation would work fairly well with future dynamic overrides, although there would likely be some other limitations to take into account.

Bastien Montagne commented

2022-03-03 10:06:14 +01:00

I will move it under the new #96144 (Dynamic Overrides) task, as some interesting usecase for this project.

Bastien Montagne changed title from ~~Missing Create Proxy Option (definitely a bug)~~ to Dynamic Overrides Use Case - Giant object-based 'rig' with a few key controlers.

2022-03-03 10:13:23 +01:00

Bastien Montagne commented

2022-03-03 10:13:23 +01:00

Changed status from 'Needs User Info' to: 'Confirmed'

Ad Morris commented

2022-03-03 10:39:41 +01:00

Would it not be more reasonable to leave the proxy system available as a fallback (which handles this so called extreme use case without issue) until said dynamic override system is actually complete then?

I believe you will you find a great many users working with CAD data are regularly required to work with object heirarchies of this nature and it isn't as much of a corner case as you think. Regardless of the character centric workflows you're likely more used encountering in the work of the Blender Studio.

Would it not be more reasonable to leave the proxy system available as a fallback (which handles this so called extreme use case without issue) until said dynamic override system is actually complete then? I believe you will you find a great many users working with CAD data are regularly required to work with object heirarchies of this nature and it isn't as much of a corner case as you think. Regardless of the character centric workflows you're likely more used encountering in the work of the Blender Studio.

Bastien Montagne commented

2022-03-03 15:02:37 +01:00

Added subscriber: @Jeroen-Bakker

Bastien Montagne commented

2022-03-03 15:02:37 +01:00

I ran some profiling this morning, unfortunately lost the results before making a screen cap, but 90% of the time is spent in pre/post processing of the last bit of remapping code not moved to new multi-processing code from @Jeroen-Bakker (namely, calls to BKE_libblock_relink_ex). So I'm going to address this, hopefully it will bring performances back into somewhat acceptable area (a few minutes and not tens of minutes). But again, such case, based on relationships between thousands of Objects, is not really supported by Blender, and we cannot guarantee anything when it comes to it.

I ran some profiling this morning, unfortunately lost the results before making a screen cap, but 90% of the time is spent in pre/post processing of the last bit of remapping code not moved to new multi-processing code from @Jeroen-Bakker (namely, calls to `BKE_libblock_relink_ex`). So I'm going to address this, hopefully it will bring performances back into somewhat acceptable area (a few minutes and not tens of minutes). But again, such case, based on relationships between thousands of Objects, is not really supported by Blender, and we cannot guarantee anything when it comes to it.

Bastien Montagne commented

2022-03-03 18:00:42 +01:00

FYI, D14240: Add 'multiple' variant of ID relink function. brings liboverride creation on Machine collection linked from this file from almost 30 minutes to about 45 seconds on my machine.

FYI, [D14240: Add 'multiple' variant of ID relink function.](https://archive.blender.org/developer/D14240) brings liboverride creation on `Machine` collection linked from this file from almost 30 minutes to about 45 seconds on my machine.

Ad Morris commented

2022-03-03 18:04:13 +01:00

That sounds much more promising :)

Raimund Klink commented

2022-03-16 16:38:39 +01:00

Added subscriber: @Raimund58

blender-admin commented

2022-03-16 17:52:16 +01:00

This issue was referenced by 7e06fc11b7

This issue was referenced by 7e06fc11b7cbf532bd8017b270f53e3a11f21523

Aaron Carlisle commented

2022-03-16 22:10:38 +01:00

@mont29 do you now consider this resolved?

Bastien Montagne commented

2022-03-16 23:09:50 +01:00

@Blendify I keep this around more as an extreme corner case reference for general optimization investigations... this is not really a bug, and in that sense there is no 'solving' it.

Satish Goda commented

2022-03-21 22:38:52 +01:00

Added subscriber: @satishgoda1

Philipp Oeser removed the

Download

What's New

Blender Studio

Manual

Developers Blog

Documentation

Benchmark

Blender Conference

Development Fund

One-time Donations

Dynamic Overrides Use Case - Giant object-based 'rig' with a few key controlers. #96092