Iterating over scene's objects while we modify those (through proxy to
override conversion code) is call for problems (use after free etc.).
Instead, all proxy objects need to be gathered first in a temporary
list, and processed all at once in a second loop.
Those cases are fairly hard to track down... Added some more checks,
also at lower levels, more generic levels of object editing, and fixed
core check in liboverride (previously code was assuming that an override
of a collection only could have overrides of objects or linked objects,
but this is not necessarily true).
Part of the resynching code would access collections' objects base
cache, which can be invalid at that point (due to previous ID remapping
and/or deletion). Use a custom recursive iterator over collections'
objects instead, since those 'raw' data like collection's objects list,
and collection's children lists, should always be valid.
Found while investigating a studio production file.
Those cases are almost always synptoms of either bug in code, or broken
files. Re-doin resync on them only costs time and causes extra trash
data as a result, without really helping in any way.
During sprite fright loading of complex scenes would spend a long time in remapping ID's
The remapping process is done on a per ID instance that resulted in a very time consuming
process that goes over every possible ID reference to find out if it needs to be updated.
If there are N of references to ID blocks and there are M ID blocks that needed to be remapped
it would take N*M checks. These checks are scattered around the place and memory.
Each reference would only be updated at most once, but most of the time no update is needed at all.
Idea: By grouping the changes together will reduce the number of checks resulting in improved performance.
This would only require N checks. Additional benefits is improved data locality as data is only loaded once
in the L2 cache.
It has be implemented for the resyncing process and UI editors.
On an Intel(R) Core(TM) i7-6700 CPU @ 3.40GHz 16Gig the resyncing process went
from 170 seconds to 145 seconds (during hotspot recording).
After this patch has been applied we could add similar approach
to references (references between data blocks) and functionality (tagged deletion).
In my understanding this could reduce the resyncing process to less than a second.
Opening the village production file between 10 and 20 seconds.
Flame graphs showing that UI remapping isn't visible anymore (`WM_main_remap_editor_id_reference`)
* Master {F12769210 size=full}
* This patch {F12769211 size=full}
Reviewed By: mont29
Maniphest Tasks: T94185
Differential Revision: https://developer.blender.org/D13615
This reverts commit 948211679f.
This commit introduced some regressions in the test suite.
As this change is a core part of blender Bastien and I decided to revert
it as the solution isn't clear and needs more investigation.
The following tests FAILED:
62 - blendfile_liblink (SEGFAULT)
63 - blendfile_library_overrides (SEGFAULT)
It fails in (id_us_ensure_real)
During sprite fright loading of complex scenes would spend a long time in remapping ID's
The remapping process is done on a per ID instance that resulted in a very time consuming
process that goes over every possible ID reference to find out if it needs to be updated.
If there are N of references to ID blocks and there are M ID blocks that needed to be remapped
it would take N*M checks. These checks are scattered around the place and memory.
Each reference would only be updated at most once, but most of the time no update is needed at all.
Idea: By grouping the changes together will reduce the number of checks resulting in improved performance.
This would only require N checks. Additional benefits is improved data locality as data is only loaded once
in the L2 cache.
It has be implemented for the resyncing process and UI editors.
On an Intel(R) Core(TM) i7-6700 CPU @ 3.40GHz 16Gig the resyncing process went
from 170 seconds to 145 seconds (during hotspot recording).
After this patch has been applied we could add similar approach
to references (references between data blocks) and functionality (tagged deletion).
In my understanding this could reduce the resyncing process to less than a second.
Opening the village production file between 10 and 20 seconds.
Flame graphs showing that UI remapping isn't visible anymore (`WM_main_remap_editor_id_reference`)
* Master {F12769210 size=full}
* This patch {F12769211 size=full}
Reviewed By: mont29
Maniphest Tasks: T94185
Differential Revision: https://developer.blender.org/D13615
Simplify signature of `BKE_lib_override_library_resync` and make it a
shallow wrapper around new internal `lib_override_library_resync` that
can then be easily extended for other internal needs.
Not functional changes expected here.
Request from studio, to help identify quickly libs that need update.
NOTE: Currently only outputing INFO log in console, display of this info
in the outliner will come in a separate commit.
Overrides that are not created as part of an override hierarchy should
not be handled through (auto)resync at all. users are responsible to
hanlde those updates if they need it.
This is achieved by flagging overrides created outside of a hierarchical
process accordingly, and skipping them during resync process.
Fully get rid of `BKE_collection_object_find` in
`lib_override_group_tag_data_object_to_collection_init`, even if only
used a few times this function was still noticeable in profiling data.
Now instead loop over collections' objects to build required
object-to-collections mapping.
Adds an extra 5-10% speed-up compared to previous commit rB0624fad0f3ff.
Related to T94059.
`BKE_collection_object_find` has extremely bad performances (very high
time complexity). While ideally this should be fixed in that API, for
now cache its results once at the beginning of the resync/creation
process.
This makes loading of complex production files with a lot of
liboverrides to resync three to four times faster.
Thanks to @brecht for the profiling in T94059.
- Added space below non doc-string comments to make it clear
these aren't comments for the symbols directly below them.
- Use doxy sections for some headers.
- Minor improvements to doc-strings.
Ref T92709
This should not be allowed in general, added some initial call to check
when user is allowed to delete a data to search for mandatory override
usages...
Proxy conversion is a fairly particular case of liboverride creation, in
which remapping all local usages of linked data probably makes more
sense, rather than only doing so whitin the overridden 'group' of IDs.
This commit also add an experimental userPreferences to prevent proxies
conversions on file load, and reporting for amount of coverted proxies
(and possible issues).
Note that potentially linked proxies from other libraries are not
hamdled here (this feature seems to be broken anyway in master
currently?).
This is used to find a valid collection in which to instantiate stray
objects and collections.
In some cases there will be no such active view layer, in which case we
can consider using the Scene's master collections children hierarchy
instead to find a valid instantiated parent collection for those stray
data.
LibOverride of scenes is not really supported currently, there are many
issues with it.
Will disable most user-accessible ways to create such overrides in a
following commit.
This makes the internal naming consistent with the public API. And also gives
us a visibility_flag rather than restrictflag that can be extended with more
flags.
Recent own rBabf3ce811f6e prevented any LayerCollection update during
the whole liboverride resync process, for both performances and feature
reasons.
However that means that the various runtime caches like the Base GHash
are not cleared anymore during ID remapping process, so we need to call
`BKE_main_collection_sync_remap` instead of `BKE_main_collection_sync`
when we finally are ready for this update.
Reported by @eyecandy (Andy Goralczyk) from Blender studio, thanks!
This is an easy & safe, yet not-so-nice way to address the
LayerCollections vs. Collections hierarchy resync problem.
Currently this resync is enforced everytime something changes in the
Collections hierarchy, which is extremely inneficient, and can even
produce 'loss' of LayerCollection data during complex Collection
processes.
Current example is during Library Overrides resync process. New code:
* Makes resync significantly faster (between 10 and 15%).
* Fixes 'disappearing' layer collections settings on sub-collections'
layers.
NOTE: This is not a proper fix for the underlying issue. However,
implementing and testing the 'lazy update' solution as proposed by
{T73411} requires a significant amount of time (especially in testing
and tracking all places where code would need to ensure LayerCollections
are up-to-date), which is not possible currently.
Differential Revision: https://developer.blender.org/D11889
Weird 'embedded for overrides' flag of embedded IDs (including ShapeKeys
in override context) was not properly cleaned up when making an override
fully local.
Reported by studio, thanks.
@jbakker should be backported to 2.93LTS if possible.
This change will ensure at least one 'local' collection can host the new
'local' override of all objects (indirectly) overridden by this
operation, such that no new override of object ends up in master
collection (which can become extremely messy in production files).
In practice, it means often at least one of the linked collection owning
those objects also has to be overridden.
NOTE: This only affect cases where root overridden linked object has
some dependencies outside of its own root linked collection. While this
situation should be avoided, it cannot always be, so we try to support
it as best as we can.
This reverts commit rB3a48147b8ab92, and fixes the issues with linking
etc.
Change compared to previous buggy commit (rBf8d219dfd4c31) is that
new `BlendFileReadReports` reports are now passed to the lowest level
function generating the `FileData` (`filedata_new()`), which ensures
(and asserts) that all code using it does have a valid non-NULL pointer
to a `BlendFileReadReport` data.
Sorry for the noise, it's always when you think a change is trivial and
do not test it well enough that you end up doing those kind of
mistakes...
Add direct user feedback (as a warning report) to user when recursive
resync of overrides was needed.
And some timing (as CLOG logs) about main readfile process steps.
This is essentially adding a new BlendFileReadReport structure that wraps
BKE_reports list, and adds some extra info (some timing, some info about
overrides and (recursive) resync, etc.).
After looking into task isolation issues with Sergey, we couldn't find the
reason behind the deadlocks that we are getting in T87938 and a Sprite Fright
file involving motion blur renders.
There is no apparent place where we adding or waiting on tasks in a task group
from different isolation regions, which is what is known to cause problems. Yet
it still hangs. Either we do not understand some limitation of TBB isolation,
or there is a bug in TBB, but we could not figure it out.
Instead the idea is to use isolation only where we know we need it: when
holding a mutex lock and then doing some multithreaded operation within that
locked region. Three places where we do this now:
* Generated images
* Cached BVH tree building
* OpenVDB lazy grid loading
Compared to the more automatic approach previously used, there is the downside
that it is easy to miss places where we need isolation. Yet doing it more
automatically is also causing unexpected issue and bugs that we found no
solution for, so this seems better.
Patch implemented by Sergey and me.
Differential Revision: https://developer.blender.org/D11603
Under some circumstances using task isolation can cause deadlocks.
Previously, our task pool implementation would run all tasks in an
isolated region. Now using task isolation is optional and can be
turned on/off for individual task pools.
Task pools that spawn new tasks recursively should never enable
task isolation. There is a new check that finds these cases at runtime.
Right now this check is disabled, so that this commit is a pure refactor.
It will be enabled in an upcoming commit.
This fixes T88598.
Differential Revision: https://developer.blender.org/D11415
In some cases e.g. only objects would actually need resync, so
collections on the override character would not be resynced, and if some
objects were sharing relationships with others those could be
lost/destroyed.
