Note that some little parts of code have been dissabled because eval_ctx
was not available there. This should be resolved once DerivedMesh is
replaced.
< Dependency graph Copy-on-Write >
--------------------------------
\ ^__^
\ (oo)\_______
(__)\ )\/\
||----w |
|| ||
This is an initial commit of Copy-on-write support added to dependency graph.
Main priority for now: get playback (Alt-A) and all operators (selection,
transform etc) to work with the new concept of clear separation between
evaluated data coming from dependency graph and original data coming from
.blend file (and stored in bmain).
= How does this work? =
The idea is to support Copy-on-Write on the ID level. This means, we duplicate
the whole ID before we cann it's evaluaiton function. This is currently done
in the following way:
- At the depsgraph construction time we create "shallow" copy of the ID
datablock, just so we know it's pointer in memory and can use for function
bindings.
- At the evaluaiton time, the copy of ID get's "expanded" (needs a better
name internally, so it does not conflict with expanding datablocks during
library linking), which means the content of the datablock is being
copied over and all IDs are getting remapped to the copied ones.
Currently we do the whole copy, in the future we will support some tricks
here to prevent duplicating geometry arrays (verts, edges, loops, faces
and polys) when we don't need that.
- Evaluation functions are operating on copied datablocks and never touching
original datablock.
- There are some cases when we need to know non-ID pointers for function
bindings. This mainly applies to scene collections and armatures. The
idea of dealing with this is to "expand" copy-on-write datablock at
the dependency graph build time. This might introduce some slowdown to the
dependency graph construction time, but allows us to have minimal changes
in the code and avoid any hash look-up from evaluation function (one of
the ideas to avoid using pointers as function bindings is to pass name
of layer or a bone to the evaluation function and look up actual data based
on that name).
Currently there is a special function in depsgraph which does such a
synchronization, in the future we might want to make it more generic.
At some point we need to synchronize copy-on-write version of datablock with
the original version. This happens, i.e., when we change active object or
change selection. We don't want any actual evaluation of update flush happening
for such thins, so now we have a special update tag:
DEG_id_tag_update((id, DEG_TAG_COPY_ON_WRITE)
- For the render engines we now have special call for the dependency graph to
give evaluated datablock for the given original one. This isn't fully ideal
but allows to have Cycles viewport render.
This is definitely a subject for further investigation / improvement.
This call will tag copy-on-write component tagged for update without causing
updates to be flushed to any other objects, causing chain reaction of updates.
This tag is handy when selection in the scene changes.
This basically summarizes ideas underneath this commit. The code should be
reasonably documented.
Here is a demo of dependency graph with all copy-on-write stuff in it:
https://developer.blender.org/F635468
= What to expect to (not) work? =
- Only meshes are properly-ish aware of copy-on-write currently, Non-mesh
geometry will probably crash or will not work at all.
- Armatures will need similar depsgraph built-time expansion of the copied
datablock.
- There are some extra tags / relations added, to keep things demo-able but
which are slowing things down for evaluation.
- Edit mode works for until click selection is used (due to the selection
code using EditDerivedMesh created ad-hoc).
- Lots of tools will lack tagging synchronization of copied datablock for
sync with original ID.
= How to move forward? =
There is some tedious work related on going over all the tools, checking
whether they need to work with original or final evaluated object and make
the required changes.
Additionally, there need synchronization tag done in fair amount of tools
and operators as well. For example, currently it's not possible to change
render engine without re-opening the file or forcing dependency graph for
re-build via python console.
There is also now some thoughts required about copying evaluated properties
between objects or from collection to a new object. Perhaps easiest way
would be to move base flag flush to Object ID node and tag new objects for
update instead of doing manual copy.
here is some WIP patch which moves such evaluaiton / flush:
https://developer.blender.org/F635479
Lots of TODOs in the code, with possible optimization.
= How to test? =
This is a feature under heavy development, so obviously it is disabled by
default. The only reason it goes to 2.8 branch is to avoid possible merge
hell.
In order to enable this feature use WITH_DEPSGRAPH_COPY_ON_WRITE CMake
configuration option.
Noisy change, but safe, and better do it sooner than later if we are to
rework copying code. Also, previous commit shows this *is* useful to
catch some mistakes.
Add face maps, needed for face-map widgets,
only data structure, widgets will be separate commit.
This comes from 'custom-manipulator' branch with only minor changes.
Objects that were using the same lamp data were having the same display matrices.
This is fixed by allowing engine to store a memory block inside the object itself.
This moves selectability/visibility flag flush from some hardcoded
places in the code to depsgraph. This way it is possible to simply
tag depsgraph to update those flags and rest it'll do on its own.
Using depsgraph for such flush is an overkill: those flags are fully
static and can not be animated, so it doesn't really make sense to
hook only those to depsgraph.
However, in the future we will have overrides on collections, which
ideally would need to be animatable and drivable and easiest way
to support this is to do this on depsgraph level, so it ensures
proper order of evaluation for animation and drivers. And it seems
logical to do both overrides and flags flush from depsgraph from
this point of view.
This commit also includes the evaluation of IDProperty for collections,
which basically are just another form of override. So once we implement
the other kind of overrides the flushing and collection evaluation won't
change.
Patch by Sergey Sharybin and Dalai Felinto
First this replace a custom data struct with IDProperty, and use
IDProperty group merge and copying functions. Which means that a collection
property setting is only created if necessary.
This implements the "Layer Collection settings" override system, as
suggested in the "Override Manifesto" document.
The core is working, with Scene, LayerCollection and Object using a
single IDProperty to store all the render settings data. Next step is to
migrate this to depsgraph.
Note: Clay engine "ssao_samples" was hardcoded to 32 for now. It will come
back as part of "Workspace Settings" later.
Many thanks for Bastien Montagne for the help with the UI template
nightmare ;)
Differential Revision: https://developer.blender.org/D2563
There are now only referenced in:
* drawobject.c
* particle_edit.c
* space_image.c (a single case to be handled on workspace branch)
* rigidbody_constraint.c (to be handled in the following commit)
This was causing blender to segfault.
We now add create a new collection and link to the layer before adding
the new object
(also included unittests, and requires updated lib/tests)
The bug T46099 no longer applies since the addition of `dist_squared_to_projected_aabb_simple`
Has also been added comments that relates to an occlusion bug with the ruler. I'll investigate this.
Note: we are now merging all the collection engines (mode and render), which eventually may get slow. But as stated before, this is to expose the functionality, while waiting for proper depsgraph integrated solution.
Every time:
* A collection settings is set
* A collection visibility changes
* An object is added/removed/ ...
We need to recalculate the "accumulated" CollectionEngineSettings that
the render engine should use for an object.
This is to be handled by the depsgraph. Meanwhile this code should allow
us to start using those settings in the render engines.
Note: We are storing this in the objects, which means we can only have
one active calculated option every time.
This is intended to get the conversation with the Depsgraph department
going.
Design Documents
----------------
* https://wiki.blender.org/index.php/Dev:2.8/Source/Layers
* https://wiki.blender.org/index.php/Dev:2.8/Source/DataDesignRevised
User Commit Log
---------------
* New Layer and Collection system to replace render layers and viewport layers.
* A layer is a set of collections of objects (and their drawing options) required for specific tasks.
* A collection is a set of objects, equivalent of the old layers in Blender. A collection can be shared across multiple layers.
* All Scenes have a master collection that all other collections are children of.
* New collection "context" tab (in Properties Editor)
* New temporary viewport "collections" panel to control per-collection
visibility
Missing User Features
---------------------
* Collection "Filter"
Option to add objects based on their names
* Collection Manager operators
The existing buttons are placeholders
* Collection Manager drawing
The editor main region is empty
* Collection Override
* Per-Collection engine settings
This will come as a separate commit, as part of the clay-engine branch
Dev Commit Log
--------------
* New DNA file (DNA_layer_types.h) with the new structs
We are replacing Base by a new extended Base while keeping it backward
compatible with some legacy settings (i.e., lay, flag_legacy).
Renamed all Base to BaseLegacy to make it clear the areas of code that
still need to be converted
Note: manual changes were required on - deg_builder_nodes.h, rna_object.c, KX_Light.cpp
* Unittesting for main syncronization requirements
- read, write, add/copy/remove objects, copy scene, collection
link/unlinking, context)
* New Editor: Collection Manager
Based on patch by Julian Eisel
This is extracted from the layer-manager branch. With the following changes:
- Renamed references of layer manager to collections manager
- I doesn't include the editors/space_collections/ draw and util files
- The drawing code itself will be implemented separately by Julian
* Base / Object:
A little note about them. Original Blender code would try to keep them
in sync through the code, juggling flags back and forth. This will now
be handled by Depsgraph, keeping Object and Bases more separated
throughout the non-rendering code.
Scene.base is being cleared in doversion, and the old viewport drawing
code was poorly converted to use the new bases while the new viewport
code doesn't get merged and replace the old one.
Python API Changes
------------------
```
- scene.layers
+ # no longer exists
- scene.objects
+ scene.scene_layers.active.objects
- scene.objects.active
+ scene.render_layers.active.objects.active
- bpy.context.scene.objects.link()
+ bpy.context.scene_collection.objects.link()
- bpy_extras.object_utils.object_data_add(context, obdata, operator=None, use_active_layer=True, name=None)
+ bpy_extras.object_utils.object_data_add(context, obdata, operator=None, name=None)
- bpy.context.object.select
+ bpy.context.object.select = True
+ bpy.context.object.select = False
+ bpy.context.object.select_get()
+ bpy.context.object.select_set(action='SELECT')
+ bpy.context.object.select_set(action='DESELECT')
-AddObjectHelper.layers
+ # no longer exists
```
The previous solution took arbitrary values to determine if the mouse was near or not to the Bound Box (it simply scaled the Bound Box).
Now the same function that detected the distance from the BVHTree nodes to the mouse is used in the Bound Box
Use Main->relations in BKE_library_foreach_ID_link(), when possible
(i.e. IDWALK_READONLY is set), and if the data is available of course.
This is quite minor optimization, no sensible improvements are expected,
but does not hurt either to avoid potentially tens of looping over e.g.
objects constraints and modifiers, or heap of drivers...
This aims at always ensuring that ID.newid (and relevant LIB_TAG_NEW)
stay in clean (i.e. cleared) state by default.
To achieve this, instead of clearing after all id copy call (would be
horribly noisy, and bad for performances), we try to completely remove
the setting of id->newid by default when copying a new ID.
This implies that areas actually needing that info (mainly, object editing
area (make single user...) and make local area) have to ensure they set
it themselves as needed.
This is far from simple change, many complex code paths to consider, so
will need some serious testing. :/
This allows appending of an entire scene from another blend file into this one,
even when that blend file contains proxified armatures.
This replaces the approach from commit 1cdc54dc7d.
Thanks @sergey for the help.
In fact, it was the whole remapping process that was broken in logic bricks area,
due to terrible design of links between those bricks...
Object copying was also broken in that case, fixed as well.
To be backported to 2.78.
Note that issue was actually probably there since ages, hidden behind dirty hacks
used in previous append code (though likely visible in some corner cases).
Listen kids: do not, never, ever, do what has been done for links between logic bricks. Never. Ever.
Even as pure runtime data it would have been bad, but as stored data...
Another great example of inconsistency in usercount handling - dupli_group was considered
as refcounted by readfile.c code (and hence by library_query.c one, which is based on it),
but not by editor/BKE_object code, which never increased group's usercount when creating
an instance of it etc.
To be backported to 2.78.
Turns out most BKE_foo_make_local datablock-specific functions are actually doing
exactly the same thing, only two currently need special additional operations
(object and brush ones). So added a BKE_id_make_local_generic instead
of copying same code over and over.
Also, changed a bit how make_local works in case we are localizing a whole library.
We need to do the 'remap' step (from old linked ID to new local one) in the second loop,
otherwise we miss some dependencies. This fixes main part of T48907.
Idea looked good, but we have too much custom situations here (some half-fake-sub-ID
being copied with their 'owner', animdata, etc.), let's let datablock copy functions
handle that themselves.
Also allows to safely call BKE_id_expand_local from all copy functions now (only when
copying linked data).
