Although this works by itself, it should actually happen after:
"Reshuffle collections base flags evaluation, make it so object is gathering
its base flags from collections."
Meanwhile we have one single hacky function (deg_flush_base_flags_and_settings)
to be removed once the task above is tackled.
Reviewers: sergey
Differential Revision: https://developer.blender.org/D2899
It will not be possible to do that after depsgraph becomes more context
oriented. Which means, all code will need to explicitly tell which graph
to free,
Before it was a compile time option which was not very easy to use or test. Now
the project is getting more mature, so very soon we will be able to call for a
public tests of limited features.
The copy-on-write (which includes animation, modifiers) is enabled using
--enable-copy-on-write command line argument.
This commit is a work forward having less updates during playback, which speeds
things up a lot here. The idea is simple: stop update all copy-on-write
datablocks (which implies full re-evaluation actually) on frame change and
re-use existing evaluated meshes as much as possible.
This brings playback speed to 24 fps on the dino test scene here. Performance
drops down a lot when armature is animated tho, but that's because of need of
tangent space calculation which we can't do much about from just a dependency
graph.
Hopefully this doesn't make copy-on-write too unstable, quick tests here are
surviving fine.
Note that some little parts of code have been dissabled because eval_ctx
was not available there. This should be resolved once DerivedMesh is
replaced.
This is a first step towards proper depsgraph "ownership", where
we would allow scene to be in multiple states dependent on active
workspace or scene layer.
This commit introduces a basic API to get proper dependency graph
for a given scene layer. It also renames scene->depsgraph to
depsgraph_legacy, so it's easier to search0-n-replace in the future.
< Dependency graph Copy-on-Write >
--------------------------------
\ ^__^
\ (oo)\_______
(__)\ )\/\
||----w |
|| ||
This is an initial commit of Copy-on-write support added to dependency graph.
Main priority for now: get playback (Alt-A) and all operators (selection,
transform etc) to work with the new concept of clear separation between
evaluated data coming from dependency graph and original data coming from
.blend file (and stored in bmain).
= How does this work? =
The idea is to support Copy-on-Write on the ID level. This means, we duplicate
the whole ID before we cann it's evaluaiton function. This is currently done
in the following way:
- At the depsgraph construction time we create "shallow" copy of the ID
datablock, just so we know it's pointer in memory and can use for function
bindings.
- At the evaluaiton time, the copy of ID get's "expanded" (needs a better
name internally, so it does not conflict with expanding datablocks during
library linking), which means the content of the datablock is being
copied over and all IDs are getting remapped to the copied ones.
Currently we do the whole copy, in the future we will support some tricks
here to prevent duplicating geometry arrays (verts, edges, loops, faces
and polys) when we don't need that.
- Evaluation functions are operating on copied datablocks and never touching
original datablock.
- There are some cases when we need to know non-ID pointers for function
bindings. This mainly applies to scene collections and armatures. The
idea of dealing with this is to "expand" copy-on-write datablock at
the dependency graph build time. This might introduce some slowdown to the
dependency graph construction time, but allows us to have minimal changes
in the code and avoid any hash look-up from evaluation function (one of
the ideas to avoid using pointers as function bindings is to pass name
of layer or a bone to the evaluation function and look up actual data based
on that name).
Currently there is a special function in depsgraph which does such a
synchronization, in the future we might want to make it more generic.
At some point we need to synchronize copy-on-write version of datablock with
the original version. This happens, i.e., when we change active object or
change selection. We don't want any actual evaluation of update flush happening
for such thins, so now we have a special update tag:
DEG_id_tag_update((id, DEG_TAG_COPY_ON_WRITE)
- For the render engines we now have special call for the dependency graph to
give evaluated datablock for the given original one. This isn't fully ideal
but allows to have Cycles viewport render.
This is definitely a subject for further investigation / improvement.
This call will tag copy-on-write component tagged for update without causing
updates to be flushed to any other objects, causing chain reaction of updates.
This tag is handy when selection in the scene changes.
This basically summarizes ideas underneath this commit. The code should be
reasonably documented.
Here is a demo of dependency graph with all copy-on-write stuff in it:
https://developer.blender.org/F635468
= What to expect to (not) work? =
- Only meshes are properly-ish aware of copy-on-write currently, Non-mesh
geometry will probably crash or will not work at all.
- Armatures will need similar depsgraph built-time expansion of the copied
datablock.
- There are some extra tags / relations added, to keep things demo-able but
which are slowing things down for evaluation.
- Edit mode works for until click selection is used (due to the selection
code using EditDerivedMesh created ad-hoc).
- Lots of tools will lack tagging synchronization of copied datablock for
sync with original ID.
= How to move forward? =
There is some tedious work related on going over all the tools, checking
whether they need to work with original or final evaluated object and make
the required changes.
Additionally, there need synchronization tag done in fair amount of tools
and operators as well. For example, currently it's not possible to change
render engine without re-opening the file or forcing dependency graph for
re-build via python console.
There is also now some thoughts required about copying evaluated properties
between objects or from collection to a new object. Perhaps easiest way
would be to move base flag flush to Object ID node and tag new objects for
update instead of doing manual copy.
here is some WIP patch which moves such evaluaiton / flush:
https://developer.blender.org/F635479
Lots of TODOs in the code, with possible optimization.
= How to test? =
This is a feature under heavy development, so obviously it is disabled by
default. The only reason it goes to 2.8 branch is to avoid possible merge
hell.
In order to enable this feature use WITH_DEPSGRAPH_COPY_ON_WRITE CMake
configuration option.
Was rather weird and only used for time source. It is simpler to make depsgraph
to keep track of time source directly.
No need to introduce extra entitites without actual need.
It was never actually used apart from being stored at a construciton time.
This caused some redundancy and ncertanty about which relation type to use
during construciton (often existing types were not close enough to particular
use case).
These bits became obsolete with the new layer system, so we can
simplify some code around them or avoid existing workarounds which
were trying to keep things working for them.
There are still work needed to be done for on_visible_change to
avoid unnecessary updates, but that can also happen later.
The idea is simple: when falling back to one of the nodes which was partially
handled we "resume" checking outgoing relations from the index which we stopped.
This gives about 15-20% depsgraph construction time save.
Current implementation more or less indiscriminately links physics
objects to colliders and forces, ignoring precise details of layer
checks and collider groups. The new depsgraph seemed to lack some
such links at all. The relevant code in modifiers suffers from a
lot of duplication.
Different physics simulations use independent implementations of
collision and similar things, which results in a lot of variance:
* Cloth collides with objects on same or visible layer with dupli.
* Softbody collides with objects on same layer without dupli.
* Non-hair particles collide on same layer with dupli.
* Smoke uses same code as cloth, but needs different modifier.
* Dynamic paint "collides" with brushes on any layer without dupli.
Force fields with absorption also imply dependency on colliders:
* For most systems, colliders are selected from same layer as field.
* For non-hair particles, it uses the same exact set as the particles.
As a special quirk, smoke ignores smoke flow force fields; on the other
hand dependency on such field implies dependency on the smoke domain.
This introduces two utility functions each for old and new depsgraph
that are flexible enough to handle all these variations, and uses them
to handle particles, cloth, smoke, softbody and dynpaint.
One thing to watch out for is that depsgraph code shouldn't rely on
any properties that don't cause a graph rebuild when changed. This
was violated in the original code that was building force field links,
while taking zero field weights into account.
This change may cause new dependency cycles in cases where necessary
dependencies were missing, but may also remove cycles in situations
where unnecessary links were previously created. It's also now possible
to solve some cycles by switching to explicit groups, since they are
now properly taken into account for dependencies.
Differential Revision: https://developer.blender.org/D2141
All in all, this patch adds an Alembic importer, an Alembic exporter,
and a new CacheFile data block which, for now, wraps around an Alembic
archive. This data block is made available through a new modifier ("Mesh
Sequence Cache") as well as a new constraint ("Transform Cache") to
somewhat properly support respectively geometric and transformation data
streaming from alembic caches.
A more in-depth documentation is to be found on the wiki, as well as a
guide to compile alembic: https://wiki.blender.org/index.php/
User:Kevindietrich/AlembicBasicIo.
Many thanks to everyone involved in this little project, and huge shout
out to "cgstrive" for the thorough testings with Maya, 3ds Max, Houdini
and Realflow as well as @fjuhec, @jensverwiebe and @jasperge for the
custom builds and compile fixes.
Reviewers: sergey, campbellbarton, mont29
Reviewed By: sergey, campbellbarton, mont29
Differential Revision: https://developer.blender.org/D2060
This is mainly a maintenance commit which was aimed to make work with
this module more pleasant and solve such issues as:
- Annoyance with looong files, which had craftload in them
- Usage of STL for the data structures we've got in BLI
- Possible symbol conflicts
- Not real clear layout of what is located where
So in this commit the following changes are done:
- STL is prohibited, it's not really predictable on various compilers,
with our BLI algorithms we can predict things much better.
There are still few usages of std::vector, but that we'll be
solving later once we've got similar thing in BLI.
- Simplify foreach loops, avoid using const_iterator all over the place.
- New directory layout, which is hopefully easier to follow.
- Some files were split, some of them will be split soon.
The idea of this is to split huge functions into own files with
good documentation and everything.
- Removed stuff which was planned for use in the future but was never
finished, tested or anything.
Let's wipe it out for now, and bring back once we really start using
it, so it'll be more clear if it solves our needs.
- All the internal routines were moved to DEG namespace to separate
them better from rest of blender.
Some places now annoyingly using DEG::foo, but that we can olve by
moving some utility functions inside of the namespace.
While working on this we've found some hotspot in updates flush, so
now playback of blenrig is few percent faster (something like 96fps
with previous master and around 99-100fps after this change).
Not saying it's something final, there is still room for cleanup and
API simplification, but those might happen as a regular development
now without doing any global changes.
Some of the tagging functions would be called for every operation, and then
in turn tag their ID nodes with all their operations again. With extensive rigs
we get ID nodes with a lot (10,000+) operation nodes, which leads to millions
of unnecessary tagging calls.
This is purely internal sanitizing/cleanup, no change in behavior is expected at all.
This change was also needed because we were getting short on ID flags, and
future enhancement of 'user_one' ID behavior requires two new ones.
id->flag remains for persistent data (fakeuser only, so far!), this also allows us
100% backward & forward compatibility.
New id->tag is used for most flags. Though written in .blend files, its content
is cleared at read time.
Note that .blend file version was bumped, so that we can clear runtimeflags from
old .blends, important in case we add new persistent flags in future.
Also, behavior of tags (either status ones, or whether they need to be cleared before/after use)
has been added as comments to their declaration.
Reviewers: sergey, campbellbarton
Differential Revision: https://developer.blender.org/D1683
This commit integrates the work done so far on the new dependency graph system,
where goal was to replace legacy depsgraph with the new one, supporting loads of
neat features like:
- More granular dependency relation nature, which solves issues with fake cycles
in the dependencies.
- Move towards all-animatable, by better integration of drivers into the system.
- Lay down some basis for upcoming copy-on-write, overrides and so on.
The new system is living side-by-side with the previous one and disabled by
default, so nothing will become suddenly broken. The way to enable new depsgraph
is to pass `--new-depsgraph` command line argument.
It's a bit early to consider the system production-ready, there are some TODOs
and issues were discovered during the merge period, they'll be addressed ASAP.
But it's important to merge, because it's the only way to attract artists to
really start testing this system.
There are number of assorted documents related on the design of the new system:
* http://wiki.blender.org/index.php/User:Aligorith/GSoC2013_Depsgraph#Design_Documents
* http://wiki.blender.org/index.php/User:Nazg-gul/DependencyGraph
There are also some user-related information online:
* http://code.blender.org/2015/02/blender-dependency-graph-branch-for-users/
* http://code.blender.org/2015/03/more-dependency-graph-tricks/
Kudos to everyone who was involved into the project:
- Joshua "Aligorith" Leung -- design specification, initial code
- Lukas "lukas_t" Toenne -- integrating code into blender, with further fixes
- Sergey "Sergey" "Sharybin" -- some mocking around, trying to wrap up the
project and so
- Bassam "slikdigit" Kurdali -- stressing the new system, reporting all the
issues and recording/writing documentation.
- Everyone else who i forgot to mention here :)
