Some runtime data that stores which sockets can be fields and which
can't is not stored in the file, but only calculated when necessary.
When opening a file, the node tree update function was called, which
recalculated this data, but that was explicily turned off for undo.
This exposes a fundamental issue with undo, the ID caching system for
undo, and how it relates to node trees in particular. Ideally this call
couldn't be necessary at all. In the future it could be removed by
adding a runtime struct to node trees, and calculating its contents
on-demand instead of preemtively.
Differential Revision: https://developer.blender.org/D12699
Copy/Paste uses its own code path for ID linking, which was not setting
`LIB_TAG_DOIT` for proper instantiation later on.
Would be nice the make this logic closer to the rest of the link/append
code at some point, but for now this fix will do.
No idea why this was done that way (it originate from initial paste
commit, rB12b642062c6f).
But the IDs 'selected' as direct paste in `BLO_library_link_copypaste`
should be 'directly' linked, it's similar case to actual append of
selected IDs by the user.
Related to T87189.
Flags controlling link/append code are split between two enums, one in
`DNA_space_types.h` and one in `BLO_readfile.h`.
This commit:
- Moves flags exclusively used in WM and BLO code to `eBLOLibLinkFlags`
in `BLO_readfile.h`. Flags in `eFileSel_Params_Flag` from
`DNA_space_types.h` are now only the ones effectively used by the
file browser editor code too.
- Fixes some internal utils in `readfile.c` still taking `short` flag
parameter instead of proper `int` one.
NOTE: there are a few other flags that could probably be moved to
`eBLOLibLinkFlags` (at the very least `FILE_LINK`, probably also
`FILE_AUTOSELECT` and `FILE_ACTIVE_COLLECTION`), since those are not
effectively used by the file browser, and control linking/appending
behavior, not filebrowser behavior.
However for now think it's safer to not touch that.
This commit adds to ID struct a new optional 'weak reference' to a
linked ID (in the form of a blend file library path and full ID name).
This can then be used on next append to try to find a matching local ID
instead of re-making the linked data local again.
Ref. T90545
NOTE: ID re-use will be disabled for regular append for the time being
(3.0 release), and only used for assets. Therefore, this commit should
not change anything user-wise.
Differential Revision: https://developer.blender.org/D12545
We now use a for_each function with callback to iterate through all sequences in the scene.
This has the benefit that we now only loop over the sequences in the scene once.
Before we would loop over them twice and allocate memory to store temporary data.
The allocation of temporary data lead to unintentional memory leaks if the code used returns to exit out of the iteration loop.
The new for_each callback method doesn't allocate any temporary data and only iterates though all sequences once.
Reviewed By: Richard Antalik, Bastien Montagne
Differential Revision: http://developer.blender.org/D12278
Compressing blendfiles can help save a lot of disk space, but the slowdown
while loading and saving is a major annoyance.
Currently Blender uses Zlib (aka gzip aka Deflate) for compression, but there
are now several more modern algorithms that outperform it in every way.
In this patch, I decided for Zstandard aka Zstd for several reasons:
- It is widely supported, both in other programs and libraries as well as in
general-purpose compression utilities on Unix
- It is extremely flexible - spanning several orders of magnitude of
compression speeds depending on the level setting.
- It is pretty much on the Pareto frontier for all of its configurations
(meaning that no other algorithm is both faster and more efficient).
One downside of course is that older versions of Blender will not be able to
read these files, but one can always just re-save them without compression or
decompress the file manually with an external tool.
The implementation here saves additional metadata into the compressed file in
order to allow for efficient seeking when loading. This is standard-compliant
and will be ignored by other tools that support Zstd.
If the metadata is not present (e.g. because you manually compressed a .blend
file with another tool), Blender will fall back to sequential reading.
Saving is multithreaded to improve performance. Loading is currently not
multithreaded since it's not easy to predict the access patterns of the
loading code when seeking is supported.
In the future, we might want to look into making this more predictable or
disabling seeking for the main .blend file, which would then allow for
multiple background threads that decompress data ahead of time.
The compression level was chosen to get sizes comparable to previous versions
at much higher speeds. In the future, this could be exposed as an option.
Reviewed By: campbellbarton, brecht, mont29
Differential Revision: https://developer.blender.org/D5799
Instead of handling mmap, compression etc. all directly in readfile.c, refactor
the code to use a generic FileReader.
This makes it easier to add new compression methods or similar, and allows to
reuse the logic in other places (e.g. thumbnail reading).
Reviewed By: campbellbarton, brecht, mont29
Differential Revision: https://developer.blender.org/D5799
Ths commit adds a new `IDNameLibMap` to `Main`, used during file reading
to quickly find already read linked IDs.
Without that, search would use string-based search over list of linked
data, which becomes extremely slow and inneficient in cases where a lot
of IDs are linked from a same library. See also {T89194}.
Extrem-usecase reported in T89194 is now about 4 times faster in linked
data reading (about 2 times faster for the whole .blend file loading).
More normal cases (like Sprites studio production files) have barely
measurable speed improvements, a few percents at best.
NOTE: `main_idmap` API was extended to support insertion and removal of
IDs from the mapping, avoids having to re-create the whole thing several
time during libraries expansion in readcode.
Differential Revision: https://developer.blender.org/D11757
Note that this commit uses a second LogRef (`blo.readfile.undo`) for undo
specific meassages. Allows to use `--log "*undo*"` cli option to match
all undo reporting.
Also did some minor tweaks to some reports on the way.
This reverts commit rB3a48147b8ab92, and fixes the issues with linking
etc.
Change compared to previous buggy commit (rBf8d219dfd4c31) is that
new `BlendFileReadReports` reports are now passed to the lowest level
function generating the `FileData` (`filedata_new()`), which ensures
(and asserts) that all code using it does have a valid non-NULL pointer
to a `BlendFileReadReport` data.
Sorry for the noise, it's always when you think a change is trivial and
do not test it well enough that you end up doing those kind of
mistakes...
This patch fixes many minor spelling mistakes, all in comments or
console output. Mostly contractions like can't, won't, don't, its/it's,
etc.
Differential Revision: https://developer.blender.org/D11663
Reviewed by Harley Acheson
Add direct user feedback (as a warning report) to user when recursive
resync of overrides was needed.
And some timing (as CLOG logs) about main readfile process steps.
This is essentially adding a new BlendFileReadReport structure that wraps
BKE_reports list, and adds some extra info (some timing, some info about
overrides and (recursive) resync, etc.).
This change was prompted by D6408 which moves thumbnail extraction into
a shared function that happens use these endian defines but only links
blenlib.
There is no need for these defines to be associated with globals
so move into their own header.
So far, linked IDs were not properly sorted at all, only the ones
explicitely linked from WM code would be, but any indirectly linked
data-blocks would end up in some random order in their lists.
While not ideal, this is not a huge issue in itself, but it had bad
side-effects, e.g. causing (recursive) resync of overrides to happen in
random order, leading to mismatches between name indices of newly-generated
override IDs and the one existings e.g.
And in general, it is much better to be consistent here.
Note that the file sub-version is bumped for this commit, since some
sorting (the directly linked IDs which we keep a reference to) should
never need to be re-done after relevant doversion process.
`SEQ_recursive_apply` and `SEQ_seqbase_recursive_apply` were incorrectly
refactored into `iterator.c` file, but they aren't part and don't use
sequencer iterator.
Functions are moved to `utils.c` file.
Text data block were not considered special in the recursive purge
function. So they would get deleted if they had no actual users.
To fix this we instead make text data block use "fake user" so that
addon authors can specify script files that should be removed if nothing
is using it anymore.
Per default, new text object have "fake user" set. So functionality
wise, the user has to explicitly specify that they want the text object
to be purge-able.
Reviewed By: Bastien
Differential Revision: http://developer.blender.org/D10983
This introduces a context path to the spreadsheet editor, which contains
information about what data is shown in the spreadsheet. The context
path (breadcrumbs) can reference a specific node in a node group
hierarchy. During object evaluation, the geometry nodes modifier checks
what data is currently requested by visible spreadsheets and stores
the corresponding geometry sets separately for later access.
The context path can be updated by the user explicitely, by clicking
on the new icon in the header of nodes. Under some circumstances,
the context path is updated automatically based on Blender's context.
This patch also consolidates the "Node" and "Final" object evaluation
mode to just "Evaluated". Based on the current context path, either
the final geometry set of an object will be displayed, or the data at
a specific node.
The new preview icon in geometry nodes now behaves more like
a toggle. It can be clicked again to clear the context path in an
open spreadsheet editor.
Previously, only an object could be pinned in the spreadsheet editor.
Now it is possible to pin the entire context path. That allows two
different spreadsheets to display geometry data from two different
nodes.
The breadcrumbs in the spreadsheet header can be collapsed by
clicking on the arrow icons. It's not ideal but works well for now.
This might be changed again, if we get a data set region on the left.
Differential Revision: https://developer.blender.org/D10931
CacheFile writing code would not write generic ID data (call for it has
been missing since the initial commit, rB61050f75b13e).
While potentially affecting other areas (mostly CustomProperties/IDProperties),
this was a critical failure for liboverrides. Also added some workaround
code to allow opening broken files (though the override of the CacheFile
data-block will be lost).
Attempt to work around some full-corruption cases created at the studio.
Not clear how those were created, so not really fixing anything here,
just detecting and 'solving' as best as possible some high corruption of
local overrides.
This is good to have in general anyway, might help prevent further
corruption to happen too.
Add new function `blo_bhead_is_id_valid_type()` to correctly check the
blend file block type.
File block type codes have four bytes, and two of those are only in use
when these blocks contain ID datablocks (like `"OB\0\0"`). However,
there are other types defined in `BLO_blend_defs.h` that have four
bytes, like `TEST`, `ENDB`, etc.
The function `BKE_idtype_idcode_is_valid(short idcode)` was used to
check for ID datablocks while reading a blend file. This only takes a
2-byte parameter, and thus its result is invalid for the 4-byte codes.
For `TEST` blocks, it would actually consider it a `TE` block, which is
a valid identifier for a Texture. This caused the heap buffer overflow,
as the datablock is not a valid ID, and thus the bytes that were
expected to form an ID name actually encode something completely
different.
Reviewed By: mont29
Differential Revision: https://developer.blender.org/D10703
This adds support for creating a `BlendFile` (internally called `Main`),
which is limited to a context.
Temporary data can now be created which can then use
`.libraries.load()` the same as with `bpy.data`.
To prevent errors caused by mixing the temporary ID's with data in
`bpy.data` they are tagged as temporary so they can't be assigned
to properties, however they can be passed as arguments to functions.
Reviewed By: mont29, sybren
Ref D10612
The `MAX_LIBARRAY` define was an annoying doublon to the `INDEX_ID_MAX` enum value
now defined in `DNA_ID.h`, and it is no more useful.
And comments were somewhat outdated. Also added an explanation about
chosen order for the `INDEX_ID_<IDTYPE>` order.
When a lot of libraries or linked IDs were missing/not found when
loading a .blend file, Blender used to show one warning report for each
missing item, potentially covering the user's screen with a giant
unuable popup.
Now it will instead generate a single warning with amount of missing lib
files and linked IDs. Each missing item is still reported individually,
but only as `INFO`, so it will still show up in the console or Info editor.
`object_in_any_collection` used during linking/appending to check
whether an object is already instanced by at least one collection, was
not taking into account embedded master collections from scenes.
Essentially, `lib_link_all` would perform some post-processing over data
in given `bmain` that **may** fail when not all data from all libraries
has been properly loaded yet.
This happens when `lib_link_all` is called from `read_libraries`, where
the bmains are split by libraries.
Now those post-processing is put into its own utils function, which
asserts that it is only called on a merged bmain.
Bonus point, this will avoid re-runing those not-so-cheap operations
more than once on the same data.
Reproducible in r1442 of Sprite repository when opening
`pro/animation_test/rex/performance/rex_crowdcamping/rex_crowdcamping.lighting.blend`
NOTE: Not so sure why we have to call `lib_link_all` several times (once
for each library, and then once on the whole merged bmain, including
local IDs then). So that it can get called for libs while we still have
that specific .blend file handle around?
In any case, the overhead here is minimal since we do ensure a data-block is
never lib-linked more than once, so this is not a serious concern right now.
Differential Revision: https://developer.blender.org/D10307
Move `eUndoStepDir` to `BKE_undo_system.h` and use its values
everywhere.
Note that this also introduce the `STEP_INVALID` value in that enum.
Finally, kept the matching struct members in some lower-level readfile
code as an `int` to avoid having to include `BKE_undo_system.h` in a lot
of unrelated files.
Instead of submitting tons of tiny IO syscalls, we can speed things up
significantly by `mmap`ing the .blend file into virtual memory and directly
accessing it.
In my local testing, this speeds up loading the Dweebs file with all its
linked files from 19sec to 10sec (on Linux).
As far as I can see, this should be supported on Linux, OSX and BSD.
For Windows, a second code path uses `CreateFileMapping` and
`MapViewOfFile` to achieve the same result.
Reviewed By: mont29, brecht
Differential Revision: https://developer.blender.org/D8246
