1
1

Compare commits

...

43 Commits

Author SHA1 Message Date
eb043d44f2 Merge branch 'master' into id-ensure-unique-memory-address 2020-02-28 10:45:29 +01:00
88e1152d4a Merge branch 'id-ensure-unique-memory-address' of git.blender.org:blender into id-ensure-unique-memory-address 2020-02-28 10:45:14 +01:00
76f381ff42 Ensure IDs get unique memory addresses withing q given editing session.
Disclaimer: this should be considered as an engeneering, explanatory patch.

Rational: for undo work, we are now manipulating IDs from potentially many different 'memory realms' (in contrast to current undo system, where, since we always reallocate the whole main database at each undo step, we only ever have to deal with two, the old one from the memfile, and the new one from re-read data from that memfile).

So far, new undo system has used ID names-based search to remap old to new pointers, but this is slower, less safe, and forces us to do full undo barrier e.g. when renaming an ID.

Idea of this patch is instead to ensure that within a given editing session, we never ever allocate a data-block at the same address twice. That way, we can consider ID pointers as valid UIDs withing that scope, and use their values to handle remapping in udo steps even in case an ID has had several memory addresses in the history of the editing session.

This patches does two things:
# Add a way to ensure a newly allocated ID gets a memory address never used before for any ID.
# Store the history of all memory addresses ever used by a given ID.

Note that point 1 above is currently using a fairly simple and naive solution, but also quiet innefficient (just allocating again and again until we get a 'free' address), we can imagine a much more efficient solution if we go deeper in MEM_ allocator system itself. Not sure if it is worth it though, as with new undo system actual re-allocation of a same ID at a different address should be fairly rare (i.e. history of used/forbidden addresses should remain quiet short).

Differential Revision: https://developer.blender.org/D6937
2020-02-26 15:02:20 +01:00
bfdb28a512 Merge branch 'master' into id-ensure-unique-memory-address 2020-02-26 14:15:35 +01:00
a5ddb3df3b Cleanup; add missing static statement. 2020-02-26 12:48:01 +01:00
6ba8c393a0 Rework a bit re-allocating code, add a lookup utils for history memaddresses. 2020-02-26 12:19:25 +01:00
1ee9fa05b6 Merge branch 'master' into id-ensure-unique-memory-address 2020-02-25 16:18:17 +01:00
79f09c6bf2 Minor fixes/cleanup. 2020-02-21 16:48:11 +01:00
726e697216 Rework the whole unique ID system to also store ID addresses history in Main's mapping.
In the end, since this is a runtime/editing session data only, makes
more sense to store it here than in ID struct itself. And it's probably
also more efficient to handle.
2020-02-21 15:49:21 +01:00
11860508bd id-unique system: add also realloc utils. 2020-02-20 14:24:10 +01:00
18ef2d5a4d Merge branch 'master' into id-ensure-unique-memory-address 2020-02-20 12:17:48 +01:00
6ee3ab5640 Merge branch 'readfile-rework-refcount-handling' into id-ensure-unique-memory-address 2020-02-19 14:08:32 +01:00
91fd432afe Merge branch 'master' into readfile-rework-refcount-handling 2020-02-19 14:07:26 +01:00
dc0f53c936 Merge branch 'master' into readfile-rework-refcount-handling 2020-02-19 12:49:34 +01:00
b7b8399c51 get rid of all remaining usage of newlibadr_us in readfile code.
Old deprecated IPOs data-blocks are not refcounted at all anymore,
however that should not be an issue since after doversion we get rid of
them anyway.
2020-02-19 10:30:50 +01:00
b45ae177d9 Merge branch 'master' into readfile-rework-refcount-handling 2020-02-19 10:17:59 +01:00
767c3899e0 Merge branch 'readfile-rework-refcount-handling' into id-ensure-unique-memory-address 2020-02-18 18:35:36 +01:00
ac346fd5fb Merge branch 'master' into readfile-rework-refcount-handling 2020-02-18 18:27:39 +01:00
537fddbf16 Remove usercount handling from ui stuff in reafile as well. 2020-02-18 15:49:36 +01:00
fcd127b815 Fix bad placement of ID refcount recomputation in readfile. 2020-02-18 15:39:29 +01:00
f359e63e2f Merge branch 'master' into readfile-rework-refcount-handling 2020-02-18 14:39:05 +01:00
3e29c2e6d0 Merge branch 'master' into readfile-rework-refcount-handling 2020-02-18 11:24:51 +01:00
2438275452 Merge branch 'readfile-rework-refcount-handling' into id-ensure-unique-memory-address 2020-02-18 10:59:02 +01:00
e752eb4ae9 Merge branch 'libquery-handle-ui-pointers' into readfile-rework-refcount-handling 2020-02-18 10:55:12 +01:00
8fbc56b32d Merge branch 'master' into libquery-handle-ui-pointers 2020-02-18 10:31:56 +01:00
388e468a04 Factorize unique ID mem address allocation into a util functions. 2020-02-14 18:58:01 +01:00
6632461a00 Merge branch 'readfile-rework-refcount-handling' into id-ensure-unique-memory-address 2020-02-14 14:56:17 +01:00
a608df8e27 Cleanup: remove timing debug calls.
We now know that extra time spent here is neglectable.
2020-02-14 14:55:03 +01:00
8a73b03fdd Fix crash on undo case.
We cannot use libquery too early in undo case, because of all the black
magic we perform over our data-blocks there...
2020-02-14 14:53:46 +01:00
6728cc6fdb Merge branch 'readfile-rework-refcount-handling' into id-ensure-unique-memory-address
Conflicts:
	source/blender/blenkernel/intern/main.c
2020-02-14 14:20:00 +01:00
7ea3e2ea82 Merge branch 'libquery-handle-ui-pointers' into readfile-rework-refcount-handling 2020-02-14 12:55:30 +01:00
0e87f42950 Merge branch 'master' into libquery-handle-ui-pointers
Conflicts:
	source/blender/blenkernel/intern/lib_query.c
2020-02-14 12:54:17 +01:00
150d5e82ea Merge branch 'libquery-handle-ui-pointers' into readfile-rework-refcount-handling 2020-02-14 12:33:58 +01:00
e3825d6373 Merge branch 'master' into libquery-handle-ui-pointers 2020-02-14 12:29:17 +01:00
53483104f7 Merge branch 'master' into libquery-handle-ui-pointers 2020-02-14 09:55:02 +01:00
587155745b Initial rough idea of 'unique ID address' change.
Goal of this code is to ensure we only ever use a given memory address
once, for all IDs generated in a bmain.

This is crucial if we want to be able to use ID pointers instead of ID
names in undo speedup work, when re-using existing IDs.
2020-02-11 17:50:53 +01:00
9ae8f97fed Merge branch 'master' into libquery-handle-ui-pointers 2020-02-10 18:49:00 +01:00
a544582c21 Merge commit 'ddad044cfe13' into libquery-handle-ui-pointers 2020-02-10 18:47:36 +01:00
f4e17b25bd libquery: add optional handling of 'UI' ID pointers.
Handling those through different ways /might/ be needed sometimes, but
in most case this is just a nest of issues, since you can easily forget
to take them into account.

Note that this should be a 'non-functional' change, as this new behavior
is not used anywhere yet.
2020-02-07 16:13:16 +01:00
ae462320b8 Better handling of usages of usercount increment in readfile liblink.
Some places could/should still need proper ID refcounting in liblink:
* UI data (spaces) - although this should be strictly 'user one' only...
* Deprectaed data/pointers which are not handled  by libquery code.

Not sure whether IPO pointers really need usercount actually...
2020-02-05 17:52:34 +01:00
ea0dc96d77 Merge branch 'master' into readfile-rework-refcount-handling
Conflicts:
	source/blender/blenloader/intern/readfile.c
2020-02-05 17:04:59 +01:00
8c7503726d readfile: Quick experiment with refcounting moved to libquery.
Having that extra ID users handling at readfile level, besides generic
one ensured by libquery, has been something bothering me for a long time
(had to fix my share of bugs due to mismatches between those two areas).

Further more, work on undo speedup will require even more complex ID
refcount management if we want to keep it in readfile.c area.

So idea is instead to generalize what we did for linked data already
when undoing: recompute properly usercount numbers after liblink step,
for all IDs.

Note that extra time required here is neglectable in a whole .blend file
reading (few extra milliseconds when loading a full production scene
e.g.).
2020-02-03 10:52:48 +01:00
9829a3c496 Refactor readfile's liblink code.
Liblink specific ID type function was so far running a loop over all IDs
of relevant type, unlike almost any other 'ID-callback-like' functions
in Blender, which usually let the looping controll to calling code.

The latter approach is more convinient when one want to add generic
(i.e. type-agnostic) code, since it typically only has to change code in
one place (caller function) instead of tens of places (all the callback
functions).

This commit also changes/sanitizes a few things that had nothing to do
in main liblink code, like mesh conversion from tessfaces to polys
(which can be done in after-linking versionning code), or scenes' cycles
detection/check regarding background 'set' scenes.

Differential Revision: https://developer.blender.org/D6727
2020-02-01 19:43:39 +01:00
7 changed files with 192 additions and 6 deletions

View File

@@ -62,7 +62,7 @@ struct PropertyRNA;
struct bContext;
size_t BKE_libblock_get_alloc_info(short type, const char **name);
void *BKE_libblock_alloc_notest(short type) ATTR_WARN_UNUSED_RESULT;
void *BKE_libblock_alloc_notest(struct Main *bmain, short type) ATTR_WARN_UNUSED_RESULT;
void *BKE_libblock_alloc(struct Main *bmain, short type, const char *name, const int flag)
ATTR_WARN_UNUSED_RESULT;
void BKE_libblock_init_empty(struct ID *id) ATTR_NONNULL(1);

View File

@@ -51,6 +51,7 @@ struct GHash;
struct GSet;
struct ImBuf;
struct Library;
struct LinkNode;
struct MainLock;
/* Blender thumbnail, as written on file (width, height, and data as char RGBA). */
@@ -100,6 +101,10 @@ typedef struct Main {
*/
char is_memfile_undo_flush_needed;
struct GHash *used_id_memhash;
struct LinkNode *used_id_memhash_history_chains;
short used_id_memhash_tag;
BlendThumbnail *blen_thumb;
struct Library *curlib;
@@ -150,12 +155,32 @@ typedef struct Main {
struct MainLock *lock;
} Main;
/* Main.used_id_memory_pointers_tag */
enum {
MAIN_IDMEMHASH_OWNER = 1 << 0,
};
struct Main *BKE_main_new(void);
void BKE_main_free(struct Main *mainvar);
void BKE_main_lock(struct Main *bmain);
void BKE_main_unlock(struct Main *bmain);
void BKE_main_idmemhash_ensure(struct Main *bmain);
void BKE_main_idmemhash_release(struct Main *bmain);
void BKE_main_idmemhash_transfer_ownership(struct Main *bmain_dst, struct Main *bmain_src);
void BKE_main_idmemhash_usefrom(struct Main *bmain_user, struct Main *bmain_src);
bool BKE_main_idmemhash_register_id(struct Main *bmain, void *old_vmemh, struct ID *id);
struct ID *BKE_main_idmemhash_lookup_id(struct Main *bmain,
void *vmemh,
struct LinkNode **r_used_id_chain);
void *BKE_main_idmemhash_unique_alloc(struct Main *bmain,
void *old_vmemh,
void *(*alloc_cb)(size_t len, const char *str),
size_t size,
const char *message);
void *BKE_main_idmemhash_unique_realloc(struct Main *bmain, void *old_vmemh, void *vmemh);
void BKE_main_relations_create(struct Main *bmain, const short flag);
void BKE_main_relations_free(struct Main *bmain);

View File

@@ -127,6 +127,7 @@ void BKE_blender_globals_init(void)
U.savetime = 1;
G_MAIN = BKE_main_new();
BKE_main_idmemhash_ensure(G_MAIN);
strcpy(G.ima, "//");

View File

@@ -1193,12 +1193,12 @@ size_t BKE_libblock_get_alloc_info(short type, const char **name)
* Allocates and returns memory of the right size for the specified block type,
* initialized to zero.
*/
void *BKE_libblock_alloc_notest(short type)
void *BKE_libblock_alloc_notest(Main *bmain, short type)
{
const char *name;
size_t size = BKE_libblock_get_alloc_info(type, &name);
if (size != 0) {
return MEM_callocN(size, name);
return BKE_main_idmemhash_unique_alloc(bmain, NULL, MEM_callocN, size, name);
}
BLI_assert(!"Request to allocate unknown data type");
return NULL;
@@ -1214,7 +1214,7 @@ void *BKE_libblock_alloc(Main *bmain, short type, const char *name, const int fl
{
BLI_assert((flag & LIB_ID_CREATE_NO_ALLOCATE) == 0);
ID *id = BKE_libblock_alloc_notest(type);
ID *id = BKE_libblock_alloc_notest(bmain, type);
if (id) {
if ((flag & LIB_ID_CREATE_NO_MAIN) != 0) {

View File

@@ -29,6 +29,7 @@
#include "BLI_blenlib.h"
#include "BLI_ghash.h"
#include "BLI_linklist.h"
#include "BLI_mempool.h"
#include "BLI_threads.h"
@@ -194,6 +195,8 @@ void BKE_main_free(Main *mainvar)
BKE_main_relations_free(mainvar);
}
BKE_main_idmemhash_release(mainvar);
BLI_spin_end((SpinLock *)mainvar->lock);
MEM_freeN(mainvar->lock);
MEM_freeN(mainvar);
@@ -209,6 +212,146 @@ void BKE_main_unlock(struct Main *bmain)
BLI_spin_unlock((SpinLock *)bmain->lock);
}
void BKE_main_idmemhash_ensure(Main *bmain)
{
if (bmain->used_id_memhash == NULL || (bmain->used_id_memhash_tag & MAIN_IDMEMHASH_OWNER) == 0) {
bmain->used_id_memhash = BLI_ghash_new(BLI_ghashutil_ptrhash, BLI_ghashutil_ptrcmp, __func__);
bmain->used_id_memhash_history_chains = NULL;
bmain->used_id_memhash_tag |= MAIN_IDMEMHASH_OWNER;
}
}
static void main_idmemhash_history_chains_free(void *linkv)
{
LinkNode *link = linkv;
BLI_linklist_free(link, NULL);
}
void BKE_main_idmemhash_release(Main *bmain)
{
if (bmain->used_id_memhash != NULL) {
if ((bmain->used_id_memhash_tag & MAIN_IDMEMHASH_OWNER) != 0) {
BLI_ghash_free(bmain->used_id_memhash, NULL, NULL);
BLI_linklist_free(bmain->used_id_memhash_history_chains, main_idmemhash_history_chains_free);
}
bmain->used_id_memhash = NULL;
bmain->used_id_memhash_history_chains = NULL;
bmain->used_id_memhash_tag &= ~MAIN_IDMEMHASH_OWNER;
}
}
void BKE_main_idmemhash_transfer_ownership(Main *bmain_dst, Main *bmain_src)
{
BKE_main_idmemhash_release(bmain_dst);
BLI_assert(bmain_src->used_id_memhash != NULL);
BLI_assert(bmain_src->used_id_memhash_tag & MAIN_IDMEMHASH_OWNER);
bmain_dst->used_id_memhash = bmain_src->used_id_memhash;
bmain_dst->used_id_memhash_history_chains = bmain_src->used_id_memhash_history_chains;
bmain_dst->used_id_memhash_tag |= MAIN_IDMEMHASH_OWNER;
bmain_src->used_id_memhash_tag &= ~MAIN_IDMEMHASH_OWNER;
}
void BKE_main_idmemhash_usefrom(Main *bmain_user, Main *bmain_src)
{
BKE_main_idmemhash_release(bmain_user);
BLI_assert(bmain_src->used_id_memhash != NULL);
bmain_user->used_id_memhash = bmain_src->used_id_memhash;
bmain_user->used_id_memhash_history_chains = bmain_src->used_id_memhash_history_chains;
}
/**
* @return true if the ID was successfully added to the memset, false if it already existed.
*/
bool BKE_main_idmemhash_register_id(Main *bmain, void *old_vmemh, ID *id)
{
BLI_assert(bmain->used_id_memhash != NULL);
BLI_assert(old_vmemh != id);
void **val;
if (!BLI_ghash_ensure_p(bmain->used_id_memhash, id, &val)) {
if (old_vmemh != NULL) {
LinkNode **chain_hook = (LinkNode **)BLI_ghash_lookup_p(bmain->used_id_memhash, old_vmemh);
BLI_assert(chain_hook != NULL);
if (*chain_hook == NULL) {
/* That datablock only ever had one address so far, we need to initialize its addresses
* history chain. */
*chain_hook = MEM_callocN(sizeof(**chain_hook), __func__);
LinkNode *old_id_entry = MEM_mallocN(sizeof(*old_id_entry), __func__);
old_id_entry->link = old_vmemh;
old_id_entry->next = NULL;
BLI_linklist_prepend_nlink(
&bmain->used_id_memhash_history_chains, old_id_entry, *chain_hook);
}
LinkNode *curr_id_entry = MEM_mallocN(sizeof(*curr_id_entry), __func__);
BLI_linklist_prepend_nlink((LinkNode **)&(*chain_hook)->link, id, curr_id_entry);
*val = *chain_hook;
}
else {
*val = NULL;
}
return true;
}
return false;
}
/**
* Lookup a random ID memory address, and return its last known valid instance, and the linked list
* of all its known addresses so far.
*
* \param r_used_id_chain If not NULL, and that address has had several previous instances, the
* linked list storing all of those.
* \return The last known instance address matching given \a vmemh pointer, or vmemh itself if it
* is unknown.
*/
ID *BKE_main_idmemhash_lookup_id(Main *bmain, void *vmemh, LinkNode **r_used_id_chain)
{
LinkNode *used_id_chain_hook = BLI_ghash_lookup(bmain->used_id_memhash, vmemh);
LinkNode *used_id_chain = used_id_chain_hook ? used_id_chain_hook->link : NULL;
if (r_used_id_chain != NULL) {
*r_used_id_chain = used_id_chain;
}
/* The last valid address should always be the first one in the chain. */
return used_id_chain != NULL ? used_id_chain->link : vmemh;
}
void *BKE_main_idmemhash_unique_alloc(Main *bmain,
void *old_vmemh,
void *(*alloc_cb)(size_t len, const char *str),
size_t size,
const char *message)
{
void *id_mem = alloc_cb(size, message);
if (bmain != NULL && bmain->used_id_memhash != NULL) {
ListBase generated_ids = {.first = NULL};
int count = 0;
while (UNLIKELY(!BKE_main_idmemhash_register_id(bmain, old_vmemh, id_mem))) {
printf("Allocating ID re-used memory address %p, trying again (%d)...\n", id_mem, ++count);
BLI_addtail(&generated_ids, id_mem);
id_mem = alloc_cb(size, message);
}
BLI_freelistN(&generated_ids);
}
return id_mem;
}
void *BKE_main_idmemhash_unique_realloc(Main *bmain, void *old_vmemh, void *vmemh)
{
void *id_mem = MEM_dupallocN(vmemh);
if (bmain != NULL && bmain->used_id_memhash != NULL) {
ListBase generated_ids = {.first = NULL};
int count = 0;
while (UNLIKELY(!BKE_main_idmemhash_register_id(bmain, old_vmemh, id_mem))) {
printf("Allocating ID re-used memory address %p, trying again (%d)...\n", id_mem, ++count);
BLI_addtail(&generated_ids, id_mem);
id_mem = MEM_dupallocN(id_mem);
}
BLI_freelistN(&generated_ids);
}
return id_mem;
}
static int main_relations_create_idlink_cb(LibraryIDLinkCallbackData *cb_data)
{
MainIDRelations *rel = cb_data->user_data;

View File

@@ -504,6 +504,10 @@ static void add_main_to_main(Main *mainvar, Main *from)
ListBase *lbarray[MAX_LIBARRAY], *fromarray[MAX_LIBARRAY];
int a;
if (!ELEM(from->used_id_memhash, NULL, mainvar->used_id_memhash)) {
BLI_assert(ELEM(from->used_id_memhash, NULL, mainvar->used_id_memhash));
}
set_listbasepointers(mainvar, lbarray);
a = set_listbasepointers(from, fromarray);
while (a--) {
@@ -561,6 +565,7 @@ void blo_split_main(ListBase *mainlist, Main *main)
int i = 0;
for (Library *lib = main->libraries.first; lib; lib = lib->id.next, i++) {
Main *libmain = BKE_main_new();
BKE_main_idmemhash_usefrom(libmain, main);
libmain->curlib = lib;
libmain->versionfile = lib->versionfile;
libmain->subversionfile = lib->subversionfile;
@@ -671,6 +676,7 @@ static Main *blo_find_main(FileData *fd, const char *filepath, const char *relab
}
m = BKE_main_new();
BKE_main_idmemhash_usefrom(m, mainlist->first);
BLI_addtail(mainlist, m);
/* Add library data-block itself to 'main' Main, since libraries are **never** linked data.
@@ -8204,6 +8210,7 @@ static void direct_link_library(FileData *fd, Library *lib, Main *main)
/* new main */
newmain = BKE_main_new();
BKE_main_idmemhash_usefrom(newmain, fd->mainlist->first);
BLI_addtail(fd->mainlist, newmain);
newmain->curlib = lib;
@@ -8788,7 +8795,7 @@ static void direct_link_linestyle(FileData *fd, FreestyleLineStyle *linestyle)
static ID *create_placeholder(Main *mainvar, const short idcode, const char *idname, const int tag)
{
ListBase *lb = which_libbase(mainvar, idcode);
ID *ph_id = BKE_libblock_alloc_notest(idcode);
ID *ph_id = BKE_libblock_alloc_notest(mainvar, idcode);
*((short *)ph_id->name) = idcode;
BLI_strncpy(ph_id->name + 2, idname, sizeof(ph_id->name) - 2);
@@ -8967,6 +8974,10 @@ static BHead *read_libblock(FileData *fd,
* like it used to be. */
BLI_remlink(fd->old_mainlist, libmain);
BLI_remlink_safe(&oldmain->libraries, libmain->curlib);
/* We also need to transfer the unique id storage system, since that libmain now belongs
* to the new Main database. */
BKE_main_idmemhash_release(libmain);
BKE_main_idmemhash_usefrom(libmain, fd->mainlist->first);
BLI_addtail(fd->mainlist, libmain);
BLI_addtail(&main->libraries, libmain->curlib);
@@ -9035,6 +9046,10 @@ static BHead *read_libblock(FileData *fd,
id->newid = NULL; /* Needed because .blend may have been saved with crap value here... */
id->orig_id = NULL;
const bool is_id_memaddress_unique = BKE_main_idmemhash_register_id(main, NULL, id);
/* Note: this is likely to fail at some point with current undo/redo code! */
BLI_assert(is_id_memaddress_unique);
/* this case cannot be direct_linked: it's just the ID part */
if (bhead->code == ID_LINK_PLACEHOLDER) {
/* That way, we know which data-lock needs do_versions (required currently for linking). */
@@ -9643,6 +9658,7 @@ BlendFileData *blo_read_file_internal(FileData *fd, const char *filepath)
bfd = MEM_callocN(sizeof(BlendFileData), "blendfiledata");
bfd->main = BKE_main_new();
BKE_main_idmemhash_ensure(bfd->main);
bfd->main->versionfile = fd->fileversion;
bfd->type = BLENFILETYPE_BLEND;
@@ -11405,6 +11421,7 @@ static void split_main_newid(Main *mainptr, Main *main_newid)
main_newid->subversionfile = mainptr->subversionfile;
BLI_strncpy(main_newid->name, mainptr->name, sizeof(main_newid->name));
main_newid->curlib = mainptr->curlib;
BKE_main_idmemhash_usefrom(main_newid, mainptr);
ListBase *lbarray[MAX_LIBARRAY];
ListBase *lbarray_newid[MAX_LIBARRAY];

View File

@@ -136,7 +136,7 @@ void IDNode::init_copy_on_write(ID *id_cow_hint)
}
}
else if (deg_copy_on_write_is_needed(id_orig)) {
id_cow = (ID *)BKE_libblock_alloc_notest(GS(id_orig->name));
id_cow = (ID *)BKE_libblock_alloc_notest(NULL, GS(id_orig->name));
DEG_COW_PRINT(
"Create shallow copy for %s: id_orig=%p id_cow=%p\n", id_orig->name, id_orig, id_cow);
deg_tag_copy_on_write_id(id_cow, id_orig);