Solving the 'missing libraries' issue #45351

New Issue

Bastien Montagne · 2015-07-06T23:01:33+02:00

Bastien Montagne commented

2015-07-06 23:01:33 +02:00

This task aims at solving the issue caused by 'missing libraries'.

Note: This is a sub-task of #46049.

Currently, when a library file is missing for some reason (removed, rename, network filesystem outline, etc.), when you open a .blend file using some of its data, you get a warning about missing data and… must absolutely not save your .blend, otherwise you'll definitively lose track of those missing datablocks.

In relation with asset project, I’ve been working on this issue, and came with this proposal, with two main steps:

Do not lose track of missing IDs anymore.
Then, we should be able (in outliner probably) to edit missing libs' paths and reload the missing datablocks at any time.

I. Do not lose track of missing IDs

Now in Master

The idea here is to add a “place holder” ID when we cannot load a real one from a library. That place holder has the same type as missing one, and is a 'lib' ID, but contains no data.

Thus, you can use it as a regular library datablock in Blender - will just contain default data (no geometry, 'Empty' type for objects, default settings, etc. etc.).

And you can save and reload your file as you want - once library becomes available again, its data will be read at load time again.

D1394 implements that (only partially, as a proof of concept), it seems to work pretty well, but would need agreement this direction is valid before going further.

II. Reload the missing datablocks

First experimental builds available, see this post

Main part here is to be able to replace some ID by another without having to save/reload… WIP code is in id-remap branch.

This task aims at solving the issue caused by 'missing libraries'. **Note:** This is a sub-task of #46049. Currently, when a library file is missing for some reason (removed, rename, network filesystem outline, etc.), when you open a .blend file using some of its data, you get a warning about missing data and… must absolutely **not save** your .blend, otherwise you'll definitively lose track of those missing datablocks. In relation with asset project, I’ve been working on this issue, and came with this proposal, with two main steps: - Do not lose track of missing IDs anymore. - Then, we should be able (in outliner probably) to edit missing libs' paths and reload the missing datablocks at any time. I. Do not lose track of missing IDs ---------- **Now in Master** The idea here is to add a “place holder” ID when we cannot load a real one from a library. That place holder has the same type as missing one, and is a 'lib' ID, but contains no data. Thus, you can use it as a regular library datablock in Blender - will just contain default data (no geometry, 'Empty' type for objects, default settings, etc. etc.). And you can save and reload your file as you want - once library becomes available again, its data will be read at load time again. [D1394](https://archive.blender.org/developer/D1394) implements that (only partially, as a proof of concept), it seems to work pretty well, but would need agreement this direction is valid before going further. II. Reload the missing datablocks ---------- **First experimental builds available, see [this post](http://code.blender.org/2015/11/test-build-live-reloading-relocating-of-libraries/)** Main part here is to be able to replace some ID by another without having to save/reload… WIP code is in [id-remap](https://developer.blender.org/diffusion/B/browse/id-remap/) branch.

Bastien Montagne commented

2015-07-06 23:01:33 +02:00

Changed status to: 'Open'

Bastien Montagne commented

2015-07-06 23:01:33 +02:00

Added subscribers: @mont29, @ideasman42, @Sergey, @fsiddi

Campbell Barton commented

2015-07-08 10:29:33 +02:00

This can of course be really useful and its great you work on this.

Note that I think reloading missing datablocks (even on a very basic level - though Python API for example),
is important to have working - in order that this feature be very useful.

Said different - if the feature is added and reload ends up being an unsolvable can of worms, Then probably its better to rethink entire patch.

Some notes...

What if someone wants to leave the data missing and edit it? (make local and edit-in-place)? Would this be supported?
What if someone wants to replace a missing data-block with another one which exists in the file? - already linked (seems reasonable and probably not too hard to support)
A way to remove missing datablocks is probably needed too.

A detail from reading the patch... you split init from add, that makes sense, but introduce possible problems where existing data-block has all sorts of data (brush mtex... scene keying sets... all sorts of inter-linking pointers).

Suggest to be strict here and add an assert if any data is non-zero after the ID struct. Otherwise we could end up in quite crappy situations with initialize data overwriting.

This can of course be really useful and its great you work on this. Note that I think reloading missing datablocks (even on a very basic level - though Python API for example), is important to have working - in order that this feature be very useful. Said different - if the feature is added and reload ends up being an unsolvable can of worms, Then probably its better to rethink entire patch. ---- Some notes... - What if someone wants to leave the data missing and edit it? (make local and edit-in-place)? Would this be supported? - What if someone wants to replace a missing data-block with another one which exists in the file? - already linked *(seems reasonable and probably not too hard to support)* - A way to remove missing datablocks is probably needed too. A detail from reading the patch... you split init from add, that makes sense, but introduce possible problems where existing data-block has all sorts of data (brush mtex... scene keying sets... all sorts of inter-linking pointers). Suggest to be strict here and add an assert if _any_ data is non-zero after the `ID` struct. Otherwise we could end up in quite crappy situations with initialize data overwriting.

Bastien Montagne commented

2015-07-08 13:07:19 +02:00

In #45351#320706, @ideasman42 wrote:
This can of course be really useful and its great you work on this.

Note that I think reloading missing datablocks (even on a very basic level - though Python API for example),
is important to have working - in order that this feature be very useful.

Said different - if the feature is added and reload ends up being an unsolvable can of worms, Then probably its better to rethink entire patch.

100% agree. Just wanted first advice on the idea of placeholder IDs. ;)

Some notes...

What if someone wants to leave the data missing and edit it? (make local and edit-in-place)? Would this be supported?

Missing data survives multiple .blend save & load. Further more, if:

you make a local copy of (aka localize) missing linked data, you just get 'empty' data, which is then completely detached from lib, as with non-missing data.
you make a proxy and edit that proxy, then proxy editing is saved and also works fine when missing lib becomes available again (note that this was checked with proxy for objects only, not sure about armature - but probably just nothing happens in this case, since there are no bones in placeholder data…).

What if someone wants to replace a missing data-block with another one which exists in the file? - already linked (seems reasonable and probably not too hard to support)

Think this is a bit an orthogonal topic (as in, could be useful even outside the 'missing lib' case), but definitively worth it, yes.

A way to remove missing datablocks is probably needed too.

Sure, that would be part of Outliner's handling of those imho.

A detail from reading the patch... you split init from add, that makes sense, but introduce possible problems where existing data-block has all sorts of data (brush mtex... scene keying sets... all sorts of inter-linking pointers).

Suggest to be strict here and add an assert if any data is non-zero after the ID struct. Otherwise we could end up in quite crappy situations with initialize data overwriting.

Good point, will do. :)

> In #45351#320706, @ideasman42 wrote: > This can of course be really useful and its great you work on this. > > Note that I think reloading missing datablocks (even on a very basic level - though Python API for example), > is important to have working - in order that this feature be very useful. > > Said different - if the feature is added and reload ends up being an unsolvable can of worms, Then probably its better to rethink entire patch. 100% agree. Just wanted first advice on the idea of placeholder IDs. ;) ---- > Some notes... > > - What if someone wants to leave the data missing and edit it? (make local and edit-in-place)? Would this be supported? Missing data survives multiple .blend save & load. Further more, if: - you make a local copy of (aka localize) missing linked data, you just get 'empty' data, which is then completely detached from lib, as with non-missing data. - you make a proxy and edit that proxy, then proxy editing is saved and also works fine when missing lib becomes available again (note that this was checked with proxy for objects only, not sure about armature - but probably just nothing happens in this case, since there are no bones in placeholder data…). > - What if someone wants to replace a missing data-block with another one which exists in the file? - already linked *(seems reasonable and probably not too hard to support)* Think this is a bit an orthogonal topic (as in, could be useful even outside the 'missing lib' case), but definitively worth it, yes. > - A way to remove missing datablocks is probably needed too. Sure, that would be part of Outliner's handling of those imho. > A detail from reading the patch... you split init from add, that makes sense, but introduce possible problems where existing data-block has all sorts of data (brush mtex... scene keying sets... all sorts of inter-linking pointers). > > Suggest to be strict here and add an assert if _any_ data is non-zero after the `ID` struct. Otherwise we could end up in quite crappy situations with initialize data overwriting. Good point, will do. :)

Francesco Siddi commented

2015-07-09 11:03:25 +02:00

Thanks for starting to work on this. This will be a major improvement and time saver when dealing with file shuffling and refactoring.

Do you plan to introduce a "Replace/Relocate" functionality to point an existing (and currently linked) library to a different endpoint? It would be extremely useful for live changes while the blendfile is open and also for a programmatic approach, where we remap some paths on startup or via a script. I guess given your design proposal this will be possible, but I just wanted to mention this.

Again, thanks for looking into this area, it will have great impact.

Thanks for starting to work on this. This will be a major improvement and time saver when dealing with file shuffling and refactoring. Do you plan to introduce a "Replace/Relocate" functionality to point an existing (and currently linked) library to a different endpoint? It would be extremely useful for live changes while the blendfile is open and also for a programmatic approach, where we remap some paths on startup or via a script. I guess given your design proposal this will be possible, but I just wanted to mention this. Again, thanks for looking into this area, it will have great impact.

Bastien Montagne commented

2015-07-09 11:09:25 +02:00

Yes, that kind of feature is definitively part of step II. :)

Bastien Montagne commented

2015-07-15 20:55:17 +02:00

Just a note that work is now in a public branch, missing-libs.

Just a note that work is now in a public branch, [missing-libs](https://developer.blender.org/diffusion/B/browse/missing-libs/).

Duarte Farrajota Ramos commented

2015-07-16 07:11:48 +02:00

Added subscriber: @DuarteRamos

Riccardo Giovanetti commented

2015-07-16 11:55:16 +02:00

Added subscriber: @Harvester

hewi commented

2015-07-17 01:37:58 +02:00

Added subscriber: @hewi

Bastien Montagne commented

2015-07-21 22:48:06 +02:00

Ok, so pushed publicly another branch, tackling the other main part of this project - being able to replace ID datablocks by another 'on the fly'.

id-remap branch contains code allowing to replace all usages of a given ID by another ID (of same type of course).

Only contains quite rough, unfinished and most likely broken code, but it does survive some very very basic tests.

Together with ID placeholder, this should give us solid foundations for a much more powerful and flexible datablocks (and libraries) handling.

Ok, so pushed publicly another branch, tackling the other main part of this project - being able to replace ID datablocks by another 'on the fly'. [id-remap](https://developer.blender.org/diffusion/B/browse/id-remap/) branch contains code allowing to replace all usages of a given ID by another ID (of same type of course). Only contains quite rough, unfinished and most likely broken code, but it does survive some very very basic tests. Together with ID placeholder, this should give us solid foundations for a much more powerful and flexible datablocks (and libraries) handling.

Aaron Carlisle commented

2015-08-23 20:27:32 +02:00

Added subscriber: @Blendify