Problems appear with huge projects (linked libs, etc.) #52205

New Issue

Alma Talp · 2017-07-27T20:00:43+02:00

Alma Talp commented

2017-07-27 20:00:43 +02:00

System Information
Win 7 64 Pro

Blender Version
2.78 hash: 4bb1e22

Short description of error

Blender forgets linked library locations accidentally, sometimes saves huge (1-2 GiB) files corruptly (cannot be opened)

Exact steps for others to reproduce the error
You will hate it: I have no idea what causes it.

Overall related to the issues:

I work on a huge scene, with many linked libraries for object, materials, Cycles node setups, etc.

For now 4-5 times happened that I saved a file, I reopened it and I got the popup about the missing files while nothing has been changed in the directory structure. After 'relocating' the linked blend file in its original location, Blender finds everything again, but the material setups are broken, I have to rewire all the textures, nodes again, set the values again, everything (while the images from the same HDD are there).

Also sometimes happen that some of the image files are loaded for a shader, some are not from my material library. The Cycles node setup shows that the image is loaded (I could read its name, no error messages on loading), but when I click it, it refers to somewhere else and it does not appear in render.

Of course I know that it does not help too much to identify the problems, but I would like to ask for help for possible causes.

What I was thinking about until now:

path length (what is the limit for it for now?). Why it works when it works?
'Matlib' HDD goes sleep somehow (it should not), Blender keeps linked stuff in 'cache', but on save it cannot find the linked stuff. There are 2 arguments against it:

All the apps I used before waked up hdds in cases like this (when I used them on a notebook as an example), and Blender should not behave differently
I should get an error message when linked stuff is not available (that is clearly something the user should be noticed about somehow during the workflow). When I model and do not render for hours it is killer when I see that all my former stuff lost its material/linked data and I saved multiple times since the start.

On the case saving huge files: sometimes I get the info that file cannot be saved
("Cannot change old file (file saved with @)" ), while I save to the same directory I used before (so no permission problem).

Could you help with some brainstorming what could cause these issues and how should I try to identify/reproduce the issues? Thanks

**System Information** Win 7 64 Pro **Blender Version** 2.78 hash: 4bb1e22 **Short description of error** Blender forgets linked library locations accidentally, sometimes saves huge (1-2 GiB) files corruptly (cannot be opened) **Exact steps for others to reproduce the error** You will hate it: I have no idea what causes it. Overall related to the issues: I work on a huge scene, with many linked libraries for object, materials, Cycles node setups, etc. For now 4-5 times happened that I saved a file, I reopened it and I got the popup about the missing files while **nothing has been changed** in the directory structure. After 'relocating' the linked blend file in its original location, Blender finds everything again, but the material setups are broken, I have to rewire all the textures, nodes again, set the values again, everything (while the images from the same HDD are there). Also sometimes happen that some of the image files are loaded for a shader, some are not from my material library. The Cycles node setup shows that the image is loaded (I could read its name, no error messages on loading), but when I click it, it refers to somewhere else and it does not appear in render. Of course I know that it does not help too much to identify the problems, but I would like to ask for help for possible causes. What I was thinking about until now: - path length (what is the limit for it for now?). Why it works when it works? - 'Matlib' HDD goes sleep somehow (it should not), Blender keeps linked stuff in 'cache', but on save it cannot find the linked stuff. There are 2 arguments against it: 1. All the apps I used before waked up hdds in cases like this (when I used them on a notebook as an example), and Blender should not behave differently 2. I should get an error message when linked stuff is not available (that is clearly something the user should be noticed about somehow during the workflow). When I model and do not render for hours it is killer when I see that all my former stuff lost its material/linked data and I saved multiple times since the start. - On the case saving huge files: sometimes I get the info that file cannot be saved ("Cannot change old file (file saved with @)" ), while I save to the same directory I used before (so no permission problem). Could you help with some brainstorming what could cause these issues and how should I try to identify/reproduce the issues? Thanks

Alma Talp commented

2017-07-27 20:00:43 +02:00

Changed status to: 'Open'

Alma Talp commented

2017-07-27 20:00:43 +02:00

Added subscriber: @AlmaTalp

Brecht Van Lommel commented

2017-07-27 23:57:03 +02:00

Added subscriber: @brecht

Brecht Van Lommel commented

2017-07-27 23:57:03 +02:00

If you encounter a missing library .blend error, the best thing to do is to not save anything, but rather ensure all the hard drives are mounted and check that the linked files exist. If they do, then you should be able to just open the .blend again without errors. If you did not intentionally move files to a different location then trying to fix the problem in Blender is usually the wrong solution.

You need to describe one problem at a time, with a description that is as exact as possible. Solving this kind of problem is done by meticulously checking things. For example if a link gets broken:

Tell us the path of the original .blend, the linked .blend, and the type of datablock that went missing.
What kind of hard drive was each .blend file on (internal, external, network, ..)?
What was the exact error message displayed (copy from Window > Console)?
Is the drive mounted?
You say the drive was sleeping, how did you determine that?
Can you browse to the missing files in explorer? Does reopening the .blend after that work again without error?
What do you mean by 'relocating', running "Find Missing Files"?

Also:

Is "Matlib" the addon, how is that related to missing linked libraries?
What do you mean by Blender keeps linked stuff in 'cache', what kind of files? Or is that the Matlib addon?
You say you get no error when linked stuff goes missing, but that seems quite odd. Or are you talking about some data other than linked libraries, maybe from an addon? Or did you miss the error message because it only appeared briefly or something?

If you encounter a missing library .blend error, the best thing to do is to not save anything, but rather ensure all the hard drives are mounted and check that the linked files exist. If they do, then you should be able to just open the .blend again without errors. If you did not intentionally move files to a different location then trying to fix the problem in Blender is usually the wrong solution. You need to describe one problem at a time, with a description that is as exact as possible. Solving this kind of problem is done by meticulously checking things. For example if a link gets broken: * Tell us the path of the original .blend, the linked .blend, and the type of datablock that went missing. * What kind of hard drive was each .blend file on (internal, external, network, ..)? * What was the exact error message displayed (copy from Window > Console)? * Is the drive mounted? * You say the drive was sleeping, how did you determine that? * Can you browse to the missing files in explorer? Does reopening the .blend after that work again without error? * What do you mean by 'relocating', running "Find Missing Files"? Also: * Is "Matlib" the addon, how is that related to missing linked libraries? * What do you mean by Blender keeps linked stuff in 'cache', what kind of files? Or is that the Matlib addon? * You say you get no error when linked stuff goes missing, but that seems quite odd. Or are you talking about some data other than linked libraries, maybe from an addon? Or did you miss the error message because it only appeared briefly or something?

Alma Talp commented

2017-07-28 00:53:41 +02:00

Hello, thanks.
I think there was some misunderstanding about the situation.
Clarifying:

'If you encounter a missing library .blend error, the best thing to do is to not save anything, but rather ensure all the hard drives are mounted and check that the linked files exist.'

Thanks, I know it.
I do not encounter it while I work on the file. Imagine this: I work for hours on a scene, no error messages detected; I regularly save and at the end of the 'workshift' I realize that linked libraries are missing when I open another file and go back to the original I worked on.

I know how to relink libraries or find missing files; the problem is that Blender does not give me any message when the 'accident' happens, so I realize it only when the files are already saved and reopened (if I don't render, just model stuff).

'You need to describe one problem at a time, with a description that is as exact as possible. Solving this kind of problem is done by meticulously checking things. For example if a link gets broken:'

Agree, but it seemed to me so special and fundamental problem that I decided to write as much info as possible and ask for help.

Answers:

'Tell us the path of the original .blend, the linked .blend, and the type of datablock that went missing.'

Example:

Blend file I work on:

E:_____FULL\SOUR_Packs\BaseScene V01\BSOUR\BaseScene V01.blend

Location of linked libraries:
E:_____FULL\SOUR_Shaders\ShaderBase.blend
E:_____FULL\SOUR_Packs_PropTech V01\BSOUR\PropTech V01.blend

Example for the secretly missing texture location:

E:____FULL\SOUR_Packs_SharedMat_AddWorn\2K\Scratched V02_LIB-SOURCE_REF_2K.png

What kind of hard drive was each .blend file on (internal, external, network, ..)?

Internal.

What was the exact error message displayed (copy from Window > Console)?

I have no idea. As I wrote it was not something what happened before my sight. I only realized it when I reopened the file; in outliner the link was broken; the textures were detached from the shader nodes.

Is the drive mounted?
It is always mounted.
You say the drive was sleeping, how did you determine that?
No, I did not say that. I was trying to guess what could cause the issue from hardware side and I tried to 'brainstorm' with you as 'what if my drive went sleep while it should not did that)
"Can you browse to the missing files in explorer? "
Always. They are there.
"Does reopening the .blend after that work again without error?"
As I wrote, NO.
The problem is that I could only realize the broke AFTER I saved and reopened the file. Then reopening cannot do anything; it just conserves the broken state.
'What do you mean by 'relocating', running "Find Missing Files

'Relocating' is that I go to the Outliner in 'Blender File' mode and choose 'relocate' for the missing blend file (the file what contains my Materials, Node Cycles Node setups, linkeg Groups of objects, etc.)

Is "Matlib" the addon, how is that related to missing linked libraries?

No, it is not an addon, I just call my Material Library blend file + directories this way (shortly).

What do you mean by Blender keeps linked stuff in 'cache', what kind of files? Or is that the Matlib addon?

No, that was a 'guessing' why Blender does not tell me anything when the link is broken during the workflow. I do not know how Blender works internally; when I hit F12, Cycles reread all the image files all the time, even if rendered preview is active, also GLSL use cache if I'm right.

You say you get no error when linked stuff goes missing, but that seems quite odd.

YES. That is why I'm here:)

"Or are you talking about some data other than linked libraries, maybe from an addon? "

NO.

"Or did you miss the error message because it only appeared briefly or something?"

NO. I would like to investigate what happens as it seems to be a bug somewhere, but I have no idea where and it drives me crazy when I have to remake stuff again and again because of it.

Thanks again for your help.

Hello, thanks. I think there was some misunderstanding about the situation. Clarifying: 'If you encounter a missing library .blend error, the best thing to do is to not save anything, but rather ensure all the hard drives are mounted and check that the linked files exist.' Thanks, I know it. I do not encounter it while I work on the file. Imagine this: I work for hours on a scene, no error messages detected; I regularly save and at the end of the 'workshift' I realize that linked libraries are missing when I open another file and go back to the original I worked on. I know how to relink libraries or find missing files; the problem is that Blender does not give me any message when the 'accident' happens, so I realize it only when the files are already saved and reopened (if I don't render, just model stuff). 'You need to describe one problem at a time, with a description that is as exact as possible. Solving this kind of problem is done by meticulously checking things. For example if a link gets broken:' Agree, but it seemed to me so special and fundamental problem that I decided to write as much info as possible and ask for help. Answers: 'Tell us the path of the original .blend, the linked .blend, and the type of datablock that went missing.' Example: Blend file I work on: E:\_____FULL\SOUR\_Packs\BaseScene V01\BSOUR\BaseScene V01.blend Location of linked libraries: E:\_____FULL\SOUR\_Shaders\ShaderBase.blend E:\_____FULL\SOUR\_Packs\_PropTech V01\BSOUR\PropTech V01.blend Example for the secretly missing texture location: E:\_____FULL\SOUR\_Packs\_SharedMat\_AddWorn\2K\Scratched V02_LIB-SOURCE_REF_2K_.png 2. What kind of hard drive was each .blend file on (internal, external, network, ..)? Internal. 3. What was the exact error message displayed (copy from Window > Console)? I have no idea. As I wrote it was not something what happened before my sight. I only realized it when I reopened the file; in outliner the link was broken; the textures were detached from the shader nodes. 4. Is the drive mounted? It is always mounted. 5. You say the drive was sleeping, how did you determine that? No, I did not say that. I was trying to guess what could cause the issue from hardware side and I tried to 'brainstorm' with you as 'what if my drive went sleep while it should not did that) 6. "Can you browse to the missing files in explorer? " Always. They are there. 7. "Does reopening the .blend after that work again without error?" As I wrote, NO. The problem is that I could only realize the broke AFTER I saved and reopened the file. Then reopening cannot do anything; it just conserves the broken state. 8. 'What do you mean by 'relocating', running "Find Missing Files 'Relocating' is that I go to the Outliner in 'Blender File' mode and choose 'relocate' for the missing blend file (the file what contains my Materials, Node Cycles Node setups, linkeg Groups of objects, etc.) 9. Is "Matlib" the addon, how is that related to missing linked libraries? No, it is not an addon, I just call my Material Library blend file + directories this way (shortly). 10. What do you mean by Blender keeps linked stuff in 'cache', what kind of files? Or is that the Matlib addon? No, that was a 'guessing' why Blender does not tell me anything when the link is broken during the workflow. I do not know how Blender works internally; when I hit F12, Cycles reread all the image files all the time, even if rendered preview is active, also GLSL use cache if I'm right. 11. You say you get no error when linked stuff goes missing, but that seems quite odd. YES. That is why I'm here:) "Or are you talking about some data other than linked libraries, maybe from an addon? " NO. "Or did you miss the error message because it only appeared briefly or something?" NO. I would like to investigate what happens as it seems to be a bug somewhere, but I have no idea where and it drives me crazy when I have to remake stuff again and again because of it. Thanks again for your help.

Brecht Van Lommel commented

2017-07-28 01:42:31 +02:00

Ok, I'm confused still.

There can be missing image files and missing linked library .blend files, and those are quite different to Blender. Are you saying it's just the image texture that are missing, or are the materials and nodes missing (and so the image textures are also missing along with them)?

You said "I reopened it and I got the popup about the missing files". What was the message in that popup? The typical error for a missing linked library .blend?

From what I can tell you are saying that something silently broke earlier, and then you only found that out later when reopening the file and getting the popup? But that is a conjecture right? Something might have broken while working on the file earlier, or something might have broken while loading the .blend, but we don't know yet which.

So you got the popup on opening the .blend, go to Blender File in the outliner, and see that the linked library is broken. The path is correct, but for some reason if you run "relocate" that fixes it, even though the path remains the same? Or at least fixes it partially, since you then still need to redo a bunch of stuff?

Ok, I'm confused still. There can be missing image files and missing linked library .blend files, and those are quite different to Blender. Are you saying it's just the image texture that are missing, or are the materials and nodes missing (and so the image textures are also missing along with them)? You said "I reopened it and I got the popup about the missing files". What was the message in that popup? The typical error for a missing linked library .blend? From what I can tell you are saying that something silently broke earlier, and then you only found that out later when reopening the file and getting the popup? But that is a conjecture right? Something might have broken while working on the file earlier, or something might have broken while loading the .blend, but we don't know yet which. So you got the popup on opening the .blend, go to Blender File in the outliner, and see that the linked library is broken. The path is correct, but for some reason if you run "relocate" that fixes it, even though the path remains the same? Or at least fixes it partially, since you then still need to redo a bunch of stuff?

Alma Talp commented

2017-07-28 02:32:56 +02:00

'There can be missing image files and missing linked library .blend files, and those are quite different to Blender. Are you saying it's just the image texture that are missing, or are the materials and nodes missing (and so the image textures are also missing along with them)?'

I have problem with both and they could be separate cases.

Missing library blend files. This cuts the local textures off from my shaders (my shaders come from an external blend). Although the textures are present and shaders relocated manually (later) in the Outliner, I have to rewire them and set the values again. In this case I have my textures (they are present), they appear in the Image Editor, only shader trees are missing when the problem exists.
Missing textures. Specific one as not all the textures are missing. For understanding I will make a schematic view for it tomorrow.

The base concept: I have an external blend file what contains my basic shader nodes (PBR shaders, texture scalers, normal map combiners, etc.) and also 'shared materials' (variations off glasses, metals, woods, etc.)

I refer to this nodes and shaders in all my 'sub-scenes' I work on separately in separate blend files (because Blender cannot handle thousands of different meshes in a file).
All of these separate blend files using the same glass materials, etc. and linked together (as linked group) in the final scene. So at the end the final scene will contain like a dozen external blend files from where the data comes to the final scene.

I use the same 5-6 'worn' textures in all the materials and blend files. They are present in a typical scene as part of the materials coming from the material library blend file. I also use them in the 'local' materials.

I attach a screenshot. The first one shows the linked texture what refers to a wrong library and does not 'work'. The second one is a 'local' texture, but technically they share the same location in the reality.

'You said "I reopened it and I got the popup about the missing files". What was the message in that popup? The typical error for a missing linked library .blend?'

YES.

'From what I can tell you are saying that something silently broke earlier, and then you only found that out later when reopening the file and getting the popup? But that is a conjecture right? Something might have broken while working on the file earlier, or something might have broken while loading the .blend, but we don't know yet which.'

Exactly. If I would be able to determine what happened and I would be able to redo it, it would be easy to fix.

'so you got the popup on opening the .blend, go to Blender File in the outliner, and see that the linked library is broken. The path is correct, but for some reason if you run "relocate" that fixes it, even though the path remains the same? '

YES.

Or at least fixes it partially, since you then still need to redo a bunch of stuff?'

Exactly. After it I have to reattach the textures to the shader inputs, change back the values (like normal strength, roughness, etc.), emission values, everything. Also I have to relink the non-local textures (locals remain untouched).

'There can be missing image files and missing linked library .blend files, and those are quite different to Blender. Are you saying it's just the image texture that are missing, or are the materials and nodes missing (and so the image textures are also missing along with them)?' I have problem with both and they could be separate cases. 1. Missing library blend files. This cuts the local textures off from my shaders (my shaders come from an external blend). Although the textures are present and shaders relocated manually (later) in the Outliner, I have to rewire them and set the values again. In this case I have my textures (they are present), they appear in the Image Editor, only shader trees are missing when the problem exists. 2. Missing textures. Specific one as not all the textures are missing. For understanding I will make a schematic view for it tomorrow. The base concept: I have an external blend file what contains my basic shader nodes (PBR shaders, texture scalers, normal map combiners, etc.) and also 'shared materials' (variations off glasses, metals, woods, etc.) I refer to this nodes and shaders in all my 'sub-scenes' I work on separately in separate blend files (because Blender cannot handle thousands of different meshes in a file). All of these separate blend files using the same glass materials, etc. and linked together (as linked group) in the final scene. So at the end the final scene will contain like a dozen external blend files from where the data comes to the final scene. I use the same 5-6 'worn' textures in all the materials and blend files. They are present in a typical scene as part of the materials coming from the material library blend file. I also use them in the 'local' materials. I attach a screenshot. The first one shows the linked texture what refers to a wrong library and does not 'work'. The second one is a 'local' texture, but technically they share the same location in the reality. ![error of textures01.png](https://archive.blender.org/developer/F681511/error_of_textures01.png) 'You said "I reopened it and I got the popup about the missing files". What was the message in that popup? The typical error for a missing linked library .blend?' YES. 'From what I can tell you are saying that something silently broke earlier, and then you only found that out later when reopening the file and getting the popup? But that is a conjecture right? Something might have broken while working on the file earlier, or something might have broken while loading the .blend, but we don't know yet which.' Exactly. If I would be able to determine what happened and I would be able to redo it, it would be easy to fix. 'so you got the popup on opening the .blend, go to Blender File in the outliner, and see that the linked library is broken. The path is correct, but for some reason if you run "relocate" that fixes it, even though the path remains the same? ' YES. Or at least fixes it partially, since you then still need to redo a bunch of stuff?' Exactly. After it I have to reattach the textures to the shader inputs, change back the values (like normal strength, roughness, etc.), emission values, everything. Also I have to relink the non-local textures (locals remain untouched).

Alma Talp commented

2017-07-29 17:55:32 +02:00

Here is the image I promised.

Here is the image I promised.![library structure.jpg](https://archive.blender.org/developer/F684669/library_structure.jpg)

Bastien Montagne commented

2017-08-08 16:27:25 +02:00

Added subscriber: @mont29

Bastien Montagne commented

2017-08-08 16:27:25 +02:00

Changed status from 'Open' to: 'Archived'

Bastien Montagne closed this issue

2017-08-08 16:27:25 +02:00

Bastien Montagne self-assigned this 2017-08-08 16:27:25 +02:00

Bastien Montagne commented

2017-08-08 16:27:25 +02:00

Am sorry, but this is not manageable issue. We need clear reproducible cases, with needed .blend files, not a monster production case involving lots of different libraries we have absolutely no access to. Further more, you are using original 2.78 release - there were three corrective releases since then (latest official one is 2.78c), and we are about to release 2.79, so… Odds the issue still happens exactly the same in current master code is rather thin anyway. So will archive the report.

Am sorry, but this is not manageable issue. We need clear reproducible cases, with needed .blend files, not a monster production case involving lots of different libraries we have absolutely no access to. Further more, you are using original 2.78 release - there were **three** corrective releases since then (latest official one is 2.78c), and we are about to release 2.79, so… Odds the issue still happens exactly the same in current master code is rather thin anyway. So will archive the report.

Alma Talp commented

2017-10-10 23:00:58 +02:00

Hello guys, since the thread I lost 5 times linked data with no warning, last time today. The linked node groups simple disappeared from the scene, no warning at all, no orphan data appeared in the Outliner, nothing. As talking about a few hundred materials with 5-6 textures per material + all the material settings it is significant time to remake, especially to do it 5 times.

So, here is a suggestion:
If you could suggest me any kind of 'logger' application what logs everything what happens in Blender, I will install it as I was not able reproduce the error manually.

Thanks

Hello guys, since the thread I lost 5 times linked data with no warning, last time today. The linked node groups simple disappeared from the scene, no warning at all, no orphan data appeared in the Outliner, nothing. As talking about a few hundred materials with 5-6 textures per material + all the material settings it is significant time to remake, especially to do it 5 times. So, here is a suggestion: If you could suggest me any kind of 'logger' application what logs everything what happens in Blender, I will install it as I was not able reproduce the error manually. Thanks

Ethan Little commented

2022-06-30 23:55:58 +02:00

Added subscriber: @Ethanicus

Ethan Little commented

2022-06-30 23:55:58 +02:00

Hi, I think I'm running into this issue in Blender 3.0.0. I was working on some animation and reloaded the file, and was met with this:

Two of the three linked collections are now broken, and won't load. I can't even find the broken collections in the outliner. The reason seems to be that somehow, the paths were saved incorrectly. The correct path is [...]\3D\- Models -\FNAF Security Breach\v2, but it has somehow been saved with an extra "loop" back around to the \- Models -\ directory.

If I had to make a complete guess, I think this could be an error in Blender's relative file path system. I think rather than saving the linked directory from the .blend file's location, it instead did it from the models' .blend locations.

To be more clear how I got to this:

I opened model_A.blend
I linked model_b and model_c into the file
I deleted model_A from the file, and linked it instead
I saved the project as a new file in a different directory

And now the directories are broken. I'm not sure what other info I could provide. I also can't seem to relocate them because they don't even appear in the file info tab.

Hi, I think I'm running into this issue in Blender 3.0.0. I was working on some animation and reloaded the file, and was met with this: ![image.png](https://archive.blender.org/developer/F13244167/image.png) Two of the three linked collections are now broken, and won't load. I can't even find the broken collections in the outliner. The reason seems to be that somehow, the paths were saved incorrectly. The correct path is `[...]\3D\- Models -\FNAF Security Breach\v2`, but it has somehow been saved with an extra "loop" back around to the `\- Models -\` directory. If I had to make a complete guess, I think this could be an error in Blender's relative file path system. I think rather than saving the linked directory from the .blend file's location, it instead did it from the models' .blend locations. To be more clear how I got to this: - I opened model_A.blend - I linked model_b and model_c into the file - I deleted model_A from the file, and linked it instead - I saved the project as a new file in a different directory And now the directories are broken. I'm not sure what other info I could provide. I also can't seem to relocate them because they don't even appear in the file info tab.

Bastien Montagne commented

2022-07-01 09:12:17 +02:00

Resurrecting a five years old report is usually not recommended, better to make a new report, preferably strictly about current versions of Blender too (LTS or latest release).

That being said, we recently fixed a very similar issue (blender/blender#98201, blender/blender@83349294b1), fix is in Blender 3.2.

Resurrecting a five years old report is usually not recommended, better to make a new report, preferably strictly about current versions of Blender too (LTS or latest release). That being said, we recently fixed a very similar issue (blender/blender#98201, blender/blender@83349294b1), fix is in Blender 3.2.

This repo is archived. You cannot comment on issues.