.blend files saved with 2.75a contain multiple file chunks (BHead) with same pointer address #45471

New Issue

Peter Quiring · 2015-07-17T15:20:27+02:00

Peter Quiring commented

2015-07-17 15:20:27 +02:00

System Information
Windows 7 SP1 64bit
Radeon HD 5400 1GB

Blender Version
Broken: 2.75a
Worked: 2.74

Short description of error
.blend files saved with 2.75a contain multiple file chunks (BHead) with the same pointer address.
I'm loading the .blend files from Java Game.

Exact steps for others to reproduce the error
Save .blend file in 2.75a
Parse the BHead entries and you will find duplicates.
All duplicate chunks are type DATA with a size of 520 bytes and appears to be UV data. Somehow blender knows which one to use but I can't figure out how.
Resave .blend file in 2.74 and there are no duplicates and everything works fine.

armor.blend

**System Information** Windows 7 SP1 64bit Radeon HD 5400 1GB **Blender Version** Broken: 2.75a Worked: 2.74 **Short description of error** .blend files saved with 2.75a contain multiple file chunks (BHead) with the same pointer address. I'm loading the .blend files from Java Game. **Exact steps for others to reproduce the error** Save .blend file in 2.75a Parse the BHead entries and you will find duplicates. All duplicate chunks are type DATA with a size of 520 bytes and appears to be UV data. Somehow blender knows which one to use but I can't figure out how. Resave .blend file in 2.74 and there are no duplicates and everything works fine. [armor.blend](https://archive.blender.org/developer/F207692/armor.blend)

Peter Quiring commented

2015-07-17 15:20:27 +02:00

Changed status to: 'Open'

Peter Quiring commented

2015-07-17 15:20:27 +02:00

Added subscriber: @pquiring

blender-admin commented

2015-07-17 15:20:27 +02:00

#45918 was marked as duplicate of this issue

Peter Quiring commented

2015-07-17 15:34:15 +02:00

I tried 2.75 (without the a) and same problem.

I've tried using the first or last occurrence of the duplicate data chunks but the results are always corrupt.

Thanks.

I tried 2.75 (without the a) and same problem. I've tried using the first or last occurrence of the duplicate data chunks but the results are always corrupt. Thanks.

Peter Quiring commented

2015-07-17 17:18:01 +02:00

The duplicate chunks are all CustomDataLayer structs (DNA# 454 in the example provided).

Thomas Dinges commented

2015-07-19 14:18:46 +02:00

Added subscribers: @ThomasDinges, @ideasman42, @Sergey

Thomas Dinges commented

2015-07-19 14:18:46 +02:00

Can you please check?

Peter Quiring commented

2015-07-19 15:11:00 +02:00

I have 50 models (.blend files) in my project. The models with only 1 object had no problem. But any other model with multiple objects seems to cause this issue.

Bastien Montagne commented

2015-07-19 16:57:34 +02:00

Added subscriber: @mont29

Bastien Montagne commented

2015-07-19 16:57:34 +02:00

Do we have a tool (like a py script or so) to easily print out a .blend file structure? Otherwise, could be worth adding this (should not be too hard, given the work already done for BAM, I hope?).

Sergey Sharybin commented

2015-07-19 17:44:24 +02:00

@mont29, you can use blendfile.py which was originally written for BlenderAID and then updated to latest .blend in BAM. See https://developer.blender.org/diffusion/BAM/browse/master/bam/blend/

Peter Quiring commented

2015-07-19 22:55:45 +02:00

I've stripped down my .blend loader and made it print out each BHead as it loads are stores them into a HashMap keyed by the oldptr value. It's a java app but I've included it compiled.
Just run "java -jar blenderFileParse.jar armor.blend"
The output is also included in armor.txt

What I find strange is that blender seems to be unaffected by these duplicate BHeads. I can open the file in blender and everything is okay (even in v2.74). Somehow in readfile.c the qsort() and oldnewmap_liblookup() function are getting the correct BHead data, possibly by chance.

blenderFileParse.zip

I've stripped down my .blend loader and made it print out each BHead as it loads are stores them into a HashMap keyed by the oldptr value. It's a java app but I've included it compiled. Just run "java -jar blenderFileParse.jar armor.blend" The output is also included in armor.txt What I find strange is that blender seems to be unaffected by these duplicate BHeads. I can open the file in blender and everything is okay (even in v2.74). Somehow in readfile.c the qsort() and oldnewmap_liblookup() function are getting the correct BHead data, possibly by chance. [blenderFileParse.zip](https://archive.blender.org/developer/F209368/blenderFileParse.zip)

Bastien Montagne self-assigned this 2015-07-19 23:25:18 +02:00

Bastien Montagne commented

2015-07-19 23:25:18 +02:00

Ok, I’ll try to understand this tomorrow

Peter Quiring commented

2015-07-20 13:52:05 +02:00

I think I found the answer. In readfile.c there is a function called oldnewmap_lookup_and_inc() that I think handles the case where multiple BHead chunks contain the same old pointer. It uses a lasthit field to remember which one it used and uses them in order.
This sucks, now I can't store the BHead's in a HashMap keyed on the old pointer.
2.74 doesn't rely on these duplicate entries but 2.75 does. Why? It's really confusing.

You should update the "mystery_of_the_blend.html" to indicate that BHead's can contain duplicate entries and are used in the order they were saved.

Sorry to bother you.

Thanks.

I think I found the answer. In readfile.c there is a function called oldnewmap_lookup_and_inc() that I think handles the case where multiple BHead chunks contain the same old pointer. It uses a lasthit field to remember which one it used and uses them in order. This sucks, now I can't store the BHead's in a HashMap keyed on the old pointer. 2.74 doesn't rely on these duplicate entries but 2.75 does. Why? It's really confusing. You should update the "mystery_of_the_blend.html" to indicate that BHead's can contain duplicate entries and are used in the order they were saved. Sorry to bother you. Thanks.

Bastien Montagne commented

2015-07-20 14:10:38 +02:00

Yeah, was coming to the same conclusion about why it works in Blender - but this is brittle and dangerous… Could break virtually anytime by a small change in some unrelated part of read code, imho.

The issue comes from writing CDLayers, in write_meshes() we make temp copy of cdlayers (to strip out temp layers we do not want to save), and we end up saving those temp copies' addresses, instead of original ones, which are guaranteed to be unique (temp copies can reuse same mem across meshes).

Should not be that hard to fix, I’m on it.

Yeah, was coming to the same conclusion about why it works in Blender - but this is brittle and dangerous… Could break virtually anytime by a small change in some unrelated part of read code, imho. The issue comes from writing CDLayers, in `write_meshes()` we make temp copy of cdlayers (to strip out temp layers we do not want to save), and we end up saving those temp copies' addresses, instead of original ones, which are guaranteed to be unique (temp copies can reuse same mem across meshes). Should not be that hard to fix, I’m on it.

Peter Quiring commented

2015-07-20 16:13:50 +02:00

Ya, I've modified my code to support reading these duplicates properly. It wasn't easy. The dups have to be read in the order they are in file but in the reverse order of the objects in their linked list order and not in the order the objects are in the file (get that?)

Anyways, thanks for helping.

Ya, I've modified my code to support reading these duplicates properly. It wasn't easy. The dups have to be read in the order they are in file but in the reverse order of the objects in their linked list order and not in the order the objects are in the file (get that?) Anyways, thanks for helping.

blender-admin commented

2015-07-21 12:02:11 +02:00

This issue was referenced by ff3d535bc2

This issue was referenced by ff3d535bc2a63092357533acbe7289556af956a0

Bastien Montagne commented

2015-07-21 12:07:56 +02:00

Changed status from 'Open' to: 'Resolved'

Bastien Montagne closed this issue

2015-07-21 12:07:56 +02:00

Bastien Montagne commented

2015-08-27 12:37:39 +02:00

Added subscribers: @Psy-Fi, @Blendify, @TheOnlyJoey