Update “Mystery Of The Blend” #52387

New Issue

Lawrence D'Oliveiro · 2017-08-14T00:04:41+02:00

2017-08-14 00:04:41 +02:00

The file doc/blender_file_format/mystery_of_the_blend.html is badly out of date, and just plain wrong in some respects. This set of patches tries to make it more useful.

ldo_blend_file_format_20170814.tar

The file doc/blender_file_format/mystery_of_the_blend.html is badly out of date, and just plain wrong in some respects. This set of patches tries to make it more useful. [ldo_blend_file_format_20170814.tar](https://archive.blender.org/developer/F709762/ldo_blend_file_format_20170814.tar)

Lawrence D'Oliveiro commented

2017-08-14 00:04:41 +02:00

Changed status to: 'Open'

Lawrence D'Oliveiro commented

2017-08-14 00:04:41 +02:00

Added subscriber: @ldo

Sergey Sharybin commented

2017-08-14 15:01:56 +02:00

Added subscribers: @Jeroen-Bakker, @Sergey

Jeroen Bakker was assigned by Sergey Sharybin

2017-08-14 15:01:56 +02:00

Sergey Sharybin commented

2017-08-14 15:01:56 +02:00

@Jeroen-Bakker, to our knowledge this doc is from you :)

Lawrence D'Oliveiro commented

2017-09-17 03:59:27 +02:00

Is anybody maintaining this? Otherwise perhaps it should be removed from the source tree.

Lawrence D'Oliveiro commented

2018-03-24 10:01:43 +01:00

Anybody there?

Jeroen Bakker commented

2018-03-24 16:56:09 +01:00

Hi Lawrence

Sorry for the long waait. It went under my radar intik thuis morning. Will check on it next week.

Jeroen

Hi Lawrence Sorry for the long waait. It went under my radar intik thuis morning. Will check on it next week. Jeroen

Jeroen Bakker commented

2018-03-27 15:01:57 +02:00

Hi Lawrence, can you elaborate on what you find Badly out of date and plainly wrong?
I reviewed the text, but expected more difference if you find something badly out of date and plainly wrong.

I do agree with the update of the links and adding more details to the doc, but would not add personal reflections of the file format to the doc as that is not the place to have discussions or opinions.

I also believe that the document by itself should be moved to perhaps wiki.blender.org or code.blender.org
I also believe that currently we have the option to move the rest to the source/tools location where currently the other tools are located.

Hi Lawrence, can you elaborate on what you find Badly out of date and plainly wrong? I reviewed the text, but expected more difference if you find something badly out of date and plainly wrong. I do agree with the update of the links and adding more details to the doc, but would not add personal reflections of the file format to the doc as that is not the place to have discussions or opinions. I also believe that the document by itself should be moved to perhaps wiki.blender.org or code.blender.org I also believe that currently we have the option to move the rest to the `source/tools` location where currently the other tools are located.

Lawrence D'Oliveiro commented

2018-03-28 00:23:50 +02:00

From the original text:

Blender is known to have excellent downward and upward compatibility

Wrong. “Upward” (forward?) compatibility was never a feature.

The article is written to be programming language independent

Wrong. It uses C-language structs.

and I've setup a web-site for support.

Out of date. The page in question has gone.

the default blend file of Blender 2.48 contains more than 400 of these file-blocks

Out of date. The default blend file of Blender 2.72 contains more than 800 of these file-blocks. Particularly since you do mention post-2.4x versions of Blender elsewhere.

Furthermore:

No mention of (rather important) GLOB block, or of the USER block.
No mention that ENDB block might be truncated.
No mention that the version number in the file header governs translation of fields and structures which may look the same as in older versions, but whose meanings have changed.
No mention that the size of a block must always be a multiple of 4 bytes.
No mention of the significant difference between principal versus subsidiary blocks, or that the SDNA index can be invalid in subsidiary blocks.
No mention of the (important!) distinction between type index and structure index.
No mention of possibly-invalid pointer fields.

Does that give you enough of a start on making sense of it all?

What do you mean by “personal reflections”?

From the original text: > Blender is known to have excellent downward and upward compatibility Wrong. “Upward” (forward?) compatibility was never a feature. > The article is written to be programming language independent Wrong. It uses C-language structs. > and I've setup a web-site for support. Out of date. The page in question has gone. > the default blend file of Blender 2.48 contains more than 400 of these file-blocks Out of date. The default blend file of Blender 2.72 contains more than 800 of these file-blocks. Particularly since you do mention post-2.4*x* versions of Blender elsewhere. Furthermore: * No mention of (rather important) GLOB block, or of the USER block. * No mention that ENDB block might be truncated. * No mention that the version number in the file header governs translation of fields and structures which may look the same as in older versions, but whose meanings have changed. * No mention that the size of a block must always be a multiple of 4 bytes. * No mention of the significant difference between *principal* versus *subsidiary* blocks, or that the SDNA index can be invalid in subsidiary blocks. * No mention of the (important!) distinction between type index and structure index. * No mention of possibly-invalid pointer fields. Does that give you enough of a start on making sense of it all? What do you mean by “personal reflections”?

Jeroen Bakker commented

2018-03-28 09:34:25 +02:00

Upwards/Forward compatibility (to extend of loading data) is a demonstrated feature. See first picture of http://zgodzinski.com/blender-prehistory/
Also it is used by production houses to temp switch for some reason to a latest version of blender to do stuff and continue with the version of blender in fixed their pipeline.

That the document uses a single C-Struct to explain stuff does not make the content of the document language specific. The article is proven to be language independent. It has been used to create blend file loaders in Java, Python, Javascript, etc. . Also the document states that it is written 'to be language independent'. Is proven to be, and you say that that is not the case?

Agree the reference to the web-site is gone. this section can be removed as it is replaced with a generator that generates the website from a blend file.

Out of date blend file data. Is not out of date. A blend file from blender 2.48 still has 400 file-blocks. You can adapt it every time blender does a new release, but at the end it is not important that this document reflects the latest version. The intention of the document is to describe the blend file format.

About the rest of your comments

You explain the GLOB and USER data blocks in more depth. The original document only described the bits and bytes. Don't mind adding that.

The meaning of the data is blender version specific. It is mentioned here:

In the source code of blender there is actually logic which can transform and translate every structure used by a Blender release to the one of the release you're using [ref: http://download.blender.org/source/blender-2.48a.tar.gz blender/blenloader/intern/readfile.c lines 4946-7960]. The more difference between releases the more logic is executed

Alignment: As DNA struct must be aligned, the file data are therefore also aligned. It is not mentioned, so we could add that.
The other points have valid value to be added. Would add more wording to it though.

The rest of your points are valid so no problem adding them.

Would remove the second paragraph of your conclusion as that should IMO be part of a different document.

Sorry about the personal reflections. I have re-read the additions and I have mis-read them previously.

Where do you suggest we could add the document including the modifications?
I would recommend to move the code parts to a blender/source/tools repository and the document to the blender developer wiki.

Upwards/Forward compatibility (to extend of loading data) is a demonstrated feature. See first picture of http://zgodzinski.com/blender-prehistory/ Also it is used by production houses to temp switch for some reason to a latest version of blender to do stuff and continue with the version of blender in fixed their pipeline. That the document uses a single C-Struct to explain stuff does not make the content of the document language specific. The article is proven to be language independent. It has been used to create blend file loaders in Java, Python, Javascript, etc. . Also the document states that it is written 'to be language independent'. Is proven to be, and you say that that is not the case? Agree the reference to the web-site is gone. this section can be removed as it is replaced with a generator that generates the website from a blend file. Out of date blend file data. Is not out of date. A blend file from blender 2.48 still has 400 file-blocks. You can adapt it every time blender does a new release, but at the end it is not important that this document reflects the latest version. The intention of the document is to describe the blend file format. About the rest of your comments You explain the GLOB and USER data blocks in more depth. The original document only described the bits and bytes. Don't mind adding that. The meaning of the data is blender version specific. It is mentioned here: > In the source code of blender there is actually logic which can transform and translate every structure used by a Blender release to the one of the release you're using [ref: http://download.blender.org/source/blender-2.48a.tar.gz blender/blenloader/intern/readfile.c lines 4946-7960]. The more difference between releases the more logic is executed Alignment: As DNA struct must be aligned, the file data are therefore also aligned. It is not mentioned, so we could add that. The other points have valid value to be added. Would add more wording to it though. The rest of your points are valid so no problem adding them. Would remove the second paragraph of your conclusion as that should IMO be part of a different document. Sorry about the personal reflections. I have re-read the additions and I have mis-read them previously. Where do you suggest we could add the document including the modifications? I would recommend to move the code parts to a blender/source/tools repository and the document to the blender developer wiki.

Brecht Van Lommel commented

2018-03-28 09:53:26 +02:00

Added subscriber: @brecht

Brecht Van Lommel commented

2018-03-28 09:53:26 +02:00

I suggest to move the document to the developer wiki where the other code documentation is. For the source code we can point here:
https://developer.blender.org/source/blender-file/

I suggest to move the document to the developer wiki where the other code documentation is. For the source code we can point here: https://developer.blender.org/source/blender-file/

Lawrence D'Oliveiro commented

2018-03-29 03:00:52 +02:00

If you want code for reading/writing .blend files, let me point out my own slightly more ambitious effort: blendhack.

If you want code for reading/writing `.blend` files, let me point out my own slightly more ambitious effort: [blendhack](https://github.com/ldo/blendhack).

Lawrence D'Oliveiro commented

2018-04-02 00:13:40 +02:00

So, is this going to be removed from the source tree?

Ulrich Windl commented

2019-01-28 21:16:24 +01:00

Added subscriber: @torus

Ulrich Windl commented

2019-01-28 21:16:24 +01:00

I' not deep into Blender (as you are), but I wrote several parsers in the past, but I fail to parse the .blend files:

The process of flattening a structure is not explained well enough, especially when the structure contains another structure. All documents I found root in one document that doesn't explain the process, to nobody else does. For example the only comment I received for How to interpret Blender's SDNA structure field type index? was "*Not really sure, but for what I remember (and doc/blender_file_format/ directory) you never refer to structure index, only to type index. Structure itself is linked with some type though, so you get that via reverse lookup.//"

Note "Not really sure, "... the public seems to do not know. So the document can be improved clearly .

I' not deep into Blender (as you are), but I wrote several parsers in the past, but I fail to parse the `.blend` files: The process of *flattening a structure* is not explained well enough, especially when the structure contains another structure. All documents I found root in one document that doesn't explain the process, to nobody else does. For example the only comment I received for [How to interpret Blender's SDNA structure field type index? ](https:*stackoverflow.com/questions/54392327/how-to-interpret-blenders-sdna-structure-field-type-index) was "*Not really sure, but for what I remember (and doc/blender_file_format/ directory) you never refer to structure index, only to type index. Structure itself is linked with some type though, so you get that via reverse lookup.//" Note "**Not really sure**, "... the public seems to do **not** know. So the document can be improved clearly .

Lawrence D'Oliveiro commented

2019-01-28 22:54:15 +01:00

Structures inside structures work the same way as in C—they have to. My rework explains exactly when the structure index is used, and when the type index is used. Which is just one reason why I think it’s a big improvement on the doc that’s already there!

Lawrence D'Oliveiro commented

2019-09-24 10:34:57 +02:00

I would like to know whether I am wasting my time.

Jeroen Bakker was unassigned by Dalai Felinto

2019-12-23 16:42:36 +01:00

Aaron Carlisle commented

2019-12-28 20:24:39 +01:00

Added subscriber: @Blendify

Aaron Carlisle commented

2019-12-28 20:24:39 +01:00

Changed status from 'Confirmed' to: 'Resolved'

Aaron Carlisle closed this issue

2019-12-28 20:24:39 +01:00

Aaron Carlisle self-assigned this 2019-12-28 20:24:39 +01:00

Aaron Carlisle commented

2019-12-28 20:24:39 +01:00

Hi thank you for submitting a patch, unfortunately, we no longer use the task subtype "Patch" please submit new patches through the differential tool: https://developer.blender.org/differential/

Sign in to join this conversation.

No Label

Download

What's New

Blender Studio

Manual

Developers Blog

Documentation

Benchmark

Blender Conference

Development Fund

One-time Donations

Update “Mystery Of The Blend” #52387