How to Handle Forward Compatibility #109151

New Issue

Forward compatibility is the ability of a software to open files generated by a more recent version of itself.

Blender has been designed since the beginning with a strong forward compatibility capacity. Its low-level data model (DNA) and its file format intrinsically support it.

On a higher level (user-level data and features) however, it is much more challenging to achieve over a long period of time. This is especially true when existing type of data or features are refactored.

Two types of forward compatibility breakage can be identified:

New feature or data type: This is the common, expected breaking case. An older version of Blender is not expected to understand and use data or feature that did not exist back then.
Refactor of existing feature or data type. There are two sub-cases here:
A. Refactor 'in place', where the internal organization and meaning of the data changes, but the general container (often an ID) remain the same. E.g. recently, the refactor of the Mesh data-block.
B. Refactor by replacing a deprecated data by a new one. E.g. recently the replacement of Proxies by LibOverrides, or the future replacement of Greasepencil v2 by the v3.
From a forward compatibility perspective, the A variant is usually way more difficult to handle properly than the B one.

Current Situation

So far, Blender has striven to ensure as much forward compatibility as possible, following these principles:

A .blend file should always open in any older Blender version.
Type 1 of breakages described above are expected and normal, they happen all the time.
- Older versions of Blender simply ignore these new features and data types.
- These new features and data types are lost if the .blend file is saved with the older version of Blender.
- There is no warning about it.
Type 2 of breakages are kept as rare as possible.
- They are only allowed when switching to a new major release.
- They involve a change in the BLENDER_FILE_MIN_VERSION/BLENDER_FILE_MIN_SUBVERSION.
- The file should still open in older versions of Blender, but there may be massive loss of data.
- There is a warning when opening such 'too new' .blend files.

Problems With Current Situation

While current situation works OK most of the time in basic cases (like appending a mesh object from a more recent .blend file version), it is usually impossible/unusable to open e.g. complex production files with older Blender versions.

Further more, the lack of documentation and testing has lead to critical breakages recently. E.g. any .blend file saved with Blender 4.0 containing a mesh will crash on any version version of Blender 3.5 or older. This includes the 3.3 LTS one (see #108491).

Proposed New Handling

Note: here, when referring to a .blend file, "incompatible" means that its min version > current Blender version, and "newer" means that its version > current Blender version.

The main proposed change is to not allow opening .blend files when their BLENDER_FILE_MIN_VERSION is higher than the current blender version.

This would integrate into this new proposed handling:

A .blend file can only be open in older Blender versions within the same major release cycle, and the latest LTS from the previous major release cycle.
- E.g. a .blend file from Blender 4.1 would be loadable by Blender 3.6 LTS, and Blender 4.0.
- The same should be true when linking or appending data.
Type 1 of breakages described above are expected and normal, they happen all the time.
- Older versions of Blender simply ignore these new features and data types.
- These new features and data types are lost if the .blend file is saved with the older version of Blender.
- There should be warning about it:
  - Have a permanent warning somewhere in the UI (footer?), instead of the popup only visible when opening.
  - Have a warning when saving a newer .blend file with an older version of Blender.
  - Later only warning when actual data loss is expected could be investigated. However, achieving a reliable detection of such cases will likely be very difficult.
Type 2 of breakages are kept as rare as possible.
- They are only allowed when switching to a new major release (so roughly every two years).
- They involve a change in the BLENDER_FILE_MIN_VERSION/BLENDER_FILE_MIN_SUBVERSION.
- When trying to open such files in older, incompatible versions of Blender, user should get a message stating the minimum Blender version expected to be able to open the file.
- Depending on the affected data and feature, the latest LTS version of the previous major release cycle can be kept compatible with both major versions, having a role of 'converter' to the older version of the data (e.g. Blender 3.6 can open meshes from 4.0, and will save them in the older 3.x compatible format).
Some automated testing for backward and forward compatibility should be created.
The expected handling regarding compatibility needs to be documented on the wiki.

Proposed Practical Changes

In case new handling described above is accepted, here are the proposed steps to implement it, by order of highest priority:

Prevent older blender versions to open incompatible .blend files, to be done in main, and backported to 3.6 and 3.3 LTS's.
- Include info message saying which minimal version of Blender to use to open the file.
Lower the minfile version in main to 3.6 (to allow 3.6 to be used as converter between both mesh data versions).
Implement more systematic warning about data loss when a newer .blend file is open, and when trying to save it.
~~Add some automated testing (CI) for forward (and backward!) compatibility.~~ Created dedicated task: #110922
Document the new version handling process in the wiki.
Add summary of this policy to the user manual (blender/blender-manual#104522).
Backport code changes to 3.6LTS (!110568).
Backport code changes to 3.3LTS (!110574).
Add an Interest/Compatibility label to projects.blender.org

Forward compatibility is the ability of a software to open files generated by a more recent version of itself. Blender has been designed since the beginning with a strong forward compatibility capacity. Its low-level data model (DNA) and its file format intrinsically support it. On a higher level (user-level data and features) however, it is much more challenging to achieve over a long period of time. This is especially true when existing type of data or features are refactored. Two types of forward compatibility breakage can be identified: 1. New feature or data type: This is the common, expected breaking case. An older version of Blender is not expected to understand and use data or feature that did not exist back then. 2. Refactor of existing feature or data type. There are two sub-cases here: A. Refactor 'in place', where the internal organization and meaning of the data changes, but the general container (often an ID) remain the same. E.g. recently, the refactor of the Mesh data-block. B. Refactor by replacing a deprecated data by a new one. E.g. recently the replacement of Proxies by LibOverrides, or the future replacement of Greasepencil v2 by the v3. From a forward compatibility perspective, the `A` variant is usually way more difficult to handle properly than the `B` one. ## Current Situation So far, Blender has striven to ensure as much forward compatibility as possible, following these principles: * A .blend file should always open in any older Blender version. * Type 1 of breakages described above are expected and normal, they happen all the time. * Older versions of Blender simply ignore these new features and data types. * These new features and data types are lost if the .blend file is saved with the older version of Blender. * There is no warning about it. * Type 2 of breakages are kept as rare as possible. * They are only allowed when switching to a new major release. * They involve a change in the `BLENDER_FILE_MIN_VERSION`/`BLENDER_FILE_MIN_SUBVERSION`. * The file should still open in older versions of Blender, but there may be massive loss of data. * There is a warning when opening such 'too new' .blend files. ## Problems With Current Situation While current situation works OK most of the time in basic cases (like appending a mesh object from a more recent .blend file version), it is usually impossible/unusable to open e.g. complex production files with older Blender versions. Further more, the lack of documentation and testing has lead to critical breakages recently. E.g. any .blend file saved with Blender 4.0 containing a mesh will crash on any version version of Blender 3.5 or older. This includes the 3.3 LTS one (see #108491). ## Proposed New Handling Note: here, when referring to a .blend file, "incompatible" means that its min version > current Blender version, and "newer" means that its version > current Blender version. The main proposed change is to not allow opening .blend files when their `BLENDER_FILE_MIN_VERSION` is higher than the current blender version. This would integrate into this new proposed handling: * A .blend file can only be open in older Blender versions within the same major release cycle, and the latest LTS from the previous major release cycle. * E.g. a .blend file from Blender 4.1 would be loadable by Blender 3.6 LTS, and Blender 4.0. * The same should be true when linking or appending data. * Type 1 of breakages described above are expected and normal, they happen all the time. * Older versions of Blender simply ignore these new features and data types. * These new features and data types are lost if the .blend file is saved with the older version of Blender. * There should be warning about it: * Have a permanent warning somewhere in the UI (footer?), instead of the popup only visible when opening. * Have a warning when saving a newer .blend file with an older version of Blender. * Later only warning when actual data loss is expected could be investigated. However, achieving a reliable detection of such cases will likely be very difficult. * Type 2 of breakages are kept as rare as possible. * They are only allowed when switching to a new major release (so roughly every two years). * They involve a change in the `BLENDER_FILE_MIN_VERSION`/`BLENDER_FILE_MIN_SUBVERSION`. * When trying to open such files in older, incompatible versions of Blender, user should get a message stating the minimum Blender version expected to be able to open the file. * Depending on the affected data and feature, the latest LTS version of the previous major release cycle can be kept compatible with both major versions, having a role of 'converter' to the older version of the data (e.g. Blender 3.6 can open meshes from 4.0, and will save them in the older 3.x compatible format). * Some automated testing for backward and forward compatibility should be created. * The expected handling regarding compatibility needs to be documented on the wiki. ## Proposed Practical Changes In case new handling described above is accepted, here are the proposed steps to implement it, by order of highest priority: - [x] Prevent older blender versions to open incompatible .blend files, to be done in main, and backported to 3.6 and 3.3 LTS's. - [x] Include info message saying which minimal version of Blender to use to open the file. - [x] Lower the minfile version in main to `3.6` (to allow 3.6 to be used as converter between both mesh data versions). - [x] Implement more systematic warning about data loss when a newer .blend file is open, and when trying to save it. - ![image](/attachments/d6994a7d-7c3b-443d-a464-0a94d87b36f7) - ![image](/attachments/f0f8f3c9-3be5-4d79-b162-9a15814b6f08) - [x] ~~Add some automated testing (CI) for forward (and backward!) compatibility.~~ Created dedicated task: #110922 - [x] [Document the new version handling process](https://wiki.blender.org/wiki/Process/Compatibility_Handling) in the wiki. - [x] Add summary of this policy to the user manual (https://projects.blender.org/blender/blender-manual/pulls/104522). - [x] Backport code changes to 3.6LTS (!110568). - [x] Backport code changes to 3.3LTS (!110574). - [x] Add an `Interest/Compatibility` label to projects.blender.org

image.png

56 KiB

@brecht @sergey @dfelinto @Ton here is the design task for the forward compatibility issue. Let me know if I forgot something or am not clear enough.

@ideasman42 think this topic may also interest you?

@brecht @sergey @dfelinto @Ton here is the design task for the forward compatibility issue. Let me know if I forgot something or am not clear enough. @ideasman42 think this topic may also interest you?

Implement more systematic warning about data loss when opening a newer blendfile and/or trying to save it.

My proposal is:

No warnings on file open (or link/append) - there is no data loss in this case.
Permanent warning on the status bar (see mockups).
Warning on saving (incorporated to the "Save" dialog).

Eventually we could be smarter about when to show the version warning (e.g., only when Blender read non-UI DNA data that was discarded).

Mockup of how the status bar could look like when we have a version forward-mismatch. When there is no warning, we can show the status bar with "plain-text" as we do now e.g., 3.5).

Detailed view + mouse to get more information:

It can still be combined with more information.

> Implement more systematic warning about data loss when opening a newer blendfile and/or trying to save it. My proposal is: * No warnings on file open (or link/append) - there is no data loss in this case. * Permanent warning on the status bar (see mockups). * Warning on saving (incorporated to the "Save" dialog). Eventually we could be smarter about when to show the version warning (e.g., only when Blender read non-UI DNA data that was discarded). --- Mockup of how the status bar could look like when we have a version forward-mismatch. When there is no warning, we can show the status bar with "plain-text" as we do now e.g., 3.5). ![image](/attachments/55da4a44-dc67-467f-a0c7-90e4435a7ed9) ---- Detailed view + mouse to get more information: ![image](/attachments/d6994a7d-7c3b-443d-a464-0a94d87b36f7) It can still be combined with more information. ![image](/attachments/45ef3417-35dc-4c89-8930-687a4575f413)

image.png

574 KiB

image.png

53 KiB

image.png

56 KiB

My proposal is:

No warnings on file open (or link/append) - there is no data loss in this case.

+1 (although you probably mean no popup or report warning, since there will be an indicator in the status bar)

Permanent warning on the status bar (see mockups).

+1
Completely offtopic but this compact layout is great for when we have Blender updates checker (in green, not red)

Warning on saving (incorporated to the "Save" dialog).

+1, this is the most important one because this is when you could lose data.

On Save/Ctrl+S, instead of simply saving, Blender should open the blocking dialog similar to when we have modified images:

With the text This file has been saved with a newer version of Blender (4.0.0), it is recommended to use that version (or newer) to prevent loss of data., or something along those lines.

> My proposal is: > * No warnings on file open (or link/append) - there is no data loss in this case. +1 (although you probably mean no *popup* or report warning, since there will be an indicator in the status bar) > * Permanent warning on the status bar (see mockups). +1 Completely offtopic but this compact layout is great for when we have Blender updates checker (in green, not red) > * Warning on saving (incorporated to the "Save" dialog). +1, this is the most important one because this is when you could lose data. On Save/Ctrl+S, instead of simply saving, Blender should open the blocking dialog similar to when we have modified images: ![image](/attachments/8587017d-a8b1-4c86-938a-05381f7cc72b) With the text `This file has been saved with a newer version of Blender (4.0.0), it is recommended to use that version (or newer) to prevent loss of data.`, or something along those lines.

image.png

56 KiB

This seems generally fine.

Am I understanding correctly that we are not planning any changes for 3.6.0 then?

A .blend file can only be open in older Blender versions within the same major release cycle, and the latest LTS from the previous major release cycle.

Handling of linking/appending here is unclear. Should it still be allowed?

In my opinion, no. The set of problems we are trying to prevent is the same as opening the .blend file.

Prevent older blender versions to open incompatible blendfiles, to be done in main, and backported to 3.6 and 3.3 LTS's.

Implement more systematic warning about data loss when opening a newer blendfile and/or trying to save it.

To be clear, "incompatible" here means the .blend file min version > current Blender version, and "newer" means .blend file version > current Blender version.

This seems generally fine. Am I understanding correctly that we are not planning any changes for 3.6.0 then? > * A .blend file can only be open in older Blender versions within the same major release cycle, and the latest LTS from the previous major release cycle. > * Handling of linking/appending here is unclear. Should it still be allowed? In my opinion, no. The set of problems we are trying to prevent is the same as opening the .blend file. > - [ ] Prevent older blender versions to open incompatible blendfiles, to be done in main, and backported to 3.6 and 3.3 LTS's. > - [ ] Implement more systematic warning about data loss when opening a newer blendfile and/or trying to save it. To be clear, "incompatible" here means the .blend file min version > current Blender version, and "newer" means .blend file version > current Blender version.

This seems generally fine.

Am I understanding correctly that we are not planning any changes for 3.6.0 then?

I think there will be changes (need to backport the 'do not open too much future files' process in there as well), but I don't think there is anything urgent, it can be done as part of a future 3.6LTS update imho.

A .blend file can only be open in older Blender versions within the same major release cycle, and the latest LTS from the previous major release cycle.

Handling of linking/appending here is unclear. Should it still be allowed?

In my opinion, no. The set of problems we are trying to prevent is the same as opening the .blend file.

Agreed.

Prevent older blender versions to open incompatible blendfiles, to be done in main, and backported to 3.6 and 3.3 LTS's.

Implement more systematic warning about data loss when opening a newer blendfile and/or trying to save it.

To be clear, "incompatible" here means the .blend file min version > current Blender version, and "newer" means .blend file version > current Blender version.

Yes, correct.

> This seems generally fine. > > Am I understanding correctly that we are not planning any changes for 3.6.0 then? I think there will be changes (need to backport the 'do not open too much future files' process in there as well), but I don't think there is anything urgent, it can be done as part of a future 3.6LTS update imho. > > * A .blend file can only be open in older Blender versions within the same major release cycle, and the latest LTS from the previous major release cycle. > > * Handling of linking/appending here is unclear. Should it still be allowed? > > In my opinion, no. The set of problems we are trying to prevent is the same as opening the .blend file. Agreed. > > - [ ] Prevent older blender versions to open incompatible blendfiles, to be done in main, and backported to 3.6 and 3.3 LTS's. > > - [ ] Implement more systematic warning about data loss when opening a newer blendfile and/or trying to save it. > > To be clear, "incompatible" here means the .blend file min version > current Blender version, and "newer" means .blend file version > current Blender version. Yes, correct.

Thanks for the feedback, updated the design task accordingly (regarding warnings when opening newer .blend file, and no support for link/append from incompatible .blend file).

Proposal looks good, am not so keen on permanent warning (adds a special case & needs to coexist with other warnings which come & go while using Blender), but I don't feel that strongly about it either.

A mockup on top of what Pablo suggested. I added the Save As option, but also made the text a bit shorter.

Override file with an older Blender version?

This file has been saved with a newer version of Blender (4.0). 
Saving with current Blender (3.6) may lead to loss of data.

I would suggest going with this and do a final pass on the text after the 3.6 release once Pablo has more time.

Edit: Overwrite may be the correct English term, and we could even use that instead of "Save" in the button label.

A mockup on top of what Pablo suggested. I added the Save As option, but also made the text a bit shorter. ![image](/attachments/d7d947e4-a8d8-4e3e-89de-84b8e91b46e9) ``` Override file with an older Blender version? This file has been saved with a newer version of Blender (4.0). Saving with current Blender (3.6) may lead to loss of data. ``` I would suggest going with this and do a final pass on the text after the 3.6 release once Pablo has more time. Edit: Overwrite may be the correct English term, and we could even use that instead of "Save" in the button label.

image.png

56 KiB

@dfelinto Since this dialog looks very close to the "Do you want to save before closing blender?" dialog, I suggest to change the icon to the warning symbol. On top of having the button say "Overwrite".

And another question: What about the autosave? Should that just save the current state as-is?

@dfelinto Since this dialog looks very close to the "Do you want to save before closing blender?" dialog, I suggest to change the icon to the warning symbol. On top of having the button say "Overwrite". And another question: What about the autosave? Should that just save the current state as-is?

@filedescriptor Auto Save doesn't overwrite existing data, it creates new files in the temp directory. It should just keep saving what it already does. So users can recover from a crash and still decide whether they want to save the changes, save as, or export to another format.

Since this dialog looks very close to the "Do you want to save before closing blender?" dialog, I suggest to change the icon to the warning symbol.

I have no strong opinion here and I'm ok deferring to Pablo to make a decision. But I find both scenarios equally alarming, so not sure I would priorizite one with exclamation mark over the other.

@filedescriptor Auto Save doesn't overwrite existing data, it creates new files in the temp directory. It should just keep saving what it already does. So users can recover from a crash and still decide whether they want to save the changes, save as, or export to another format. > Since this dialog looks very close to the "Do you want to save before closing blender?" dialog, I suggest to change the icon to the warning symbol. I have no strong opinion here and I'm ok deferring to Pablo to make a decision. But I find both scenarios equally alarming, so not sure I would priorizite one with exclamation mark over the other.

Per the request for feedback from the weeklies on devatalk: I am pro the UX of adding more warnings, either permenant with versioning warnings on overwrite or save as. Good proposal and solution to the issue.

Added a WIP wiki page about Blender compatibility handling: https://wiki.blender.org/wiki/Process/Compatibility_Handling

Also had to add a section about backward compatibility, though this is a more straight-forward topic I think it's good to be documented as well.

Still very much WIP, though.

Added a WIP wiki page about Blender compatibility handling: https://wiki.blender.org/wiki/Process/Compatibility_Handling Also had to add a section about backward compatibility, though this is a more straight-forward topic I think it's good to be documented as well. Still very much WIP, though.

This looks great guys, and a very much needed feature!
If I may add my two cents based on my personal experience (coming from India where English is non-native)

Override file with an older Blender version?

This file has been saved with a newer version of Blender (4.0).
Saving with current Blender (3.6) may lead to loss of data.

Artists are not very receptive of long, wordy messages. It will be great if we can use a shorter but precise message.

Since this dialog looks very close to the "Do you want to save before closing blender?" dialog, I suggest to change the icon to the warning symbol. On top of having the button say "Overwrite".

I agree with @filedescriptor. Responding to dialog boxes becomes a part of muscle memory and no one really reads the actual message every time. A slight change in color/symbol will be much appreciated.
For the same reason, I would also suggest the default option should be Save As, and the alternative should be Save/Overwrite.

But I find both scenarios equally alarming,

@dfelinto agreed, but this scenario will be rarer and more destructive, hence requires more discretion IMO.

Mockup based on suggestions (looks messy because I'm throwing in all ideas... we can pick only what works for us) :

This looks great guys, and a very much needed feature! If I may add my two cents based on my personal experience (coming from India where English is non-native) > Override file with an older Blender version? > > This file has been saved with a newer version of Blender (4.0). > Saving with current Blender (3.6) may lead to loss of data. - Artists are not very receptive of long, wordy messages. It will be great if we can use a shorter but precise message. ----------------------------- > Since this dialog looks very close to the "Do you want to save before closing blender?" dialog, I suggest to change the icon to the warning symbol. On top of having the button say "Overwrite". - I agree with @filedescriptor. Responding to dialog boxes becomes a part of muscle memory and no one really reads the actual message every time. A slight change in color/symbol will be much appreciated. For the same reason, I would also suggest the default option should be Save As, and the alternative should be Save/Overwrite. ----------------------------- > But I find both scenarios equally alarming, - @dfelinto agreed, but this scenario will be rarer and more destructive, hence requires more discretion IMO. ----------------------------- Mockup based on suggestions (looks messy because I'm throwing in all ideas... we can pick only what works for us) : ![version check.jpg](/attachments/c2a36925-9ec1-4d0a-b5cb-521e8b7cb2a9)

version check.jpg

63 KiB

To be clear, "incompatible" here means the .blend file min version > current Blender version, and "newer" means .blend file version > current Blender version.

For extra clarity, I'd replace "current Blender" with "currently running Blender". I had to read a few times before I understood this, as "current Blender version" could also be read as "the latest release".

Mockup based on suggestions (looks messy because I'm throwing in all ideas... we can pick only what works for us) :

I think the "will be converted" is too promising. This mockup starts with a positive message that your data will be converted. The warning that this might not be a complete conversion is only shown at the end, and in the smallest font. I'd be afraid people are going to miss it.

I do like the red "Overwrite" button on the right, and the blue "Save As" button on the left. This makes the non-overwriting, safe option the default, and the dangerous, lossy option the special one. If people don't read and just click the right-most button (as that's typically the "next next next until it's done" spot) they don't lose their data.

> To be clear, "incompatible" here means the .blend file min version > current Blender version, and "newer" means .blend file version > current Blender version. For extra clarity, I'd replace "current Blender" with "currently running Blender". I had to read a few times before I understood this, as "current Blender version" could also be read as "the latest release". > Mockup based on suggestions (looks messy because I'm throwing in all ideas... we can pick only what works for us) : > > ![version check.jpg](/attachments/c2a36925-9ec1-4d0a-b5cb-521e8b7cb2a9) I think the "will be converted" is too promising. This mockup starts with a positive message that your data will be converted. The warning that this might not be a complete conversion is only shown at the end, and in the smallest font. I'd be afraid people are going to miss it. I do like the red "Overwrite" button on the right, and the blue "Save As" button on the left. This makes the non-overwriting, safe option the default, and the dangerous, lossy option the special one. If people don't read and just click the right-most button (as that's typically the "next next next until it's done" spot) they don't lose their data.

Note that there is now an actual implementation in #110109 to test and give UI feedback on.

There should also be user level documentation making it clear what users can expect regarding file compatibility. On a quick search I couldn't find anything in the current manual.

There should also be user level documentation making it clear what users can expect regarding file compatibility. On a quick search I couldn't find anything in the current manual.

Good point, added another entry to the TODO list in the task above.

> There should also be user level documentation making it clear what users can expect regarding file compatibility. On a quick search I couldn't find anything in the current manual. Good point, added another entry to the TODO list in the task above.

Think we can close this one now, testing compatibility has its own dedicated task: #110922.

Download

What's New

Blender Studio

Manual

Developers Blog

Documentation

Benchmark

Blender Conference

Development Fund

One-time Donations

How to Handle Forward Compatibility #109151

Current Situation

Problems With Current Situation

Proposed New Handling

Proposed Practical Changes