Add Backward and Forward Compatibility Unit tests #110922

New Issue

Bastien Montagne · 2023-08-08T13:04:51+02:00

Bastien Montagne commented

2023-08-08 13:04:51 +02:00

This task is a proposal to implement some more consistent and covering compatibility testing.

Currently, we only get some flaky, fuzzy backward compatibility testing as indirect by-product of some other unittests, which often open test blendfiles that were generated by older Blender versions. This is however totally inconsistent and unreliable.

And there is no forward compatibility test at all.

Type of Blendfile for testing

Testing could use simple small, few-features testing files, complex 'full-featured' files (e.g. from Blender studio productions), or a mix of both.

Small, few-features files:
- + Have a small file size.
- + Cover a well-defined area.
- + Make finding actual issue in case of failure easier.
- + Are fast to process.
- - Are (very) tedious to create and maintain.
- - Will never cover as much as complex files, if nothing else because they will not cover complex interactions between different areas of code.
Big, full-featured production files:
- + Are already available (to some extent)
- + Have lower maintenance overhead.
- + Cover a lot with only a couple of files.
- + Cover complex interactions between different areas of Blender.
- + Cover issues that would only show up with bigger set of data.
- - Are (very) big files
- - Are slower to process
- - Will not be as easy to investigate in case of failure.

In any case, this set of file should be clearly grouped (in a specific folder, with their blender version in their file names).
Every time a new release is done, matching files should be added (or updated, in case of corrective releases).

Compatibility Testing Process

A good part of the issues should be detected by simply opening the blendfile. A few common operations could be added on top to cover even more potential cases:

Opening.
Changing current frame.
Select an armature object and switch to Pose mode.
Select one object for all available object types and switch to Edit mode.
Render one frame.
...

Backward Compatibility

Supporting this is relatively easy. the testing code (likely a python script) simply list all testing files, filters them based on the expected supported versions, and process them one after the other.

In case a failure is detected, and it is decided to not fix it (and it does not violates the compatibility policy, the minimum supported/tested version can be raised.

Forward Compatibility

This is slightly more involved to get working. The process is essentially the same as for the backward compatibility case, but the future versions of Blender need to be available.

The simplest solution would probably be for the tool to download the official release builds (within the expected range), and open the current version's test blend files with all these older builds.

Potential Issues

This proposal can raise some practical issues:

File size: especially if targeting production files for tests, this will likely add over 10Gb or more to the testing repository.
Processing time: opening heavy (or many) blendfiles in many Blender versions will take time, potentially a lot of time.
Maintenance:
- Need to often update file(s) for the Blender version(s) in development.
- Need to update file(s) for every release (including corrective releases, also for all LTS ones).

This task is a proposal to implement some more consistent and covering compatibility testing. Currently, we only get some flaky, fuzzy backward compatibility testing as indirect by-product of some other unittests, which often open test blendfiles that were generated by older Blender versions. This is however totally inconsistent and unreliable. And there is no forward compatibility test at all. ## Type of Blendfile for testing Testing could use simple small, few-features testing files, complex 'full-featured' files (e.g. from Blender studio productions), or a mix of both. * Small, few-features files: * `+` Have a small file size. * `+` Cover a well-defined area. * `+` Make finding actual issue in case of failure easier. * `+` Are fast to process. * `-` Are (very) tedious to create and maintain. * `-` Will never cover as much as complex files, if nothing else because they will not cover complex interactions between different areas of code. * Big, full-featured production files: * `+` Are already available (to some extent) * `+` Have lower maintenance overhead. * `+` Cover a lot with only a couple of files. * `+` Cover complex interactions between different areas of Blender. * `+` Cover issues that would only show up with bigger set of data. * `-` Are (very) big files * `-` Are slower to process * `-` Will not be as easy to investigate in case of failure. In any case, this set of file should be clearly grouped (in a specific folder, with their blender version in their file names). Every time a new release is done, matching files should be added (or updated, in case of corrective releases). ## Compatibility Testing Process A good part of the issues should be detected by simply opening the blendfile. A few common operations could be added on top to cover even more potential cases: * Opening. * Changing current frame. * Select an armature object and switch to Pose mode. * Select one object for all available object types and switch to Edit mode. * Render one frame. * ... ## Backward Compatibility Supporting this is relatively easy. the testing code (likely a python script) simply list all testing files, filters them based on the expected supported versions, and process them one after the other. In case a failure is detected, and it is decided to not fix it (and it does not violates the [compatibility policy](https://wiki.blender.org/wiki/Process/Compatibility_Handling), the minimum supported/tested version can be raised. ## Forward Compatibility This is slightly more involved to get working. The process is essentially the same as for the backward compatibility case, but the future versions of Blender need to be available. The simplest solution would probably be for the tool to download the official release builds (within the expected range), and open the current version's test blend files with all these older builds. ## Potential Issues This proposal can raise some practical issues: * File size: especially if targeting production files for tests, this will likely add over 10Gb or more to the testing repository. * Processing time: opening heavy (or many) blendfiles in many Blender versions will take time, potentially a lot of time. * Maintenance: * Need to often update file(s) for the Blender version(s) in development. * Need to update file(s) for every release (including corrective releases, also for all LTS ones).

👍 1

Bastien Montagne added the

Download

What's New

Blender Studio

Manual

Developers Blog

Documentation

Benchmark

Blender Conference

Development Fund

One-time Donations

Add Backward and Forward Compatibility Unit tests #110922

Type of Blendfile for testing

Compatibility Testing Process

Backward Compatibility

Forward Compatibility

Potential Issues