Add Backward and Forward Compatibility Unittests #110922
This task is a proposal to implement some more consistent and covering compatibility testing.
Currently, we only get some flaky, fuzzy backward compatibility testing as indirect by-product of some other unittests, which often open test blendfiles that were generated by older Blender versions. This is however totally inconsistent and unreliable.
And there is no forward compatibility test at all.
Type of Blendfile for testing
Testing could use simple small, few-features testing files, complex 'full-featured' files (e.g. from Blender studio productions), or a mix of both.
- Small, few-features files:
+Have a small file size.
+Cover a well-defined area.
+Make finding actual issue in case of failure easier.
+Are fast to process.
-Are (very) tedious to create and maintain.
-Will never cover as much as complex files, if nothing else because they will not cover complex interactions between different areas of code.
- Big, full-featured production files:
+Are already available (to some extent)
+Have lower maintenance overhead.
+Cover a lot with only a couple of files.
+Cover complex interactions between different areas of Blender.
+Cover issues that would only show up with bigger set of data.
-Are (very) big files
-Are slower to process
-Will not be as easy to investigate in case of failure.
In any case, this set of file should be clearly grouped (in a specific folder, with their blender version in their file names).
Every time a new release is done, matching files should be added (or updated, in case of corrective releases).
Compatibility Testing Process
A good part of the issues should be detected by simply opening the blendfile. A few common operations could be added on top to cover even more potential cases:
- Changing current frame.
- Select an armature object and switch to Pose mode.
- Select one object for all available object types and switch to Edit mode.
- Render one frame.
Supporting this is relatively easy. the testing code (likely a python script) simply list all testing files, filters them based on the expected supported versions, and process them one after the other.
In case a failure is detected, and it is decided to not fix it (and it does not violates the compatibility policy, the minimum supported/tested version can be raised.
This is slightly more involved to get working. The process is essentially the same as for the backward compatibility case, but the future versions of Blender need to be available.
The simplest solution would probably be for the tool to download the official release builds (within the expected range), and open the current version's test blend files with all these older builds.
This proposal can raise some practical issues:
- File size: especially if targeting production files for tests, this will likely add over 10Gb or more to the testing repository.
- Processing time: opening heavy (or many) blendfiles in many Blender versions will take time, potentially a lot of time.
- Need to often update file(s) for the Blender version(s) in development.
- Need to update file(s) for every release (including corrective releases, also for all LTS ones).
No due date set.
No dependencies set.
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?