Simulation Nodes: File Format for cached/baked simulation states #105251

New Issue

Jacques Lucke · 2023-02-27T15:40:45+01:00

Jacques Lucke commented

2023-02-27 15:40:45 +01:00

For simulation nodes (#103032) we need to store caches on disk. The task here is to define the file format and directory structure for the cache.

Requirements:

Support complex simulation states including instances, volumes, single values and linked materials.
Easy append after simulating more frames.
Random access to specific frames.
Avoid storing the same data multiple times (e.g. only store topology and other attributes once, if we only deform an object).

Considerations

There are three main options with their own pros and cons which are explained in more detail below.

.blend Files

The main benefit of using .blend files is that they could easily be opended in Blender and we already have code to serialize and deserialize e.g. meshes to this format. There are a couple of different ways this could work:

Have a standalone .blend file per frame.
Have all frames in one .blend file.
Have multiple .blend files that can reference each other to avoid storing the same array data multiple times.

Currently, .blend files cannot store instances and volumes. Appending to an existing .blend files is likely very slow. The files also store way more than would be necessary for a simulation. It's hard to optimize reading and writing .blend files for the simulation cache use-case because of a lot of existing code surrounding the file format that needs to keep working. We'd likely have to do a fair amount of changes to make .blend files suitable for simulation cache that does not waste lots of memory and fullfills our performance expectations, making the whole code even more complex.

Common Interchange Formats

Another possibility is to use e.g. Alembic or USD to store the simulation state. The main benefit here is that the individual frames can be opened in Blender and also in other software. While it is of course important to be able to export the simulated data in these formats, for exporting the "raw geometry data" they are not necessarily the right solution. With other file formats we always have to consider out to map Blender data to these and back in a hopefully lossless way.

From what we've seen so far, these formats don't work well with compressing data over multiple frames. Also I heard that we ran into performance problems trying to read animations cached into alembic in real-time.

Using a common interchange format would make it impossible for us to change the format to better suit our needs if that becomes necessary.

Custom Format

With a custom format with have the most flexibility. Also it's likely simpler to do than any of the other approaches, because we wouldn't have to figure out how to fit all the data we want to store into those formats. Shared data over multiple frames can just be part of the format description. The main concern with this approach is that we have to maintain a new file format in the long term, hence it should be kept as simple and straight forward as possible. Another downside is that we have to write new code to serialize geometry data like a mesh. Luckily, not all data in a Mesh data-block has to be stored, and all the important data generally is stored in plain arrays already. Volumes would like still be stored as openvdb files.

There are of course many ways to design a custom format given the requirements. Currently, I have the following in mind:

The data is split into two parts: The "big data" (e.g. attribute arrays) and a description for how to assemble the that data into the simulation state (e.g. a GeometrySet).
Both parts are saved in separate files.
For every frame, a new (relatively small) file is written that contains all the information for what is in the simulation state of that frame. This file usually has references to data stored in other files. The file format of that has not been decided upon. It's likely easiest to use some human readable format like json. This data could also easily be kept in memory at all times because it's small.
The "big data" that is new in every frame (has not been stored previously) is stored in one or multiple "raw data" files.

One issue that none of these formats automatically solve is how to deal with referenced id data blocks like materials. However, a fairly straight forward and likely good-enough option can be to store just the library and material name on disk.

Conclusion

While there are still details to be figured out when actually implementing this, using a custom format seems to be the best approach, so we will focus on that from now on.

For simulation nodes (#103032) we need to store caches on disk. The task here is to define the file format and directory structure for the cache. Requirements: * Support complex simulation states including instances, volumes, single values and linked materials. * Easy append after simulating more frames. * Random access to specific frames. * Avoid storing the same data multiple times (e.g. only store topology and other attributes once, if we only deform an object). ## Considerations There are three main options with their own pros and cons which are explained in more detail below. ### .blend Files The main benefit of using .blend files is that they could easily be opended in Blender and we already have code to serialize and deserialize e.g. meshes to this format. There are a couple of different ways this could work: * Have a standalone .blend file per frame. * Have all frames in one .blend file. * Have multiple .blend files that can reference each other to avoid storing the same array data multiple times. Currently, .blend files cannot store instances and volumes. Appending to an existing .blend files is likely very slow. The files also store way more than would be necessary for a simulation. It's hard to optimize reading and writing .blend files for the simulation cache use-case because of a lot of existing code surrounding the file format that needs to keep working. We'd likely have to do a fair amount of changes to make .blend files suitable for simulation cache that does not waste lots of memory and fullfills our performance expectations, making the whole code even more complex. ### Common Interchange Formats Another possibility is to use e.g. Alembic or USD to store the simulation state. The main benefit here is that the individual frames can be opened in Blender and also in other software. While it is of course important to be able to export the simulated data in these formats, for exporting the "raw geometry data" they are not necessarily the right solution. With other file formats we always have to consider out to map Blender data to these and back in a hopefully lossless way. From what we've seen so far, these formats don't work well with compressing data over multiple frames. Also I heard that we ran into performance problems trying to read animations cached into alembic in real-time. Using a common interchange format would make it impossible for us to change the format to better suit our needs if that becomes necessary. ### Custom Format With a custom format with have the most flexibility. Also it's likely simpler to do than any of the other approaches, because we wouldn't have to figure out how to fit all the data we want to store into those formats. Shared data over multiple frames can just be part of the format description. The main concern with this approach is that we have to maintain a new file format in the long term, hence it should be kept as simple and straight forward as possible. Another downside is that we have to write new code to serialize geometry data like a mesh. Luckily, not all data in a `Mesh` data-block has to be stored, and all the important data generally is stored in plain arrays already. Volumes would like still be stored as openvdb files. There are of course many ways to design a custom format given the requirements. Currently, I have the following in mind: * The data is split into two parts: The "big data" (e.g. attribute arrays) and a description for how to assemble the that data into the simulation state (e.g. a `GeometrySet`). * Both parts are saved in separate files. * For every frame, a new (relatively small) file is written that contains all the information for what is in the simulation state of that frame. This file usually has references to data stored in other files. The file format of that has not been decided upon. It's likely easiest to use some human readable format like json. This data could also easily be kept in memory at all times because it's small. * The "big data" that is new in every frame (has not been stored previously) is stored in one or multiple "raw data" files. ---- One issue that none of these formats automatically solve is how to deal with referenced id data blocks like materials. However, a fairly straight forward and likely good-enough option can be to store just the library and material name on disk. ## Conclusion While there are still details to be figured out when actually implementing this, using a custom format seems to be the best approach, so we will focus on that from now on.

🎉 1 🚀 1

Jacques Lucke added the

Type

Design

label 2023-02-27 15:40:45 +01:00

Jacques Lucke added this to the Nodes & Physics project 2023-02-27 15:40:46 +01:00

Iliya Katushenock commented

2023-02-27 15:43:27 +01:00

Data blocks cannot be a simulation layer