Custom Data: support implicit sharing for custom data layers #106228

This integrates the new implicit-sharing system (from fbcddfcd68) with CustomData. Now the potentially long arrays referenced by custom data layers can be shared between different systems but most importantly between different geometries. This makes e.g. copying a mesh much cheaper because none of the attributes has to be copied. Only when an attribute is modified does it have to be copied.

Also see the original design task: #95845.

This reduces memory and improves performance by avoiding unnecessary data copies. For example, the used memory after loading a highly subdivided mesh is reduced from 2.4GB to 1.79GB. This is about 25% less which is the expected amount because in main there are 4 copies of the data:

The original data which is allocated when the file is loaded.
The copy for the depsgraph allocated during depsgraph evaluation.
The copy for the undo system allocated when the first undo step is created right after loading the file.
GPU buffers allocated for drawing.

This patch only gets rid of copy number 2 for the depsgraph. In theory the other copies can be removed as part of follow up PRs as well though.

The patch has three main components:

Slightly modified CustomData API to make it work better with implicit sharing:
- CD_REFERENCE and CD_DUPLICATE have been removed because they are meaningless when implicit-sharing is used.
- CD_ASSIGN has been removed as well because it's not an allocation type anyway. The functionality of using existing arrays as custom data layers has not been removed though.
- This can still be done with CustomData_add_layer_with_data which also has a new argument that allows passing in information about whether the array is shared.
- CD_FLAG_NOFREE has been removed because it's no longer necessary. It only existed because of CD_REFERENCE.
- CustomData_copy and CustomData_merge have been split up into a functions that do copy the actual attribute values and those that do not. The latter functions now have the _layout suffix (e.g. CustomData_copy_layout).
Changes in customdata.cc to make it actually use implicit-sharing.
Changes in various other files to adapt to the changes in BKE_customdata.h.

This integrates the new implicit-sharing system (from fbcddfcd68adc72f4dde6c1b769659ba8e9f086b) with `CustomData`. Now the potentially long arrays referenced by custom data layers can be shared between different systems but most importantly between different geometries. This makes e.g. copying a mesh much cheaper because none of the attributes has to be copied. Only when an attribute is modified does it have to be copied. Also see the original design task: #95845. This reduces memory and improves performance by avoiding unnecessary data copies. For example, the used memory after loading a highly subdivided mesh is reduced from 2.4GB to 1.79GB. This is about 25% less which is the expected amount because in `main` there are 4 copies of the data: 1. The original data which is allocated when the file is loaded. 2. The copy for the depsgraph allocated during depsgraph evaluation. 3. The copy for the undo system allocated when the first undo step is created right after loading the file. 4. GPU buffers allocated for drawing. This patch only gets rid of copy number 2 for the depsgraph. In theory the other copies can be removed as part of follow up PRs as well though. ----- The patch has three main components: * Slightly modified `CustomData` API to make it work better with implicit sharing: * `CD_REFERENCE` and `CD_DUPLICATE` have been removed because they are meaningless when implicit-sharing is used. * `CD_ASSIGN` has been removed as well because it's not an allocation type anyway. The functionality of using existing arrays as custom data layers has not been removed though. * This can still be done with `CustomData_add_layer_with_data` which also has a new argument that allows passing in information about whether the array is shared. * `CD_FLAG_NOFREE` has been removed because it's no longer necessary. It only existed because of `CD_REFERENCE`. * `CustomData_copy` and `CustomData_merge` have been split up into a functions that do copy the actual attribute values and those that do not. The latter functions now have the `_layout` suffix (e.g. `CustomData_copy_layout`). * Changes in `customdata.cc` to make it actually use implicit-sharing. * Changes in various other files to adapt to the changes in `BKE_customdata.h`.

@blender-bot build

A very nice improvement in the custom data API and more obviously in memory usage! This all looks nice to me. I just added a few inline comments.

I think this actually will improve memory usage in sculpt undo system when full copies are made, since SCULPT_UNDO_GEOMETRY is implemented basically just with CustomData_copy. It's nice when just changing the API slightly will give that improvement.

To get used to this I'll probably try to implement it implicit sharing for mesh and curve offsets next, if that makes sense to you.

A very nice improvement in the custom data API and more obviously in memory usage! This all looks nice to me. I just added a few inline comments. I think this actually will improve memory usage in sculpt undo system when full copies are made, since `SCULPT_UNDO_GEOMETRY` is implemented basically just with `CustomData_copy`. It's nice when just changing the API slightly will give that improvement. To get used to this I'll probably try to implement it implicit sharing for mesh and curve offsets next, if that makes sense to you.

@blender-bot build

@blender-bot build windows

@blender-bot build

Great memory optimization! I also like how the CustomData functions are documented and have more clear naming :)

Took me a bit to understand what the BLI_implicit_sharing.h is about. Maybe add a comment that it is for forward declaration a class defined in the BLI_implicit_sharing.hh but which is available for both C and C++? Just a thought.

From the code side seems good. Did you test it with some complex files like from some of the open movies (like, the current one)? Do you want us to do some testing here at the studio?

Great memory optimization! I also like how the CustomData functions are documented and have more clear naming :) Took me a bit to understand what the `BLI_implicit_sharing.h` is about. Maybe add a comment that it is for forward declaration a class defined in the `BLI_implicit_sharing.hh` but which is available for both C and C++? Just a thought. From the code side seems good. Did you test it with some complex files like from some of the open movies (like, the current one)? Do you want us to do some testing here at the studio?

I did open a few Heist files, changed some small things, and hit undo. That worked as expected. I don't yet have access to the current open movie. Any additional testing at the studio would be appreciated!

@blender-bot package

Package build started. Download here when ready.

Package build started. [Download here](https://builder.blender.org/download/patch/PR106228) when ready.

Gave it a test with the production file I've been looking into recently. So far so good!

I'm mainly waiting for final approval by @brecht now.

 @ -163,1 +156,3 @@
  * if a layer should be copied or not. alloctype must be one of the above.
  * Initializes a CustomData object with the same layer setup as source. `mask` is a bit-field where
  * `(mask & (1 << (layer type)))` indicates if a layer should be copied or not. The data layers
  * will be shared or copied depending on whether the layer uses COW.

 @ -233,2 +242,2 @@
  * backed by an external data array. the different allocation types are
  * defined above. returns the data of the layer.
  * Adds a layer of the given type to the #CustomData object. The new layer is initialized based on
  * the given alloctype. \return The layer data.

 @ -2314,0 +2376,4 @@
 };
 /** Create a #ImplicitSharingInfo that takes ownership of the data. */
 static ImplicitSharingInfo *make_cow_for_array(const eCustomDataType type,

 @ -2732,3 +2840,3 @@
 }
 static bool customData_resize(CustomData *data, const int amount)
 static void customData_resize(CustomData *data, const int grow_amount)

 @ -626,3 +626,3 @@
   CustomData_free(&pointcloud->pdata, pointcloud->totpoint);
   pointcloud->totpoint = me->totvert;
   CustomData_merge(&me->vdata, &pointcloud->pdata, CD_MASK_PROP_ALL, CD_DUPLICATE, me->totvert);

 @ -274,2 +275,4 @@
       if (layer->data) {
         if (layer->sharing_info) {
           /* This assumes that the layer is not shared, which it is not here because it has just

 @ -240,0 +250,4 @@
 /**
  * Adds a layer of the given type to the #CustomData object. The new layer takes ownership of the
  * passed in `layer_data`. If a #ImplicitSharingInfoHandle is passed in, it's user count is

Download

What's New

Blender Studio

Manual

Developers Blog

Documentation

Benchmark

Blender Conference

Development Fund

One-time Donations

Custom Data: support implicit sharing for custom data layers #106228

Reviewers