WIP : Use an extra step of indirection in accessing CustomDataLayers via rna. #108319

Open
Martijn Versteegh wants to merge 3 commits from Baardaap/blender:fix_dangling_customdata_pointers into main

When changing the target branch, be careful to rebase the branch in your fork to match. See documentation.

3 Commits

Author SHA1 Message Date
Martijn Versteegh 2ad8de628b Mark indirection table entries as unused on layer remove. 2023-05-26 16:45:34 +02:00
Martijn Versteegh 3b94c39085 Remove extraneous initialisation of indirection table. 2023-05-26 16:44:55 +02:00
Martijn Versteegh b7e2b37d25 WIP : Use an extra step of indirection in accessing CustomDataLayers via rna.
The RNA CustomDataLayer api stores pointers to CustomDataLayer structs in the
PointerRNA, but when CustomData layers are added/removed those get reordered.
This leads to pythons MeshUVLoopLayer referencing the wrong layer. Potentially
even pointing into freed memory.

see #107500

This patch is a 'proof of concept' and is unfinished in a few ways:
- Only MeshUVLoopLayers are handled, other CustomDataLayer accesses
(attributes,  vertex_create, vertex_color etc etc) are still broken, possibly
*more* broken.
- The indirection table used is currently hardcoded to 64 entries. This will
crash & burn whenever there are more than 64 simultaneous customdata layers.

Possible ways to move forward:
- Don't use an indirection table, use a unique id on every CustomDataLayer
and do a search for every access. This is slower, but the speeds maybe not
too important for the layer lookups, as long as the element access still is
fast.
- Store the indirection table in a (to be created) C++ array-like type without
 reallocation.
- Ignore this problem for 3.6 and for 4.0 redesign attribute/customdata storage
in a way that the layer(-header)s don't move around.
2023-05-26 16:23:48 +02:00