Mesh: Replace MLoop struct with generic attributes #104424

Merged
Hans Goudey merged 261 commits from refactor-mesh-corners-generic into main 2023-03-20 15:55:25 +01:00
Member

Implements #102359.

Split the MLoop struct into two separate integer arrays called
corner_verts and corner_edges, referring to the vertex each corner
is attached to and the next edge around the face at each corner. These
arrays can be sliced to give access to the edges or vertices in a face.
Then they are often referred to as "poly_verts" or "poly_edges".

The main benefits are halving the necessary memory bandwidth when only
one array is used and simplifications from using regular integer indices
instead of a special-purpose struct.

The commit also starts a renaming from "loop" to "corner" in mesh code.

Like the other mesh struct of array refactors, forward compatibility is
kept by writing files with the older format. This will be done until 4.0
to ease the transition process.

Looking at a small portion of the patch should give a good impression
for the rest of the changes. I tried to make the changes as small as
possible so it's easy to tell the correctness from the diff. Though I
found Blender developers have been very inventive over the last decade
when finding different ways to loop over the corners in a face.

For performance, nearly every piece of code that deals with Mesh is
slightly impacted. Any algorithm that is memory bottle-necked should
see an improvement. For example, here is a comparison of interpolating
a vertex float attribute to face corners (Ryzen 3700x):

Before (Average: 3.7 ms, Min: 3.4 ms)

threading::parallel_for(loops.index_range(), 4096, [&](IndexRange range) {
  for (const int64_t i : range) {
    dst[i] = src[loops[i].v];
  }
});

After (Average: 2.9 ms, Min: 2.6 ms)

array_utils::gather(src, corner_verts, dst);

That's an improvement of 28% to the average timings, and it's also a
simplification, since an index-based routine can be used instead.
For more examples using the new arrays, see the design task.

Implements #102359. Split the `MLoop` struct into two separate integer arrays called `corner_verts` and `corner_edges`, referring to the vertex each corner is attached to and the next edge around the face at each corner. These arrays can be sliced to give access to the edges or vertices in a face. Then they are often referred to as "poly_verts" or "poly_edges". The main benefits are halving the necessary memory bandwidth when only one array is used and simplifications from using regular integer indices instead of a special-purpose struct. The commit also starts a renaming from "loop" to "corner" in mesh code. Like the other mesh struct of array refactors, forward compatibility is kept by writing files with the older format. This will be done until 4.0 to ease the transition process. Looking at a small portion of the patch should give a good impression for the rest of the changes. I tried to make the changes as small as possible so it's easy to tell the correctness from the diff. Though I found Blender developers have been very inventive over the last decade when finding different ways to loop over the corners in a face. For performance, nearly every piece of code that deals with `Mesh` is slightly impacted. Any algorithm that is memory bottle-necked should see an improvement. For example, here is a comparison of interpolating a vertex float attribute to face corners (Ryzen 3700x): **Before** (Average: 3.7 ms, Min: 3.4 ms) ``` threading::parallel_for(loops.index_range(), 4096, [&](IndexRange range) { for (const int64_t i : range) { dst[i] = src[loops[i].v]; } }); ``` **After** (Average: 2.9 ms, Min: 2.6 ms) ``` array_utils::gather(src, corner_verts, dst); ``` That's an improvement of 28% to the average timings, and it's also a simplification, since an index-based routine can be used instead. For more examples using the new arrays, see the design task.
Hans Goudey added 207 commits 2023-02-07 19:39:52 +01:00
9726bb4e51 Initial progress splitting MLoop in two arrays
About halfway through removing all uses of `MLoop`. See T102359
d300fe01ac Cleanup: Make naming more consistent
Avoid _i prefix which doesn't really help
Hans Goudey added 1 commit 2023-02-07 22:04:43 +01:00
Hans Goudey added 2 commits 2023-02-09 23:08:12 +01:00
Hans Goudey added 2 commits 2023-02-10 20:08:02 +01:00
Hans Goudey added 1 commit 2023-02-13 04:21:52 +01:00
Brecht Van Lommel added this to the Modeling project 2023-02-13 09:20:25 +01:00
Hans Goudey added 1 commit 2023-02-14 05:26:40 +01:00
Hans Goudey added 1 commit 2023-02-14 16:34:37 +01:00
Hans Goudey added 1 commit 2023-02-16 03:57:05 +01:00
Hans Goudey added 2 commits 2023-02-16 23:51:38 +01:00
Hans Goudey added 1 commit 2023-02-18 16:55:19 +01:00
Hans Goudey added this to the 3.6 LTS milestone 2023-02-21 15:27:10 +01:00
Hans Goudey added 1 commit 2023-02-22 04:43:37 +01:00
Hans Goudey added 1 commit 2023-02-23 19:16:55 +01:00
Hans Goudey added 1 commit 2023-02-23 19:26:23 +01:00
Hans Goudey added 1 commit 2023-02-27 21:40:16 +01:00
Hans Goudey added 1 commit 2023-02-27 22:12:33 +01:00
Hans Goudey added 1 commit 2023-02-28 00:36:26 +01:00
Hans Goudey added 1 commit 2023-02-28 01:46:27 +01:00
Hans Goudey added 1 commit 2023-02-28 19:34:21 +01:00
Hans Goudey added 1 commit 2023-02-28 23:56:55 +01:00
Hans Goudey added 1 commit 2023-03-01 14:53:51 +01:00
Hans Goudey added 1 commit 2023-03-03 14:36:18 +01:00
Hans Goudey added 1 commit 2023-03-03 18:49:15 +01:00
Hans Goudey added 1 commit 2023-03-03 23:22:12 +01:00
Hans Goudey added 1 commit 2023-03-06 04:26:28 +01:00
Hans Goudey added 1 commit 2023-03-06 04:31:14 +01:00
Hans Goudey added 2 commits 2023-03-06 18:43:34 +01:00
Hans Goudey requested review from Jacques Lucke 2023-03-06 18:44:03 +01:00
Hans Goudey requested review from Campbell Barton 2023-03-06 18:44:26 +01:00
Hans Goudey added 1 commit 2023-03-06 21:13:24 +01:00
Hans Goudey added 1 commit 2023-03-06 21:14:25 +01:00
buildbot/vexp-code-patch-coordinator Build done. Details
b7071bdaa5
Merge branch 'main' into refactor-mesh-corners-generic
Author
Member

@blender-bot build

@blender-bot build
Hans Goudey added 2 commits 2023-03-08 14:33:48 +01:00
Hans Goudey added 1 commit 2023-03-08 15:55:18 +01:00
Hans Goudey added 1 commit 2023-03-08 17:14:45 +01:00
buildbot/vexp-code-patch-coordinator Build done. Details
9959b431dc
Small cleanups and variable renames
Member

@blender-bot build

@blender-bot build
Hans Goudey added 1 commit 2023-03-09 17:15:50 +01:00
Hans Goudey added 1 commit 2023-03-09 21:59:37 +01:00
Hans Goudey added 1 commit 2023-03-09 22:41:36 +01:00
Hans Goudey added 1 commit 2023-03-10 14:06:49 +01:00
Hans Goudey added 1 commit 2023-03-13 00:05:51 +01:00
Jacques Lucke approved these changes 2023-03-13 11:30:18 +01:00
Jacques Lucke left a comment
Member

Besides the things mentioned, this looks good to me.
Make sure to run geometry nodes tests in a debug build after merging master.

Besides the things mentioned, this looks good to me. Make sure to run geometry nodes tests in a debug build after merging master.
@ -1751,3 +1770,2 @@
if (lnors_spacearr.lspacearr[i]->flags & MLNOR_SPACE_IS_SINGLE) {
BLI_assert(POINTER_AS_INT(loop_link) == i);
const int nidx = use_vertices ? int(loops[i].v) : i;
BLI_assert(POINTER_AS_INT(loops) == i);
Member

This does not build in a debug build.

This does not build in a debug build.
HooglyBoogly marked this conversation as resolved
@ -246,2 +242,2 @@
/** Write access to loop data. */
blender::MutableSpan<MLoop> loops_for_write();
blender::Span<int> corner_verts() const;
Member

Some comment here and below would be good.

Some comment here and below would be good.
HooglyBoogly marked this conversation as resolved
Hans Goudey added 3 commits 2023-03-13 17:38:06 +01:00
Hans Goudey added 1 commit 2023-03-14 17:20:13 +01:00
Campbell Barton requested changes 2023-03-17 10:56:22 +01:00
Campbell Barton left a comment
Owner

Generally seems fine - minor issues noted inline.

Testing I ran into two issues though:

  • Mesh::mloop pointer is left set after writing a file, shouldn't this be cleared after writing, so the dangling pointer isn't left?

  • Saving and loading the default startup file is crashing too, attached ASAN log.

Generally seems fine - minor issues noted inline. Testing I ran into two issues though: - `Mesh::mloop` pointer is left set after writing a file, shouldn't this be cleared after writing, so the dangling pointer isn't left? - Saving and loading the default startup file is crashing too, attached ASAN log.
7.5 KiB
@ -1665,3 +1683,3 @@
LinkNode *loop_link = lnors_spacearr.lspacearr[i]->loops;
const MLoop *prev_ml = nullptr;
int corner_prev = -1;

*picky*: reverse prefix/suffix, inconsistent with existing naming.

\*picky\*: reverse prefix/suffix, inconsistent with existing naming.
HooglyBoogly marked this conversation as resolved
@ -65,3 +65,3 @@
{
const Span<float3> input_positions = input_mesh->vert_positions();
const Span<MLoop> input_loops = input_mesh->loops();
const Span<int> src_corner_verts = input_mesh->corner_verts();

*picky* renames input_ to src_, a prefix which remains used elsewhere. Use input_ here too?

\*picky\* renames `input_` to `src_`, a prefix which remains used elsewhere. Use `input_` here too?
HooglyBoogly marked this conversation as resolved
@ -641,1 +641,3 @@
v2 = mloops[sp->loopstart + (j + 1) % poly.totloop].v;
for (j = 0; j < poly.totloop; j++) {
const int corner = sp->loopstart + j;
const int vert = corner_verts[corner];

*picky*: odd not to call this vert_i ? (used elsewhere).

\*picky\*: odd not to call this `vert_i` ? (used elsewhere).
@ -3740,4 +3741,3 @@
pbvh_face_iter_verts_reserve(fd, poly.totloop);
const MLoop *ml = fd->mloop_ + poly.loopstart;
const int grid_area = fd->subdiv_key_.grid_area;

*minor*: assignment to int *poly_verts = &fd->corner_verts_[poly.loopstart]; can be kept here.

Then fd->verts[i].i = poly_verts[i]; later on.

\*minor\*: assignment to `int *poly_verts = &fd->corner_verts_[poly.loopstart];` can be kept here. Then `fd->verts[i].i = poly_verts[i];` later on.
HooglyBoogly marked this conversation as resolved
Hans Goudey reviewed 2023-03-17 21:02:54 +01:00
@ -641,1 +641,3 @@
v2 = mloops[sp->loopstart + (j + 1) % poly.totloop].v;
for (j = 0; j < poly.totloop; j++) {
const int corner = sp->loopstart + j;
const int vert = corner_verts[corner];
Author
Member

I'm trying to use the vert name consistently actually (I just didn't want to change some existing variable names). The idea is that the index is all the vertex is really-- a marker in an array used to access various attributes. So it makes sense to use the simplest name for that. It's also shorter, easier to say and read, etc.

I'm trying to use the `vert` name consistently actually (I just didn't want to change some existing variable names). The idea is that the index is all the vertex is really-- a marker in an array used to access various attributes. So it makes sense to use the simplest name for that. It's also shorter, easier to say and read, etc.
Author
Member

Generally seems fine - minor issues noted inline.

Testing I ran into two issues though:

  • Mesh::mloop pointer is left set after writing a file, shouldn't this be cleared after writing, so the dangling pointer isn't left?

The blend_write calls are run on local copies of the ID data-block, so any changes to the mesh pointer there have no effect elsewhere. That's why we can do things like mesh->runtime = nullptr; in that function.

  • Saving and loading the default startup file is crashing too, attached ASAN log.

Hmm, I'm not able to reproduce this here. Is loading and saving the default startup file and loading it again enough to trigger the problem for you?

> Generally seems fine - minor issues noted inline. > > Testing I ran into two issues though: > > - `Mesh::mloop` pointer is left set after writing a file, shouldn't this be cleared after writing, so the dangling pointer isn't left? The `blend_write` calls are run on local copies of the ID data-block, so any changes to the `mesh` pointer there have no effect elsewhere. That's why we can do things like `mesh->runtime = nullptr;` in that function. > - Saving and loading the default startup file is crashing too, attached ASAN log. Hmm, I'm not able to reproduce this here. Is loading and saving the default startup file and loading it again enough to trigger the problem for you?
Hans Goudey added 4 commits 2023-03-17 22:03:44 +01:00
Hans Goudey requested review from Campbell Barton 2023-03-17 22:09:30 +01:00
Author
Member

@blender-bot build

@blender-bot build

Generally seems fine - minor issues noted inline.

Testing I ran into two issues though:

  • Mesh::mloop pointer is left set after writing a file, shouldn't this be cleared after writing, so the dangling pointer isn't left?

The blend_write calls are run on local copies of the ID data-block, so any changes to the mesh pointer there have no effect elsewhere. That's why we can do things like mesh->runtime = nullptr; in that function.

Of course, no issue with this then.

  • Saving and loading the default startup file is crashing too, attached ASAN log.

Hmm, I'm not able to reproduce this here. Is loading and saving the default startup file and loading it again enough to trigger the problem for you?

It is: blender --factory-startup save, load, save (crash).
Edit it's not crashing with ASAN every time, but often enough, at least 1 in 5 times.

Looking into this - it's caused by BKE_mesh_legacy_convert_uvs_to_struct calling CustomData_get_layer_named(...) with an mesh->ldata.totlayer that has already been set to a fake value by CustomData_blend_write_prepare (which may not match mesh->ldata.layers allocation, causing out of bounds access).

It looks like this bug is exposed by this patch (as the UV conversion is running with a potentially invalid totlayer in the main branch too).

Attached a proof of concept fix, although it would be simpler if all custom-data handling can be done before CustomData_blend_write_prepare(..) as doing so seems quite error prone.

> > Generally seems fine - minor issues noted inline. > > > > Testing I ran into two issues though: > > > > - `Mesh::mloop` pointer is left set after writing a file, shouldn't this be cleared after writing, so the dangling pointer isn't left? > > The `blend_write` calls are run on local copies of the ID data-block, so any changes to the `mesh` pointer there have no effect elsewhere. That's why we can do things like `mesh->runtime = nullptr;` in that function. Of course, no issue with this then. > > - Saving and loading the default startup file is crashing too, attached ASAN log. > > Hmm, I'm not able to reproduce this here. Is loading and saving the default startup file and loading it again enough to trigger the problem for you? It is: `blender --factory-startup` save, load, save (crash). *Edit* it's not crashing with ASAN every time, but often enough, at least 1 in 5 times. Looking into this - it's caused by `BKE_mesh_legacy_convert_uvs_to_struct` calling `CustomData_get_layer_named(...)` with an `mesh->ldata.totlayer` that has already been set to a *fake* value by `CustomData_blend_write_prepare` (which may not match `mesh->ldata.layers` allocation, causing out of bounds access). It looks like this bug is exposed by this patch (as the UV conversion is running with a potentially invalid totlayer in the `main` branch too). Attached a proof of concept fix, although it would be simpler if all custom-data handling can be done before `CustomData_blend_write_prepare(..)` as doing so seems quite error prone.
Campbell Barton requested changes 2023-03-20 01:04:00 +01:00
Campbell Barton left a comment
Owner

Issue with layer-writing noted in reply.

Issue with layer-writing noted in reply.
Author
Member

Looking into this - it's caused by BKE_mesh_legacy_convert_uvs_to_struct calling CustomData_get_layer_named(...) with an mesh->ldata.totlayer that has already been set to a fake value by CustomData_blend_write_prepare (which may not match mesh->ldata.layers allocation, causing out of bounds access).

It looks like this bug is exposed by this patch (as the UV conversion is running with a potentially invalid totlayer in the main branch too).

Thanks, that makes sense! Should be fixed in the release branch and main with de49d18af5.

> Looking into this - it's caused by `BKE_mesh_legacy_convert_uvs_to_struct` calling `CustomData_get_layer_named(...)` with an `mesh->ldata.totlayer` that has already been set to a *fake* value by `CustomData_blend_write_prepare` (which may not match `mesh->ldata.layers` allocation, causing out of bounds access). > >It looks like this bug is exposed by this patch (as the UV conversion is running with a potentially invalid totlayer in the `main` branch too). Thanks, that makes sense! Should be fixed in the release branch and `main` with de49d18af52589cea5cc66d03aa57b0d76ecd357.
Hans Goudey requested review from Campbell Barton 2023-03-20 04:27:17 +01:00
Hans Goudey added 4 commits 2023-03-20 04:54:22 +01:00
Campbell Barton approved these changes 2023-03-20 10:40:56 +01:00
Member

@blender-bot build

@blender-bot build
Hans Goudey added 2 commits 2023-03-20 13:45:01 +01:00
Author
Member

@blender-bot build

@blender-bot build
Hans Goudey merged commit 16fbadde36 into main 2023-03-20 15:55:25 +01:00
Hans Goudey deleted branch refactor-mesh-corners-generic 2023-03-20 15:55:26 +01:00
Sign in to join this conversation.
No Label
Interest
Alembic
Interest
Animation & Rigging
Interest
Asset Browser
Interest
Asset Browser Project Overview
Interest
Audio
Interest
Automated Testing
Interest
Blender Asset Bundle
Interest
BlendFile
Interest
Collada
Interest
Compatibility
Interest
Compositing
Interest
Core
Interest
Cycles
Interest
Dependency Graph
Interest
Development Management
Interest
EEVEE
Interest
EEVEE & Viewport
Interest
Freestyle
Interest
Geometry Nodes
Interest
Grease Pencil
Interest
ID Management
Interest
Images & Movies
Interest
Import Export
Interest
Line Art
Interest
Masking
Interest
Metal
Interest
Modeling
Interest
Modifiers
Interest
Motion Tracking
Interest
Nodes & Physics
Interest
OpenGL
Interest
Overlay
Interest
Overrides
Interest
Performance
Interest
Physics
Interest
Pipeline, Assets & IO
Interest
Platforms, Builds & Tests
Interest
Python API
Interest
Render & Cycles
Interest
Render Pipeline
Interest
Sculpt, Paint & Texture
Interest
Text Editor
Interest
Translations
Interest
Triaging
Interest
Undo
Interest
USD
Interest
User Interface
Interest
UV Editing
Interest
VFX & Video
Interest
Video Sequencer
Interest
Virtual Reality
Interest
Vulkan
Interest
Wayland
Interest
Workbench
Interest: X11
Legacy
Blender 2.8 Project
Legacy
Milestone 1: Basic, Local Asset Browser
Legacy
OpenGL Error
Meta
Good First Issue
Meta
Papercut
Meta
Retrospective
Meta
Security
Module
Animation & Rigging
Module
Core
Module
Development Management
Module
EEVEE & Viewport
Module
Grease Pencil
Module
Modeling
Module
Nodes & Physics
Module
Pipeline, Assets & IO
Module
Platforms, Builds & Tests
Module
Python API
Module
Render & Cycles
Module
Sculpt, Paint & Texture
Module
Triaging
Module
User Interface
Module
VFX & Video
Platform
FreeBSD
Platform
Linux
Platform
macOS
Platform
Windows
Priority
High
Priority
Low
Priority
Normal
Priority
Unbreak Now!
Status
Archived
Status
Confirmed
Status
Duplicate
Status
Needs Info from Developers
Status
Needs Information from User
Status
Needs Triage
Status
Resolved
Type
Bug
Type
Design
Type
Known Issue
Type
Patch
Type
Report
Type
To Do
No Milestone
No project
No Assignees
3 Participants
Notifications
Due Date
The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: blender/blender#104424
No description provided.