Crash when baking to vertex colors while "Normals-> Auto Smooth" is enabled. #88756

New Issue

marc · 2021-06-02T21:22:52+02:00

marc commented

2021-06-02 21:22:52 +02:00

System Information
Operating system: Linux-5.8.0-53-generic-x86_64-with-glibc2.31 64 Bits
Graphics card: Mesa DRI Intel(R) HD Graphics 2000 (SNB GT1) Intel Open Source Technology Center 3.3 (Core Profile) Mesa 20.2.6

Blender Version
Broken: version: 2.93.0, branch: master, commit date: 2021-06-02 11:21, hash: 84da05a8b8
Worked: 2.92

Caused by 85be72c1cc

Short description of error
Selecting 'Bake' with Output->Target set to 'Vertex Colors' crashes blender when "Normals-> Auto Smooth" is enabled.

Exact steps for others to reproduce the error
From a new scene:
1 - Set render engine to Cycles
2 - Create new vertex colors on 'Object Data Properties' for the default cube.
3 - Enable Normals->Auto Smooth on the cube.
3 - Set Bake->Output->Target to 'Vertex Colors'
4 - With the cube selected, press bake.

From blender file:
1 - With the cube selected, press bake.
bug.blend

**System Information** Operating system: Linux-5.8.0-53-generic-x86_64-with-glibc2.31 64 Bits Graphics card: Mesa DRI Intel(R) HD Graphics 2000 (SNB GT1) Intel Open Source Technology Center 3.3 (Core Profile) Mesa 20.2.6 **Blender Version** Broken: version: 2.93.0, branch: master, commit date: 2021-06-02 11:21, hash: `84da05a8b8` Worked: 2.92 Caused by 85be72c1cc **Short description of error** Selecting 'Bake' with Output->Target set to 'Vertex Colors' crashes blender when "Normals-> Auto Smooth" is enabled. **Exact steps for others to reproduce the error** From a new scene: 1 - Set render engine to Cycles 2 - Create new vertex colors on 'Object Data Properties' for the default cube. 3 - Enable Normals->Auto Smooth on the cube. 3 - Set Bake->Output->Target to 'Vertex Colors' 4 - With the cube selected, press bake. From blender file: 1 - With the cube selected, press bake. [bug.blend](https://archive.blender.org/developer/F10154823/bug.blend)

marc commented

2021-06-02 21:22:52 +02:00

Added subscriber: @marcslashnum

blender-admin commented

2021-06-02 21:22:52 +02:00

#89842 was marked as duplicate of this issue

Philipp Oeser commented

2021-06-03 10:32:00 +02:00

Added subscriber: @lichtwerk

Philipp Oeser commented

2021-06-03 10:32:00 +02:00

Changed status from 'Needs Triage' to: 'Confirmed'

Philipp Oeser commented

2021-06-03 10:32:00 +02:00

Can confirm, will check

Philipp Oeser commented

2021-06-03 12:08:30 +02:00

Caused by 85be72c1cc

Caused by 85be72c1cc

Kévin Dietrich commented

2021-06-08 18:04:13 +02:00

Added subscriber: @kevindietrich

Kévin Dietrich commented

2021-06-08 18:04:13 +02:00

After some investigation it would appear that the Mesh is corrupted somehow. Edges and polygons have garbage data which is causing the crash due to out-of-bounds memory accesses.

林虎 commented

2021-06-21 10:02:05 +02:00

Added subscriber: @Werk

林虎 commented

2021-06-21 10:02:05 +02:00

i got the same problem,when i use "auto smooth"

Kévin Dietrich commented

2021-06-24 12:28:38 +02:00

Added subscriber: @brecht

Kévin Dietrich commented

2021-06-24 12:28:38 +02:00

When baking some data, we create a new Mesh with edits and modifiers applied (bake_mesh_new_from_object). This will evaluate the input mesh and return us a copy with the final data. Since there is no modifier here, the Mesh returned from mesh_calc_modifiers is actually referencing data layers from the original Mesh. When autosmooth is enabled, we also ensure that the Mesh is split (still in bake_mesh_new_from_object). However, since the new Mesh is referencing the original one, although BKE_mesh_split_faces is creating new vertices and edges, the reallocation of the vertex and edge data layers is preempted because of the reference, so adding the new vertices and edges overwrites valid data.

Prior to 85be72c1cc, we would apparently always receive a new Mesh which owns its data layers so the Mesh split went fine. It is the addition of the preserve_origindex parameter which forces us down the code path of modifier evaluation.

Note that this bug has no relation to vertex colors, the crash also happens if the bake target is set to images.

This patch solves the issue, but I don't know whether a better, more robust solution exists:

diff --git a/source/blender/editors/object/object_bake_api.c b/source/blender/editors/object/object_bake_api.c
index 7f26d44a4ed..bd7c355bd6a 100644
--- a/source/blender/editors/object/object_bake_api.c
+++ b/source/blender/editors/object/object_bake_api.c
@@ -671,8 +671,11 @@ static void bake_targets_clear(Main *bmain, const bool is_tangent)
 /* create new mesh with edit mode changes and modifiers applied */
 static Mesh *bake_mesh_new_from_object(Depsgraph *depsgraph,
                                        Object *object,
-                                       const bool preserve_origindex)
+                                       const bool should_preserve_origindex)
 {
+  /* Only ask to preserve origindex if we have modifiers, otherwise we will receive a referenced
+   * copy of the Mesh which will make BKE_mesh_split_faces fail to allocate new data layers. */
+  const bool preserve_origindex = should_preserve_origindex && object->modifiers.first != NULL;
   Mesh *me = BKE_mesh_new_from_object(depsgraph, object, false, preserve_origindex);
 
   if (me->flag & ME_AUTOSMOOTH) {

CC @brecht

When baking some data, we create a new Mesh with edits and modifiers applied (`bake_mesh_new_from_object`). This will evaluate the input mesh and return us a copy with the final data. Since there is no modifier here, the Mesh returned from `mesh_calc_modifiers` is actually referencing data layers from the original Mesh. When autosmooth is enabled, we also ensure that the Mesh is split (still in `bake_mesh_new_from_object`). However, since the new Mesh is referencing the original one, although `BKE_mesh_split_faces` is creating new vertices and edges, the reallocation of the vertex and edge data layers is preempted because of the reference, so adding the new vertices and edges overwrites valid data. Prior to 85be72c1cc, we would apparently always receive a new Mesh which owns its data layers so the Mesh split went fine. It is the addition of the `preserve_origindex` parameter which forces us down the code path of modifier evaluation. Note that this bug has no relation to vertex colors, the crash also happens if the bake target is set to images. This patch solves the issue, but I don't know whether a better, more robust solution exists: ``` diff --git a/source/blender/editors/object/object_bake_api.c b/source/blender/editors/object/object_bake_api.c index 7f26d44a4ed..bd7c355bd6a 100644 --- a/source/blender/editors/object/object_bake_api.c +++ b/source/blender/editors/object/object_bake_api.c @@ -671,8 +671,11 @@ static void bake_targets_clear(Main *bmain, const bool is_tangent) /* create new mesh with edit mode changes and modifiers applied */ static Mesh *bake_mesh_new_from_object(Depsgraph *depsgraph, Object *object, - const bool preserve_origindex) + const bool should_preserve_origindex) { + /* Only ask to preserve origindex if we have modifiers, otherwise we will receive a referenced + * copy of the Mesh which will make BKE_mesh_split_faces fail to allocate new data layers. */ + const bool preserve_origindex = should_preserve_origindex && object->modifiers.first != NULL; Mesh *me = BKE_mesh_new_from_object(depsgraph, object, false, preserve_origindex); if (me->flag & ME_AUTOSMOOTH) { ``` CC @brecht

Brecht Van Lommel commented

2021-06-24 18:10:29 +02:00

Relying on a object->modifiers.first != NULL test is not reliable, since modifiers may also be e.g. disabled, or maybe something else in depsgraph evaluation could change the mesh.

This is what sounds wrong to me:

although BKE_mesh_split_faces is creating new vertices and edges, the reallocation of the vertex and edge data layers is preempted because of the reference, so adding the new vertices and edges overwrites valid data.

Referenced data layers are supposed to use a copy-on-write mechanism, where you call CustomData_duplicate_referenced_layer before modifying the layer. Is it possible to modify BKE_mesh_split_faces to do that?

Relying on a `object->modifiers.first != NULL` test is not reliable, since modifiers may also be e.g. disabled, or maybe something else in depsgraph evaluation could change the mesh. This is what sounds wrong to me: > although BKE_mesh_split_faces is creating new vertices and edges, the reallocation of the vertex and edge data layers is preempted because of the reference, so adding the new vertices and edges overwrites valid data. Referenced data layers are supposed to use a copy-on-write mechanism, where you call `CustomData_duplicate_referenced_layer` before modifying the layer. Is it possible to modify `BKE_mesh_split_faces` to do that?

blender-admin commented

2021-06-25 12:56:01 +02:00

This issue was referenced by d070cce778

This issue was referenced by d070cce77881c0f2c1bd7aa05fea4281a1cafce1

blender-admin commented

2021-06-25 12:56:01 +02:00

This issue was referenced by 5f9677fe0c

This issue was referenced by 5f9677fe0c533b008b815d7fee0b56509a414ab7

Kévin Dietrich commented

2021-06-25 14:17:41 +02:00

Changed status from 'Confirmed' to: 'Resolved'

Kévin Dietrich closed this issue