Fix #117824: Sculpt color filter crash #118885

Merged
Raul Fernandez Hernandez merged 11 commits from farsthary/blender:Sculpt-Color-Filter-crash into blender-v4.1-release 2024-03-04 16:32:03 +01:00

Fix for Sculpt: Color Filter crash #117824

This change fixes the crash due to invalid active color layer
in the Undo/Redo stack when redoing from the original state.

Previously when performing a redo to a filter color state from the
original state, it cased a crash because
&((SculptUndoStep *)us_iter)->active_color_start did not contain valid
color data in the original state.
In this PR we check if is the original state and use the
&((SculptUndoStep *)us_iter)->active_color_end instead which is
de correct data to use.

  • Fix color update when redo from initial state
Fix for Sculpt: Color Filter crash #117824 This change fixes the crash due to invalid active color layer in the Undo/Redo stack when redoing from the original state. Previously when performing a redo to a filter color state from the original state, it cased a crash because &((SculptUndoStep *)us_iter)->active_color_start did not contain valid color data in the original state. In this PR we check if is the original state and use the &((SculptUndoStep *)us_iter)->active_color_end instead which is de correct data to use. - [x] Fix color update when redo from initial state
Raul Fernandez Hernandez added 4 commits 2024-02-29 05:10:32 +01:00
Hans Goudey changed title from WIP: Fix for Sculpt: Color Filter crash #117824 to WIP: Fix #117824: Sculpt color filter crash 2024-02-29 14:38:56 +01:00
Hans Goudey reviewed 2024-02-29 14:40:03 +01:00
Hans Goudey left a comment
Member

I find this area a bit confusing, maybe the PR description could give a brief description of how the logic is fixed?

I find this area a bit confusing, maybe the PR description could give a brief description of how the logic is fixed?
@ -30,3 +30,3 @@
enum class AttrDomain : int8_t {
/* Use for to choose automatically based on other data. */
/* Used to choose automatically based on other data. */
Member

This can be committed separately, it's not related to this PR

This can be committed separately, it's not related to this PR
Author
Member

Addressed in #118913

Addressed in https://projects.blender.org/blender/blender/pulls/118913
farsthary marked this conversation as resolved
Raul Fernandez Hernandez added 1 commit 2024-02-29 17:27:25 +01:00
Raul Fernandez Hernandez added 1 commit 2024-03-01 00:21:59 +01:00
Raul Fernandez Hernandez changed title from WIP: Fix #117824: Sculpt color filter crash to Fix #117824: Sculpt color filter crash 2024-03-01 00:29:13 +01:00
Raul Fernandez Hernandez requested review from Hans Goudey 2024-03-01 00:30:58 +01:00
Raul Fernandez Hernandez requested review from Sergey Sharybin 2024-03-01 00:31:22 +01:00
Author
Member

Fixed now

Fixed now
Sergey Sharybin requested changes 2024-03-01 12:07:19 +01:00
Dismissed
@ -1053,0 +1053,4 @@
if (changed_color) {
/* Force a redraw to fix color update
when redo from initial state */
SCULPT_pbvh_clear(ob);

Changes in color should not imply re-creation of an entire PBVH. I think the root problems comes from the fact that some of drawing code used evaluated state of mesh for decision making, while some data comes from the original mesh.

While it would be nice to make such things consistent and avoid unnecessary copies/tags it would be a much bigger task. For the purposes of a crash fix I think it would suffice to do DEG_id_tag_update(&ob->id, ID_RECALC_GEOMETRY); in set_active_layer after mesh->attributes_for_write().add. Doing so will keep tagging consistent with what it happens in sculpt mode when color layer is automatically added.

Changes in color should not imply re-creation of an entire PBVH. I think the root problems comes from the fact that some of drawing code used evaluated state of mesh for decision making, while some data comes from the original mesh. While it would be nice to make such things consistent and avoid unnecessary copies/tags it would be a much bigger task. For the purposes of a crash fix I think it would suffice to do `DEG_id_tag_update(&ob->id, ID_RECALC_GEOMETRY);` in `set_active_layer` after `mesh->attributes_for_write().add`. Doing so will keep tagging consistent with what it happens in sculpt mode when color layer is automatically added.
Author
Member

Thanks for the deep explanation! this fixed the color update too.

Thanks for the deep explanation! this fixed the color update too.
farsthary marked this conversation as resolved
@ -1848,3 +1854,3 @@
}
while (us_iter && (us_iter->step.is_applied == false)) {
set_active_layer(C, &((SculptUndoStep *)us_iter)->active_color_start);
if (us_iter != us) {

This seems a bit strange to have such condition here. Intuitively the layer activation and data modification should be symmetrical between undo and redo. Wouldn't it be more correct to do

    set_active_layer(C, &((SculptUndoStep *)us_iter)->active_color_end);
    sculpt_undosys_step_decode_redo_impl(C, depsgraph, us_iter);

    if (us_iter == us) {
      set_active_layer(C, &((SculptUndoStep *)us_iter)->active_color_start);
      break;
    }

``` ?
This seems a bit strange to have such condition here. Intuitively the layer activation and data modification should be symmetrical between undo and redo. Wouldn't it be more correct to do ``` set_active_layer(C, &((SculptUndoStep *)us_iter)->active_color_end); sculpt_undosys_step_decode_redo_impl(C, depsgraph, us_iter); if (us_iter == us) { set_active_layer(C, &((SculptUndoStep *)us_iter)->active_color_start); break; } ``` ?
Author
Member

Thanks! that worked and is simpler. Submitting in next commit

Thanks! that worked and is simpler. Submitting in next commit
farsthary marked this conversation as resolved
Raul Fernandez Hernandez added 1 commit 2024-03-01 15:51:57 +01:00
Raul Fernandez Hernandez added 1 commit 2024-03-01 16:03:42 +01:00
Raul Fernandez Hernandez added 1 commit 2024-03-01 17:43:43 +01:00
Raul Fernandez Hernandez added 1 commit 2024-03-01 17:44:10 +01:00
Raul Fernandez Hernandez requested review from Sergey Sharybin 2024-03-01 17:47:38 +01:00
Raul Fernandez Hernandez added the
Module
Sculpt, Paint & Texture
label 2024-03-01 22:24:44 +01:00
Sergey Sharybin approved these changes 2024-03-04 14:56:35 +01:00
Raul Fernandez Hernandez merged commit 3c6a9cb5a5 into blender-v4.1-release 2024-03-04 16:32:03 +01:00
Raul Fernandez Hernandez deleted branch Sculpt-Color-Filter-crash 2024-03-04 16:32:09 +01:00
Sign in to join this conversation.
No Label
Interest
Alembic
Interest
Animation & Rigging
Interest
Asset System
Interest
Audio
Interest
Automated Testing
Interest
Blender Asset Bundle
Interest
BlendFile
Interest
Collada
Interest
Compatibility
Interest
Compositing
Interest
Core
Interest
Cycles
Interest
Dependency Graph
Interest
Development Management
Interest
EEVEE
Interest
EEVEE & Viewport
Interest
Freestyle
Interest
Geometry Nodes
Interest
Grease Pencil
Interest
ID Management
Interest
Images & Movies
Interest
Import Export
Interest
Line Art
Interest
Masking
Interest
Metal
Interest
Modeling
Interest
Modifiers
Interest
Motion Tracking
Interest
Nodes & Physics
Interest
OpenGL
Interest
Overlay
Interest
Overrides
Interest
Performance
Interest
Physics
Interest
Pipeline, Assets & IO
Interest
Platforms, Builds & Tests
Interest
Python API
Interest
Render & Cycles
Interest
Render Pipeline
Interest
Sculpt, Paint & Texture
Interest
Text Editor
Interest
Translations
Interest
Triaging
Interest
Undo
Interest
USD
Interest
User Interface
Interest
UV Editing
Interest
VFX & Video
Interest
Video Sequencer
Interest
Virtual Reality
Interest
Vulkan
Interest
Wayland
Interest
Workbench
Interest: X11
Legacy
Asset Browser Project
Legacy
Blender 2.8 Project
Legacy
Milestone 1: Basic, Local Asset Browser
Legacy
OpenGL Error
Meta
Good First Issue
Meta
Papercut
Meta
Retrospective
Meta
Security
Module
Animation & Rigging
Module
Core
Module
Development Management
Module
EEVEE & Viewport
Module
Grease Pencil
Module
Modeling
Module
Nodes & Physics
Module
Pipeline, Assets & IO
Module
Platforms, Builds & Tests
Module
Python API
Module
Render & Cycles
Module
Sculpt, Paint & Texture
Module
Triaging
Module
User Interface
Module
VFX & Video
Platform
FreeBSD
Platform
Linux
Platform
macOS
Platform
Windows
Priority
High
Priority
Low
Priority
Normal
Priority
Unbreak Now!
Status
Archived
Status
Confirmed
Status
Duplicate
Status
Needs Info from Developers
Status
Needs Information from User
Status
Needs Triage
Status
Resolved
Type
Bug
Type
Design
Type
Known Issue
Type
Patch
Type
Report
Type
To Do
No Milestone
No project
No Assignees
3 Participants
Notifications
Due Date
The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: blender/blender#118885
No description provided.