Invert Masking behaviour for Sculpt/Paint Mode #97903

New Issue

Julien Kaspar · 2022-05-06T13:35:55+02:00

Julien Kaspar commented

2022-05-06 13:35:55 +02:00

This is a design related to the implementation of Paint Mode.
It outlines the possibility to invert the behavior & visualization of masks.
Current = Masked vertices cannot be edited
Proposed = Masks & Selections are the same -> They can be edited.

Current Issues

There are a few different issues that exist the with current way of how masks work.

An Inverted Workflow

A typical workflow issue when using masks is that users mask a section of their mesh and then invert the mask.
In a lot of cases the areas you mask are the ones you DO want to edit.
Masking is very much designed with this workflow in mind. But this leads to using Invert on the mask potentially hundreds of times per day.

Syncing Selections across Modes

For the implementation of Paint Mode it will be very beneficial to start syncing selections between various modes.
So if a selection is made in Edit Mode it will also appear as masked in Sculpt/Paint Mode (This would likely be a direct conversion every time you switch modes)
This will be less confusing and disconnected if both the selection in Edit Mode and the mask in Sculpt/Paint Mode are areas you CAN affect.

Masks vs Selections

A current plan for the implementation of Paint Mode and the new Curves object type is to use "Selections".
The use case for sculpt mode masks and selections is essentially the same:
You define a temporary area that should be affected by your operations.

The issue is that "Masks" also refer to the concept of for example creating an attribute (like a vertex group or color attribute)
or a grayscale image texture and using it as a factor for shader/geometry nodes. The use case here is quite different:
You save an area for repeated use in materials and procedural networks.
For example layering textures.

Replacing the current concept of sculpt mode masks with selections would avoid confusion and make the use case of selections and masks very clear.

Proposal

Selection instead of Masking

We need to test the following behavior:

Rename "Mask" to "Selection" or "Selection Mask"
Unselected areas are darkened with an overlay
Selected areas are affected by brushes & other operations

There are use cases that are important to keep in mind with this change:

When entering sculpt mode, everything should already be selected.
Previous workflows (Mask brush) to create a deselected area and should still be accessible.

Other areas of Blender will still refer to "Masks" like a Texture Mask in the brush settings, mask nodes, grease pencil layer masks and motion tracking masks.

Offer a preference to use the old behavior

There should be an option in the preferences to invert the selection behavior globally in sculpt/paint mode and get the old behavior back.
To what extend the inverted behavior would be done needs to be tested.

This is a design related to the implementation of Paint Mode. It outlines the possibility to invert the behavior & visualization of masks. **Current** = Masked vertices cannot be edited **Proposed** = Masks & Selections are the same -> They can be edited. # Current Issues There are a few different issues that exist the with current way of how masks work. ### An Inverted Workflow A typical workflow issue when using masks is that users mask a section of their mesh and then invert the mask. In a lot of cases the areas you mask are the ones you DO want to edit. Masking is very much designed with this workflow in mind. But this leads to using `Invert` on the mask potentially hundreds of times per day. ### Syncing Selections across Modes For the implementation of Paint Mode it will be very beneficial to start syncing selections between various modes. So if a selection is made in Edit Mode it will also appear as masked in Sculpt/Paint Mode (This would likely be a direct conversion every time you switch modes) This will be less confusing and disconnected if both the selection in Edit Mode and the mask in Sculpt/Paint Mode are areas you CAN affect. ### Masks vs Selections A current plan for the implementation of Paint Mode and the new Curves object type is to use "Selections". The use case for sculpt mode masks and selections is essentially the same: **You define a temporary area that should be affected by your operations.** The issue is that "Masks" also refer to the concept of for example creating an attribute (like a vertex group or color attribute) or a grayscale image texture and using it as a factor for shader/geometry nodes. The use case here is quite different: **You save an area for repeated use in materials and procedural networks.** For example layering textures. Replacing the current concept of sculpt mode masks with selections would avoid confusion and make the use case of selections and masks very clear. # Proposal ### Selection instead of Masking We need to test the following behavior: - Rename "Mask" to "Selection" or "Selection Mask" - Unselected areas are darkened with an overlay - Selected areas are affected by brushes & other operations There are use cases that are important to keep in mind with this change: - When entering sculpt mode, everything should already be selected. - Previous workflows (Mask brush) to create a deselected area and should still be accessible. Other areas of Blender will still refer to "Masks" like a Texture Mask in the brush settings, mask nodes, grease pencil layer masks and motion tracking masks. ### Offer a preference to use the old behavior There should be an option in the preferences to invert the selection behavior globally in sculpt/paint mode and get the old behavior back. To what extend the inverted behavior would be done needs to be tested.

Julien Kaspar commented

2022-05-06 13:35:55 +02:00

Added subscribers: @JulienKaspar, @JosephEagar, @Jeroen-Bakker, @Limarest

Julien Kaspar commented

2022-05-06 13:41:16 +02:00

I discussed this task with @jbakker. While this is not a high priority for the implementation of Paint Mode, it would be good to test this design further.

@Limarest If you are interested you could create work on a patch to try out the inverted behaviour. It would help a lot in prototyping this before going further.

Sorry if the task is not clear enough from the description. I will work it out more once I have time. You can also write me on blender.chat.

I discussed this task with @jbakker. While this is not a high priority for the implementation of Paint Mode, it would be good to test this design further. @Limarest If you are interested you could create work on a patch to try out the inverted behaviour. It would help a lot in prototyping this before going further. Sorry if the task is not clear enough from the description. I will work it out more once I have time. You can also write me on blender.chat.

Julien Kaspar commented

2022-05-06 13:46:13 +02:00

Changed status from 'Needs Triage' to: 'Confirmed'

Julien Kaspar commented

2022-05-14 12:58:19 +02:00

Added subscriber: @dfelinto

Julien Kaspar commented

2022-05-14 12:58:19 +02:00

Here's an update from recent experiments and discussions.

@dfelinto Already started experiemnting with new selection tool icons for the hair system. These can be reused for Paint/Sculpt Mode any time.

@Limarest Started working on a prototype and inverting the behaviour and visualisation is not a huge task. (D14978)
blender_fHyN0wQJ5q.mp4

The biggest issue that this highlights that needs to be solved:

The default state of selection/masking in sculpt mode is that nothing is darkened

The main conflict in logic that we are trying to solve is this:

Edit Mode Selection = Areas you can edit
Sculpt/Paint Mode Masking = Areas you cannot edit

Se we are trying to make this more consistent (For the reasons mentioned in the main description like selection syncing, workflow issues and feature naming)
But we ignored another conflict between the two modes:

Default Edit Mode state= Nothing is selected
Default Sculpt/Paint Mode state = Everything is selected (Nothing is masked)

This ensures that in both modes everything is clearly visible by default. You also only select areas in edit mode once you want to do an operation on something.
But with brushes you don't depend on selections. You can just sculpt/paint anywhere anytime. You only need selections/masking if you want to restrict areas from getting edited.

We'll discuss this further to perhaps reach a solution to make this still just as intuitive as the old behaviour.

Here's an update from recent experiments and discussions. @dfelinto Already started experiemnting with new selection tool icons for the hair system. These can be reused for Paint/Sculpt Mode any time. ![Clipboard - May 12.jpg](https://archive.blender.org/developer/F13073769/Clipboard_-_May_12.jpg) @Limarest Started working on a prototype and inverting the behaviour and visualisation is not a huge task. ([D14978](https://archive.blender.org/developer/D14978)) [blender_fHyN0wQJ5q.mp4](https://archive.blender.org/developer/F13073765/blender_fHyN0wQJ5q.mp4) The biggest issue that this highlights that needs to be solved: ### The default state of selection/masking in sculpt mode is that nothing is darkened The main conflict in logic that we are trying to solve is this: > Edit Mode Selection = Areas you can edit > Sculpt/Paint Mode Masking = Areas you cannot edit Se we are trying to make this more consistent (For the reasons mentioned in the main description like selection syncing, workflow issues and feature naming) But we ignored another conflict between the two modes: > Default Edit Mode state= Nothing is selected > Default Sculpt/Paint Mode state = Everything is selected (Nothing is masked) This ensures that in both modes everything is clearly visible by default. You also only select areas in edit mode once you want to do an operation on something. But with brushes you don't depend on selections. You can just sculpt/paint anywhere anytime. You only need selections/masking if you want to restrict areas from getting edited. We'll discuss this further to perhaps reach a solution to make this still just as intuitive as the old behaviour.

Julien Kaspar commented

2022-05-18 21:52:20 +02:00

Changed status from 'Confirmed' to: 'Needs Developer To Reproduce'

Julien Kaspar commented

2022-05-18 21:52:20 +02:00

From more discussion and conclusions from the recent module meetings it is wise to go for the following for now:

Do not invert the masking workflow for both Sculpt and Paint Mode (keep it as is until we have a solid design)
Sync unmasked and selected areas between modes (Convert them upon mode switching)
Further discuss potential of making connection of masks & selections clear

It's clear that the cons heavily outweigh the pros in any current design ideas of inverting the workflow of creating masks.
The change will make the connection between selections in modes clearer. And it will remove the very regular step of inverting the mask after creating it.
But it will also create a lot of difficult visualisation, complexity or unpredictability while masking/selecting.

I will revisit this design at a later point. Until then the syncing of selections & masks between modes is the more important task and still part of #97955.

From more discussion and conclusions from the recent module meetings it is wise to go for the following for now: - Do not invert the masking workflow for both Sculpt and Paint Mode (keep it as is until we have a solid design) - Sync unmasked and selected areas between modes (Convert them upon mode switching) - Further discuss potential of making connection of masks & selections clear It's clear that the cons heavily outweigh the pros in any current design ideas of inverting the workflow of creating masks. The change will make the connection between selections in modes clearer. And it will remove the very regular step of inverting the mask after creating it. But it will also create a lot of difficult visualisation, complexity or unpredictability while masking/selecting. I will revisit this design at a later point. Until then the syncing of selections & masks between modes is the more important task and still part of #97955.

Julien Kaspar commented

2022-05-19 15:07:39 +02:00

But it will also create a lot of difficult visualisation, complexity or unpredictability while masking/selecting.

I want to elaborate on this and share 3 approaches from discussions here that could be considered to invert the masking behaviour.
For this I will refer to masks as "selections", because we are still moving forward with this proposal to sync edit mode selections with unmasked areas in sculpt/paint mode.

Selected = Can be sculpted
Deselected = Cannot be sculpted (visually darkened)

Calling them selections here will make the desired behaviour clearer.

Approach 1 - Set if all visible is selected

(Closer to original proposal)

If everything visible is selected

Using a selection operation (brush, box, lasso, expand) will deselect everything and create a new selection (set a selection)
Any subsequent operations will extend the selection
Holding Ctrl will deselect

If something that is visible is not selected

Using a selection operation will extend (visibly do nothing)

If something is hidden and unselected

Same behaviour as 1

Issues with proposal:

When selecting vertices in sculpt mode but hiding a face set, the select brush would not set a new selection?
This behaviour might lead to users using the “Select All” operation - just to be sure - before doing any new selection.Otherwise selection operators might not set but extend instead.
They might assume that something is deselected and try to add it to the selection -> Accidentally deselect everything?

Approach 2 - Always set selection. Hold Shift to Extend

(More consistent way with edit mode selection tools)

Use any selection operation to set a new selection (Darken everything and select affected areas)
Hold Shift to extend selection
Hold Ctrl to subtract from selection
Hold other modifier combination (like Alt Shift) to smooth selection

Issues with proposal:

This will lead to users having to hold modifier keys much more often
It introduces an inconsistency with all other brushes (Shift is for smoothing)

Approach 3 - Never Set

This is the simplest approach but is lacking any advantages of the original proposal. We shouldn't go for this imo.

Using any selection operation will extend the current selection
Holding Ctrl while doing so will subtract.
Holding Shift will smooth the selection

All of these suggestions are either adding new inconsistencies, create the necessity to invert the selection or select all regularly or make it hard to predict if the selection will be set or extended.
So they are not an improvement to the current behaviour.

> But it will also create a lot of difficult visualisation, complexity or unpredictability while masking/selecting. I want to elaborate on this and share 3 approaches from discussions here that could be considered to invert the masking behaviour. For this I will refer to masks as "selections", because we are still moving forward with this proposal to sync edit mode selections with unmasked areas in sculpt/paint mode. Selected = Can be sculpted Deselected = Cannot be sculpted (visually darkened) Calling them selections here will make the desired behaviour clearer. ### Approach 1 - Set if all visible is selected (Closer to original proposal) 1. If everything visible is selected - Using a selection operation (brush, box, lasso, expand) will deselect everything and create a new selection (set a selection) - Any subsequent operations will extend the selection - Holding Ctrl will deselect 2. If something that is visible is not selected - Using a selection operation will extend (visibly do nothing) 3. If something is hidden and unselected - Same behaviour as 1 Issues with proposal: - When selecting vertices in sculpt mode but hiding a face set, the select brush would not set a new selection? - This behaviour might lead to users using the “Select All” operation - just to be sure - before doing any new selection.Otherwise selection operators might not set but extend instead. - They might assume that something is deselected and try to add it to the selection -> Accidentally deselect everything? ### Approach 2 - Always set selection. Hold Shift to Extend (More consistent way with edit mode selection tools) 1. Use any selection operation to set a new selection (Darken everything and select affected areas) 2. Hold Shift to extend selection 3. Hold Ctrl to subtract from selection 4. Hold other modifier combination (like Alt Shift) to smooth selection Issues with proposal: - This will lead to users having to hold modifier keys much more often - It introduces an inconsistency with all other brushes (Shift is for smoothing) ### Approach 3 - Never Set This is the simplest approach but is lacking any advantages of the original proposal. We shouldn't go for this imo. 1. Using any selection operation will extend the current selection 2. Holding Ctrl while doing so will subtract. 3. Holding Shift will smooth the selection --- All of these suggestions are either adding new inconsistencies, create the necessity to invert the selection or select all regularly or make it hard to predict if the selection will be set or extended. So they are not an improvement to the current behaviour.

Albert commented

2022-05-21 08:19:38 +02:00

Added subscriber: @wevon-2

Albert commented

2022-05-21 08:19:38 +02:00

The third approach might work if:
·By default the selection is not synchronized.
·There is a quick way to select and deselect all ("a" and "Alt+a")
·There is a quick way to invert the selection ("Ctrl+a" or "Click on empty screen space")
·There is a quick mode for Select Linked (double click on the element with selection tool)
·There is a quick mode for Unselect Linked (Ctrl+double click on the element with selection tool)

A fourth approach would be to keep the current interaction while a selection tool is active, and invert the visibility of the mask/selection when passing to a sculpt tool.

The **third approach** might work if: ·By default the selection is not synchronized. ·There is a quick way to select and deselect all ("a" and "Alt+a") ·There is a quick way to invert the selection ("Ctrl+a" or "Click on empty screen space") ·There is a quick mode for Select Linked (double click on the element with selection tool) ·There is a quick mode for Unselect Linked (Ctrl+double click on the element with selection tool) A **fourth approach** would be to keep the current interaction while a selection tool is active, and invert the visibility of the mask/selection when passing to a sculpt tool.

RedMser commented

2022-06-19 12:04:53 +02:00

Added subscriber: @RedMser

Julien Kaspar commented

2023-01-17 11:55:34 +01:00

I simplified the description a bit.
Once we test this behavior, like mentioned in a previous comment, the current tool/brush behavior shouldn't change too much.
So brushes and tools never Set a new selection but always extend or subtract from it.

I simplified the description a bit. Once we test this behavior, like mentioned in a previous comment, the current tool/brush behavior shouldn't change too much. So brushes and tools never Set a new selection but always extend or subtract from it.

Julien Kaspar added this to the Sculpt, Paint & Texture project 2023-02-08 10:20:48 +01:00

Philipp Oeser removed the

Download

What's New

Blender Studio

Manual

Developers Blog

Documentation

Benchmark

Blender Conference

Development Fund

One-time Donations

Invert Masking behaviour for Sculpt/Paint Mode #97903

Current Issues

An Inverted Workflow

Syncing Selections across Modes

Masks vs Selections

Proposal

Selection instead of Masking

Offer a preference to use the old behavior

The default state of selection/masking in sculpt mode is that nothing is darkened

Approach 1 - Set if all visible is selected

Approach 2 - Always set selection. Hold Shift to Extend

Approach 3 - Never Set