Support for tools with interactive geometry preview #83115

New Issue

Pablo Dobarro · 2020-11-27T16:47:31+01:00

Pablo Dobarro commented

2020-11-27 16:47:31 +01:00

Some of the current or planned tools (add object tool, insert mesh brushes...) and other functionality requested project members require adding new geometry to the scene.

As far as I know, these tools can't render new geometry without creating a new object or adding that geometry to an existing mesh. This makes the gizmo previews of the add object tool not ideal and some of the requested functionality for sculpt mode is not possible to implement. It would require to modify the mesh and rebuild the PBVH in real time each time geometry changes.

The only tool so far that requires this functionality in sculpt mode is a limited version of the insert mesh brush in D9380. That tool joins the insert mesh to the active mesh using BMesh when the stroke starts and then rebuilds the PBVH so the new geometry can be rendered. This makes the tool lag each time a stroke starts in a high poly mesh, and it limits the functionality to only an insert operation per stroke. (The ideal behavior should be that the insert mesh brush should insert a new mesh per stroke step, so you can insert meshes with spacing over the mesh).

The main goal of this design of to have a fast way to create previews for tools that modify geometry which look like the mesh is actually being modified. This is just a implementation detail, so the user should not know if the mesh geometry is actually being modified or if it is just a preview, as both options should ideally look the same.

Proposed solution

The ideal solution would be to have a generic function(s) that can render arbitrary geometry using the current scene shaders and render mode (as it was a new mesh object). This function should be possible to use from any operator and mode.

In the case of the insert mesh brush, the tool can manage a list of triangles and indices created by combining the geometry of the insert mesh and use this function to render them in real time without modifying the PBVH or the active object geometry. Then after the stroke finishes, all the new geometry can be joined to the active mesh in a single operation.

Workarounds

If implemented that functionality is not possible, some workaround for sculpt mode could be:

When PBVH rendering is used, add a "fake" node to the PBVH and add the new geometry there. If this fake node exists, render it as a normal node. When the operation finishes, delete this node, join the meshes and rebuild the tree. This approach won't work when PBVH rendering is disabled, so these tools will still lag.
Exit sculpt mode when the operation starts and create a new temporally object to do the geometry preview there. After the operation finishes, join the geometry, delete the preview object and enter sculpt mode in the initial object. Not sure what potential problems this approach could have (like undo).

Some of the current or planned tools (add object tool, insert mesh brushes...) and other functionality requested project members require adding new geometry to the scene. As far as I know, these tools can't render new geometry without creating a new object or adding that geometry to an existing mesh. This makes the gizmo previews of the add object tool not ideal and some of the requested functionality for sculpt mode is not possible to implement. It would require to modify the mesh and rebuild the PBVH in real time each time geometry changes. The only tool so far that requires this functionality in sculpt mode is a limited version of the insert mesh brush in [D9380](https://archive.blender.org/developer/D9380). That tool joins the insert mesh to the active mesh using BMesh when the stroke starts and then rebuilds the PBVH so the new geometry can be rendered. This makes the tool lag each time a stroke starts in a high poly mesh, and it limits the functionality to only an insert operation per stroke. (The ideal behavior should be that the insert mesh brush should insert a new mesh per stroke step, so you can insert meshes with spacing over the mesh). The main goal of this design of to have a fast way to create previews for tools that modify geometry which look like the mesh is actually being modified. This is just a implementation detail, so the user should not know if the mesh geometry is actually being modified or if it is just a preview, as both options should ideally look the same. ## Proposed solution The ideal solution would be to have a generic function(s) that can render arbitrary geometry using the current scene shaders and render mode (as it was a new mesh object). This function should be possible to use from any operator and mode. In the case of the insert mesh brush, the tool can manage a list of triangles and indices created by combining the geometry of the insert mesh and use this function to render them in real time without modifying the PBVH or the active object geometry. Then after the stroke finishes, all the new geometry can be joined to the active mesh in a single operation. ## Workarounds If implemented that functionality is not possible, some workaround for sculpt mode could be: - When PBVH rendering is used, add a "fake" node to the PBVH and add the new geometry there. If this fake node exists, render it as a normal node. When the operation finishes, delete this node, join the meshes and rebuild the tree. This approach won't work when PBVH rendering is disabled, so these tools will still lag. - Exit sculpt mode when the operation starts and create a new temporally object to do the geometry preview there. After the operation finishes, join the geometry, delete the preview object and enter sculpt mode in the initial object. Not sure what potential problems this approach could have (like undo).

Pablo Dobarro commented

2020-11-27 16:47:31 +01:00

Added subscribers: @PabloDobarro, @brecht, @Sergey

Przemyslaw Golab (SirPigeonz) commented

2020-11-28 02:11:21 +01:00

Added subscriber: @SirPigeonz

Sergey Sharybin commented

2020-11-30 11:14:21 +01:00

I see where it is coming from. The decision on whether this is something we really need/want in Blender's interaction language I'd leave to Brecht and the UI/UX team.

To me this task is more of an idea than a real strong design task. Missing points are:

Who is the owner of such temporary objects? What their lifetime is?
How does it integrate with draw manager and render engines?
What is the format of the data?
What is the material of such temporary objects?
How it is indicated to user that it is a temporary object (as opposite of "real" one being transformed)?

I see where it is coming from. The decision on whether this is something we really need/want in Blender's interaction language I'd leave to Brecht and the UI/UX team. To me this task is more of an idea than a real strong design task. Missing points are: - Who is the owner of such temporary objects? What their lifetime is? - How does it integrate with draw manager and render engines? - What is the format of the data? - What is the material of such temporary objects? - How it is indicated to user that it is a temporary object (as opposite of "real" one being transformed)?

Pablo Dobarro commented

2020-11-30 20:12:23 +01:00

Who is the owner of such temporary objects? What their lifetime is?

In the proposed solution, the idea would be to not have any temporary object.

How does it integrate with draw manager and render engines?

This is the part I would like to discuss, I don't know how the draw manager works.

What is the format of the data?
What is the material of such temporary objects?

I would expect something that works as similar as possible to drawing geometry with immediate mode. For example, a function that takes a list of points, a list of triangle indices and an optional object to use its render and material settings (if null, it can use some default ones). The idea of this design is just having a better preview of that geometry in the scene instead of something drawn with random wireframes and triangles. All tools we have currently that need this functionality use the wireframe approach, so all of them look different and they look completely out of context compared to the rest of the UI and interactive tools.

How it is indicated to user that it is a temporary object (as opposite of "real" one being transformed)?

There should no be any need for this. This is more like an implementation detail of how the mesh modification is handled in drawing, the user should not be aware that this is happening. In the end, the tools should look just like the videos in D9380.

> Who is the owner of such temporary objects? What their lifetime is? In the proposed solution, the idea would be to not have any temporary object. > How does it integrate with draw manager and render engines? This is the part I would like to discuss, I don't know how the draw manager works. > What is the format of the data? > What is the material of such temporary objects? I would expect something that works as similar as possible to drawing geometry with immediate mode. For example, a function that takes a list of points, a list of triangle indices and an optional object to use its render and material settings (if null, it can use some default ones). The idea of this design is just having a better preview of that geometry in the scene instead of something drawn with random wireframes and triangles. All tools we have currently that need this functionality use the wireframe approach, so all of them look different and they look completely out of context compared to the rest of the UI and interactive tools. > How it is indicated to user that it is a temporary object (as opposite of "real" one being transformed)? There should no be any need for this. This is more like an implementation detail of how the mesh modification is handled in drawing, the user should not be aware that this is happening. In the end, the tools should look just like the videos in [D9380](https://archive.blender.org/developer/D9380).

Sergey Sharybin commented

2020-12-01 15:44:17 +01:00

Who is the owner of such temporary objects? What their lifetime is?

In the proposed solution, the idea would be to not have any temporary object.

The naming gets in a way. I was not talking about Object as in Object from DNA_object_types.h. Meant those temporary entities with geometry.

Someone needs to be an owner of them. And they need to be recognized by render engines somehow.

This is the part I would like to discuss, I don't know how the draw manager works.

There are quite some unknowns about whats' possible and what's not in the draw manager.

Brecht, not sure what's best. Keep you fully involved here, or try to offload some of tasks from you and involve Clement/Jeroen?

How it is indicated to user that it is a temporary object (as opposite of "real" one being transformed)?

There should no be any need for this. This is more like an implementation detail of how the mesh modification is handled in drawing, the user should not be aware that this is happening. In the end, the tools should look just like the videos in D9380.

Ok, good to know. I'd say such information should be stated in the task description, so that the latest agreements and decisions are reflected there for ease of discoverability.

>> Who is the owner of such temporary objects? What their lifetime is? > In the proposed solution, the idea would be to not have any temporary object. The naming gets in a way. I was not talking about `Object` as in `Object` from `DNA_object_types.h`. Meant those temporary entities with geometry. Someone needs to be an owner of them. And they need to be recognized by render engines somehow. > This is the part I would like to discuss, I don't know how the draw manager works. There are quite some unknowns about whats' possible and what's not in the draw manager. Brecht, not sure what's best. Keep you fully involved here, or try to offload some of tasks from you and involve Clement/Jeroen? >> How it is indicated to user that it is a temporary object (as opposite of "real" one being transformed)? > There should no be any need for this. This is more like an implementation detail of how the mesh modification is handled in drawing, the user should not be aware that this is happening. In the end, the tools should look just like the videos in [D9380](https://archive.blender.org/developer/D9380). Ok, good to know. I'd say such information should be stated in the task description, so that the latest agreements and decisions are reflected there for ease of discoverability.

Brecht Van Lommel commented

2020-12-01 16:06:51 +01:00

In principle you can avoid rebuilding the full PBVH, and instead do a partial rebuild by keeping the existing nodes and adding geometry in new nodes. You would have to reallocate the Mesh arrays though, and probably refactor PBVH code to ensure pointers remain valid. Not sure how practical that is.

I guess the easiest solution may be to put the new geometry in a separate Mesh datablock owned by the sculpt session, and pass that to the draw manager? That way the draw manager can simply read from the same data structures it already knows, and use the same materials, custom data, etc.

In principle you can avoid rebuilding the full PBVH, and instead do a partial rebuild by keeping the existing nodes and adding geometry in new nodes. You would have to reallocate the Mesh arrays though, and probably refactor PBVH code to ensure pointers remain valid. Not sure how practical that is. I guess the easiest solution may be to put the new geometry in a separate `Mesh` datablock owned by the sculpt session, and pass that to the draw manager? That way the draw manager can simply read from the same data structures it already knows, and use the same materials, custom data, etc.

TheRedWaxPolice commented

2020-12-01 16:40:06 +01:00

Added subscriber: @TheRedWaxPolice

Pablo Dobarro commented

2020-12-01 21:01:35 +01:00

I guess the easiest solution may be to put the new geometry in a separate Mesh datablock owned by the sculpt session, and pass that to the draw manager?

@brecht Imagine the Insert mesh brush in D9380 but working with stroke method spacing. That means that per stroke step over the surface, it is going to insert a mesh in the location of the brush tip. Lets consider about 500 vertices as average for a mesh that will be inserted with that brush, and a stroke has an average of 200 samples (counting all symmetry passes).

With my proposed approach, I can create a list of points an indices from the looptris of the insert mesh when the stroke starts. The main brush function will just need to store an new matrix per stroke step to store the transform of the new instance. Then creating the drawing commands from this data using something similar to immediate mode is trivial.
After the stroke ends, it is possible to combine all inserted meshes and join them to the sculpt mesh in a single step. This is the most performance intensive operation as it also requires the rebuild, but because it happens at the end of the stroke, the tool should be much more responsive.

With your proposal, the tool can work in a similar way but it will need to create a new mesh datablock with all the combined inserted geometry each time the viewport needs to be rendered. Joining this mesh with the sculpt mesh can still happen at the end of the stroke. Do you thing the performance of this will be enough?

> I guess the easiest solution may be to put the new geometry in a separate Mesh datablock owned by the sculpt session, and pass that to the draw manager? @brecht Imagine the Insert mesh brush in [D9380](https://archive.blender.org/developer/D9380) but working with stroke method spacing. That means that per stroke step over the surface, it is going to insert a mesh in the location of the brush tip. Lets consider about 500 vertices as average for a mesh that will be inserted with that brush, and a stroke has an average of 200 samples (counting all symmetry passes). With my proposed approach, I can create a list of points an indices from the looptris of the insert mesh when the stroke starts. The main brush function will just need to store an new matrix per stroke step to store the transform of the new instance. Then creating the drawing commands from this data using something similar to immediate mode is trivial. After the stroke ends, it is possible to combine all inserted meshes and join them to the sculpt mesh in a single step. This is the most performance intensive operation as it also requires the rebuild, but because it happens at the end of the stroke, the tool should be much more responsive. With your proposal, the tool can work in a similar way but it will need to create a new mesh datablock with all the combined inserted geometry each time the viewport needs to be rendered. Joining this mesh with the sculpt mesh can still happen at the end of the stroke. Do you thing the performance of this will be enough?

Brecht Van Lommel commented

2020-12-02 01:57:49 +01:00

The appropriate draw commands for Eevee, workbench and overlays are not simple as far as I know. Immediate mode also is not supported in the draw manager.

There is no need to create a new mesh datablock for every viewport render, you can create it in the sculpting code and keep it during the stroke. When there are no geometry changes, the GPU batch caches would also be kept. I don't see where the performance issue would be.

The appropriate draw commands for Eevee, workbench and overlays are not simple as far as I know. Immediate mode also is not supported in the draw manager. There is no need to create a new mesh datablock for every viewport render, you can create it in the sculpting code and keep it during the stroke. When there are no geometry changes, the GPU batch caches would also be kept. I don't see where the performance issue would be.

Pablo Dobarro commented

2020-12-02 16:13:01 +01:00

What I mean is, for each viewport render, I need to discard all mesh arrays and build them again with the new data from the tool. This means going trough a similar process that the voxel remesher does when building the mesh from the OpenVDB points and indices each time the viewport renders. For the particular case of the insert mesh brush it may be possible to join the geometry to the previous mesh directly, but for many other tools that change the previous modified topology this won't be possible.

Dejan Pejacki commented

2021-09-03 18:40:06 +02:00

Added subscriber: @dlc17

Pratik Borhade commented

2022-01-19 10:26:21 +01:00

Changed status from 'Needs Triage' to: 'Confirmed'

Julien Kaspar added this to the Sculpt, Paint & Texture project 2023-02-08 10:20:48 +01:00

Philipp Oeser removed the

Download

What's New

Blender Studio

Manual

Developers Blog

Documentation

Benchmark

Blender Conference

Development Fund

One-time Donations

Support for tools with interactive geometry preview #83115

Proposed solution

Workarounds