Texture Displacement Sculpting design #73935

New Issue

Pablo Dobarro · 2020-02-17T19:01:40+01:00

Pablo Dobarro commented

2020-02-17 19:01:40 +01:00

Texture Displacement Sculpting

Some things originally in this task have moved to #96223 (PBVH image texture painting technical design) and #96225 (PBVH image texture painting implementation).

The idea here is to build on the above tasks, and make sculpt tools work with displacement textures and normal maps.

For some workflows this is a huge improvement. Game artist that rely on high poly sculpts and baking tools to create details in assets for game engines won’t need any of that anymore. They will be able to use a fully feature sculpting system to add the detail to the textures of the model directly.

This is obviously going to fail and create a lot of artifacts if the sculpted deformation is too big, but that is not the initial idea (that is also going to fail if the the high res sculpt and the low res mesh shape differ too much). The idea is to avoid using high poly sculpts to add surface detail when it is not necessary, and for surface detail, this approach should work fine.

I can’t evaluate the performance of this sculpting mode, but if we add UDIM support to the ##PBVH_TEXTURE## (should not be hard to do, specially if we don’t support faces across different UDIM tiles) and the performance is OK, it could potentially allow to sculpt surface detail on models with a much higher resolution than any other solution based on mesh geometry and vertex deformation.

To implement this, the idea is to add a second buffer that stores the vectors to get the “deformed” 3D coordinates, and its normals. When iterating over the vertices in ##PBVH_ITER_BEGIN##, the ##vd.co## will be calculated by using the original 3D position of the vertex + displacement vector stored in the buffer. It should be possible to write this information to the texture to get a displacement texture and it should also be possible to write the normals calculated when the ##PBVH_UpdateNormals## flag is processed to get a normal map.

Possible Issues

I’m still not sure how this is going to work across seams. I think that if we add some extra padding pixels to the node which contains that face as if we were extending the face in 3D space it should be fine, but hard to know without trying it.
I’m also not sure what kind of performance we can expect out of this, but even if performance is not the best, we can still try to code it to support some of its features, like sculpting information to the textures.

Technical Requirements

Implement the neighbor iterator in the sculpt API
Implement the deformed coordinates buffer and the normal map/vector displacement calculations
Update the Sculpt API to read data from the coordinates buffer
Add the Sculpt Mode UI to support textures
Add normal map/displacement support to workbench (If possible, we can use EEVEE instead if that is hard to do)

## Texture Displacement Sculpting Some things originally in this task have moved to #96223 (PBVH image texture painting technical design) and #96225 (PBVH image texture painting implementation). The idea here is to build on the above tasks, and make sculpt tools work with displacement textures and normal maps. For some workflows this is a huge improvement. Game artist that rely on high poly sculpts and baking tools to create details in assets for game engines won’t need any of that anymore. They will be able to use a fully feature sculpting system to add the detail to the textures of the model directly. This is obviously going to fail and create a lot of artifacts if the sculpted deformation is too big, but that is not the initial idea (that is also going to fail if the the high res sculpt and the low res mesh shape differ too much). The idea is to avoid using high poly sculpts to add surface detail when it is not necessary, and for surface detail, this approach should work fine. I can’t evaluate the performance of this sculpting mode, but if we add UDIM support to the ##PBVH_TEXTURE## (should not be hard to do, specially if we don’t support faces across different UDIM tiles) and the performance is OK, it could potentially allow to sculpt surface detail on models with a much higher resolution than any other solution based on mesh geometry and vertex deformation. To implement this, the idea is to add a second buffer that stores the vectors to get the “deformed” 3D coordinates, and its normals. When iterating over the vertices in ##PBVH_ITER_BEGIN##, the ##vd.co## will be calculated by using the original 3D position of the vertex + displacement vector stored in the buffer. It should be possible to write this information to the texture to get a displacement texture and it should also be possible to write the normals calculated when the ##PBVH_UpdateNormals## flag is processed to get a normal map. ### Possible Issues - I’m still not sure how this is going to work across seams. I think that if we add some extra padding pixels to the node which contains that face as if we were extending the face in 3D space it should be fine, but hard to know without trying it. - I’m also not sure what kind of performance we can expect out of this, but even if performance is not the best, we can still try to code it to support some of its features, like sculpting information to the textures. ### Technical Requirements - Implement the neighbor iterator in the sculpt API - Implement the deformed coordinates buffer and the normal map/vector displacement calculations - Update the Sculpt API to read data from the coordinates buffer - Add the Sculpt Mode UI to support textures - Add normal map/displacement support to workbench (If possible, we can use EEVEE instead if that is hard to do)

Pablo Dobarro self-assigned this 2020-02-17 19:01:40 +01:00

Pablo Dobarro commented

2020-02-17 19:01:40 +01:00

Added subscriber: @PabloDobarro

Konstantina Anagnostou commented

2020-02-17 19:26:31 +01:00

Added subscriber: @Constantina32

Andrey commented

2020-02-17 20:16:25 +01:00

Added subscriber: @Andruxa696

Aaron Carlisle commented

2020-02-17 20:31:28 +01:00

Changed status from 'Needs Triage' to: 'Confirmed'

Petter Lundh commented

2020-02-17 20:57:08 +01:00

Added subscriber: @PetterLundh

Demeter Dzadik commented

2020-02-18 12:33:55 +01:00

Added subscriber: @Mets

Demeter Dzadik commented

2020-02-18 12:33:55 +01:00

Sculpt Mode, Texture paint and Vertex Paint will run on the exact same code

Is it fair to assume that this also includes Weight Paint, since it's probably already sharing most code with Vertex Paint?

> Sculpt Mode, Texture paint and Vertex Paint will run on the exact same code Is it fair to assume that this also includes Weight Paint, since it's probably already sharing most code with Vertex Paint?

Alan Redd commented

2020-02-20 00:58:17 +01:00

Added subscriber: @AlRedd-3

Alan Redd commented

2020-02-20 00:58:17 +01:00

This is a game changer for realistic character workflows!!! Especially texture displacement Sculpting!!

kfir merlaub commented

2020-05-01 05:43:30 +02:00

Added subscriber: @kfir

Alex commented

2020-06-09 17:11:43 +02:00

Added subscriber: @SpectreFirst

Mindinsomnia commented

2020-09-08 10:11:52 +02:00

Added subscriber: @Grady

Mindinsomnia commented

2020-09-08 10:11:52 +02:00

@PabloDobarro Would this change make it possible to draw to multiple textures concurrently with different colour values?

Like for example, creating a custom brush for painting "bolts", with textures for albedo, metallic, roughness, specular, displacement, etc, and painting those textures directly onto a surface in one go?

@PabloDobarro Would this change make it possible to draw to multiple textures concurrently with different colour values? Like for example, creating a custom brush for painting "bolts", with textures for albedo, metallic, roughness, specular, displacement, etc, and painting those textures directly onto a surface in one go?

Przemyslaw Golab (SirPigeonz) commented

2020-09-08 13:49:39 +02:00

Added subscriber: @SirPigeonz