Realtime Compositor: Implement Inpaint node #111792

Omar Emara · 2023-09-01T16:35:58+02:00

Omar Emara commented

2023-09-01 16:35:58 +02:00

This patch implements the Inpaint node for the Realtime Compositor. The
inpainting region is filled by sampling the closest pixel at the inpainting
boundary, which is done using the Jump Flooding algorithm. To smooth the
inpainting region, a lateral blur is applied in the direction tangent to the
inpainting boundary.

The implementation is not identical to the existing CPU implementation due to
technical infeasibility. In particular, the CPU implementation uses a Manhattan
distance transform, while the GPU implementation uses an Euclidean one, which is
a consequence of the use of the Jump Flooding algorithm. Furthermore, the CPU
uses a serial convolution starting from the boundary outwards, while the GPU
uses a lateral Gaussian blur in the direction tangent to the boundary.

Depends on #112223. For the Jump Flooding implementation.

This patch implements the Inpaint node for the Realtime Compositor. The inpainting region is filled by sampling the closest pixel at the inpainting boundary, which is done using the Jump Flooding algorithm. To smooth the inpainting region, a lateral blur is applied in the direction tangent to the inpainting boundary. The implementation is not identical to the existing CPU implementation due to technical infeasibility. In particular, the CPU implementation uses a Manhattan distance transform, while the GPU implementation uses an Euclidean one, which is a consequence of the use of the Jump Flooding algorithm. Furthermore, the CPU uses a serial convolution starting from the boundary outwards, while the GPU uses a lateral Gaussian blur in the direction tangent to the boundary. Depends on #112223. For the Jump Flooding implementation.

Omar Emara added 1 commit 2023-09-01 16:36:10 +02:00

604ba2871a Realtime Compositor: Implement Inpaint node

This patch implements the Inpaint node for the Realtime Compositor.

Omar Emara added 2 commits 2023-09-12 10:55:16 +02:00

bad5bb7154 Merge branch 'main' into gpu-inpaint

8d02f25243 Add a lateral blur pass

Omar Emara changed title from ~~WIP: Realtime Compositor: Implement Inpaint node~~ to Realtime Compositor: Implement Inpaint node

2023-09-12 11:10:53 +02:00

Omar Emara commented

2023-09-12 11:24:18 +02:00

Comparison between CPU and GPU implementations with a 30px inpainting distance. The weird shape of the CPU one is due to the use of Manhatten distance.

Comparison between CPU and GPU implementations with a 30px inpainting distance. The weird shape of the CPU one is due to the use of Manhatten distance. ![CPUvsGPU.png](/attachments/9aa3140c-5498-47c8-8e03-0458d9d8daff)

CPUvsGPU.png

50 KiB

Omar Emara requested review from Clément Foucault 2023-09-12 11:24:32 +02:00

Omar Emara requested review from Sergey Sharybin 2023-09-12 11:24:40 +02:00

Clément Foucault commented

2023-09-12 12:59:53 +02:00

Looks very good and super useful! I was just wondering if the left tearing artifact is to be expected

Looks very good and super useful! I was just wondering if the left tearing artifact is to be expected ![image](/attachments/6a2f403c-dee8-42c1-9eb2-30bac3ee99ee)

tearing_artifacts.blend

930 KiB

image.png

729 KiB

Clément Foucault requested changes 2023-09-12 13:03:08 +02:00

source/blender/compositor/realtime_compositor/shaders/compositor_jump_flooding.glsl Outdated

						
				@ -0,0 +48,4 @@

				      if (squared_distance < minimum_squared_distance) {

				        minimum_squared_distance = squared_distance;

				        closest_seed_position = position;

Clément Foucault commented

2023-09-12 13:03:06 +02:00

Fails on OSX

ERROR (gpu.shader): compositor_jump_flooding Compute Shader: 
      | 
      | Error: assigning to 'vec2' (aka 'float2') from incompatible type 'ivec2' (aka 'int2')
      | 
      | closest_seed_position = position;
      | ^~~~~~~~

Fails on OSX ``` ERROR (gpu.shader): compositor_jump_flooding Compute Shader: | | Error: assigning to 'vec2' (aka 'float2') from incompatible type 'ivec2' (aka 'int2') | | closest_seed_position = position; | ^~~~~~~~ ```

Omar Emara added 1 commit 2023-09-12 13:12:12 +02:00

d0f0263e26 Fix compile error on Metal

Omar Emara commented

2023-09-12 13:16:10 +02:00

@fclem Not sure, looking into it.

Omar Emara commented

2023-09-12 14:41:51 +02:00

Apparently, those banding effects are due to a sudden change of the tangent vector because two neighbouring pixels picks up two boundary pixels that are far way from each other, which in this case happens due to the aliasing at the cube edges, as can be seen in this illustration.

A possible workaround is to recompute the tangent at each integration step, at the cost of many more texture fetches, I could look into that.

Apparently, those banding effects are due to a sudden change of the tangent vector because two neighbouring pixels picks up two boundary pixels that are far way from each other, which in this case happens due to the aliasing at the cube edges, as can be seen in this illustration. ![20230912-152854.png](/attachments/21d04882-3e33-4fc2-8f57-f311a9faa282) A possible workaround is to recompute the tangent at each integration step, at the cost of many more texture fetches, I could look into that.

20230912-152854.png

3.1 KiB

Omar Emara added 1 commit 2023-09-13 09:12:51 +02:00

42adb0f020 Blur along a dynamic tangential path

Omar Emara requested review from Clément Foucault 2023-09-13 09:14:39 +02:00

Clément Foucault approved these changes 2023-09-14 11:45:12 +02:00

Clément Foucault left a comment

The step artifact are indeed fixed! To me everything is good.

Sergey Sharybin approved these changes 2023-09-18 22:37:53 +02:00

Sergey Sharybin left a comment

I think the difference in the algorithm is fine for the main purpose of the node. I don't think it was really designed to close large gaps.
That being said, do you think it feasible to use euclidean distance on CPU to align its result more closely to GPU?

I think the difference in the algorithm is fine for the main purpose of the node. I don't think it was really designed to close large gaps. That being said, do you think it feasible to use euclidean distance on CPU to align its result more closely to GPU?

Omar Emara commented

2023-09-22 13:31:30 +02:00

@Sergey It is feasible to use euclidean distance for the CPU as well. I just haven't done my homework yet on what the best way to do that is. The 1+JFA algorithm we use on the GPU was chosen because it works well on massively parallel hardware, which might not be the best on CPUs.

Sergey Sharybin commented

2023-09-22 17:40:00 +02:00

@OmarEmaraDev I see. Well, for the main intended use-case I think the difference will be minimal. So don't think it is practical to consider the difference a stopper here.

Omar Emara added 1 commit 2023-09-25 08:39:34 +02:00

76d098db49 Merge branch 'main' into gpu-inpaint

Omar Emara merged commit acb88528a5 into main

2023-09-25 08:45:54 +02:00

Omar Emara referenced this issue from a commit

2023-09-25 08:45:55 +02:00

Realtime Compositor: Implement Inpaint node

Omar Emara deleted branch gpu-inpaint

2023-09-25 08:45:56 +02:00

Colin Basnett referenced this issue from a commit

2023-09-27 07:30:22 +02:00

Realtime Compositor: Implement Inpaint node