Realtime compositor: bokeh blur do not work with transformed image #102252

New Issue

Vyacheslav Kobozev · 2022-11-03T16:34:27+01:00

2022-11-03 16:34:27 +01:00

System Information
Operating system: Windows-10-10.0.17763-SP0 64 Bits
Graphics card: NVIDIA GeForce RTX 3060/PCIe/SSE2 NVIDIA Corporation 4.5.0 NVIDIA 512.95

Blender Version
Broken: version: 3.4.0 Alpha, branch: master, commit date: 2022-11-02 22:23, hash: bad55d56bc

Short description of error
For example rotation node do not rotate bokeh.
Same issue with transform node.
{F13850198 size=full}
2022-11-03_18-25-00.mp4
rtcompose.blend

Exact steps for others to reproduce the error
Enable realtime compositor,
Connect custom bokeh image to bokeh blur.
Insert rotation node between (after image) and try to rotate bokeh

**System Information** Operating system: Windows-10-10.0.17763-SP0 64 Bits Graphics card: NVIDIA GeForce RTX 3060/PCIe/SSE2 NVIDIA Corporation 4.5.0 NVIDIA 512.95 **Blender Version** Broken: version: 3.4.0 Alpha, branch: master, commit date: 2022-11-02 22:23, hash: `bad55d56bc` **Short description of error** For example rotation node do not rotate bokeh. Same issue with transform node. {[F13850198](https://archive.blender.org/developer/F13850198/изображение.png) size=full} [2022-11-03_18-25-00.mp4](https://archive.blender.org/developer/F13850201/2022-11-03_18-25-00.mp4) [rtcompose.blend](https://archive.blender.org/developer/F13850202/rtcompose.blend) **Exact steps for others to reproduce the error** Enable realtime compositor, Connect custom bokeh image to bokeh blur. Insert rotation node between (after image) and try to rotate bokeh

Vyacheslav Kobozev commented

2022-11-03 16:34:27 +01:00

Added subscriber: @Vyach

Vyacheslav Kobozev commented

2022-11-03 16:36:40 +01:00

It is fun, but composed image allow to use scale and translation
2022-11-03_18-32-32.mp4
rtcompose1.blend

It is fun, but composed image allow to use scale and translation [2022-11-03_18-32-32.mp4](https://archive.blender.org/developer/F13850238/2022-11-03_18-32-32.mp4) [rtcompose1.blend](https://archive.blender.org/developer/F13850258/rtcompose1.blend)

Vyacheslav Kobozev commented

2022-11-03 16:37:24 +01:00

Added subscriber: @OmarEmaraDev

Vyacheslav Kobozev commented

2022-11-03 16:37:24 +01:00

call for @OmarEmaraDev

Iliya Katushenock commented

2022-11-03 19:34:07 +01:00

Added subscriber: @mod_moder

Omar Emara commented

2022-11-04 09:35:38 +01:00

Changed status from 'Needs Triage' to: 'Confirmed'

Omar Emara self-assigned this 2022-11-04 09:35:38 +01:00

Omar Emara commented

2022-11-10 09:34:55 +01:00

@Vyach I am planning to take rotation into consideration, but I think I will ignore the scale and translation of the bokeh image. Scaling is analogous to changing the blur radius, so I don't see a point in taking it into consideration. Both translation and scaling can clip the bokeh image, but I would argue that this is undesirable in most cases. If the user wants to clip the bokeh some why, manual masking should be used instead. Do you think this is reasonable?

Jeroen Bakker commented

2022-11-10 10:45:35 +01:00

Added subscriber: @Jeroen-Bakker

Jeroen Bakker commented

2022-11-10 10:45:35 +01:00

For this we should keep to the same logic as the regular compositor. Don't supporting a feature without being able to clearly communicate should be avoided. Although for mentioned cases you're right it might become unmanageable for bigger setups or when users want to prototype differnet styles or setups. I would also still support them as there are many composite setups currently using it, when loading them it is not clear to the user why there composites in the viewport will look differently.

If something is time consuming we should report that to the user (similar to the geo nodes project performance overlay). We should keep the artistic freedom as much as possible to the user. They can optimize afterwards when they have chosen a style.

For this we should keep to the same logic as the regular compositor. Don't supporting a feature without being able to clearly communicate should be avoided. Although for mentioned cases you're right it might become unmanageable for bigger setups or when users want to prototype differnet styles or setups. I would also still support them as there are many composite setups currently using it, when loading them it is not clear to the user why there composites in the viewport will look differently. If something is time consuming we should report that to the user (similar to the geo nodes project performance overlay). We should keep the artistic freedom as much as possible to the user. They can optimize afterwards when they have chosen a style.

Omar Emara commented

2022-11-10 11:58:28 +01:00

@Jeroen-Bakker My reasoning was less about performance and more about what makes more sense in the context of infinite canvas compositing. I designed the realtime compositor putting in mind that transforming an image shouldn't clip it like the CPU compositor does. So I am just skeptical of breaking this requirement for this specific case. And I am arguing that users don't typically want the clipping behavior anyways in most cases. For instance, when the user rotates a white square 45 degrees, the user expects to get a rhombus, but instead get this hexagonal due to clipping:

The same for translation and scaling, trying to achieve shifted bokeh will result in the bokeh shape clipping, which is also unlikely to be the desired outcome by the user:

I do recognize that the clipping behavior might be used in some setups, perhaps to get hexagonal like above, but that clipping can be achieved using other means as I mentioned, and those means can be implemented in versioning for backward compatibility. On the other hand, enforcing clipping takes away functionality from the user that are otherwise hard to achieve.

We can find middle ground, perhaps by defining a specific domain for the bokeh input and clip it to that, but that will be hard to grasp for most users, and it is not clear what domain we should define as it seems arbitrary.

@Jeroen-Bakker My reasoning was less about performance and more about what makes more sense in the context of infinite canvas compositing. I designed the realtime compositor putting in mind that transforming an image shouldn't clip it like the CPU compositor does. So I am just skeptical of breaking this requirement for this specific case. And I am arguing that users don't typically want the clipping behavior anyways in most cases. For instance, when the user rotates a white square 45 degrees, the user expects to get a rhombus, but instead get this hexagonal due to clipping: ![20221110-124337.png](https://archive.blender.org/developer/F13882866/20221110-124337.png) The same for translation and scaling, trying to achieve shifted bokeh will result in the bokeh shape clipping, which is also unlikely to be the desired outcome by the user: ![20221110-125624.png](https://archive.blender.org/developer/F13882938/20221110-125624.png) I do recognize that the clipping behavior might be used in some setups, perhaps to get hexagonal like above, but that clipping can be achieved using other means as I mentioned, and those means can be implemented in versioning for backward compatibility. On the other hand, enforcing clipping takes away functionality from the user that are otherwise hard to achieve. We can find middle ground, perhaps by defining a specific domain for the bokeh input and clip it to that, but that will be hard to grasp for most users, and it is not clear what domain we should define as it seems arbitrary.

Vyacheslav Kobozev commented

2022-11-10 20:37:53 +01:00

In #102252#1443589, @OmarEmaraDev wrote:
@Vyach I am planning to take rotation into consideration, but I think I will ignore the scale and translation of the bokeh image. Scaling is analogous to changing the blur radius, so I don't see a point in taking it into consideration. Both translation and scaling can clip the bokeh image, but I would argue that this is undesirable in most cases. If the user wants to clip the bokeh some why, manual masking should be used instead. Do you think this is reasonable?

Scaling part of bokeh is important to make complex bokeh images, translation too. It is not about masking.
In example i can use three triangles with different scale and rotation to make stars.
Also translation can allow to fix a bit misaligned image.
Here:
bokeh test.blend

So if you can suggest another and easier way to do the same — tell me.

I agree with @Jeroen-Bakker . Artistic freedom. So user can block out, try variants with nodes fast and only then screenshot result and remake it as single proper image for the sake of performance.

> In #102252#1443589, @OmarEmaraDev wrote: > @Vyach I am planning to take rotation into consideration, but I think I will ignore the scale and translation of the bokeh image. Scaling is analogous to changing the blur radius, so I don't see a point in taking it into consideration. Both translation and scaling can clip the bokeh image, but I would argue that this is undesirable in most cases. If the user wants to clip the bokeh some why, manual masking should be used instead. Do you think this is reasonable? Scaling part of bokeh is important to make complex bokeh images, translation too. It is not about masking. In example i can use three triangles with different scale and rotation to make stars. Also translation can allow to fix a bit misaligned image. Here: [bokeh test.blend](https://archive.blender.org/developer/F13885836/bokeh_test.blend) So if you can suggest another and easier way to do the same — tell me. I agree with @Jeroen-Bakker . Artistic freedom. So user can block out, try variants with nodes fast and only then screenshot result and remake it as single proper image for the sake of performance.

Omar Emara commented

2022-11-11 08:34:02 +01:00

@Vyach The file you shared seems to give the expected outcome in the realtime compositor, what do you expect to happen?

Jeroen Bakker commented

2022-11-11 09:24:21 +01:00

@OmarEmaraDev Thanks for the explanation. I am not sure why infinite canvas compositing couldn't solve this problem. But that might be something we should review in more depth design-wise outside this patch.

I don't think that a technical solution should limit the way how users work. Perhaps the code might be more complex and we should check on the tradeoffs. But those aren't clear at this point. In the final implementation limitations should become clear to users. Perhaps there are different approaches to infinite canvas compositing that doesn't require these limitations. For example the full frame compositor has support for canvas compositing and infinite canvas compositing is also working (patches are not in master yet) there without the limitation.

Perhaps for now we should have a list of know limitations so we can collect these and discuss the implications of those limitations and design there. When designing infinite canvas compositing these limitations could then be reviewed to check which are important to have and drive the design based on collected feedback.

@OmarEmaraDev Thanks for the explanation. I am not sure why infinite canvas compositing couldn't solve this problem. But that might be something we should review in more depth design-wise outside this patch. I don't think that a technical solution should limit the way how users work. Perhaps the code might be more complex and we should check on the tradeoffs. But those aren't clear at this point. In the final implementation limitations should become clear to users. Perhaps there are different approaches to infinite canvas compositing that doesn't require these limitations. For example the full frame compositor has support for canvas compositing and infinite canvas compositing is also working (patches are not in master yet) there without the limitation. Perhaps for now we should have a list of know limitations so we can collect these and discuss the implications of those limitations and design there. When designing infinite canvas compositing these limitations could then be reviewed to check which are important to have and drive the design based on collected feedback.

Omar Emara commented

2022-11-11 09:53:12 +01:00

@Jeroen-Bakker To clarify, this is not a limitation of the implementation or the design, we can replicate the existing behavior if that was decided without much consideration.

I am talking about what makes sense to the user in the context of infinite canvas compositing. The problem with the current approach is that it defines a certain domain for the bokeh input that the user have to work around, for instance, in the cases presented above. Why should a rotated square produce a hexagonal bokeh result? Why does a shift in the bokeh change its shape? When one thinks about infinite canvas compositing, one shouldn't worry about those questions, which is my concern in this discussion.

The full frame compositor actually breaks backward compatibility to solve some of those concerns, for instance, rotation doesn't introduce clipping in the full frame compositor, so a rotated square produces a rhombus correctly. I think this is a step in the right direction, that behavior should be kept and appropriate versioning should be implemented instead. Translation and scaling still doesn't work for the full frame compositor though.

Ideally, I think the bokeh operation should have specializations to handle transformations itself. For instance, translations could corresponds to an offset in the blur window in the shader, while scaling could corresponds to scaling of the weights sampler in the shader. So translations and scaling can be ignored in the compositor but gets passed to the shader for special behavior.

@Jeroen-Bakker To clarify, this is not a limitation of the implementation or the design, we can replicate the existing behavior if that was decided without much consideration. I am talking about what makes sense to the user in the context of infinite canvas compositing. The problem with the current approach is that it defines a certain domain for the bokeh input that the user have to work around, for instance, in the cases presented above. Why should a rotated square produce a hexagonal bokeh result? Why does a shift in the bokeh change its shape? When one thinks about infinite canvas compositing, one shouldn't worry about those questions, which is my concern in this discussion. The full frame compositor actually breaks backward compatibility to solve some of those concerns, for instance, rotation doesn't introduce clipping in the full frame compositor, so a rotated square produces a rhombus correctly. I think this is a step in the right direction, that behavior should be kept and appropriate versioning should be implemented instead. Translation and scaling still doesn't work for the full frame compositor though. Ideally, I think the bokeh operation should have specializations to handle transformations itself. For instance, translations could corresponds to an offset in the blur window in the shader, while scaling could corresponds to scaling of the weights sampler in the shader. So translations and scaling can be ignored in the compositor but gets passed to the shader for special behavior.

Vyacheslav Kobozev commented

2022-11-11 19:46:47 +01:00

In #102252#1444165, @OmarEmaraDev wrote:
@Vyach The file you shared seems to give the expected outcome in the realtime compositor, what do you expect to happen?

I expect the same, but without initial mixing with itself only for achieving ability to transform.
And it is just a demo, that shows, that rotation/translation/scale are useful for any image, including bokeh mask.

> In #102252#1444165, @OmarEmaraDev wrote: > @Vyach The file you shared seems to give the expected outcome in the realtime compositor, what do you expect to happen? I expect the same, but without initial mixing with itself only for achieving ability to transform. And it is just a demo, that shows, that rotation/translation/scale are useful for any image, including bokeh mask.

Philipp Oeser removed the

Download

What's New

Blender Studio

Manual

Developers Blog

Documentation

Benchmark

Blender Conference

Development Fund

One-time Donations

Realtime compositor: bokeh blur do not work with transformed image #102252