Fix #114260: Compositor sometimes produces straight alpha #114305

Omar Emara · 2023-10-31T09:29:29+01:00

Omar Emara commented

2023-10-31 09:29:29 +01:00

The compositor sometimes produces straigh alpha even though
premultiplied alpha is expected. Moreover, there is an inconsitiency
between the CPU and GPU compositors.

For the GPU compositor, this is because GPU textures sometimes store
straight alpha, while the compositor always expects premultiplied alpha,
so we need to premultiply the alpha in those cases.

For the CPU compositor, this is because the image operation didn't
premultiply the alpha of byte textures, so we need to ensure
premultiplied alpha in those cases.

There is a data loss issue in case of byte images, since the IMB module
unpremultiplies premultiplied images then the compositior premultiplies
it again. But this will be handled in a different patch since it require
some design and refactoring first.

The compositor sometimes produces straigh alpha even though premultiplied alpha is expected. Moreover, there is an inconsitiency between the CPU and GPU compositors. For the GPU compositor, this is because GPU textures sometimes store straight alpha, while the compositor always expects premultiplied alpha, so we need to premultiply the alpha in those cases. For the CPU compositor, this is because the image operation didn't premultiply the alpha of byte textures, so we need to ensure premultiplied alpha in those cases. There is a data loss issue in case of byte images, since the IMB module unpremultiplies premultiplied images then the compositior premultiplies it again. But this will be handled in a different patch since it require some design and refactoring first.

❤️ 1

Omar Emara added 1 commit 2023-10-31 09:29:40 +01:00

0bae72783d Fix #114260 : Alpha handling is different in GPU compositor

Alpha in loaded differently in the GPU compositor relative to CPU
compositors.

This is because GPU textures sometimes store straigh alpha, while the
compositor always expects premultiplied alpha, so we need to premultiply
the alpha in those cases.

Omar Emara commented

2023-10-31 12:04:33 +01:00

@brecht @Sergey

If we have an 8-bit PNG image with emissive transparent color [1, 1, 1, 0] and an alpha mode of premultiplied. One would expect the final output to be [1, 1, 1, 0], but is in fact [0, 0, 0, 0] at the moment.

That's because the alpha handling will be as follows:

The PNG reader will load the color directly as [1, 1, 1, 0].
imb_handle_alpha will unpremultiply it to be [1, 1, 1, 0].
IMB_colormanagement_imbuf_to_float_texture in imb_gpu_get_data will premultiply the alpha to get [0, 0, 0, 0].

So reading such images involves some lossy operations. Any pointers on how we should handle that?

@brecht @Sergey If we have an 8-bit PNG image with emissive transparent color `[1, 1, 1, 0]` and an alpha mode of premultiplied. One would expect the final output to be `[1, 1, 1, 0]`, but is in fact `[0, 0, 0, 0]` at the moment. That's because the alpha handling will be as follows: 1. The PNG reader will load the color directly as `[1, 1, 1, 0]`. 2. `imb_handle_alpha` will unpremultiply it to be `[1, 1, 1, 0]`. 3. `IMB_colormanagement_imbuf_to_float_texture` in `imb_gpu_get_data` will premultiply the alpha to get `[0, 0, 0, 0]`. So reading such images involves some lossy operations. Any pointers on how we should handle that?

Brecht Van Lommel commented

2023-10-31 18:30:10 +01:00

Conversion between straight and premultiplied alpha is lossy in both directions. Both have values that the other can't represent. The compositor works with premultiplied alpha, which is the natural format for renders and EXR files. Following the current design, the lossy operation is expected.

Changing the Alpha setting on the image can be used to load the straight alpha unmodified, but it is not very clear and compositing nodes by default will not work correctly with such images.

With that in mind, I'm not sure I understand the question. Are you asking if we should improve support for loading straight alpha without data loss in the compositor, both for the CPU and GPU?

Conversion between straight and premultiplied alpha is lossy in both directions. Both have values that the other can't represent. The compositor works with premultiplied alpha, which is the natural format for renders and EXR files. Following the current design, the lossy operation is expected. Changing the Alpha setting on the image can be used to load the straight alpha unmodified, but it is not very clear and compositing nodes by default will not work correctly with such images. With that in mind, I'm not sure I understand the question. Are you asking if we should improve support for loading straight alpha without data loss in the compositor, both for the CPU and GPU?

Omar Emara commented

2023-10-31 19:18:19 +01:00

With that in mind, I'm not sure I understand the question. Are you asking if we should improve support for loading straight alpha without data loss in the compositor, both for the CPU and GPU?

Yes, but the problem I described above is about premultiplied alpha, not straight one. I realize that the conversion is an intrinsically lossy operation, my point is that the conversion is redundant, and thus the loss can be avoided.

In my example, the PNG image already contains premultiplied alpha, which the user indicated by choosing the premultiplied alpha mode. And since the compositor needs premultiplied alpha anyways, no conversion is needed here. Yet, we do two opposite operations to get the same data, except with data loss.

> With that in mind, I'm not sure I understand the question. Are you asking if we should improve support for loading straight alpha without data loss in the compositor, both for the CPU and GPU? Yes, but the problem I described above is about premultiplied alpha, not straight one. I realize that the conversion is an intrinsically lossy operation, my point is that the conversion is redundant, and thus the loss can be avoided. In my example, the PNG image already contains premultiplied alpha, which the user indicated by choosing the premultiplied alpha mode. And since the compositor needs premultiplied alpha anyways, no conversion is needed here. Yet, we do two opposite operations to get the same data, except with data loss.

Brecht Van Lommel commented

2023-10-31 20:01:59 +01:00

Ah yes. The issue is that by all byte buffers in ImBuf are straight alpha. Ideally we would support both premul and straight alpha images in these byte buffers, but that requires many changes throughout the code.

I can't immediately think of a good way to bypass that byte buffer.

Ah yes. The issue is that by all byte buffers in `ImBuf` are straight alpha. Ideally we would support both premul and straight alpha images in these byte buffers, but that requires many changes throughout the code. I can't immediately think of a good way to bypass that byte buffer.

Sergey Sharybin commented

2023-11-01 10:50:07 +01:00

To me it seems to be two separate topics. One of them being aligning the GPU compositor to the CPU within the current limitations of the 8bit image storage in Blender. And this is what the path is about, unless I am missing something? It marked as WIP, so not sure if it is ready for code/functional review?

The other one is the 8bit and alpha handling in Blender. Which is a very valid topic, but needs more clear design and will be a much bigger change in the code side.

To me it seems to be two separate topics. One of them being aligning the GPU compositor to the CPU within the current limitations of the 8bit image storage in Blender. And this is what the path is about, unless I am missing something? It marked as WIP, so not sure if it is ready for code/functional review? The other one is the 8bit and alpha handling in Blender. Which is a very valid topic, but needs more clear design and will be a much bigger change in the code side.

Omar Emara commented

2023-11-01 11:59:35 +01:00

@Sergey The two topics are related in that the GPU texture generation from images causes the aforementioned data loss, while the same data loss doesn't happen for the CPU compositor, considering it doesn't use that GPU code path. I still have this as a WIP because I am still investigating why the CPU compositor produces results with no data loss, and I suspect that's because its output is wrong, and a fix would produce the same data loss. More on this when I finish investigating.

Sergey Sharybin commented

2023-11-01 12:13:26 +01:00

@OmarEmaraDev COM_ImageOperation.cc, sample_image_at_location. It seems the the CPU compositor outputs image as-is, without associating the alpha channel sigh :'(

@OmarEmaraDev `COM_ImageOperation.cc`, `sample_image_at_location`. It seems the the CPU compositor outputs image as-is, without associating the alpha channel *sigh* :'(

Omar Emara commented

2023-11-01 12:20:04 +01:00

I guess that confirms my suspension. :)
So I will probably have to also update the CPU compositor to do the same thing as this patch.
Then we can handle the data loss separately.

I guess that confirms my suspension. :) So I will probably have to also update the CPU compositor to do the same thing as this patch. Then we can handle the data loss separately.

Omar Emara added 2 commits 2023-11-01 14:38:53 +01:00

80213b9e3d Merge branch 'main' into fix-114260

7677cfa2a7 Fix straight alpha output in CPU compositor

Omar Emara changed title from ~~WIP: Fix #114260: Alpha handling is different in GPU compositor~~ to Fix #114260: Compositor sometimes produces straight alpha

2023-11-01 14:40:04 +01:00

Omar Emara requested review from Sergey Sharybin 2023-11-01 14:57:15 +01:00

Omar Emara added the

Download

What's New

Blender Studio

Manual

Developers Blog

Documentation

Benchmark

Blender Conference

Development Fund

One-time Donations

Fix #114260: Compositor sometimes produces straight alpha #114305