Compositor: canvas and transforms #108944

New Issue

Open

opened 2023-06-13 18:05:58 +02:00 by Brecht Van Lommel · 20 comments

Brecht Van Lommel commented 2023-06-13 18:05:58 +02:00

Owner

Copy Link

Motivation

The tiled, full-frame and realtime compositors currently behave differently regarding transforms. See A Comparative Look At Compositor Systems In Blender created by @OmarEmaraDev. COM_domain.hh in the realtime compositor also has good information on the current workings of that system.

We need consistent behavior across CPU and GPU compositing, and decide on the right behavior. Ideally we would like support all of the following:

• 3D viewport and final render should give consistent results relative to the camera frame.
• Automatic resolution independence for previewing or outputting and different resolutions.
• Overscan, where pixels exist beyond the frame for flexibility in compositing and avoiding border artifacts.
• Border render, where a render is only a subset of the full frame.
• Combining multiple images with different transforms on a shared canvas.
• Avoid clipping and losing image resolution where possible.
• OpenEXR data and display windows.
• Execute node graph in easy to understand way, without unexpected behavior.

Proposal

Data and Display Windows

Every image used in compositing is placed on an infinite canvas, and has the following properties:

• Data window: bounding box for the part of the canvas that contains pixel data.
• Display window: bounding box for the part of the image that will be displayed or output.

The data window may be bigger (for overscan) or smaller (for border render) than the display window. See the OpenEXR docs for illustrations.

By making this distinction, transform nodes can translate, rotate and scale images without clipping data that falls outside the display window. Such transforms would change the data window but not the display window.

Combining Images

All images that go into the compositor are placed on the same infinite canvas. Nodes like mix and alpha over by default output a data window that is the union of the two bounding boxes. However there can also be an option to use the intersection, or use the data window from either image.

The display window would be taken from the primary image input.

Realizing Transforms

The realtime compositor has a mechanism where affine transforms and wrapping are not immediately applied after every node. Rather they are stored as part of the image, and realized when an operation is performed that needs it.

This mechanism can remain, but clipping can be avoided by writing to a data window that may be partially or fully outside the display window.

One thing that we found confusing in its current behavior is that the transform is not realized before filter nodes like blur. That means rotating an image 90° and then applying a horizontal blur, the image is blurred vertically in global space. We suggest realizing such transforms before filter nodes. If users want the existing behavior, they can place the blur node before the rotation.

Performance and User Control

While the data window avoids losing data, it can also lead to high memory usage as users may (accidentally) make node setups that occupy many pixels outside the display window.

There are no ideal solutions for this, but we can think of ways to mitigate this:

• Nodes that work on a single image could have settings to keep the data window unchanged.
• Nodes that work on multiple images could have a setting to use the data window from just one of them.
• A default limit on data windows, set to some % beyond the display window?

Resolution Independence

Nodes should generally not use pixel coordinates, but rather coordinates relative to the image size or output size. The reason being that while changing the render resolution percentage could scale the size in pixel coordinates, this would not work when the user changes the render resolution.

Compositing nodes should be evaluated at a known resolution factor, either taken from the render resolution percentage, or computed automatically from the 3D viewport size. Images coming into the compositor should then be scaled and translated appropriately to match.

Viewing and Editing Windows

When using the viewer nodes, there should be a visual indication of what the data and display windows are in the resulting image.

There should also be nodes to modify the display window, and to clip the data window to the display window or non-empty pixels.

Open Questions

• Data windows and display windows may be difficult to understand for users. Are there ways to make this easier to understand, or defaults that mean users don't have to think about them much in simple cases?
• Can we find better terminology? The current terms come mainly from OpenEXR, and are also used in e.g. OpenImageIO. Another compositing application calls data window "bounding box" and display window "format". Image editors have some similar concepts where a layer may only take up part of the canvas, but I could not find a clear enough correspondence between concept to share terminology.
• This design assumes the display window is coming from the input render data or camera footage, and the compositor output will just follow this. But render output may expect a particular resolution, and some things may be simplified if the whole compositing nodes would use the display window from the render output. Still some other compositors seem to offer flexibility where the display window does not have to be the same throughout. What is the best solution here? Would it make sense to adapt any input images to the render output display window by default, and then make it an option to not do that for advanced users?
• Should nodes like blur also extend the data window automatically? This could be helpful for example if you want to blur an element you want to composite over another image, without having to manually increase its size first. On the other hand if you apply depth of field or motion blur to a complete render and not just an individual element, that's not what you want.

## Motivation The tiled, full-frame and realtime compositors currently behave differently regarding transforms. See [A Comparative Look At Compositor Systems In Blender](/attachments/92ec0f92-851e-4468-a463-a23bb2baf79d) created by @OmarEmaraDev. `COM_domain.hh` in the realtime compositor also has good information on the current workings of that system. We need consistent behavior across CPU and GPU compositing, and decide on the right behavior. Ideally we would like support all of the following: * 3D viewport and final render should give consistent results relative to the camera frame. * Automatic resolution independence for previewing or outputting and different resolutions. * Overscan, where pixels exist beyond the frame for flexibility in compositing and avoiding border artifacts. * Border render, where a render is only a subset of the full frame. * Combining multiple images with different transforms on a shared canvas. * Avoid clipping and losing image resolution where possible. * OpenEXR data and display windows. * Execute node graph in easy to understand way, without unexpected behavior. ## Proposal ### Data and Display Windows Every image used in compositing is placed on an infinite canvas, and has the following properties: * Data window: bounding box for the part of the canvas that contains pixel data. * Display window: bounding box for the part of the image that will be displayed or output. The data window may be bigger (for overscan) or smaller (for border render) than the display window. See the [OpenEXR docs](https://openexr.com/en/latest/TechnicalIntroduction.html#overview-of-the-openexr-file-format) for illustrations. By making this distinction, transform nodes can translate, rotate and scale images without clipping data that falls outside the display window. Such transforms would change the data window but not the display window. ### Combining Images All images that go into the compositor are placed on the same infinite canvas. Nodes like mix and alpha over by default output a data window that is the union of the two bounding boxes. However there can also be an option to use the intersection, or use the data window from either image. The display window would be taken from the primary image input. ### Realizing Transforms The realtime compositor has a mechanism where affine transforms and wrapping are not immediately applied after every node. Rather they are stored as part of the image, and realized when an operation is performed that needs it. This mechanism can remain, but clipping can be avoided by writing to a data window that may be partially or fully outside the display window. One thing that we found confusing in its current behavior is that the transform is not realized before filter nodes like blur. That means rotating an image 90° and then applying a horizontal blur, the image is blurred vertically in global space. We suggest realizing such transforms before filter nodes. If users want the existing behavior, they can place the blur node before the rotation. ### Performance and User Control While the data window avoids losing data, it can also lead to high memory usage as users may (accidentally) make node setups that occupy many pixels outside the display window. There are no ideal solutions for this, but we can think of ways to mitigate this: * Nodes that work on a single image could have settings to keep the data window unchanged. * Nodes that work on multiple images could have a setting to use the data window from just one of them. * A default limit on data windows, set to some % beyond the display window? ### Resolution Independence Nodes should generally not use pixel coordinates, but rather coordinates relative to the image size or output size. The reason being that while changing the render resolution percentage could scale the size in pixel coordinates, this would not work when the user changes the render resolution. Compositing nodes should be evaluated at a known resolution factor, either taken from the render resolution percentage, or computed automatically from the 3D viewport size. Images coming into the compositor should then be scaled and translated appropriately to match. ### Viewing and Editing Windows When using the viewer nodes, there should be a visual indication of what the data and display windows are in the resulting image. There should also be nodes to modify the display window, and to clip the data window to the display window or non-empty pixels. ### Open Questions * Data windows and display windows may be difficult to understand for users. Are there ways to make this easier to understand, or defaults that mean users don't have to think about them much in simple cases? * Can we find better terminology? The current terms come mainly from OpenEXR, and are also used in e.g. OpenImageIO. Another compositing application calls data window "bounding box" and display window "format". Image editors have some similar concepts where a layer may only take up part of the canvas, but I could not find a clear enough correspondence between concept to share terminology. * This design assumes the display window is coming from the input render data or camera footage, and the compositor output will just follow this. But render output may expect a particular resolution, and some things may be simplified if the whole compositing nodes would use the display window from the render output. Still some other compositors seem to offer flexibility where the display window does not have to be the same throughout. What is the best solution here? Would it make sense to adapt any input images to the render output display window by default, and then make it an option to not do that for advanced users? * Should nodes like blur also extend the data window automatically? This could be helpful for example if you want to blur an element you want to composite over another image, without having to manually increase its size first. On the other hand if you apply depth of field or motion blur to a complete render and not just an individual element, that's not what you want.

aComparativeLookAtCompositorSystemsInBlender.pdf

22 MiB

👀 3 ❤️ 1

Brecht Van Lommel added the

Interest

Compositing

Type

Design

Module

VFX & Video

labels 2023-06-13 18:05:58 +02:00

Brecht Van Lommel added this to the Compositing project 2023-06-13 18:05:59 +02:00

Brecht Van Lommel commented 2023-06-13 18:10:16 +02:00

Author

Owner

Copy Link

CC @Sergey @OmarEmaraDev @Jeroen-Bakker @zazizizou.

Sergey and I reviewed Omar's document and discussed the design. This is my view on things but I'm not so confident on the right design here yet.

I think the way the realtime compositor delays realizing transforms for some types of nodes is unwanted, and we could add user control over this. But then if you have to do that you can be back to the situation where things get clipped when you don't want to. Data windows can provide a solution to that.

CC @Sergey @OmarEmaraDev @Jeroen-Bakker @zazizizou. Sergey and I reviewed Omar's document and discussed the design. This is my view on things but I'm not so confident on the right design here yet. I think the way the realtime compositor delays realizing transforms for some types of nodes is unwanted, and we could add user control over this. But then if you have to do that you can be back to the situation where things get clipped when you don't want to. Data windows can provide a solution to that.

Brecht Van Lommel commented 2023-06-20 18:23:15 +02:00

Author

Owner

Copy Link

A few thoughts after the meeting we had about this today.

Realizing Transforms before Filters

An alternative to always realizing the transform before filter operations like blur would be to have an explicit realize transform node. The effective difference with this proposal would be in the defaults. Either filter operations work in local space by default and to making them work in global space requires adding realize transform node. Or filter options always work in global space and users may have to reorder nodes to get the behavior they want.

Besides a realize transform node, the transform nodes could also have an option to be realized immediately or not.

Rotations could be always realized, but translations and scale could be deferred. Operations like blur could scale the filter size for example, and avoid resolution loss in some cases where immediately realizing would be a problem.

Practical Use Cases

We need some artists feedback and evaluate things in the context of practical use cases. We can try to be clever with regions of interest like the tiled based compositor or deferring realizing transforms, but the complexity these things add is not necessarily helpful. Avoiding unexpectedly high memory usage and resolution loss is a concern, but not the only one.

A few different example scenarios:

• Combining render layers that all have the same display and data window, without using any transforms. This is a common case and comparatively easy. Support for overscan can help here to get better quality at the edges.
• Compositing elements (e.g. smoke, explosions, speed lines, ..) on top of a camera footage that matches the output resolution. In this case we want to be able to transform the elements to position within the camera frame. We want to avoid clipping of the elements as they are rotated, scaled and blurred. Only clipping outside the output camera frame + overscan could be acceptable. A smart region of interest mechanism could reduce the computation cost and memory usage by discarding parts outside the camera frame, but it's also not super likely to be a big factor, and manual cropping before scaling could also work. In this case we may also want to apply operations like distortion or motion blur on all elements in the same global space, with coordinates relative to the camera frame.
• Using the compositor as a general image processor, where a bunch of file output nodes write out images at varying resolutions. For example someone might write out sprites for a game, write out the same image at multiple resolutions, stitch together a cubemap, etc. In this case the output camera frame is irrelevant. If we want to support this use case, the display window would need to be per image rather than purely defined by the scene or compositor or viewer nodes, since it would be more convenient to just write out an image with whatever display and data window was fed into the file output node.

Different Types of Transforms

In some cases you want a transform to affect both the display and data window, and in others you want only the data window to be affected.

If the display window of an image is set to the output camera frame, most likely want to only change the data window and keep the camera frame intact. For example if you recorded camera footage with overscan, you can add some camera shake.

On the other hand if you load in an image of e.g. smoke to composite over camera footage, the display window could just be the dimensions of the image itself, in which case you do want to change both the display and data window. On the other hand if the element already includes the display window matching the camera output, this would not be so helpful.

Transform nodes could have an option for this.

An alternative could be that images can either have their own display window, or follow the output camera frame. In the former case the display window would be affected by transforms, in the latter case not. Image input notes then would need some setting to control if they should get the display window from the scene, or keep their own. This may be a difficult concept to explain to users.

A few thoughts after the meeting we had about this today. ### Realizing Transforms before Filters An alternative to always realizing the transform before filter operations like blur would be to have an explicit realize transform node. The effective difference with this proposal would be in the defaults. Either filter operations work in local space by default and to making them work in global space requires adding realize transform node. Or filter options always work in global space and users may have to reorder nodes to get the behavior they want. Besides a realize transform node, the transform nodes could also have an option to be realized immediately or not. Rotations could be always realized, but translations and scale could be deferred. Operations like blur could scale the filter size for example, and avoid resolution loss in some cases where immediately realizing would be a problem. ### Practical Use Cases We need some artists feedback and evaluate things in the context of practical use cases. We can try to be clever with regions of interest like the tiled based compositor or deferring realizing transforms, but the complexity these things add is not necessarily helpful. Avoiding unexpectedly high memory usage and resolution loss is a concern, but not the only one. A few different example scenarios: * Combining render layers that all have the same display and data window, without using any transforms. This is a common case and comparatively easy. Support for overscan can help here to get better quality at the edges. * Compositing elements (e.g. smoke, explosions, speed lines, ..) on top of a camera footage that matches the output resolution. In this case we want to be able to transform the elements to position within the camera frame. We want to avoid clipping of the elements as they are rotated, scaled and blurred. Only clipping outside the output camera frame + overscan could be acceptable. A smart region of interest mechanism could reduce the computation cost and memory usage by discarding parts outside the camera frame, but it's also not super likely to be a big factor, and manual cropping before scaling could also work. In this case we may also want to apply operations like distortion or motion blur on all elements in the same global space, with coordinates relative to the camera frame. * Using the compositor as a general image processor, where a bunch of file output nodes write out images at varying resolutions. For example someone might write out sprites for a game, write out the same image at multiple resolutions, stitch together a cubemap, etc. In this case the output camera frame is irrelevant. If we want to support this use case, the display window would need to be per image rather than purely defined by the scene or compositor or viewer nodes, since it would be more convenient to just write out an image with whatever display and data window was fed into the file output node. ### Different Types of Transforms In some cases you want a transform to affect both the display and data window, and in others you want only the data window to be affected. If the display window of an image is set to the output camera frame, most likely want to only change the data window and keep the camera frame intact. For example if you recorded camera footage with overscan, you can add some camera shake. On the other hand if you load in an image of e.g. smoke to composite over camera footage, the display window could just be the dimensions of the image itself, in which case you do want to change both the display and data window. On the other hand if the element already includes the display window matching the camera output, this would not be so helpful. Transform nodes could have an option for this. An alternative could be that images can either have their own display window, or follow the output camera frame. In the former case the display window would be affected by transforms, in the latter case not. Image input notes then would need some setting to control if they should get the display window from the scene, or keep their own. This may be a difficult concept to explain to users.

Frank Meier commented 2023-06-20 18:51:53 +02:00

Copy Link

The data/display windows are a common concept many canvas compositors are based on. They are usually called region-of-definition (ROD) and region-of-interest (ROI), e.g. in Nuke's documentation or in OpenFX. The latter also covers an algorithm to determine minimum ROD/ROI sizes so that clipping by default can be avoided (which would be preferable). In Nuke the ROD size can also be set by the user, but that is optional.

This paper has some more details.

The data/display windows are a common concept many canvas compositors are based on. They are usually called region-of-definition (ROD) and region-of-interest (ROI), e.g. in Nuke's documentation or in [OpenFX](https://openfx.readthedocs.io/en/master/Reference/ofxProcessingArch.html). The latter also covers an algorithm to determine minimum ROD/ROI sizes so that clipping by default can be avoided (which would be preferable). In Nuke the ROD size can also be set by the user, but that is optional. [This paper](https://dl.acm.org/doi/10.1145/192161.192191) has some more details.

Brecht Van Lommel commented 2023-06-20 19:51:22 +02:00

Author

Owner

Copy Link

Yes, in the Nuke user interface it is called Format and Bounding Box, which is what I was alluding to earlier.

The current tiled compositor is actually based on that paper you linked. But the architecture is inefficient on today's CPUs and GPUs, which is the main reason we are switching. I haven't checked to what extent this inefficiency is due to the paper or the Blender implementation. But out-of-core processing and realtime are not compatible with each other, and nowadays there is enough memory to have multiple full resolution images in memory.

Yes, in the Nuke user interface it is called Format and Bounding Box, which is what I was alluding to earlier. The current tiled compositor is actually based on that paper you linked. But the architecture is inefficient on today's CPUs and GPUs, which is the main reason we are switching. I haven't checked to what extent this inefficiency is due to the paper or the Blender implementation. But out-of-core processing and realtime are not compatible with each other, and nowadays there is enough memory to have multiple full resolution images in memory.

Omar Emara commented 2023-08-18 15:01:26 +02:00

Member

Copy Link

We created two test builds in order to have some practical testing grounds for some of the proposals here.

Realized Rotation Build

This build is built from #111213 and can be downloaded from here.

In this first build, we now always realize the rotation of the primary input before executing operations that are not rotation invariant, like most filter operations. This realization takes the form of creating a new image that completely contains the rotated image with transparency everywhere else. This is illustrated in the following example:

Those operations will now operate on their inputs in the rotational global space. For instance, a 45 degree rotated image that is blurred along its x axis will now be blurred along it diagonal, instead of its local x axis. This is illustrated in the following example:

Realized Rotation And Scale Build

This build is built from #111216 and can be downloaded from here.

In this second build, in addition to rotation, we now also always realize the scale of the primary input before executing operations that are not scale invariant, like most filter operations. This realization takes the form of actually increasing or decreasing the resolution of the images based on their scale factor. So for instance, a 512x512 image that is scaled down by a factor of 0.2 will be of resolution 103x103. This is illustrated in the following example:

---

---

Automatic realization of translation is probably not something that should be done, so we have't considered it in our tests builds. But if there are concerns regarding translations please bring them up as well.

In our previous discussions, there was a consensus that automatic realization of rotation was necessary, so this change will likely make it to the main build soon enough. If there are objections or concerns, please also bring them up.

Automatic realization of scale is the main point of discussion here due to the concerns and alternatives outlined above, so we would to hear feedback on this with practical examples in its favor or against it.

We created two test builds in order to have some practical testing grounds for some of the proposals here. ## Realized Rotation Build This build is built from #111213 and can be downloaded from [here](https://builder.blender.org/download/patch/PR111213/). In this first build, we now always realize the rotation of the primary input before executing operations that are not rotation invariant, like most filter operations. This realization takes the form of creating a new image that completely contains the rotated image with transparency everywhere else. This is illustrated in the following example:  Those operations will now operate on their inputs in the rotational global space. For instance, a 45 degree rotated image that is blurred along its x axis will now be blurred along it diagonal, instead of its local x axis. This is illustrated in the following example:  ## Realized Rotation And Scale Build This build is built from #111216 and can be downloaded from [here](https://builder.blender.org/download/patch/PR111216/). In this second build, in addition to rotation, we now also always realize the scale of the primary input before executing operations that are not scale invariant, like most filter operations. This realization takes the form of actually increasing or decreasing the resolution of the images based on their scale factor. So for instance, a 512x512 image that is scaled down by a factor of 0.2 will be of resolution 103x103. This is illustrated in the following example:  ----- ----- Automatic realization of translation is probably not something that should be done, so we have't considered it in our tests builds. But if there are concerns regarding translations please bring them up as well. In our previous discussions, there was a consensus that automatic realization of rotation was necessary, so this change will likely make it to the main build soon enough. If there are objections or concerns, please also bring them up. Automatic realization of scale is the main point of discussion here due to the concerns and alternatives outlined above, so we would to hear feedback on this with practical examples in its favor or against it.

20230818-152713.png

1.2 MiB

20230818-153321.png

755 KiB

20230818-154357.png

498 KiB

Omar Emara commented 2023-08-18 15:45:54 +02:00

Member

Copy Link

Here are some notes and concerns on the aforementioned builds.

Deferred Automatic Realization Of Rotation Is Not Side Effect Free

Deferring realization of rotation but at the same time automatically realizing it on non rotation invariant operations would result in the following behavior. Pixel-wise operations that care about transparency and follow the non rotation invariant operations would produce different results had the realization not happened. Because, as mentioned, realizing rotation produces an image with a bigger bounding box that has transparency at the boundary. This examples should clarify this point:

Muting the filter nodes removes the white background, which is probably unexpected to the user. This is because the realization no longer happens, so the transparent areas no longer exist for the Alpha Over node to act on.

As far as I can see, the only solution would be to immediately realize the rotation at the transform nodes and not defer them. Though I have not studied the consequences of this yet.

The Size Of Rotated Images Changes As Rotation Changes

The size of images oscillate between 1x and sqrt(2)x as the user rotates the image. To realize why this might be a problem. Consider the example where a user uses a rotated image as Bokeh weights in the Bokeh Blur node. As the user changes the rotation amount, the size of the blur will apparently oscillate, which might not be clear to the user. This is illustrated in the following example:

Notice how the size of the Bokeh blur oscillates as the image rotates. Please ignore the fact that the Bokeh Image node has a rotation option, it is just a demonstration. :)

Here are some notes and concerns on the aforementioned builds. ### Deferred Automatic Realization Of Rotation Is Not Side Effect Free Deferring realization of rotation but at the same time automatically realizing it on non rotation invariant operations would result in the following behavior. Pixel-wise operations that care about transparency and follow the non rotation invariant operations would produce different results had the realization not happened. Because, as mentioned, realizing rotation produces an image with a bigger bounding box that has transparency at the boundary. This examples should clarify this point: <video src="/attachments/cb5c188d-25f4-4a09-8454-5182492cf60c" loop controls></video> Muting the filter nodes removes the white background, which is probably unexpected to the user. This is because the realization no longer happens, so the transparent areas no longer exist for the Alpha Over node to act on. As far as I can see, the only solution would be to immediately realize the rotation at the transform nodes and not defer them. Though I have not studied the consequences of this yet. ### The Size Of Rotated Images Changes As Rotation Changes The size of images oscillate between `1x` and `sqrt(2)x` as the user rotates the image. To realize why this might be a problem. Consider the example where a user uses a rotated image as Bokeh weights in the Bokeh Blur node. As the user changes the rotation amount, the size of the blur will apparently oscillate, which might not be clear to the user. This is illustrated in the following example: <video src="/attachments/2c6cb431-65f0-4701-8c32-34632e329151" loop controls></video> Notice how the size of the Bokeh blur oscillates as the image rotates. Please ignore the fact that the Bokeh Image node has a rotation option, it is just a demonstration. :)

20230818-162919.mp4

56 KiB

20230818-161249.mp4

450 KiB

Sergey Sharybin referenced this issue 2023-08-28 11:03:04 +02:00

Compositor: Re-organize Add menu #111481

Sergey Sharybin commented 2023-09-06 12:25:43 +02:00

Owner

Copy Link

To me the behavior of the #111216 seems to be the most intuitive on a user level.

As for the issue of "disappearing" white background, intuitively I'd always expect the white background. So, perhaps any socket which is not from a transform node should do realization?

To me the behavior of the #111216 seems to be the most intuitive on a user level. As for the issue of "disappearing" white background, intuitively I'd always expect the white background. So, perhaps any socket which is not from a transform node should do realization?

Omar Emara commented 2023-09-06 18:05:45 +02:00

Member

Copy Link

So, perhaps any socket which is not from a transform node should do realization?

I assume you mean to realize the rotation automatically for all nodes except Transform nodes?
Wouldn't this practically be the same as realizing immediately at the transform node that introduced the rotation? Since it wouldn't really make sense for the user to have two consecutive rotate nodes.

> So, perhaps any socket which is not from a transform node should do realization? I assume you mean to realize the rotation automatically for all nodes except Transform nodes? Wouldn't this practically be the same as realizing immediately at the transform node that introduced the rotation? Since it wouldn't really make sense for the user to have two consecutive rotate nodes.

Sergey Sharybin commented 2023-09-06 18:17:27 +02:00

Owner

Copy Link

If we can combine all consecutive transform nodes and realize transform only once this sounds like a good idea from the performance perspective. So if we have a chain like Image -> Scale -> Rotate -> Blur -> Output ideally we'll only perform actual image transform once, which will include scale and rotation in its transform.

If we can combine all consecutive transform nodes and realize transform only once this sounds like a good idea from the performance perspective. So if we have a chain like `Image -> Scale -> Rotate -> Blur -> Output` ideally we'll only perform actual image transform once, which will include scale and rotation in its transform.

Brecht Van Lommel commented 2023-09-06 19:31:38 +02:00

Author

Owner

Copy Link

It makes sense to me to realize before any non-transform node, to avoid those side effects. I still think that from the user point of view evaluation should be from left to right without side effects, and if users wants something else they can reorder the nodes.

For the size changes with bokeh, I think we need an option on transform nodes to control if the size should change or not. To me both behaviors seems useful.

If we add data/display windows there can be an additional option to keep data outside the display window, so that the bokeh node is unaffected by the data window size change and only looks at the display window.

It makes sense to me to realize before any non-transform node, to avoid those side effects. I still think that from the user point of view evaluation should be from left to right without side effects, and if users wants something else they can reorder the nodes. For the size changes with bokeh, I think we need an option on transform nodes to control if the size should change or not. To me both behaviors seems useful. If we add data/display windows there can be an additional option to keep data outside the display window, so that the bokeh node is unaffected by the data window size change and only looks at the display window.

Sebastian Koenig commented 2023-09-07 13:03:49 +02:00

Member

Copy Link

From the perspective from a user, who is not smart enough to fully grasp the concept of deferred realization, it should just be how Brecht described: evaluation from left to right without side-effects. Probably easier said than done :D
It's great that the direction of blur after a rotation node is now correct.
Though rotating an image still behaves weird:

1. Alpha-overing a rotated image over a bigger image works fine.
2. As soon as I put a filter behind the rotate, I get an unwanted result.
3. Even weirder when I look at the image directly. I suppose this is because each image has it's own canvas window or so. But for a user it's really annoying.

But you probably have that on your agenda anyway, so nevermind. :)

From the perspective from a user, who is not smart enough to fully grasp the concept of deferred realization, it should just be how Brecht described: evaluation from left to right without side-effects. Probably easier said than done :D It's great that the direction of blur after a rotation node is now correct. Though rotating an image still behaves weird: 1. Alpha-overing a rotated image over a bigger image works fine.  2. As soon as I put a filter behind the rotate, I get an unwanted result.  3. Even weirder when I look at the image directly. I suppose this is because each image has it's own canvas window or so. But for a user it's really annoying. But you probably have that on your agenda anyway, so nevermind. :)

2_compositor_canvas_rotate_blur.jpg

69 KiB

1_compositor_canvas_rotate_no_filter.jpg

111 KiB

3_compositor_canvas_rotate_direct.jpg

84 KiB

Omar Emara commented 2023-09-07 17:53:27 +02:00

Member

Copy Link

@sebastian_k Are you sure you are using the Realtime Compositor for those tests? Since it shouldn't behave in the way you showed in the second and third images.

@brecht @Sergey Sounds good then. Design-wise, it should be clear. Though technically, realizing before any non-transform node would be problematic because if a rotate node has multiple outgoing links, the exact realization might happen multiple times, so I need to figure out a way to optimize this out.

@sebastian_k Are you sure you are using the Realtime Compositor for those tests? Since it shouldn't behave in the way you showed in the second and third images. @brecht @Sergey Sounds good then. Design-wise, it should be clear. Though technically, realizing before any non-transform node would be problematic because if a rotate node has multiple outgoing links, the exact realization might happen multiple times, so I need to figure out a way to optimize this out.

Sean Kennedy commented 2023-09-08 00:43:27 +02:00

Member

Copy Link

I just did the same test as @sebastian_k but in the realtime compositor, and yes, the bounding box on the foreground layer doesn't seem to be an issue. But the deferred realization definitely is (which is expected after reading all the above).

I just did the same test as @sebastian_k but in the realtime compositor, and yes, the bounding box on the foreground layer doesn't seem to be an issue. But the deferred realization definitely is (which is expected after reading all the above).

blender_compositor_2023_screenshot_01.jpg

959 KiB

blender_compositor_2023_screenshot_02.jpg

598 KiB

Sebastian Koenig commented 2023-09-08 13:34:05 +02:00

Member

Copy Link

@sebastian_k Are you sure you are using the Realtime Compositor for those tests? Since it shouldn't behave in the way you showed in the second and third images.

Oops, sorry, indeed I was testing the normal compositor. Sorry for that.

Now I tried again in the realtime compositor in the viewport, but still getting strange results.
The blur direction seems to be fine with both of the builds you posted, but a large blur still results in artifacts at the edges.

Strangely blurring first and then rotating works okay in the normal compositor, but not in realtime.

> @sebastian_k Are you sure you are using the Realtime Compositor for those tests? Since it shouldn't behave in the way you showed in the second and third images. > Oops, sorry, indeed I was testing the normal compositor. Sorry for that. Now I tried again in the realtime compositor in the viewport, but still getting strange results. The blur direction seems to be fine with both of the builds you posted, but a large blur still results in artifacts at the edges.  Strangely blurring first and then rotating works okay in the normal compositor, but not in realtime. 

2_realtime_compositor_blur_rotate.jpg

172 KiB

1_realtime_compositor_rotate_blur.jpg

167 KiB

Brecht Van Lommel commented 2023-09-08 15:14:46 +02:00

Author

Owner

Copy Link

These results are expected, knowing that the realtime compositor just applies one operation after the other to the image left-to-right, while the old compositor reaches backwards in some cases.

For both issues, I think what you need is an option to either enlarge the image based on the blur radius and blur the edges, or keep the image size fixed and the edges sharp.

These results are expected, knowing that the realtime compositor just applies one operation after the other to the image left-to-right, while the old compositor reaches backwards in some cases. For both issues, I think what you need is an option to either enlarge the image based on the blur radius and blur the edges, or keep the image size fixed and the edges sharp.

Omar Emara commented 2023-09-08 16:05:05 +02:00

Member

Copy Link

@brecht @sebastian_k I think this is already implemented by the Extend Bounds option of the Blur node, at least for the first issue.

@brecht @sebastian_k I think this is already implemented by the Extend Bounds option of the Blur node, at least for the first issue.

Brecht Van Lommel commented 2023-09-08 16:19:19 +02:00

Author

Owner

Copy Link

@OmarEmaraDev ah indeed. It doesn't seem to work correctly with the GPU compositor, the edges get blurred but image size is unchanged.

@OmarEmaraDev ah indeed. It doesn't seem to work correctly with the GPU compositor, the edges get blurred but image size is unchanged.

Omar Emara commented 2023-09-08 16:23:17 +02:00

Member

Copy Link

It seems to work for me, maybe it is something else?

It seems to work for me, maybe it is something else?

20230908-172206.png

788 KiB

Brecht Van Lommel commented 2023-09-08 16:38:52 +02:00

Author

Owner

Copy Link

You're right. What I found is different behavior of the Viewer node when using the render compositor, nothing related to blur specifically. For GPU it seems to clip to the render resolution, for Tiled it uses the image as is.

You're right. What I found is different behavior of the Viewer node when using the render compositor, nothing related to blur specifically. For GPU it seems to clip to the render resolution, for Tiled it uses the image as is.

Omar Emara referenced this issue 2023-09-13 16:29:55 +02:00

Realtime Compositor: Immediately realize transformations #112332

Omar Emara referenced this issue 2023-09-25 10:25:20 +02:00

Compositor: Procedural texturing, Step 1 #112732

Habib Gahbiche commented 2023-10-31 21:09:14 +01:00

Member

Copy Link

I started looking into making the split viewer node a regular node, see #114245

I started looking into making the split viewer node a regular node, see #114245

Sergey Sharybin referenced this issue 2024-01-11 11:26:04 +01:00

Compositor: Short term planning #106533

Sergey Sharybin referenced this issue 2024-01-18 17:50:31 +01:00

Fix Split node in full-frame compositor when no inputs are connected #117282

Sign in to join this conversation.

No Branch/Tag Specified

Branches Tags

main

brush-assets-project

fix-121021

blender-v4.1-release

blender-v3.3-release

blender-v3.6-release

universal-scene-description

blender-v3.6-temp_wmoss_animrig_public

temp-sculpt-dyntopo

gpencil-next

anim/animation-id-113594

blender-v4.0-release

blender-projects-basics

bridge-curves

sculpt-blender

asset-browser-frontend-split

asset-shelf

tmp-usd-python-mtl

tmp-usd-3.6

blender-v3.5-release

blender-v2.93-release

realtime-clock

sculpt-dev

bevelv2

xr-dev

v4.1.1

v3.6.11

v3.3.18

v4.1.0

v3.3.17

v3.6.10

v3.6.9

v3.3.16

v3.6.8

v3.6.7

v3.3.14

v4.0.2

v4.0.1

v4.0.0

v3.6.5

v3.3.12

v3.6.4

v3.6.3

v3.3.11

v3.6.2

v3.3.10

v3.6.1

v3.3.9

v3.6.0

v3.3.8

v3.3.7

v2.93.18

v3.5.1

v3.3.6

v2.93.17

v3.5.0

v2.93.16

v3.3.5

v3.3.4

v2.93.15

v2.93.14

v3.3.3

v2.93.13

v2.93.12

v3.4.1

v3.3.2

v3.4.0

v3.3.1

v2.93.11

v3.3.0

v3.2.2

v2.93.10

v3.2.1

v3.2.0

v2.83.20

v2.93.9

v3.1.2

v3.1.1

v3.1.0

v2.83.19

v2.93.8

v3.0.1

v2.93.7

v3.0.0

v2.93.6

v2.93.5

v2.83.18

v2.93.4

v2.93.3

v2.83.17

v2.93.2

v2.93.1

v2.83.16

v2.93.0

v2.83.15

v2.83.14

v2.83.13

v2.92.0

v2.83.12

v2.91.2

v2.83.10

v2.91.0

v2.83.9

v2.83.8

v2.83.7

v2.90.1

v2.83.6.1

v2.83.6

v2.90.0

v2.83.5

v2.83.4

v2.83.3

v2.83.2

v2.83.1

v2.83

v2.82a

v2.82

v2.81a

v2.81

v2.80

v2.80-rc3

v2.80-rc2

v2.80-rc1

v2.79b

v2.79a

v2.79

v2.79-rc2

v2.79-rc1

v2.78c

v2.78b

v2.78a

v2.78

v2.78-rc2

v2.78-rc1

v2.77a

v2.77

v2.77-rc2

v2.77-rc1

v2.76b

v2.76a

v2.76

v2.76-rc3

v2.76-rc2

v2.76-rc1

v2.75a

v2.75

v2.75-rc2

v2.75-rc1

v2.74

v2.74-rc4

v2.74-rc3

v2.74-rc2

v2.74-rc1

v2.73a

v2.73

v2.73-rc1

v2.72b

2.72b

v2.72a

v2.72

v2.72-rc1

v2.71

v2.71-rc2

v2.71-rc1

v2.70a

v2.70

v2.70-rc2

v2.70-rc

v2.69

v2.68a

v2.68

v2.67b

v2.67a

v2.67

v2.66a

v2.66

v2.65a

v2.65

v2.64a

v2.64

v2.63a

v2.63

v2.61

v2.60a

v2.60

v2.59

v2.58a

v2.58

v2.57b

v2.57a

v2.57

v2.56a

v2.56

v2.55

v2.54

v2.53

v2.52

v2.51

v2.50

v2.49b

v2.49a

v2.49

v2.48a

v2.48

v2.47

v2.46

v2.45

v2.44

v2.43

v2.42a

v2.42

v2.41

v2.40

v2.37a

v2.37

v2.36

v2.35a

v2.35

v2.34

v2.33a

v2.33

v2.32

v2.31a

v2.31

v2.30

v2.28c

v2.28a

v2.28

v2.27

v2.26

v2.25

Labels

Clear labels   

Interest

Alembic

  

Interest

Animation & Rigging

  

Interest

Asset Browser

  

Interest

Asset Browser Project Overview

  

Interest

Audio

  

Interest

Automated Testing

  

Interest

Blender Asset Bundle

  

Interest

BlendFile

  

Interest

Collada

  

Interest

Compatibility

This issue affects/is about backward or forward compatibility

  

Interest

Compositing

  

Interest

Core

  

Interest

Cycles

  

Interest

Dependency Graph

  

Interest

Development Management

  

Interest

EEVEE

  

Interest

EEVEE & Viewport

  

Interest

Freestyle

  

Interest

Geometry Nodes

  

Interest

Grease Pencil

  

Interest

ID Management

  

Interest

Images & Movies

  

Interest

Import Export

  

Interest

Line Art

  

Interest

Masking

  

Interest

Metal

  

Interest

Modeling

  

Interest

Modifiers

  

Interest

Motion Tracking

  

Interest

Nodes & Physics

  

Interest

OpenGL

  

Interest

Overlay

  

Interest

Overrides

  

Interest

Performance

  

Interest

Physics

  

Interest

Pipeline, Assets & IO

  

Interest

Platforms, Builds & Tests

  

Interest

Python API

  

Interest

Render & Cycles

  

Interest

Render Pipeline

  

Interest

Sculpt, Paint & Texture

  

Interest

Text Editor

  

Interest

Translations

  

Interest

Triaging

  

Interest

Undo

  

Interest

USD

  

Interest

User Interface

  

Interest

UV Editing

  

Interest

VFX & Video

  

Interest

Video Sequencer

  

Interest

Virtual Reality

  

Interest

Vulkan

  

Interest

Wayland

Issues relating to Wayland windowing support.

  

Interest

Workbench

  

Interest: X11

Issues relating to Xorg/X11 support.

  

Legacy

Blender 2.8 Project

  

Legacy

Milestone 1: Basic, Local Asset Browser

  

Legacy

OpenGL Error

  

Meta

Good First Issue

  

Meta

Papercut

  

Meta

Retrospective

  

Meta

Security

Issues relating to security: https://wiki.blender.org/wiki/Process/Vulnerability_Reports

  

Module

Animation & Rigging

  

Module

Core

  

Module

Development Management

  

Module

EEVEE & Viewport

  

Module

Grease Pencil

  

Module

Modeling

  

Module

Nodes & Physics

  

Module

Pipeline, Assets & IO

  

Module

Platforms, Builds & Tests

  

Module

Python API

  

Module

Render & Cycles

  

Module

Sculpt, Paint & Texture

  

Module

Triaging

  

Module

User Interface

  

Module

VFX & Video

  

Platform

FreeBSD

  

Platform

Linux

  

Platform

macOS

  

Platform

Windows

  

Priority

High

  

Priority

Low

  

Priority

Normal

  

Priority

Unbreak Now!

  

Status

Archived

  

Status

Confirmed

  

Status

Duplicate

  

Status

Needs Info from Developers

  

Status

Needs Information from User

  

Status

Needs Triage

  

Status

Resolved

  

Type

Bug

  

Type

Design

  

Type

Known Issue

  

Type

Patch

  

Type

Report

  

Type

To Do

No Label

Interest

Alembic

Interest

Animation & Rigging

Interest

Asset Browser

Interest

Asset Browser Project Overview

Interest

Audio

Interest

Automated Testing

Interest

Blender Asset Bundle

Interest

BlendFile

Interest

Collada

Interest

Compatibility

Interest

Compositing

Interest

Core

Interest

Cycles

Interest

Dependency Graph

Interest

Development Management

Interest

EEVEE

Interest

EEVEE & Viewport

Interest

Freestyle

Interest

Geometry Nodes

Interest

Grease Pencil

Interest

ID Management

Interest

Images & Movies

Interest

Import Export

Interest

Line Art

Interest

Masking

Interest

Metal

Interest

Modeling

Interest

Modifiers

Interest

Motion Tracking

Interest

Nodes & Physics

Interest

OpenGL

Interest

Overlay

Interest

Overrides

Interest

Performance

Interest

Physics

Interest

Pipeline, Assets & IO

Interest

Platforms, Builds & Tests

Interest

Python API

Interest

Render & Cycles

Interest

Render Pipeline

Interest

Sculpt, Paint & Texture

Interest

Text Editor

Interest

Translations

Interest

Triaging

Interest

Undo

Interest

USD

Interest

User Interface

Interest

UV Editing

Interest

VFX & Video

Interest

Video Sequencer

Interest

Virtual Reality

Interest

Vulkan

Interest

Wayland

Interest

Workbench

Interest: X11

Legacy

Blender 2.8 Project

Legacy

Milestone 1: Basic, Local Asset Browser

Legacy

OpenGL Error

Meta

Good First Issue

Meta

Papercut

Meta

Retrospective

Meta

Security

Module

Animation & Rigging

Module

Core

Module

Development Management

Module

EEVEE & Viewport

Module

Grease Pencil

Module

Modeling

Module

Nodes & Physics

Module

Pipeline, Assets & IO

Module

Platforms, Builds & Tests

Module

Python API

Module

Render & Cycles

Module

Sculpt, Paint & Texture

Module

Triaging

Module

User Interface

Module

VFX & Video

Platform

FreeBSD

Platform

Linux

Platform

macOS

Platform

Windows

Priority

High

Priority

Low

Priority

Normal

Priority

Unbreak Now!

Status

Archived

Status

Confirmed

Status

Duplicate

Status

Needs Info from Developers

Status

Needs Information from User

Status

Needs Triage

Status

Resolved

Type

Bug

Type

Design

Type

Known Issue

Type

Patch

Type

Report

Type

To Do

Milestone

Clear milestone

No items

No Milestone

Projects

Clear projects

No project

Compositing

Assignees

Clear assignees

No Assignees

7 Participants

Notifications

Subscribe

Due Date

The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: blender/blender#108944

No description provided.