Viewport Compositor: Render passes support #108618

New Issue

Sergey Sharybin · 2023-06-05T12:57:48+02:00

Sergey Sharybin commented

2023-06-05 12:57:48 +02:00

Scope

This design task describes integration of multiple render passes of the same view layer to the viewport compositor.

While ultimately multiple view layers would need to be supported as well, it will be handled separately as there more design considerations to be solved to keep performance and memory usage under control/

Overview

The overall overview of the proposal is quite simple:

Render engine provides GPU textures for all render passes.
The compositing is done by the draw manager from the Blender side.

Breakdown

GPU texture management

The render engine manages GPU textures on the engine side, and allows them to be bound for the viewport compositor shaders. This allows the engine to have full control over data management and synchronization for updates. For example, it allows engine to do proper initialization order for the graphics interoperability.

Overlays

Some of the data which a render engine wants to visualize does not fit into the compositing pipeline, and is better suitable as an overlay. On example of such visualization is the Active Pixels (available with Cycles Debug).

Such overlays will be moved from "being" baked into the combined pass and instead will become a dedicated call in the render engine draw_overlay().

On the render engine side it will behave the same way as the current draw(): "just" draw a texture(s) with overlays.

Data structures

The proposal is to base the data passed in the API on the RenderResult, and not to create a dedicated specialized data structures. This will allow maximum code re-usaiblity between the viewport, viewport compositor, final renders, and eventually GPU compositor for the final renders.

There are changes which needs to be done in the RenderResult in order to support this, with the final goal is to allow RenderResult to point to a GPU texture instead of CPU-side memory buffer. The proposal here is to:

Make RenderResult use ImBuf to store passes data.
Make ImBuf have GPU texture without CPU side buffer.

Considerations for such decision:

The RenderResult is heavily integrated into the image drawing (for the Render Result image data-block, multi-layer EXR images). If the RenderResult stores data as ImBuf then it will simplify drawing, which will no longer need to do shallow copies.
The Image data-block already uses GPU textures, which can be moved to the ImBuf.

Synchronization

Update of the GPU texture and its use by the draw manager happens from different thread, . This is already the case where DisplayDriver takes care of synchronization.

To start with the synchronization strategy will not change:

The render engine will lock the GPU textures at the moment when it knows its internal render buffer is in a consistent state (in case of Cycles: when the entire wavefront finished sampling), and will copy data to the GPU textures.
From within the same lock the render engine will copy data needed for the overlays.
Once the data is copied, the render engine releases the textures lock.
When draw manager requests RenderResult it will lock the textures, and will hold the lock until the drawing and compositing pipeline is finished.

This potentially introduces wait-on-a-lock for the render thread: if the compositing takes longer than sampling.

When CPU rendering is used it is relatively easy and beneficial to allow "interleaving": render more samples while the draw manager is still busy with compositing. In the case of Cycles it will mean that the PathTrace will need to take time spent in drawing into account and schedule more samples to fill it in.

For the GPU rendering the story is more complicated:

If the rendering and compositing done on the same GPU the interleaving might not be desired at all, as a lot of supported GPUs do not allow interleaving GPGPU and viewport rendering without performance impact.
If the rendering is happening on a different GPU than viewport rendering then interleaving is possible.

For the latter either engine could do a guess which GPU is used for viewport, or it would need to be a dedicated API call to provide this information explicitly.

Used render passes

Currently for the viewport render the used render passes are in the full control of the render engine. This needs to be changed to something:

Render engine provides list of supported render passes
The viewport compositor provides information which of them are actually needed for the viewport

The former is probably already covered with the render passes integration used to create sockets in the Render Layers nodes.

The latter one is needed to make it so unused passes do not take memory: for example if some passes are only used to write to file.

The render pass display option (currently available in Cycles, and allows to choose which render pass is displayed in the viewport) will behave more like it does in the Image Editor where it is possible to choose between Composite output and View Layer / Passes, can then can work for any render engine.

# Scope This design task describes integration of multiple render passes of the same view layer to the viewport compositor. While ultimately multiple view layers would need to be supported as well, it will be handled separately as there more design considerations to be solved to keep performance and memory usage under control/ # Overview The overall overview of the proposal is quite simple: - Render engine provides GPU textures for all render passes. - The compositing is done by the draw manager from the Blender side. # Breakdown ## GPU texture management The render engine manages GPU textures on the engine side, and allows them to be bound for the viewport compositor shaders. This allows the engine to have full control over data management and synchronization for updates. For example, it allows engine to do proper initialization order for the graphics interoperability. ## Overlays Some of the data which a render engine wants to visualize does not fit into the compositing pipeline, and is better suitable as an overlay. On example of such visualization is the `Active Pixels` (available with `Cycles Debug`). Such overlays will be moved from "being" baked into the combined pass and instead will become a dedicated call in the render engine `draw_overlay()`. On the render engine side it will behave the same way as the current `draw()`: "just" draw a texture(s) with overlays. ## Data structures The proposal is to base the data passed in the API on the `RenderResult`, and not to create a dedicated specialized data structures. This will allow maximum code re-usaiblity between the viewport, viewport compositor, final renders, and eventually GPU compositor for the final renders. There are changes which needs to be done in the `RenderResult` in order to support this, with the final goal is to allow `RenderResult` to point to a GPU texture instead of CPU-side memory buffer. The proposal here is to: * Make `RenderResult` use `ImBuf` to store passes data. * Make `ImBuf` have GPU texture without CPU side buffer. Considerations for such decision: * The `RenderResult` is heavily integrated into the image drawing (for the `Render Result` image data-block, multi-layer EXR images). If the `RenderResult` stores data as `ImBuf` then it will simplify drawing, which will no longer need to do shallow copies. * The `Image` data-block already uses GPU textures, which can be moved to the `ImBuf`. ## Synchronization Update of the GPU texture and its use by the draw manager happens from different thread, . This is already the case where `DisplayDriver` takes care of synchronization. To start with the synchronization strategy will not change: * The render engine will lock the GPU textures at the moment when it knows its internal render buffer is in a consistent state (in case of Cycles: when the entire wavefront finished sampling), and will copy data to the GPU textures. * From within the same lock the render engine will copy data needed for the overlays. * Once the data is copied, the render engine releases the textures lock. * When draw manager requests `RenderResult` it will lock the textures, and will hold the lock until the drawing and compositing pipeline is finished. This potentially introduces wait-on-a-lock for the render thread: if the compositing takes longer than sampling. When CPU rendering is used it is relatively easy and beneficial to allow "interleaving": render more samples while the draw manager is still busy with compositing. In the case of Cycles it will mean that the `PathTrace` will need to take time spent in drawing into account and schedule more samples to fill it in. For the GPU rendering the story is more complicated: * If the rendering and compositing done on the same GPU the interleaving might not be desired at all, as a lot of supported GPUs do not allow interleaving GPGPU and viewport rendering without performance impact. * If the rendering is happening on a different GPU than viewport rendering then interleaving is possible. For the latter either engine could do a guess which GPU is used for viewport, or it would need to be a dedicated API call to provide this information explicitly. ## Used render passes Currently for the viewport render the used render passes are in the full control of the render engine. This needs to be changed to something: * Render engine provides list of supported render passes * The viewport compositor provides information which of them are actually needed for the viewport The former is probably already covered with the render passes integration used to create sockets in the Render Layers nodes. The latter one is needed to make it so unused passes do not take memory: for example if some passes are only used to write to file. The render pass display option (currently available in Cycles, and allows to choose which render pass is displayed in the viewport) will behave more like it does in the Image Editor where it is possible to choose between Composite output and View Layer / Passes, can then can work for any render engine.

❤️ 5 🎉 4

Sergey Sharybin added the

Type

Design

label 2023-06-05 12:57:48 +02:00

Sergey Sharybin added this to the Compositing project 2023-06-05 12:57:55 +02:00

Sergey Sharybin added the

Download

What's New

Blender Studio

Manual

Developers Blog

Documentation

Benchmark

Blender Conference

Development Fund

One-time Donations

Viewport Compositor: Render passes support #108618