Realtime Compositor #99210

This is a task that tracks the current state of the realtime compositor project and lists some of its to-dos. For user-level overview and demos, see the blog post on the matter. For feedback and discussions, use the devtalk topic for the project.

Description

The aim of this project is to develop a new GPU accelerated compositor that is both realtime and interactive. As a first step, this new compositor will be used to power the Viewport Compositor, which is a new feature that applies the compositor directly in the viewport. In the long term, the realtime compositor will be unified with the existing compositor, effectively accelerating the existing compositing workflow as well. This task is only concerned about the realtime compositor in the context of the viewport compositor.

Milestones

{icon circle color = green} Add a usable viewport compositor with basic input, processing, transform, and output nodes.
{icon circle color = yellow} Support basic filter nodes like blur, filter, lens distortion, glare, and other frequently used nodes.
{icon circle color = red} Support multi-pass compositing.
{icon circle color = red} Support the rest of the nodes with some exceptions.

The first milestone is already done and the target is to land the project in master as an experimental feature, which is currently in progress and undergoing review.

Code

Most of the code is in the temp-viewport-compositor-merge branch and a test build is available, but the code have since been split up into many smaller patches where development and review are taking place.

With other smaller patches that are not strictly related to the realtime compositor.

To Do

The following to-dos are not part of the first milestone and are thus not blockers for its merge to master.

Features
- - Single-axis domain repetition. Wait for D14366.
- - Bicubic interpolation. Reuse logic in node_tex_image_cubic.
- - Multi-pass compositing. Needs sizable changes in the draw module and render engines.
Robustness
- - Consider hardware capabilities when constructing shader compile units. Exceeding limits is unlikely due to scheduling strategy.
Performance
- - Allow nodes to be compiled into either node operations or shader operations depending on its options.
- - Cache GPU textures for multiple passes at the same time.
Portability
- - MacOS support. Needs #96261 (Metal Viewport ).
Quality
- - Specialized domain realization shaders to avoid fuzziness introduced by interpolation.
Documentation
- - Document the viewport compositor, its supported nodes, and its compatibility with the existing compositor.
Nodes
- See the following section.

Supported Nodes

Input
- - Render Layers.
- - Image.
- - Movie Clip.
- - RGB.
- - Value.
- - Time Curve.
- - Scene Time.
- - Bokeh Image.
- - Track Position.
- - ~~Texture~~.
- - Mask.
Output
- - Composite.
- - Viewer.
- - Split Viewer.
- - ~~Output File~~.
- - Levels.
Color
- - Mix RGB.
- - Alpha Over.
- - Invert.
- - RGB Curves.
- - Hue Saturation Value.
- - Color Balance.
- - Hue Correct.
- - Bright Contrast.
- - Gamma.
- - Exposure.
- - Color Correction.
- - Posterize.
- - Tone Map.
- - Z Combine.
Converter
- - Math.
- - Color Ramp.
- - Set Alpha.
- - Alpha Convert.
- - RGB To BW.
- - Separate Color.
- - Combine Color.
- - Separate XYZ.
- - Combine XYZ.
- - Switch View.
- - ID Mask.
- - Convert Color Space.
Filter
- - Bilateral Blur.
- - Despeckle.
- - Filter.
- - Directional Blur.
- - Dilate Erode.
- - Pixelate.
- - Bokeh Blur.
- - Blur.
- - Glare.
  - - Fog Glow.
- - Vector Blur.
- - Defocus.
- - Inpaint.
- - Sun Beams.
- - Denoise.
- - Anti-Aliasing.
Vector
- - Normal.
- - Map Value.
- - Map Range.
- - Vector Curves.
- - Normalize.
Matte
- - Channel Key.
- - Color Spill.
- - Box Mask.
- - Ellipse Mask.
- - Luminance Key.
- - Difference Key.
- - Distance Key.
- - Chroma Key.
- - Color Key.
- - ~~Legacy Cryptomatte~~.
- - Keying.
- - Double Edge Mask.
- - Keying Screen.
- - Cryptomatte.
Distort
- - Lens Distortion.
- - Translate.
- - Rotate.
- - Flip.
- - Crop.
- - Transform.
- - Scale.
- - Movie Distortion.
- - Displace.
- - Map UV.
- - Stabilize.
- - Plane Track Deform.
- - Corner Pin.

This is a task that tracks the current state of the realtime compositor project and lists some of its to-dos. For user-level overview and demos, see the [blog post](https:*code.blender.org/2022/07/real-time-compositor/) on the matter. For feedback and discussions, use the [devtalk topic](https:*devtalk.blender.org/t/real-time-compositor-feedback-and-discussion/) for the project. ## Description The aim of this project is to develop a new GPU accelerated compositor that is both realtime and interactive. As a first step, this new compositor will be used to power the *Viewport Compositor*, which is a new feature that applies the compositor directly in the viewport. In the long term, the realtime compositor will be unified with the existing compositor, effectively accelerating the existing compositing workflow as well. This task is only concerned about the realtime compositor in the context of the viewport compositor. ## Milestones - {icon circle color = green} Add a usable viewport compositor with basic input, processing, transform, and output nodes. - {icon circle color = yellow} Support basic filter nodes like blur, filter, lens distortion, glare, and other frequently used nodes. - {icon circle color = red} Support multi-pass compositing. - {icon circle color = red} Support the rest of the nodes with some exceptions. The first milestone is already done and the target is to land the project in master as an experimental feature, which is currently in progress and undergoing review. ## Code Most of the code is in the [temp-viewport-compositor-merge](https:*developer.blender.org/diffusion/B/history/temp-viewport-compositor-merge/) branch and a [test build](https:*builder.blender.org/download/experimental/temp-viewport-compositor-merge/) is available, but the code have since been split up into many smaller patches where development and review are taking place. - [D14690: Realtime Compositor: Add needed GPU module changes](https://archive.blender.org/developer/D14690). - [D15206: Realtime Compositor: Add initial evaluator and engine](https://archive.blender.org/developer/D15206). - [D15226: Realtime Compositor: Add basic output nodes](https://archive.blender.org/developer/D15226). - [D15227: Realtime Compositor: Add basic input nodes](https://archive.blender.org/developer/D15227). - [D15228: Realtime Compositor: Add basic color nodes](https://archive.blender.org/developer/D15228). - [D15229: Realtime Compositor: Add basic convert and vector nodes](https://archive.blender.org/developer/D15229). - [D15230: Realtime Compositor: Add basic matte nodes](https://archive.blender.org/developer/D15230). - [D15231: Realtime Compositor: Add basic distort nodes](https://archive.blender.org/developer/D15231). - [D15464: Realtime Compositor: Add stub unsupported nodes](https://archive.blender.org/developer/D15464). With other smaller patches that are not strictly related to the realtime compositor. ## To Do The following to-dos are not part of the first milestone and are thus not blockers for its merge to master. - Features - - [ ] Single-axis domain repetition. Wait for [D14366](https://archive.blender.org/developer/D14366). - - [ ] Bicubic interpolation. Reuse logic in `node_tex_image_cubic`. - - [ ] Multi-pass compositing. Needs sizable changes in the draw module and render engines. - Robustness - - [ ] Consider hardware capabilities when constructing shader compile units. Exceeding limits is unlikely due to scheduling strategy. - Performance - - [ ] Allow nodes to be compiled into either node operations or shader operations depending on its options. - - [ ] Cache GPU textures for multiple passes at the same time. - Portability - - [ ] MacOS support. Needs #96261 (Metal Viewport ). - Quality - - [x] Specialized domain realization shaders to avoid fuzziness introduced by interpolation. - Documentation - - [x] Document the viewport compositor, its supported nodes, and its compatibility with the existing compositor. - Nodes - See the following section. ## Supported Nodes - Input - - [x] Render Layers. - - [x] Image. - - [x] Movie Clip. - - [x] RGB. - - [x] Value. - - [x] Time Curve. - - [x] Scene Time. - - [x] Bokeh Image. - - [x] Track Position. - - [x] ~~Texture~~. - - [ ] Mask. - Output - - [x] Composite. - - [x] Viewer. - - [x] Split Viewer. - - [x] ~~Output File~~. - - [x] Levels. - Color - - [x] Mix RGB. - - [x] Alpha Over. - - [x] Invert. - - [x] RGB Curves. - - [x] Hue Saturation Value. - - [x] Color Balance. - - [x] Hue Correct. - - [x] Bright Contrast. - - [x] Gamma. - - [x] Exposure. - - [x] Color Correction. - - [x] Posterize. - - [x] Tone Map. - - [ ] Z Combine. - Converter - - [x] Math. - - [x] Color Ramp. - - [x] Set Alpha. - - [x] Alpha Convert. - - [x] RGB To BW. - - [x] Separate Color. - - [x] Combine Color. - - [x] Separate XYZ. - - [x] Combine XYZ. - - [x] Switch View. - - [ ] ID Mask. - - [ ] Convert Color Space. - Filter - - [x] Bilateral Blur. - - [x] Despeckle. - - [x] Filter. - - [x] Directional Blur. - - [x] Dilate Erode. - - [x] Pixelate. - - [x] Bokeh Blur. - - [x] Blur. - - [x] Glare. - - [ ] Fog Glow. - - [ ] Vector Blur. - - [ ] Defocus. - - [ ] Inpaint. - - [ ] Sun Beams. - - [ ] Denoise. - - [ ] Anti-Aliasing. - Vector - - [x] Normal. - - [x] Map Value. - - [x] Map Range. - - [x] Vector Curves. - - [x] Normalize. - Matte - - [x] Channel Key. - - [x] Color Spill. - - [x] Box Mask. - - [x] Ellipse Mask. - - [x] Luminance Key. - - [x] Difference Key. - - [x] Distance Key. - - [x] Chroma Key. - - [x] Color Key. - - [x] ~~Legacy Cryptomatte~~. - - [ ] Keying. - - [ ] Double Edge Mask. - - [ ] Keying Screen. - - [ ] Cryptomatte. - Distort - - [x] Lens Distortion. - - [x] Translate. - - [x] Rotate. - - [x] Flip. - - [x] Crop. - - [x] Transform. - - [x] Scale. - - [ ] Movie Distortion. - - [ ] Displace. - - [ ] Map UV. - - [ ] Stabilize. - - [ ] Plane Track Deform. - - [ ] Corner Pin.

Changed status from 'Needs Triage' to: 'Confirmed'

Added subscriber: @OmarEmaraDev

Added subscriber: @Emi_Martinez

Added subscriber: @DuarteRamos

Added subscriber: @Timothy_m

Added subscriber: @AndyCuccaro

Added subscriber: @zanqdo

Added subscriber: @Dangry

Added subscriber: @Yuro

Added subscriber: @VDC

Added subscriber: @Alaska

Added subscriber: @lsscpp

Added subscriber: @AlexeyAdamitsky

Added subscriber: @bent

Added subscriber: @TheRedWaxPolice

Added subscriber: @Pipeliner

Added subscriber: @JacobMerrill-1

I recommend adding a way to store the result of a buffer offscreen, to be used next frame too*
(motion blur and a few other techniques use it)

ie 'Output nodes'

Added subscriber: @PierreSchiller

Is it possible to implement recognizing Cryptomatte ID's?
That would be a huge plus, to apply specific nodes (filters) to a material or a mesh in the scene. The user can even re-map textures and materials
on the fly.

Added subscriber: @pixeltrain3d

I created a devtalk topic for feedback and discussions regarding this project, let us use that topic instead.

https://devtalk.blender.org/t/real-time-compositor-feedback-and-discussion/

Added subscriber: @Nurb2Kea

Added subscriber: @Sparazza

Added subscriber: @ParallelMayhem

Added subscriber: @YegorSmirnov

Added subscriber: @Eliot-Mack

Removed subscriber: @Eliot-Mack

Added subscriber: @Eliot-Mack

Added subscriber: @gaonirico

Added subscriber: @GeorgiaPacific

Added subscriber: @vignette

Added subscriber: @Festivity

Added subscriber: @Vyach

Very cool project. Please implement UV/position node for compositing. That will allow to distort image easier.
Now I use two blend textures as X and Y.

Added subscriber: @Likkez-2

Added subscriber: @heini

Added subscriber: @Nik-9

Added subscriber: @1936403010

Just for info !

Omar Emara (OmarSquircleArt) :

MacOS versions which don’t support OpenGL 4.3 are expected not to work.

The real-time compositor is built over the GPU module, which uses whatever back-end that is preferred by the underlying system. So when Apple finishes Metal support in Blender, it will automatically work on Metal

Added subscriber: @ShyDugong

Added subscriber: @mattli911

In #99210#1388196, @Nurb2Kea wrote:
Just for info !

Omar Emara (OmarSquircleArt) :

MacOS versions which don’t support OpenGL 4.3 are expected not to work.

The real-time compositor is built over the GPU module, which uses whatever back-end that is preferred by the underlying system. So when Apple finishes Metal support in Blender, it will automatically work on Metal

Yeah, I'm curious when Metal support would be "done", or support this new feature. Since M1 Pro chip/other versions crash instantly when going into Render View Mode.

Added subscriber: @aaronfang

@mattli911 The state and progress of the Metal backend can be tracked in #96261 (Metal Viewport ).

Added subscriber: @Mounir

Added subscriber: @harisreedhar

Added subscriber: @SteffenD

Added subscriber: @dursunumit

Added subscriber: @finalflasher

Added subscriber: @PrettyFireNOI7

please add glare filter, thank you :)

Added subscriber: @igiornotigi

Added subscriber: @Healtrap

Removed subscriber: @Healtrap

Added subscriber: @Healtrap

Added subscriber: @RedMser

Added subscriber: @Taumich

Added subscriber: @MrAlwis

Added subscriber: @Ulf3000

image node doesnt scale with the realtime viewport. means it will always overlay too small or too large and in wrong position. It is fixed in size while the viewport is not fixed in size.

Added subscriber: @marklio

Added subscriber: @EAW

@Ulf3000 See:
https://devtalk.blender.org/t/real-time-compositor-feedback-and-discussion/25018/90

and

https://devtalk.blender.org/t/real-time-compositor-feedback-and-discussion/25018/117

Added subscriber: @viadvena

Added subscriber: @SergOrc

I really look forward to multi-pass. no way without it ))

Added subscriber: @eyecandy

Should I make a real bug, if enabling compositor on Mac, can instant crash Blender? Or should we still expect that/is not supported yet.

If I enable it in the Preferences, then enable in the UI dropdown. I said "Use Nodes" in the compositor Workspace, then I tried to go to back to Default Workspace and it closes.

In #99210#1412027, @mattli911 wrote:
Should I make a real bug, if enabling compositor on Mac, can instant crash Blender? Or should we still expect that/is not supported yet.

If I enable it in the Preferences, then enable in the UI dropdown. I said "Use Nodes" in the compositor Workspace, then I tried to go to back to Default Workspace and it closes.

I am not 100% sure but, I think this will require the metal viewport *

ah yes*
see

   Portability
     MacOS support. Needs T96261: Metal Viewport .

Added subscriber: @Baga-4

Added subscriber: @Clarence-Brown

Added subscriber: @costa

Why is vector math not listed here? 🤔

In #99210#1413637, @costa wrote:
Why is vector math not listed here? 🤔

100% agree

using normal input +dot can be used for all sorts of hacks.

In #99210#1413673, @JacobMerrill-1 wrote:

In #99210#1413637, @costa wrote:
Why is vector math not listed here? 🤔

100% agree

using normal input +dot can be used for all sorts of hacks.

As Omar said, and I am agree, the first there should be all that already exist.
And yes, there is a plan for vector math for compositor

Added subscriber: @MarcelLegindi

Added subscriber: @system-system

Is it possible for freestyle to join the Realtime Compositor?

Added subscriber: @zx

Added subscriber: @thomlicous

please glare filter please, we need a lot of ☺

Added subscriber: @hoxabel

Will the new Light groups option be available in the Real-time Compositor?

In #99210#1429360, @hoxabel wrote:
Will the new Light groups option be available in the Real-time Compositor?

I agree...

Z depth pass is not working atm or AOV
depth_blur_diff_outline_2.blend

/test file
fast outline using a bit of math works at render time but not in viewport.

old upbge 0.2.5x had a commit called 'multiple render targets' or 'render attachments'

maybe we can cherry pick some of it?

Added subscriber: @3di

Hi, any plans to have this working with viewport render animation?

Added subscriber: @Memento

Added subscriber: @ww123td

since many versions blender always crashes when i duplicate an object and the viewport compositor is turned on .. boom crash. the nodes im using are only filter:soften and hue/sat/val.. nothing else

In #99210#1436394, @Ulf3000 wrote:
since many versions blender always crashes when i duplicate an object and the viewport compositor is turned on .. boom crash. the nodes im using are only filter:soften and hue/sat/val.. nothing else

make tests, if it is crashes always, or with certain filters
make report, attach your file, screencast video, crashlogs (start blender with blender_debug_log.cmd before)

All these will increase chance of fix very much

This comment was removed by @Ulf3000

Added subscriber: @MarcM0

Removed subscriber: @MarcM0

Added subscriber: @MoonMoon

This is amazing, now I just need the convert color space node

Added subscriber: @samranxx

Added subscriber: @Harti

Can the viewport compositior be enabled in solid mode? Is this planned?
Would be awesome.

In #99210#1446465, @Harti wrote:
Can the viewport compositior be enabled in solid mode? Is this planned?
Would be awesome.

I suppose it. because Workbench is render engine too.

@OmarEmaraDev was Texture support disabled forever? Heavy reason?
A bit strange, because RTC already supports procedurally generated textures.

Added subscriber: @htuncay-4

Added subscriber: @Forian-Naki

Added subscriber: @VortexTheBreadMan

thank you Omer Emara.

Added subscriber: @AD_Blend-69420

Added subscriber: @Peter-Jackson

Added subscribers: @omar-1, @Bachadam

since tracking position node is working for realtime compositor it wouldn't be a right time to implement PLANE TRACK DIFORM and CORNER PIN nodes?
@omar-1 Emara
Now the most waited nodes are:

DISPLACE node useful for vfx effects creation
KEYING node is the most used for green screen removal.
and Mask node

Added subscriber: @mhmmdmnrkn

Added subscriber: @mega789

Added subscriber: @kadir1975

Added subscriber: @ktycnpkcn

Added subscriber: @Enesaytac

Added subscriber: @Pixxellent

This issue was referenced by 57552f52b2

Download

What's New

Blender Studio

Manual

Developers Blog

Documentation

Benchmark

Blender Conference

Development Fund

One-time Donations