Viewport Compositor #99210

This is a task that tracks the current state of the Realtime Compositor project and lists some of its to-dos. For user-level overview and demos, see the blog post on the matter. For feedback and discussions, use the devtalk topic for the project.

Description

The aim of this project is to develop a new GPU accelerated compositor that is both realtime and interactive. As a first step, this new compositor will be used to power the Viewport Compositor, which is a new feature that applies the compositor directly in the viewport. In the long term, the realtime compositor will be unified with the existing compositor, effectively accelerating the existing compositing workflow as well. This task is only concerned about the realtime compositor in the context of the viewport compositor.

Milestones

Add a usable viewport compositor with basic input, processing, transform, and output nodes.
Support basic filter nodes like blur, filter, lens distortion, glare, and other frequently used nodes.
Support multi-pass compositing.
Support the rest of the nodes.

To Do

The following to-dos are not part of the first milestone and are thus not blockers for its merge to master.

Features
- - Single-axis domain repetition. #105642
- - Bicubic interpolation. #105533
- - Allow access to multiple EXR layers and passes. #105547
- - Multi-pass compositing.
Portability
- - MacOS support. #96261
Quality
- - Specialized domain realization shaders to avoid fuzziness introduced by interpolation.
Documentation
- - Document the viewport compositor, its supported nodes, and its compatibility with the existing compositor.
Nodes
- See the following section.

Supported Nodes

Input
- - Render Layers.
- - Image.
- - Movie Clip.
- - RGB.
- - Value.
- - Time Curve.
- - Scene Time.
- - Bokeh Image.
- - Track Position.
- - Texture.
- - Mask.
Output
- - Composite.
- - Viewer.
- - Split Viewer.
- - ~~Output File~~.
- - Levels.
Color
- - Mix RGB.
- - Alpha Over.
- - Invert.
- - RGB Curves.
- - Hue Saturation Value.
- - Color Balance.
- - Hue Correct.
- - Bright Contrast.
- - Gamma.
- - Exposure.
- - Color Correction.
- - Posterize.
- - Tone Map.
- - Z Combine.
Converter
- - Math.
- - Color Ramp.
- - Set Alpha.
- - Alpha Convert.
- - RGB To BW.
- - Separate Color.
- - Combine Color.
- - Separate XYZ.
- - Combine XYZ.
- - Switch View.
- - ID Mask.
- - Convert Color Space.
Filter
- - Bilateral Blur.
- - Despeckle.
- - Filter.
- - Directional Blur.
- - Dilate Erode.
- - Pixelate.
- - Bokeh Blur.
- - Blur.
- - Glare.
- - Anti-Aliasing.
- - Denoise.
- - Vector Blur.
- - Defocus.
- - Inpaint.
- - Sun Beams.
Vector
- - Normal.
- - Map Value.
- - Map Range.
- - Vector Curves.
- - Normalize.
Matte
- - Channel Key.
- - Color Spill.
- - Box Mask.
- - Ellipse Mask.
- - Luminance Key.
- - Difference Key.
- - Distance Key.
- - Chroma Key.
- - Color Key.
- - ~~Legacy Cryptomatte~~.
- - Keying.
- - Double Edge Mask.
- - Keying Screen.
- - Cryptomatte.
Distort
- - Lens Distortion.
- - Translate.
- - Rotate.
- - Flip.
- - Crop.
- - Transform.
- - Scale.
- - Displace.
- - Map UV.
- - Corner Pin.
- - Stabilize.
- - Plane Track Deform.
- - Movie Distortion.

This is a task that tracks the current state of the Realtime Compositor project and lists some of its to-dos. For user-level overview and demos, see the [blog post](https:*code.blender.org/2022/07/real-time-compositor/) on the matter. For feedback and discussions, use the [devtalk topic](https:*devtalk.blender.org/t/real-time-compositor-feedback-and-discussion/) for the project. ## Description The aim of this project is to develop a new GPU accelerated compositor that is both realtime and interactive. As a first step, this new compositor will be used to power the *Viewport Compositor*, which is a new feature that applies the compositor directly in the viewport. In the long term, the realtime compositor will be unified with the existing compositor, effectively accelerating the existing compositing workflow as well. This task is only concerned about the realtime compositor in the context of the viewport compositor. ## Milestones - [x] Add a usable viewport compositor with basic input, processing, transform, and output nodes. - [x] Support basic filter nodes like blur, filter, lens distortion, glare, and other frequently used nodes. - [ ] Support multi-pass compositing. - [ ] Support the rest of the nodes. ## To Do The following to-dos are not part of the first milestone and are thus not blockers for its merge to master. - Features - - [x] Single-axis domain repetition. #105642 - - [x] Bicubic interpolation. #105533 - - [x] Allow access to multiple EXR layers and passes. #105547 - - [ ] Multi-pass compositing. - Portability - - [x] MacOS support. #96261 - Quality - - [x] Specialized domain realization shaders to avoid fuzziness introduced by interpolation. - Documentation - - [x] Document the viewport compositor, its supported nodes, and its compatibility with the existing compositor. - Nodes - See the following section. ## Supported Nodes - Input - - [x] Render Layers. - - [x] Image. - - [x] Movie Clip. - - [x] RGB. - - [x] Value. - - [x] Time Curve. - - [x] Scene Time. - - [x] Bokeh Image. - - [x] Track Position. - - [x] Texture. - - [x] Mask. - Output - - [x] Composite. - - [x] Viewer. - - [x] Split Viewer. - - [x] ~~Output File~~. - - [x] Levels. - Color - - [x] Mix RGB. - - [x] Alpha Over. - - [x] Invert. - - [x] RGB Curves. - - [x] Hue Saturation Value. - - [x] Color Balance. - - [x] Hue Correct. - - [x] Bright Contrast. - - [x] Gamma. - - [x] Exposure. - - [x] Color Correction. - - [x] Posterize. - - [x] Tone Map. - - [x] Z Combine. - Converter - - [x] Math. - - [x] Color Ramp. - - [x] Set Alpha. - - [x] Alpha Convert. - - [x] RGB To BW. - - [x] Separate Color. - - [x] Combine Color. - - [x] Separate XYZ. - - [x] Combine XYZ. - - [x] Switch View. - - [x] ID Mask. - - [x] Convert Color Space. - Filter - - [x] Bilateral Blur. - - [x] Despeckle. - - [x] Filter. - - [x] Directional Blur. - - [x] Dilate Erode. - - [x] Pixelate. - - [x] Bokeh Blur. - - [x] Blur. - - [x] Glare. - - [x] Anti-Aliasing. - - [x] Denoise. - - [ ] Vector Blur. - - [ ] Defocus. - - [ ] Inpaint. - - [ ] Sun Beams. - Vector - - [x] Normal. - - [x] Map Value. - - [x] Map Range. - - [x] Vector Curves. - - [x] Normalize. - Matte - - [x] Channel Key. - - [x] Color Spill. - - [x] Box Mask. - - [x] Ellipse Mask. - - [x] Luminance Key. - - [x] Difference Key. - - [x] Distance Key. - - [x] Chroma Key. - - [x] Color Key. - - [x] ~~Legacy Cryptomatte~~. - - [ ] Keying. - - [ ] Double Edge Mask. - - [ ] Keying Screen. - - [ ] Cryptomatte. - Distort - - [x] Lens Distortion. - - [x] Translate. - - [x] Rotate. - - [x] Flip. - - [x] Crop. - - [x] Transform. - - [x] Scale. - - [x] Displace. - - [x] Map UV. - - [x] Corner Pin. - - [x] Stabilize. - - [x] Plane Track Deform. - - [x] Movie Distortion.

I recommend adding a way to store the result of a buffer offscreen, to be used next frame too*
(motion blur and a few other techniques use it)

ie 'Output nodes'

I recommend adding a way to store the result of a buffer offscreen, to be used next frame too* (motion blur and a few other techniques use it) ie 'Output nodes'

Is it possible to implement recognizing Cryptomatte ID's?
That would be a huge plus, to apply specific nodes (filters) to a material or a mesh in the scene. The user can even re-map textures and materials
on the fly.

Is it possible to implement recognizing Cryptomatte ID's? That would be a huge plus, to apply specific nodes (filters) to a material or a mesh in the scene. The user can even re-map textures and materials on the fly.

I created a devtalk topic for feedback and discussions regarding this project, let us use that topic instead.

https://devtalk.blender.org/t/real-time-compositor-feedback-and-discussion/

I created a devtalk topic for feedback and discussions regarding this project, let us use that topic instead. https://devtalk.blender.org/t/real-time-compositor-feedback-and-discussion/

Very cool project. Please implement UV/position node for compositing. That will allow to distort image easier.
Now I use two blend textures as X and Y.

Very cool project. Please implement UV/position node for compositing. That will allow to distort image easier. Now I use two blend textures as X and Y.

Just for info !

Omar Emara (OmarSquircleArt) :

MacOS versions which don’t support OpenGL 4.3 are expected not to work.

The real-time compositor is built over the GPU module, which uses whatever back-end that is preferred by the underlying system. So when Apple finishes Metal support in Blender, it will automatically work on Metal

Just for info ! Omar Emara (OmarSquircleArt) : > MacOS versions which don’t support OpenGL 4.3 are expected not to work. > The real-time compositor is built over the GPU module, which uses whatever back-end that is preferred by the underlying system. So when Apple finishes Metal support in Blender, it will automatically work on Metal

In #99210#1388196, @Nurb2Kea wrote:
Just for info !

Omar Emara (OmarSquircleArt) :

MacOS versions which don’t support OpenGL 4.3 are expected not to work.

The real-time compositor is built over the GPU module, which uses whatever back-end that is preferred by the underlying system. So when Apple finishes Metal support in Blender, it will automatically work on Metal

Yeah, I'm curious when Metal support would be "done", or support this new feature. Since M1 Pro chip/other versions crash instantly when going into Render View Mode.

> In #99210#1388196, @Nurb2Kea wrote: > Just for info ! > > Omar Emara (OmarSquircleArt) : > >> MacOS versions which don’t support OpenGL 4.3 are expected not to work. > > > >> The real-time compositor is built over the GPU module, which uses whatever back-end that is preferred by the underlying system. So when Apple finishes Metal support in Blender, it will automatically work on Metal Yeah, I'm curious when Metal support would be "done", or support this new feature. Since M1 Pro chip/other versions crash instantly when going into Render View Mode.

@mattli911 The state and progress of the Metal backend can be tracked in #96261 (Metal Viewport ).

please add glare filter, thank you :)

image node doesnt scale with the realtime viewport. means it will always overlay too small or too large and in wrong position. It is fixed in size while the viewport is not fixed in size.

@Ulf3000 See:
https://devtalk.blender.org/t/real-time-compositor-feedback-and-discussion/25018/90

and

https://devtalk.blender.org/t/real-time-compositor-feedback-and-discussion/25018/117

@Ulf3000 See: https://devtalk.blender.org/t/real-time-compositor-feedback-and-discussion/25018/90 and https://devtalk.blender.org/t/real-time-compositor-feedback-and-discussion/25018/117

I really look forward to multi-pass. no way without it ))

Should I make a real bug, if enabling compositor on Mac, can instant crash Blender? Or should we still expect that/is not supported yet.

If I enable it in the Preferences, then enable in the UI dropdown. I said "Use Nodes" in the compositor Workspace, then I tried to go to back to Default Workspace and it closes.

Should I make a real bug, if enabling compositor on Mac, can instant crash Blender? Or should we still expect that/is not supported yet. If I enable it in the Preferences, then enable in the UI dropdown. I said "Use Nodes" in the compositor Workspace, then I tried to go to back to Default Workspace and it closes. ![CleanShot 2022-09-03 at 07.47.19.png](https://archive.blender.org/developer/F13455455/CleanShot_2022-09-03_at_07.47.19.png)

In #99210#1412027, @mattli911 wrote:
Should I make a real bug, if enabling compositor on Mac, can instant crash Blender? Or should we still expect that/is not supported yet.

If I enable it in the Preferences, then enable in the UI dropdown. I said "Use Nodes" in the compositor Workspace, then I tried to go to back to Default Workspace and it closes.

I am not 100% sure but, I think this will require the metal viewport *

> In #99210#1412027, @mattli911 wrote: > Should I make a real bug, if enabling compositor on Mac, can instant crash Blender? Or should we still expect that/is not supported yet. > > If I enable it in the Preferences, then enable in the UI dropdown. I said "Use Nodes" in the compositor Workspace, then I tried to go to back to Default Workspace and it closes. > > ![CleanShot 2022-09-03 at 07.47.19.png](https://archive.blender.org/developer/F13455455/CleanShot_2022-09-03_at_07.47.19.png) I am not 100% sure but, I think this will require the metal viewport *

ah yes*
see

   Portability
     MacOS support. Needs T96261: Metal Viewport .

ah yes* see ``` Portability MacOS support. Needs T96261: Metal Viewport . ```

Why is vector math not listed here? 🤔

In #99210#1413637, @costa wrote:
Why is vector math not listed here? 🤔

100% agree

using normal input +dot can be used for all sorts of hacks.

> In #99210#1413637, @costa wrote: > Why is vector math not listed here? 🤔 100% agree using normal input +dot can be used for all sorts of hacks.

In #99210#1413673, @JacobMerrill-1 wrote:

In #99210#1413637, @costa wrote:
Why is vector math not listed here? 🤔

100% agree

using normal input +dot can be used for all sorts of hacks.

As Omar said, and I am agree, the first there should be all that already exist.
And yes, there is a plan for vector math for compositor

> In #99210#1413673, @JacobMerrill-1 wrote: >> In #99210#1413637, @costa wrote: >> Why is vector math not listed here? 🤔 > > 100% agree > > using normal input +dot can be used for all sorts of hacks. As Omar said, and I am agree, the first there should be all that already exist. And yes, there is a plan for vector math for compositor

Is it possible for freestyle to join the Realtime Compositor?

please glare filter please, we need a lot of ☺

Will the new Light groups option be available in the Real-time Compositor?

In #99210#1429360, @hoxabel wrote:
Will the new Light groups option be available in the Real-time Compositor?

I agree...

> In #99210#1429360, @hoxabel wrote: > Will the new Light groups option be available in the Real-time Compositor? I agree...

Z depth pass is not working atm or AOV
depth_blur_diff_outline_2.blend

/test file
fast outline using a bit of math works at render time but not in viewport.

old upbge 0.2.5x had a commit called 'multiple render targets' or 'render attachments'

maybe we can cherry pick some of it?

Z depth pass is not working atm or AOV [depth_blur_diff_outline_2.blend](https://archive.blender.org/developer/F13639211/depth_blur_diff_outline_2.blend) /test file fast outline using a bit of math works at render time but not in viewport. ![image.png](https://archive.blender.org/developer/F13639217/image.png) old upbge 0.2.5x had a commit called 'multiple render targets' or 'render attachments' maybe we can cherry pick some of it?

Hi, any plans to have this working with viewport render animation?

Hi, any plans to have this working with viewport render animation? ![image.png](https://archive.blender.org/developer/F13676012/image.png)

since many versions blender always crashes when i duplicate an object and the viewport compositor is turned on .. boom crash. the nodes im using are only filter:soften and hue/sat/val.. nothing else

In #99210#1436394, @Ulf3000 wrote:
since many versions blender always crashes when i duplicate an object and the viewport compositor is turned on .. boom crash. the nodes im using are only filter:soften and hue/sat/val.. nothing else

make tests, if it is crashes always, or with certain filters
make report, attach your file, screencast video, crashlogs (start blender with blender_debug_log.cmd before)

All these will increase chance of fix very much

> In #99210#1436394, @Ulf3000 wrote: > since many versions blender always crashes when i duplicate an object and the viewport compositor is turned on .. boom crash. the nodes im using are only filter:soften and hue/sat/val.. nothing else 1) make tests, if it is crashes always, or with certain filters 2) make report, attach your file, screencast video, crashlogs (start blender with blender_debug_log.cmd before) All these will increase chance of fix very much

This is amazing, now I just need the convert color space node

Can the viewport compositior be enabled in solid mode? Is this planned?
Would be awesome.

Can the viewport compositior be enabled in solid mode? Is this planned? Would be awesome.

In #99210#1446465, @Harti wrote:
Can the viewport compositior be enabled in solid mode? Is this planned?
Would be awesome.

I suppose it. because Workbench is render engine too.

@OmarEmaraDev was Texture support disabled forever? Heavy reason?
A bit strange, because RTC already supports procedurally generated textures.

> In #99210#1446465, @Harti wrote: > Can the viewport compositior be enabled in solid mode? Is this planned? > Would be awesome. I suppose it. because Workbench is render engine too. @OmarEmaraDev was Texture support disabled forever? Heavy reason? A bit strange, because RTC already supports procedurally generated textures.

thank you Omer Emara.

since tracking position node is working for realtime compositor it wouldn't be a right time to implement PLANE TRACK DIFORM and CORNER PIN nodes?
@omar-1 Emara
Now the most waited nodes are:

DISPLACE node useful for vfx effects creation
KEYING node is the most used for green screen removal.
and Mask node

since tracking position node is working for realtime compositor it wouldn't be a right time to implement PLANE TRACK DIFORM and CORNER PIN nodes? @omar-1 Emara Now the most waited nodes are: - DISPLACE node useful for vfx effects creation - KEYING node is the most used for green screen removal. - and Mask node

This issue was referenced by 57552f52b2

This issue was referenced by 57552f52b2987adfd1525c3e4109565e3641ef5d

@OmarEmaraDev I wanted to bring up that while the Lens Distortion node is supported, its Fit option is not working (tested in Blender 3.6 alpha 7cda559d7c). Not sure if I should create a bug as the feature is still in development, please point me to the best channel to report these.

@OmarEmaraDev I wanted to bring up that while the Lens Distortion node is supported, its Fit option is not working (tested in Blender 3.6 alpha 7cda559d7c1b). Not sure if I should create a bug as the feature is still in development, please point me to the best channel to report these.

@chafouin You can file a bug report here: https://projects.blender.org/blender/blender/issues/new/choose

The two links at the top are broken and the release announcement links here.

https://code.blender.org/2022/07/real-time-compositor/

The two links at the top are broken and the release announcement links here. https://code.blender.org/2022/07/real-time-compositor/

the glare node r still not working in viewport anyone know whats going on plz?

the glare node r still not working in viewport anyone know whats going on plz?

This is a developer platform for coordinating development task, for user support and issues use a community forum or other platforms

> the glare node r still not working in viewport anyone know whats going on plz? This is a developer platform for coordinating development task, for user support and issues use a community forum or other platforms

the glare node r still not working in viewport anyone know whats going on plz?

if you have a bug, please report properly and you can call the developer, (tag them, link relative topics)

> the glare node r still not working in viewport anyone know whats going on plz? if you have a bug, please report properly and you can call the developer, (tag them, link relative topics)

Is the compositor supposed to run on every frame in solid mode viewport? I can't imagine this is the intended results because it makes an almost instant frame change turn into a 1-2 second frame change with a compositor denoiser. If so, is there a way to disable it specifically only for viewport solid mode instead of render as well?

P.S.

the glare node r still not working in viewport anyone know whats going on plz?

I haven't tested since 3.5.1 and I doubt it's been changed, but when 3.5.0 released the only glare option that didn't work in viewport was Fog. Every other glare option works in viewport for me personally, unless I'm mistaken.

Is the compositor supposed to run on every frame in solid mode viewport? I can't imagine this is the intended results because it makes an almost instant frame change turn into a 1-2 second frame change with a compositor denoiser. If so, is there a way to disable it specifically _only_ for viewport solid mode instead of render as well? P.S. > the glare node r still not working in viewport anyone know whats going on plz? I haven't tested since 3.5.1 and I doubt it's been changed, but when 3.5.0 released the only glare option that didn't work in viewport was Fog. Every other glare option works in viewport for me personally, unless I'm mistaken.

@Calibrator It is best to give feedback on https://devtalk.blender.org/t/real-time-compositor-feedback-and-discussion/.

We don't enable the compositor for solid mode as far as I remember, so I am not entirely sure what you mean by that.

@Calibrator It is best to give feedback on https://devtalk.blender.org/t/real-time-compositor-feedback-and-discussion/. We don't enable the compositor for solid mode as far as I remember, so I am not entirely sure what you mean by that.

Ah thanks. I'm new here and it seems blind. I'll head over there.

When is the keying node coming? Please I need it.

And I want to know how can we render viewport composition since it is faster than cpu one.

When is the keying node coming? Please I need it. And I want to know how can we render viewport composition since it is faster than cpu one.

@Bachadam as soon as it will come. This is development tracker, not for requests or such questions.

@Bachadam I will look into both of those this week. But as @Vyach said, this is better suited for the devtalk thread:
https://devtalk.blender.org/t/real-time-compositor-feedback-and-discussion/

@Bachadam I will look into both of those this week. But as @Vyach said, this is better suited for the devtalk thread: https://devtalk.blender.org/t/real-time-compositor-feedback-and-discussion/

Download

What's New

Blender Studio

Manual

Developers Blog

Documentation

Benchmark

Blender Conference

Development Fund

One-time Donations

Labels Milestones

Viewport Compositor #99210

Description

Milestones

To Do

Supported Nodes