Compositor improvement plan #74491

New Issue

Sergey Sharybin · 2020-03-06T12:46:11+01:00

Sergey Sharybin commented

2020-03-06 12:46:11 +01:00

Overview

This is an initial pass on formalizing proposal how to improve performance and user experience of Compositor.

There are three aspects which are aimed to be addressed:

Ease of use
Performance
Memory consumption

Ease of use

Currently there are many settings which needs to be set and tweak to have best performance: tile size, OpenCL, buffer use and so on. Settings also have some implicit dependencies under the hood: for example, OpenCL needs big tile size, but big tile size might make some nodes measurably slower, and bigger tile size "breaks" the initial intention of compositor design to show tiles appearing as quick as possible.

For OpenCL case it's also not clear when it's actually being engaged: it will just fail silently, falling back to CPU, giving false impression of having GPU accelerated compute.

Performance

Performance of compositor is not up to date. Partially due to its scheduler design, partially due to technical implementation which is per-pixel virtual call (which ruins all sort of coherency).

Memory usage

This is something what is absolutely out of artists control: some operations require memory buffers before/after the node and those are being created automatically. This makes it hard to predict how much memory node setup requires, and how much extra memory is needed when increasing final resolution.

Solution

Proposed end goal is: deliver final image as fast as possible.

This is different from being tile-based, where goal was to have first tiles to appear as quick as possible, with giving gradual updates. The downside of this is that overall frame time is higher than giving an entire frame at once. Additionally, tile-based nature complicates task scheduler a lot, and makes it required to keep track of more memory at a time.

It should be possible to transform current design to proposed one in incremental steps:

[Temporarily] Remove code which unnecessarily complicates scheduler and memory manager which is GPU support.
Convert all operations to operate in a relative space rather than pixel space (basically, make it possible to change final render resolution without changing compositor network setup).
Make compositor to operate on final resolution which closely matches resolution of the current "viewer": there is no need to do full 8K compositing when final result is viewed as a tiny backdrop on Full HD monitor.
Modify operations to operate on an entire frame (or on a given area).
Modify scheduler to do bottom-to-top scheduling, operating on the entire image.
Modify memory manager to allocate buffers once is needed and discard them as soon as possible.
Vectorize (SIMD) all operations where possible.

Look into GPU support with the following requirements:

Minimize memory throughput, which implies the following point.
Have all operations implemented on GPU, which again implies following point.
Share implementation between CPU and GPU as much as possible.

The steps can be gradual and formulated well-enough to happens as code quality days in #73586.

# Overview This is an initial pass on formalizing proposal how to improve performance and user experience of Compositor. There are three aspects which are aimed to be addressed: * Ease of use * Performance * Memory consumption ## Ease of use Currently there are many settings which needs to be set and tweak to have best performance: tile size, OpenCL, buffer use and so on. Settings also have some implicit dependencies under the hood: for example, OpenCL needs big tile size, but big tile size might make some nodes measurably slower, and bigger tile size "breaks" the initial intention of compositor design to show tiles appearing as quick as possible. For OpenCL case it's also not clear when it's actually being engaged: it will just fail silently, falling back to CPU, giving false impression of having GPU accelerated compute. # Performance Performance of compositor is not up to date. Partially due to its scheduler design, partially due to technical implementation which is per-pixel virtual call (which ruins all sort of coherency). # Memory usage This is something what is absolutely out of artists control: some operations require memory buffers before/after the node and those are being created automatically. This makes it hard to predict how much memory node setup requires, and how much extra memory is needed when increasing final resolution. # Solution Proposed end goal is: deliver final image as fast as possible. This is different from being tile-based, where goal was to have first tiles to appear as quick as possible, with giving gradual updates. The downside of this is that overall frame time is higher than giving an entire frame at once. Additionally, tile-based nature complicates task scheduler a lot, and makes it required to keep track of more memory at a time. It should be possible to transform current design to proposed one in incremental steps: * [Temporarily] Remove code which unnecessarily complicates scheduler and memory manager which is GPU support. * Convert all operations to operate in a relative space rather than pixel space (basically, make it possible to change final render resolution without changing compositor network setup). * Make compositor to operate on final resolution which closely matches resolution of the current "viewer": there is no need to do full 8K compositing when final result is viewed as a tiny backdrop on Full HD monitor. * Modify operations to operate on an entire frame (or on a given area). * Modify scheduler to do bottom-to-top scheduling, operating on the entire image. * Modify memory manager to allocate buffers once is needed and discard them as soon as possible. * Vectorize (SIMD) all operations where possible. Look into GPU support with the following requirements: * Minimize memory throughput, which implies the following point. * Have all operations implemented on GPU, which again implies following point. * Share implementation between CPU and GPU as much as possible. The steps can be gradual and formulated well-enough to happens as code quality days in #73586.

Sergey Sharybin commented

2020-03-06 12:46:11 +01:00

Changed status from 'Needs Triage' to: 'Confirmed'

Sergey Sharybin commented

2020-03-06 12:46:11 +01:00

Added subscribers: @Sergey, @Jeroen-Bakker, @brecht

Ray molenkamp commented

2020-03-06 16:33:06 +01:00

Added subscriber: @LazyDodo

Ray molenkamp commented

2020-03-06 16:33:06 +01:00

This is going to be a controversial suggestion given it's a chunky new dependency, however...halide is a DSL designed for exactly this kind of problem (takes in account locality, parallelism, vectorization etc) , it's mature (8+ years old, and still actively maintained and in use by both google and adobe) it supports CPU/GPU/SIMD/Threading/Metal/OpenGL out of the box, it is based on LLVM however which generally has a high startup cost, but they mitigated that by allowing to pre-build the kernels at compile time (rather than runtime) It has a python API so you can make kernels in python (still with all hardware support mentioned earlier) which could be good for us addon-wise (but you're back to a small run-time cost at that point).

If anything watch the video on the bottom of their home page to see what it is about, I feel this is a really good match for the compositor and should be considered.

This is going to be a controversial suggestion given it's a chunky new dependency, however...[halide ](https://halide-lang.org/) is a DSL designed for exactly this kind of problem (takes in account locality, parallelism, vectorization etc) , it's mature (8+ years old, and still actively maintained and in use by both google and adobe) it supports CPU/GPU/SIMD/Threading/Metal/OpenGL out of the box, it is based on LLVM however which generally has a high startup cost, but they mitigated that by allowing to pre-build the kernels at compile time (rather than runtime) It has a python API so you can make kernels in python (still with all hardware support mentioned earlier) which could be good for us addon-wise (but you're back to a small run-time cost at that point). If anything watch the video on the bottom of their home page to see what it is about, I feel this is a *really* good match for the compositor and should be considered.

Sergey Sharybin commented

2020-03-06 17:27:35 +01:00

@LazyDodo Thanks for pointing it out. I am aware of Halide and was considering it for tracker/compositor a while. Nowadays not that much sold. With the function nodes it feels like we can fill in missing gaps and achieve same level of functionality with building blocks which are native to Blender. But will see.

Ray molenkamp commented

2020-03-06 18:07:25 +01:00

I have experimented with it in the past, where it really shines is how easily you can change a schedule add/changes data-layout/threading/caching/vectorization or go, "now run it on the GPU! now do CUDA! no OpenCL! use DX12!" by changing a line of 2 of code. It is by far the best performing code I have written with the least amount of effort.

However not everything they offer is a good match for us, fusion across the whole nodegraph while theoretically awesome will just never work for us, the run-time cost to schedule and jit them is just too high (once done the perf is great.. however spending several seconds in schedule and jit so you can run a graph under 3ms just ruins any gains to be had) if we use it, it be best to thread every node as it's own block and optimize that at build time rather than trying to run-time optimize the graph the user puts together.

Funciton nodes is still to 'unknown' for me to make a call on, I think there's pro's and con's for both solutions, however gaining a dep as large as halide is definitely hanging out on the con side of things.

I have experimented with it in the past, where it really shines is how easily you can change a schedule add/changes data-layout/threading/caching/vectorization or go, "now run it on the GPU! now do CUDA! no OpenCL! use DX12!" by changing a line of 2 of code. It is by far the best performing code I have written with the least amount of effort. However not everything they offer is a good match for us, fusion across the whole nodegraph while theoretically awesome will just never work for us, the run-time cost to schedule and jit them is just too high (once done the perf is great.. however spending several seconds in schedule and jit so you can run a graph under 3ms just ruins any gains to be had) if we use it, it be best to thread every node as it's own block and optimize that at build time rather than trying to run-time optimize the graph the user puts together. Funciton nodes is still to 'unknown' for me to make a call on, I think there's pro's and con's for both solutions, however gaining a dep as large as halide is definitely hanging out on the con side of things.

Sam Van Hulle commented

2020-03-07 00:33:38 +01:00

Added subscriber: @frameshift

Bintang Pratama commented

2020-03-07 00:43:28 +01:00

Added subscriber: @BintangPratama

Evan Wilson commented

2020-03-07 01:23:27 +01:00

Added subscriber: @EAW

Silas Opel commented

2020-03-07 02:38:56 +01:00

Added subscriber: @DarkKnight

Fahad Hasan commented

2020-03-10 10:15:05 +01:00

Added subscriber: @MD.FahadHassan

Fahad Hasan commented

2020-03-10 10:37:58 +01:00

If Possible:
Updating only the nodes that has been changed and later connected nodes in the chain, avoiding the upstream.

If Possible: Updating only the nodes that has been changed and later connected nodes in the chain, avoiding the upstream.

Ray molenkamp commented

2020-03-12 00:26:39 +01:00

In #74491#888273, @MD.FahadHassan wrote:
If Possible:
Updating only the nodes that has been changed and later connected nodes in the chain, avoiding the upstream.

That assumes you keep the output buffers of each of the nodes, which if you are compositing at high resolution or you have a massive amount of nodes gets expensive memory wise real fast.

> In #74491#888273, @MD.FahadHassan wrote: > If Possible: > Updating only the nodes that has been changed and later connected nodes in the chain, avoiding the upstream. That assumes you keep the output buffers of each of the nodes, which if you are compositing at high resolution or you have a massive amount of nodes gets expensive memory wise real fast.

Fahad Hasan commented

2020-03-12 02:14:48 +01:00

In #74491#889620, @LazyDodo wrote:
That assumes you keep the output buffers of each of the nodes, which if you are compositing at high resolution or you have a massive amount of nodes gets expensive memory wise real fast.

That's true as well.

> In #74491#889620, @LazyDodo wrote: >That assumes you keep the output buffers of each of the nodes, which if you are compositing at high resolution or you have a massive amount of nodes gets expensive memory wise real fast. That's true as well.

Pipeliner commented

2020-03-12 11:18:36 +01:00

Added subscriber: @Pipeliner

Tiago Chaves commented

2020-03-28 14:08:01 +01:00

Added subscriber: @laurelkeys

michael campbell commented

2020-04-20 17:13:10 +02:00

Added subscriber: @3di

michael campbell commented

2020-04-20 17:13:10 +02:00

To avoid keeping the output buffers of each node, perhaps a cache node would be a good first step, so the user could decide which points in the tree to cache (after a denoise node for example). Something similar to the houdini approach, additionally a freeze toggle on each node in case the user doesn't want to use the output of a file cache node in a different branch (or different compositor if multiple compositors become possible with a compositor node). This way anything below the file cache node would not need to be recalculated each time an upstream parameter change occurs.

{F8485301}

To avoid keeping the output buffers of each node, perhaps a cache node would be a good first step, so the user could decide which points in the tree to cache (after a denoise node for example). Something similar to the houdini approach, additionally a freeze toggle on each node in case the user doesn't want to use the output of a file cache node in a different branch (or different compositor if multiple compositors become possible with a compositor node). This way anything below the file cache node would not need to be recalculated each time an upstream parameter change occurs. {F8485301}

Bartosz Moniewski commented

2020-05-19 20:31:05 +02:00

Added subscriber: @BartekMoniewski

Jason Clarke commented

2020-05-19 21:13:20 +02:00

Added subscriber: @JasonClarke

Sean Kennedy commented

2020-05-19 21:18:40 +02:00

Added subscriber: @SeanKennedy

Joao Morgado commented

2020-05-19 21:24:06 +02:00

Added subscriber: @MrJoMo

Michael Hermann commented

2020-05-19 21:44:20 +02:00

Added subscriber: @MichaelHermann

mantissa commented

2020-05-20 14:31:41 +02:00

Added subscriber: @Mantissa

Bent Hillerkus commented

2020-06-12 21:19:28 +02:00

Added subscriber: @bent

Dylan Neill commented

2020-08-05 00:12:17 +02:00

Added subscriber: @semimetallic

Monique Dewanchand commented

2020-08-24 21:03:27 +02:00

Added subscriber: @monique

Monique Dewanchand commented

2020-08-24 21:03:27 +02:00

Regarding the first bullet point in this list of improvements, relative space, I've came up with a possible solution: https://devtalk.blender.org/t/compositor-improvement-plan-relative-space/14874

Your input would be much appreciated.

Regarding the first bullet point in this list of improvements, relative space, I've came up with a possible solution: https://devtalk.blender.org/t/compositor-improvement-plan-relative-space/14874 Your input would be much appreciated.

Bent Hillerkus commented

2020-08-24 21:58:35 +02:00

It'd be nice if nodes could have Gizmos on the preview, when they are active. I'm mostly thinking about the transform gizmo for mask and transform nodes, but a point gizmo would be nice too, for example for the sun glare node.
If OFX plugins were to be supported in the VSE and / or in the Compositor gizmos would be needed too.

It'd be nice if nodes could have Gizmos on the preview, when they are active. I'm mostly thinking about the transform gizmo for mask and transform nodes, but a point gizmo would be nice too, for example for the sun glare node. If OFX plugins were to be supported in the VSE and / or in the Compositor gizmos would be needed too.

Fahad Hasan commented

2020-08-24 22:12:17 +02:00

My suggestion would be:
For Viewer:

Discard the backdrop first. It's a cluttered feature and probably will not go with relative space.
Instead creating a new viewer explicitly or making the Movie Editor as the standard viewer for compositing. Which has:
- Masking system (fetching from existing Movie Editor)
- A timeline and cache system (possible to fetch from Movie Editor)
- Trackers (Fetch from Movie Editor)
- Relative space render buffer (Which is used by Natron as well and it is efficient for playback memory cache too)
- Should have a switcher to render entire pixel too. Because of anti aliasing like hair splits or texts or GPU based nodes
- A color-space switcher (Linear/sRGB/REC709/Filmic)
  For Nodes:
- Making all nodes relative space aware
- Focusing on Property panel more than nodes channels
- Transferring procedural textures into compositor from texture editor
- Giving a GLSL authoring node like this (https://github.com/bitsawer/blender-custom-nodes). This is the fastest possible node in the entire Blender compositor after input nodes. And as GLSL is just like scripting language the growth of scripts by the community will be massive. If anything is not in Blender, there will be.
- Giving a Python script based node is also cool for AI based future compositing industry built with pytorch, numpy and openCV. (https://github.com/bitsawer/blender-custom-nodes)
- A Gizmo API is needed for nodes. Specially Transform and Screen dependent nodes. Probably a simple 2D transform gizmo and a single point gizmo is more than enough.

For Node Cache system:

After doing some pipeline dev I emulated a cache system with "File node". Where I plugged it after doing denoising and using those denoised file image for later operations. Similarly after doing some heavy blur I plugged another file cache. >>
Instead of ram cache for nodes can we usedisc cache instead? In this way viewer will get all the ram for playback. If so then Natron like "value change aware cache" system can be put in. (Updating only the nodes that has been changed and later connected nodes in the chain, avoiding the upstream.)

Hope it helps. Thanks.

My suggestion would be: For Viewer: - Discard the backdrop first. It's a cluttered feature and probably will not go with relative space. - Instead creating a new viewer explicitly or making the Movie Editor as the standard viewer for compositing. Which has: - Masking system (fetching from existing Movie Editor) - A timeline and cache system (possible to fetch from Movie Editor) - Trackers (Fetch from Movie Editor) - Relative space render buffer (Which is used by Natron as well and it is efficient for playback memory cache too) - Should have a switcher to render entire pixel too. Because of anti aliasing like hair splits or texts or GPU based nodes - A color-space switcher (Linear/sRGB/REC709/Filmic) For Nodes: - Making all nodes relative space aware - Focusing on Property panel more than nodes channels - Transferring procedural textures into compositor from texture editor - Giving a GLSL authoring node like this (https://github.com/bitsawer/blender-custom-nodes). This is the fastest possible node in the entire Blender compositor after input nodes. And as GLSL is just like scripting language the growth of scripts by the community will be massive. If anything is not in Blender, there will be. - Giving a Python script based node is also cool for AI based future compositing industry built with pytorch, numpy and openCV. (https://github.com/bitsawer/blender-custom-nodes) - A Gizmo API is needed for nodes. Specially Transform and Screen dependent nodes. Probably a simple 2D transform gizmo and a single point gizmo is more than enough. For Node Cache system: - After doing some pipeline dev I emulated a cache system with "File node". Where I plugged it after doing denoising and using those denoised file image for later operations. Similarly after doing some heavy blur I plugged another file cache. >> - Instead of **ram cache** for nodes can we use**disc cache** instead? In this way viewer will get all the ram for playback. If so then Natron like "value change aware cache" system can be put in. (Updating only the nodes that has been changed and later connected nodes in the chain, avoiding the upstream.) Hope it helps. Thanks.

Sergey Sharybin commented

2020-09-07 13:00:23 +02:00

@monique, do you mind moving the design/code proposal from the devtalk.b.o to the developer.b.o as a subtask of this one? Will make it easier to keep track of it.

Andrea Monzini commented

2020-09-25 13:57:14 +02:00

Added subscriber: @AndreaMonzini

Andrea Monzini commented

2020-09-25 14:03:20 +02:00

Hello, just to inform ( if you not already know it ) that there is an experimental and unofficial Blender branch by "Manuel Castilla" to improve the performance of the compositor.

Repository:
https://github.com/m-castilla/blender/tree/compositor-up

Blender Chat channel :
https://blender.chat/channel/compositor-up

Hello, just to inform ( if you not already know it ) that there is an experimental and unofficial Blender branch by "Manuel Castilla" to improve the performance of the compositor. Repository: https://github.com/m-castilla/blender/tree/compositor-up Blender Chat channel : https://blender.chat/channel/compositor-up

Daniel Vesterbæk Jensen commented

2020-10-25 11:25:14 +01:00

Added subscriber: @DanielVesterbaekJensen

Przemyslaw Golab (SirPigeonz) commented

2020-10-30 12:40:10 +01:00

Added subscriber: @SirPigeonz

Keith Boshoff commented

2020-12-14 11:38:10 +01:00

Added subscriber: @Wahooney

Keith Boshoff commented

2020-12-14 11:38:10 +01:00

I don't know if this has been proposed, but moving Compositor data to a data-block structure would have benefits: including moving comps easily between files/projects, switching out comps in batch renders (my current use case), allowing scratch pads for comps, and probably more I can't think of.

Christoph Werner commented

2021-01-08 11:51:44 +01:00

Added subscriber: @ChristophWerner

Vladislav Macíček commented

2021-04-01 22:23:56 +02:00

Added subscriber: @Aeraglyx

Joao Morgado commented

2021-04-12 15:30:43 +02:00

Hi @Jeroen-Bakker,

Looks like some work might be happening on the compositor (exciting!). I started collecting some ideas in a DevTalk thread a while ago, but unfortunately got busy and mostly abandoned it. It might be worth checking it out again.

https://devtalk.blender.org/t/compositor-improvements/13264

Good luck with the improvements and, as always, thank you for the hard work!

Hi @Jeroen-Bakker, Looks like some work might be happening on the compositor (exciting!). I started collecting some ideas in a DevTalk thread a while ago, but unfortunately got busy and mostly abandoned it. It might be worth checking it out again. https://devtalk.blender.org/t/compositor-improvements/13264 Good luck with the improvements and, as always, thank you for the hard work!

Tas commented

2021-04-20 17:45:54 +02:00

Added subscriber: @Tasch