Compositor: Use RBF Interpolation in Keying Screen node #112480

Merged

Omar Emara merged 5 commits from OmarEmaraDev/blender:rbf-keying-screen into main

2023-10-04 07:07:14 +02:00

Omar Emara commented

2023-09-17 17:12:44 +02:00

Member

This patch changes the interpolation algorithm utilized by the Keying
Screen node to a Gaussian Radial Basis Function Interpolation. This is
proposed because the current Voronoi triangulation based interpolation
has the following properties:

Not temporally stable since the triangulation can abruptly change as
tracking markers change position.
Not smooth in the mathematical sense, which is also readily visible in
the artists sense.
Computationally expensive due to the triangulation and naive
rasterization algorithm.

On the other hand, the RBF interpolation method is temporally stable and
continuous, smooth and infinitely differentiable, and relatively simple
to compute assuming low number of markers, which is typically the case
for keying screen objects.

This breaks backward compatibility, but the keying screen is only used
as a secondary input for keying in typical compositor setups, so one
should expect minimal difference in outputs.

This patch changes the interpolation algorithm utilized by the Keying Screen node to a Gaussian Radial Basis Function Interpolation. This is proposed because the current Voronoi triangulation based interpolation has the following properties: - Not temporally stable since the triangulation can abruptly change as tracking markers change position. - Not smooth in the mathematical sense, which is also readily visible in the artists sense. - Computationally expensive due to the triangulation and naive rasterization algorithm. On the other hand, the RBF interpolation method is temporally stable and continuous, smooth and infinitely differentiable, and relatively simple to compute assuming low number of markers, which is typically the case for keying screen objects. This breaks backward compatibility, but the keying screen is only used as a secondary input for keying in typical compositor setups, so one should expect minimal difference in outputs.

Omar Emara added the

labels 2023-09-17 17:12:44 +02:00

Omar Emara added 1 commit 2023-09-17 17:12:56 +02:00

buildbot/vexp-code-patch-coordinator Build done. Details

f35ed754f8 Compositor: Use RBF Interpolation in Keying Screen node

This patch changes the interpolation algorithm utilized by the Keying
Screen node to a Gaussian Radial Basis Function Interpolation. This is
proposed because the current Voronoi triangulation based interpolation
has the following properties:

- Not temporally stable since the triangulation can abruptly change as
  tracking markers change position.
- Not smooth in the mathematical sense, which is also readily visible in
  the artists sense.
- Computationally expensive due to the triangulation and naive
  cauterization algorithm.

On the other hand, the RBF interpolation method is temporally stable and
continuous, smooth and infinitely differentiable, and relatively simple
to compute assuming low number of markers, which is typically the case
for keying screen objects.

This breaks backward compatibility, but the keying screen is only used
as a secondary input for keying in typical compositor setups, so one
should expect minimal difference in outputs.

Omar Emara requested review from Sergey Sharybin 2023-09-17 17:13:13 +02:00

Omar Emara commented

2023-09-17 17:20:25 +02:00

Author

Member

Here is a comparison between both the triangulation method and the RBF method.

The patch still needs to be cleaned up and versioned, but it is done at its core. I wanted to get some feedback first.

Here is a comparison between both the triangulation method and the RBF method. ![triangles.png](/attachments/2d4fbe52-75b5-40d8-a466-2aacbf9aaccb) ![RBF.png](/attachments/54d14437-3b8e-4f73-8c60-dbf6d5e5e35b) The patch still needs to be cleaned up and versioned, but it is done at its core. I wanted to get some feedback first.

226 KiB

triangles.png

610 KiB

Sergey Sharybin commented

2023-09-18 20:32:12 +02:00

Owner

That is a very interesting results! It does sound like it should give much better results for keying, but extra testing would be very welcome.

@sebastian_k Do you have some existing configurations to test this change?

@OmarEmaraDev Since you've mentioned you still plan to do some cleanup I did not dig deep into the code. Also with rapid frame changes I've managed to crash Blender. Not sure yet if that's related to this change. Maybe you can try to confirm it on your side?

That is a very interesting results! It does sound like it should give much better results for keying, but extra testing would be very welcome. @sebastian_k Do you have some existing configurations to test this change? @OmarEmaraDev Since you've mentioned you still plan to do some cleanup I did not dig deep into the code. Also with rapid frame changes I've managed to crash Blender. Not sure yet if that's related to this change. Maybe you can try to confirm it on your side?

Sergey Sharybin commented

2023-09-18 20:34:27 +02:00

Owner

@blender-bot package

@blender-bot package

Blender Bot commented

2023-09-18 20:34:30 +02:00

Member

Package build started. Download here when ready.

Package build started. [Download here](https://builder.blender.org/download/patch/PR112480) when ready.

Omar Emara commented

2023-09-22 13:33:25 +02:00

Author

Member

@Sergey I can't reproduce the crash. Can you share the file or a stacktrace?

@Sergey I can't reproduce the crash. Can you share the file or a stacktrace?

Sergey Sharybin commented

2023-09-22 17:38:27 +02:00

Owner

The crash actually is not related to this patch, i can reproduce it on the main branch and CPU tiled compositor. I can have a look into it next week, no need to consider it a stopper for this patch.

P.S. To reproduce the crash track some simple footage with handful of markers, add those as a keying screen, and scrub timeline while compositor backdrop is enabled.

The crash actually is not related to this patch, i can reproduce it on the main branch and CPU tiled compositor. I can have a look into it next week, no need to consider it a stopper for this patch. P.S. To reproduce the crash track some simple footage with handful of markers, add those as a keying screen, and scrub timeline while compositor backdrop is enabled.

Eliot Mack commented

2023-09-26 10:04:39 +02:00

First-time contributor

The RBF results look better and much closer to the lighting variation patterns seen in actual green screens.

How soon can the GPU version of this be done?

The RBF results look better and much closer to the lighting variation patterns seen in actual green screens. How soon can the GPU version of this be done?

Omar Emara referenced this pull request

2023-09-29 12:53:00 +02:00

Realtime Compositor: Implement Keying Screen node #113055

Omar Emara commented

2023-09-29 13:06:29 +02:00

Author

Member

I can't seem to understand why the crash happens. Maybe @iss can give me some pointers?

* thread #1, name = 'blender', stop reason = signal SIGSEGV: invalid address (fault address: 0x0)
    frame #0: 0x000000000ccb8610 blender`av_freep
    frame #1: 0x000000000c422ecd blender`av_packet_free_side_data + 45
    frame #2: 0x000000000c4235a9 blender`av_packet_unref + 9
    frame #3: 0x000000000c423626 blender`av_packet_free + 22
    frame #4: 0x000000000c42297d blender`avcodec_close + 153
    frame #5: 0x000000000c859189 blender`avcodec_free_context + 25
    frame #6: 0x0000000006709dce blender`free_anim_ffmpeg(anim=0x00007fff6580e008) at anim_movie.cc:1449:5
    frame #7: 0x0000000006709cc5 blender`IMB_free_anim(anim=0x00007fff6580e008) at anim_movie.cc:157:3
  * frame #8: 0x0000000005a3855d blender`free_buffers(clip=0x00007fff90ecfc08) at movieclip.cc:1610:5
    frame #9: 0x0000000005a37ebd blender`movie_clip_free_data(id=0x00007fff90ecfc08) at movieclip.cc:115:3
    frame #10: 0x0000000005900cc4 blender`BKE_libblock_free_datablock(id=0x00007fff90ecfc08, (null)=0) at lib_id_delete.cc:76:7
    frame #11: 0x0000000006487b96 blender`blender::deg::deg_free_copy_on_write_datablock(id_cow=0x00007fff90ecfc08) at deg_eval_copy_on_write.cc:1016:3
    frame #12: 0x000000000649f120 blender`blender::deg::IDNode::destroy(this=0x00007fffa1282308) at deg_node_id.cc:125:5
    frame #13: 0x00000000064a2754 blender`void blender::deg::clear_id_nodes_conditional<blender::deg::Depsgraph::clear_id_nodes()::$_1>(id_nodes=0x00007fffa2e1bcc0, filter=0x00007fffffffe4c8) at depsgraph.cc:150:16
    frame #14: 0x00000000064a22b5 blender`blender::deg::Depsgraph::clear_id_nodes(this=0x00007fffa2e1bc08) at depsgraph.cc:161:3
    frame #15: 0x00000000064a21c9 blender`blender::deg::Depsgraph::~Depsgraph(this=0x00007fffa2e1bc08) at depsgraph.cc:77:3
    frame #16: 0x00000000064a2cf8 blender`DEG_graph_free(graph=0x00007fffa2e1bc08) at depsgraph.cc:307:3
    frame #17: 0x0000000007d44e4a blender`blender::ed::space_node::compo_freejob(cjv=0x00007fff93d591a8) at node_edit.cc:222:5
    frame #18: 0x000000000604ac77 blender`wm_jobs_timer(wm=0x00007ffff059d208, wt=0x00007fff93d58cd8) at wm_jobs.cc:651:9
    frame #19: 0x00000000060720da blender`wm_window_timers_process(C=0x00007ffff0dd4c08, sleep_us_p=0x00007fffffffe6e4) at wm_window.cc:1640:7
    frame #20: 0x0000000006071d6a blender`wm_window_events_process(C=0x00007ffff0dd4c08) at wm_window.cc:1698:16
    frame #21: 0x0000000006017c5e blender`WM_main(C=0x00007ffff0dd4c08) at wm.cc:606:5
    frame #22: 0x0000000005415c05 blender`main(argc=1, argv=0x00007fffffffe8a8) at creator.cc:590:5
    frame #23: 0x00007ffff7c27cd0 libc.so.6`___lldb_unnamed_symbol3190 + 128
    frame #24: 0x00007ffff7c27d8a libc.so.6`__libc_start_main + 138
    frame #25: 0x00000000054156a5 blender`_start + 37

I can't seem to understand why the crash happens. Maybe @iss can give me some pointers? ``` * thread #1, name = 'blender', stop reason = signal SIGSEGV: invalid address (fault address: 0x0) frame #0: 0x000000000ccb8610 blender`av_freep frame #1: 0x000000000c422ecd blender`av_packet_free_side_data + 45 frame #2: 0x000000000c4235a9 blender`av_packet_unref + 9 frame #3: 0x000000000c423626 blender`av_packet_free + 22 frame #4: 0x000000000c42297d blender`avcodec_close + 153 frame #5: 0x000000000c859189 blender`avcodec_free_context + 25 frame #6: 0x0000000006709dce blender`free_anim_ffmpeg(anim=0x00007fff6580e008) at anim_movie.cc:1449:5 frame #7: 0x0000000006709cc5 blender`IMB_free_anim(anim=0x00007fff6580e008) at anim_movie.cc:157:3 * frame #8: 0x0000000005a3855d blender`free_buffers(clip=0x00007fff90ecfc08) at movieclip.cc:1610:5 frame #9: 0x0000000005a37ebd blender`movie_clip_free_data(id=0x00007fff90ecfc08) at movieclip.cc:115:3 frame #10: 0x0000000005900cc4 blender`BKE_libblock_free_datablock(id=0x00007fff90ecfc08, (null)=0) at lib_id_delete.cc:76:7 frame #11: 0x0000000006487b96 blender`blender::deg::deg_free_copy_on_write_datablock(id_cow=0x00007fff90ecfc08) at deg_eval_copy_on_write.cc:1016:3 frame #12: 0x000000000649f120 blender`blender::deg::IDNode::destroy(this=0x00007fffa1282308) at deg_node_id.cc:125:5 frame #13: 0x00000000064a2754 blender`void blender::deg::clear_id_nodes_conditional<blender::deg::Depsgraph::clear_id_nodes()::$_1>(id_nodes=0x00007fffa2e1bcc0, filter=0x00007fffffffe4c8) at depsgraph.cc:150:16 frame #14: 0x00000000064a22b5 blender`blender::deg::Depsgraph::clear_id_nodes(this=0x00007fffa2e1bc08) at depsgraph.cc:161:3 frame #15: 0x00000000064a21c9 blender`blender::deg::Depsgraph::~Depsgraph(this=0x00007fffa2e1bc08) at depsgraph.cc:77:3 frame #16: 0x00000000064a2cf8 blender`DEG_graph_free(graph=0x00007fffa2e1bc08) at depsgraph.cc:307:3 frame #17: 0x0000000007d44e4a blender`blender::ed::space_node::compo_freejob(cjv=0x00007fff93d591a8) at node_edit.cc:222:5 frame #18: 0x000000000604ac77 blender`wm_jobs_timer(wm=0x00007ffff059d208, wt=0x00007fff93d58cd8) at wm_jobs.cc:651:9 frame #19: 0x00000000060720da blender`wm_window_timers_process(C=0x00007ffff0dd4c08, sleep_us_p=0x00007fffffffe6e4) at wm_window.cc:1640:7 frame #20: 0x0000000006071d6a blender`wm_window_events_process(C=0x00007ffff0dd4c08) at wm_window.cc:1698:16 frame #21: 0x0000000006017c5e blender`WM_main(C=0x00007ffff0dd4c08) at wm.cc:606:5 frame #22: 0x0000000005415c05 blender`main(argc=1, argv=0x00007fffffffe8a8) at creator.cc:590:5 frame #23: 0x00007ffff7c27cd0 libc.so.6`___lldb_unnamed_symbol3190 + 128 frame #24: 0x00007ffff7c27d8a libc.so.6`__libc_start_main + 138 frame #25: 0x00000000054156a5 blender`_start + 37 ```

Sebastian Koenig commented

2023-09-29 15:20:58 +02:00

Member

Hey, this does look like a very cool improvement!
The keying screen looks much more even than before and matches better the way light behaves on the actual greenscreen.
I do not have an existing config though because I rarely use the keying screen. Partly due to the artifacts that sometimes occur with the triangulation method in fact...
So yes, in those cases where a keyscreen is needed this should give better results. The smoothness control is also very useful!

Hey, this does look like a very cool improvement! The keying screen looks much more even than before and matches better the way light behaves on the actual greenscreen. I do not have an existing config though because I rarely use the keying screen. Partly due to the artifacts that sometimes occur with the triangulation method in fact... So yes, in those cases where a keyscreen is needed this should give better results. The smoothness control is also very useful!

new_keying_screen.jpg

165 KiB

Omar Emara added 2 commits 2023-09-29 16:35:43 +02:00

f5c09b3ca9 Merge branch 'main' into rbf-keying-screen

fdfa6a8dd9 Cleanup and add versioning

Omar Emara commented

2023-09-29 16:38:17 +02:00

Author

Member

@Sergey I added versioning, so patch should be ready.

@Sergey I added versioning, so patch should be ready.

Richard Antalik commented

2023-09-30 04:56:12 +02:00

Member

I can't seem to understand why the crash happens. Maybe @iss can give me some pointers?

Looks like crash in ffmpeg, Can you make report for that?

> I can't seem to understand why the crash happens. Maybe @iss can give me some pointers? Looks like crash in ffmpeg, Can you make report for that?

Omar Emara added 2 commits 2023-10-02 09:59:03 +02:00

b508aa4d4d Merge branch 'main' into rbf-keying-screen

240534e557 Change smoothness default to zero

Sergey Sharybin approved these changes 2023-10-02 11:46:38 +02:00

Omar Emara merged commit 75c947a467 into main

2023-10-04 07:07:14 +02:00

Omar Emara referenced this issue from a commit

2023-10-04 07:07:15 +02:00

Compositor: Use RBF Interpolation in Keying Screen node

Omar Emara deleted branch rbf-keying-screen

2023-10-04 07:07:16 +02:00

Vasilis-Milios referenced this issue from a commit

2023-10-04 14:15:43 +02:00

Compositor: Use RBF Interpolation in Keying Screen node

Vasilis-Milios referenced this issue from a commit

2023-10-04 14:26:48 +02:00

Compositor: Use RBF Interpolation in Keying Screen node

Sign in to join this conversation.

No reviewers

No Label

Animation & Rigging

Asset Browser Project Overview

Automated Testing

Blender Asset Bundle

Dependency Graph

Development Management

EEVEE & Viewport

Images & Movies

Motion Tracking

Nodes & Physics

Pipeline, Assets & IO

Platforms, Builds & Tests

Render & Cycles

Render Pipeline

Sculpt, Paint & Texture

Video Sequencer

Virtual Reality

Blender 2.8 Project

Milestone 1: Basic, Local Asset Browser

Good First Issue

Animation & Rigging

Development Management

EEVEE & Viewport

Nodes & Physics

Pipeline, Assets & IO

Platforms, Builds & Tests

Render & Cycles

Sculpt, Paint & Texture

Needs Info from Developers

Needs Information from User

No Milestone

No project

No Assignees

6 Participants

Notifications

Due Date

The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: blender/blender#112480

No description provided.