VSE: replace Subsampled3x3 filter by a general Box filter #117584

Aras Pranckevicius · 2024-01-28T10:36:41+01:00

2024-01-28 10:36:41 +01:00

Part of overall "improve filtering situation" (#116980): replace Subsampled3x3 (added for blender 3.5 in f210842a72 et al.) strip scaling filter with a general Box filter.

Subsampled3x3 is really a Box filter ("average pixel values over NxM region"), hardcoded to 3x3 size. As such, it works pretty well when downscaling images by 3x on each axis. But starts to break down and introduce aliasing at other scaling factors. When scaling up or scaling down by less than 3x, using total of 9 samples is a bit of overkill and hurts performance.

So instead, calculate the amount of NxM samples needed by looking at scaling factors on X/Y axes. Note: use at least 2 samples on each axis, so that when rotation is present, the result edges will get some anti-aliasing, just like it was happening in previous filter implementation.

Notes

The (very recently added) sequencer_render_filter.box test is failing, as expected; the expected image needs to be updated with merge of this PR.

Example images

Source: this perspective grid pattern image. The result images will be the same input, scaled down in VSE and then scaled up again with an adjustment layer using Nearest filtering:

Scaling that down by 3x using currently default Bilinear filtering produces a lot of aliasing, which the current Subsampled3x3 improves quite a bit. That is good so far:

However, at 4x downscaling, previous Subsampled3x3 vs the new Box:

At 15x downscaling, old vs new:

Or another case, this image:

Downscaling it exactly by 12x looks almost correct:

However downscaling it by 30x is complete nonsense!

The new Box filter, downscaling by 30x looks like this which is way more sensible:

Part of overall "improve filtering situation" (#116980): replace Subsampled3x3 (added for blender 3.5 in f210842a7259 et al.) strip scaling filter with a general Box filter. Subsampled3x3 is really a Box filter ("average pixel values over NxM region"), hardcoded to 3x3 size. As such, it works pretty well when downscaling images by 3x on each axis. But starts to break down and introduce aliasing at other scaling factors. When scaling up or scaling down by less than 3x, using total of 9 samples is a bit of overkill and hurts performance. So instead, calculate the amount of NxM samples needed by looking at scaling factors on X/Y axes. Note: use at least 2 samples on each axis, so that when rotation is present, the result edges will get some anti-aliasing, just like it was happening in previous filter implementation. ### Notes The (very recently added) `sequencer_render_filter.box` test is failing, as expected; the expected image needs to be updated with merge of this PR. ### Example images Source: this perspective grid pattern image. The result images will be the same input, scaled down in VSE and then scaled up again with an adjustment layer using Nearest filtering: ![subsgrid0_source.png](/attachments/8ae069fe-e67d-463e-88d3-4e45a39af114) Scaling that down by 3x using currently default Bilinear filtering produces a lot of aliasing, which the current Subsampled3x3 improves quite a bit. That is good so far: ![subsgrid1_3x_bilin.png](/attachments/7671e521-781c-42a8-809b-3ffa8bee978a) ![subsgrid1_3x_subs.png](/attachments/827f2a55-8984-4c60-9e23-3eafec9d9f6d) However, at 4x downscaling, previous Subsampled3x3 vs the new Box: ![subsgrid1_4x_subs.png](/attachments/1ff05a63-dbff-4a9d-b0dc-a76615746ca9) ![subsgrid1_4x_subsBox.png](/attachments/706142c2-0128-4eb1-8d1c-42cac42cb202) At 15x downscaling, old vs new: ![subsgrid1_15x_subs.png](/attachments/92bdd43a-530b-4e58-8a4e-20a5d17a60d2) ![subsgrid1_15x_subsBox.png](/attachments/5d719682-1f70-4448-80c4-c3043227decf) Or another case, this image: ![subs0_source.png](/attachments/26c9a6b7-54d9-415d-a38c-0f1af8b7c807) Downscaling it exactly by 12x looks almost correct: ![subs1_12x.png](/attachments/c8d4f3c3-cd11-41fc-a63f-c0cf14328afe) However downscaling it by 30x is complete nonsense! ![subs1_30x.png](/attachments/5e96f6fe-2daa-48e1-91e9-1c9d2ed72aee) The new Box filter, downscaling by 30x looks like this which is way more sensible: ![subs1_30xBox.png](/attachments/432708f5-960e-4ae9-9d34-7a4a3dbde9d6)

subs0_source.png

1.5 KiB

subs1_12x.png

910 B

subsgrid1_15x_subsBox.png

2.9 KiB

subsgrid0_source.png

161 KiB

subsgrid1_3x_bilin.png

30 KiB

subsgrid1_4x_subs.png

21 KiB

subsgrid1_4x_subsBox.png

20 KiB

subs1_30x.png

1.2 KiB

subsgrid1_3x_subs.png

33 KiB

subs1_30xBox.png

1.1 KiB

subsgrid1_15x_subs.png

3.3 KiB

👍 3 ❤️ 1

Aras Pranckevicius force-pushed vse_box_filter from 5dcc8e4c74 to c25412142d

2024-01-28 20:27:59 +01:00

Compare

Aras Pranckevicius commented

2024-01-28 20:39:18 +01:00

@blender-bot build

Aras Pranckevicius changed title from ~~WIP: VSE: replace Subsampled3x3 filter by a general Box filter~~ to VSE: replace Subsampled3x3 filter by a general Box filter

2024-01-28 20:54:51 +01:00

Aras Pranckevicius added this to the Video Sequencer project 2024-01-28 20:54:56 +01:00

Aras Pranckevicius referenced this pull request

2024-01-28 20:55:59 +01:00

VSE: Improve image filtering defaults/options #116980

Aras Pranckevicius requested review from Sergey Sharybin 2024-01-29 05:59:59 +01:00

Sergey Sharybin requested changes 2024-01-29 11:10:35 +01:00

Sergey Sharybin left a comment

Having O(2) complexity could be good for the 3x3 size, but if we allow larger filter sizes we need to avoid it.

The box filter is separable, so you can apply it independently along the X and Y axis, significantly lowering the computational cost.

Having O(2) complexity could be good for the 3x3 size, but if we allow larger filter sizes we need to avoid it. The box filter is [separable](https://en.wikipedia.org/wiki/Separable_filter), so you can apply it independently along the X and Y axis, significantly lowering the computational cost.

Aras Pranckevicius commented

2024-01-29 14:02:40 +01:00

Having O(2) complexity could be good for the 3x3 size, but if we allow larger filter sizes we need to avoid it.

@Sergey I skipped on that so far since I thought that "hey, the upper count of total amount of samples done is bounded by the size of the source media", but you're right, I should at least try a separable filter. Will do!

> Having O(2) complexity could be good for the 3x3 size, but if we allow larger filter sizes we need to avoid it. @Sergey I skipped on that so far since I thought that "hey, the upper count of total amount of samples done is bounded by the size of the source media", but you're right, I should at least try a separable filter. Will do!

Aras Pranckevicius commented

2024-01-29 16:27:41 +01:00

@Sergey though now that I think about it, I'm not quite sure that separable approach would even work here at all. This is not convolution, this is resampling. Each NxM block of source pixels gets averaged into one destination pixel. As a result, all source pixels are read once, and they have to be. There's no way to rearrange or optimize anything; all the source pixels have to be sampled. If this was convolution (e.g. "box blur"), then sure, doing convolution in one direction and then in the other direction would save a lot of sampling, but this is not a convolution.

@Sergey though now that I think about it, I'm not quite sure that separable approach would even work here at all. This is not _convolution_, this is _resampling_. Each NxM block of source pixels gets averaged into one destination pixel. As a result, all source pixels are read once, and they have to be. There's no way to rearrange or optimize anything; all the source pixels have to be sampled. If this was convolution (e.g. "box blur"), then sure, doing convolution in one direction and then in the other direction would save a lot of sampling, but this is not a convolution.

Sergey Sharybin approved these changes 2024-01-29 18:16:11 +01:00

Sergey Sharybin left a comment

This is not convolution, this is resampling

This is a good point. Somehow wasn't that obvious, and thought it is first doing some sort of LPF before resampling. Sorry for the confusion.

> This is not convolution, this is resampling This is a good point. Somehow wasn't that obvious, and thought it is first doing some sort of LPF before resampling. Sorry for the confusion.

Aras Pranckevicius merged commit 5bd1e0bb22 into main

2024-01-29 18:41:40 +01:00

Aras Pranckevicius referenced this issue from a commit

2024-01-29 18:41:40 +01:00

VSE: replace Subsampled3x3 filter by a general Box filter

Aras Pranckevicius deleted branch vse_box_filter

2024-01-29 18:41:42 +01:00

Jonas Dichelle referenced this issue from a commit

2024-02-08 17:29:06 +01:00

VSE: replace Subsampled3x3 filter by a general Box filter

Sign in to join this conversation.

No reviewers

No Label

Download

What's New

Blender Studio

Manual

Developers Blog

Documentation

Benchmark

Blender Conference

Development Fund

One-time Donations

VSE: replace Subsampled3x3 filter by a general Box filter #117584

Notes

Example images