VSE: expand "occlusion culling" via opaque strips #117790

New Issue

Aras Pranckevicius · 2024-02-03T21:04:25+01:00

2024-02-03 21:04:25 +01:00

VSE system has optimizations in place where a "known to be opaque" strip, when it covers the whole rendering area, effectively "culls" the lower strips, so they are not rendered at all (including their image/movies data is not loaded, etc.). So that's nice, and in timelines where you'd have many "layers" of media variations for editorial/review purposes with mostly the topmost one visible this gets a lot of both render and playback performance.

However, all of that stops working as soon as your media does not cover the whole screen. E.g. if you have two solid color strips for "letterbox" at top/bottom, with "content" strips in the middle. The content might still be opaque and fully cover "other variants" of content under it, but since it does not cover whole screen, everything gets evaluated.

Idea: track "opaque rectangles" and occlusion cull via them

Proposal is to extend the "opaque strip that covers everything" culling with more fine grained culling:

For a strip that is known to be opaque, record the quad that it covers (note: the quad might be placed/rotated at arbitrary place on screen).
Whenever processing a strip, check if any opaque quads above it fully cover it. If they do, skip processing the strip since it will not contribute anything to the final image.

Prototype results

On this somewhat problematic camera cut from Gold production:

That frame takes 295ms while playing back (budget for 24fps playback: 40ms), because it starts four new ffmpeg contexts -- since there are four new movie clips that start to play all at once. During playing of 200 frames, 15 of them do not fit into 24fps playback budget.

With the system above, this gets down to two movie clips. The "actually finally visible" clip is only one though, but one of the non-visible clips is shifted vertically, and so it does not fall under any single clip from above it (it is partially obscured by the letterbox, and partially by the actually visible clip). However, the two lower clips are completely at the same location as the topmost one, and they are "culled" away from processing. This frame takes 175ms then, which is still way too long, but better than before. And during playback of 200 frames, only 5 frames do not fit into budget.

Questions

How to properly do this. One issue is that by the time a strip image gets "preprocessed" (to account for transform), if it does not cover the full screen it is marked as "this has alpha channel" (near end of sequencer_preprocess_transform_crop). In my prototype impl I extended struct SeqRenderState to store a Map<ImBuf*, int> planes_before_pp, i.e. record, per image buffer, the planes information before it got preprocessed to account for transform. All of this is transient info during rendering, and seems to work, but I have no idea what I'm doing.
How to properly skip strips that are occluded. The seq_render_strip_stack function has two loops over the strips, for reasons that are not entirely clear to me. What I do right now, is add "is this occluded?" check before /* Early out for alpha over part, and for occluded strips set early out to EARLY_USE_INPUT_1. And then also, in the second loop, do occlusion check again. Not sure if that's the right approach at all.

VSE system has optimizations in place where a "known to be opaque" strip, when it covers the whole rendering area, effectively "culls" the lower strips, so they are not rendered at all (including their image/movies data is not loaded, etc.). So that's nice, and in timelines where you'd have many "layers" of media variations for editorial/review purposes with mostly the topmost one visible this gets a lot of both render and playback performance. However, all of that stops working as soon as your media does not cover the whole screen. E.g. if you have two solid color strips for "letterbox" at top/bottom, with "content" strips in the middle. The content might still be opaque and fully cover "other variants" of content under it, but since it does not cover whole screen, everything gets evaluated. ### Idea: track "opaque rectangles" and occlusion cull via them Proposal is to extend the "opaque strip that covers everything" culling with more fine grained culling: - For a strip that is known to be opaque, record the quad that it covers (note: the quad might be placed/rotated at arbitrary place on screen). - Whenever processing a strip, check if any opaque quads above it fully cover it. If they do, skip processing the strip since it will not contribute anything to the final image. ### Prototype results On this somewhat problematic camera cut from Gold production: ![Clipboard02.png](/attachments/bc70215b-3644-4408-b900-f487ffc63644) That frame takes **295ms** while playing back (budget for 24fps playback: 40ms), because it starts four new ffmpeg contexts -- since there are four new movie clips that start to play all at once. During playing of 200 frames, 15 of them do not fit into 24fps playback budget. With the system above, this gets down to two movie clips. The "actually finally visible" clip is only one though, but one of the non-visible clips is shifted vertically, and so it does not fall under any single clip from above it (it is partially obscured by the letterbox, and partially by the actually visible clip). However, the two lower clips are completely at the same location as the topmost one, and they are "culled" away from processing. This frame takes **175ms** then, which is still way too long, but better than before. And during playback of 200 frames, only 5 frames do not fit into budget. ### Questions 1. How to properly do this. One issue is that by the time a strip image gets "preprocessed" (to account for transform), if it does not cover the full screen it is marked as "this has alpha channel" (near end of `sequencer_preprocess_transform_crop`). In my prototype impl I extended `struct SeqRenderState` to store a `Map<ImBuf*, int> planes_before_pp`, i.e. record, per image buffer, the planes information before it got preprocessed to account for transform. All of this is transient info during rendering, and seems to work, but I have no idea what I'm doing. 1. How to properly skip strips that are occluded. The `seq_render_strip_stack` function has two loops over the strips, for reasons that are not entirely clear to me. What I do right now, is add "is this occluded?" check before `/* Early out for alpha over` part, and for occluded strips set early out to `EARLY_USE_INPUT_1`. And then also, in the second loop, do occlusion check again. Not sure if that's the right approach at all.

Clipboard02.png

36 KiB

Aras Pranckevicius added the

Type

Design

label 2024-02-03 21:04:25 +01:00

Aras Pranckevicius self-assigned this 2024-02-03 21:04:34 +01:00

Aras Pranckevicius added this to the Video Sequencer project 2024-02-03 21:04:38 +01:00

Aras Pranckevicius changed title from ~~VSE: expanded "occlusion culling" via opaque strips~~ to VSE: expand "occlusion culling" via opaque strips

2024-02-03 21:10:22 +01:00

Aras Pranckevicius referenced this issue

2024-02-12 15:50:39 +01:00

ffmpeg: cache swscale contexts instead of re-creating them #118130

Richard Antalik referenced this issue

2024-02-12 16:06:13 +01:00

ffmpeg: cache swscale contexts instead of re-creating them #118130

Richard Antalik commented

2024-02-12 17:01:59 +01:00

IMO Ideally you wouldn't even include occluded strip in seq_get_shown_sequences(), but for that you would need to know, if there is any transparency in advance, and that is not quite possible. Technically you can store data about transparency before preprocessing in the strip itself. This would have to be handled when file is changed, but we do that already with resolution, however, this data is re-validated on rendering, which means, that strip needs to be included in seq_get_shown_sequences() and thus it needs at least to load raw image. Not great, but fine.

Now depending on how much you want this optimization to exist, you can assume, that movie strip will never have alpha channel, which changes previous conclusion and things can be simple for this case.

For other strip types, it is a bit more complicated. Will have to over the code...

Rendering starts from topmost strip, which is great. Got alphaover blending, black square 500x500
Not knowing much better at this point, let's go via StripEarlyOut::DoEffect case. So seq_render_strip() is called for next strip
I get the image, it's 32x32 white square. There is seq_input_have_to_preprocess().This one could check things, but it would be best to have dedicated function for this optimization. seq_render_preprocess_ibuf() has some conditions anyway (bit mad...).

Question is what info the function need:
Eeh so lets walk backwards I guess... It needs to know, what operation is invoking this strip rendering - so alpha blending, and size of fully opaque area. I guess it could really be simplified to "opaque area, if none is given, continue checking". Then it needs to know what area the rendered image of current strip would be. Simple enough.

Then question is, what to do with the image - it must not be cached, so you kinda need to return null, which would work I guess..
Not sure if this answered your questions, did not check the code yet.

Also on completely unrelated note, Ages ago, I wanted to implement making text effect to produce images only as large as the text itself. Not sure if this would improve performance (I guess it should a bit). Back then you could move,scale and rotate only with transform effect strip, so now this could be much smoother.

IMO Ideally you wouldn't even include occluded strip in `seq_get_shown_sequences()`, but for that you would need to know, if there is any transparency in advance, and that is not quite possible. Technically you can store data about transparency before preprocessing in the strip itself. This would have to be handled when file is changed, but we do that already with resolution, however, this data is re-validated on rendering, which means, that strip needs to be included in `seq_get_shown_sequences()` and thus it needs at least to load raw image. Not great, but fine. Now depending on how much you want this optimization to exist, you can assume, that movie strip will never have alpha channel, which changes previous conclusion and things can be simple for this case. For other strip types, it is a bit more complicated. Will have to over the code... - Rendering starts from topmost strip, which is great. Got alphaover blending, black square 500x500 - Not knowing much better at this point, let's go via `StripEarlyOut::DoEffect` case. So `seq_render_strip()` is called for next strip - I get the image, it's 32x32 white square. There is `seq_input_have_to_preprocess()`.This one could check things, but it would be best to have dedicated function for this optimization. `seq_render_preprocess_ibuf()` has some conditions anyway (bit mad...). Question is what info the function need: Eeh so lets walk backwards I guess... It needs to know, what operation is invoking this strip rendering - so alpha blending, and size of fully opaque area. I guess it could really be simplified to "opaque area, if none is given, continue checking". Then it needs to know what area the rendered image of current strip would be. Simple enough. Then question is, what to do with the image - it must not be cached, so you kinda need to return null, which would work I guess.. Not sure if this answered your questions, did not check the code yet. Also on completely unrelated note, Ages ago, I wanted to implement making text effect to produce images only as large as the text itself. Not sure if this would improve performance (I guess it should a bit). Back then you could move,scale and rotate only with transform effect strip, so now this could be much smoother.

👍 1

Aras Pranckevicius referenced this issue

2024-02-16 21:25:43 +01:00

VSE: Skip rendering lower strips that are behind opaque strips above them #118396

Aras Pranckevicius referenced this issue from a commit

2024-02-21 20:16:54 +01:00

VSE: Skip rendering lower strips that are behind opaque strips above them

Aras Pranckevicius commented

2024-03-22 09:56:21 +01:00

Implemented in #118396

Aras Pranckevicius closed this issue

2024-03-22 09:56:24 +01:00