ffmpeg: cache swscale contexts instead of re-creating them #118130

Aras Pranckevicius · 2024-02-12T10:35:19+01:00

2024-02-12 10:35:19 +01:00

ffmpeg's libswscale is used to do RGB<->YUV conversions on movie reading
and writing. The "context" for the scale operation was being created
and destroyed for each movie clip / animation. Now, maintain a cache
of the scale contexts instead.

E.g. in Gold edit, it only ever needs two contexts (one for reading
source movie clips since they are all exactly the same resolution
and format; and one for rendering the resulting movie).

During playback, on some of the "slow" frames (camera cuts) this
saves 10-20ms (Windows, Ryzen 5950X). Rendering whole movie goes
from 390sec to 376sec.

ffmpeg's libswscale is used to do RGB<->YUV conversions on movie reading and writing. The "context" for the scale operation was being created and destroyed for each movie clip / animation. Now, maintain a cache of the scale contexts instead. E.g. in Gold edit, it only ever needs two contexts (one for reading source movie clips since they are all exactly the same resolution and format; and one for rendering the resulting movie). During playback, on some of the "slow" frames (camera cuts) this saves 10-20ms (Windows, Ryzen 5950X). Rendering whole movie goes from 390sec to 376sec.

❤️ 2

Aras Pranckevicius added 1 commit 2024-02-12 10:35:29 +01:00

buildbot/vexp-code-patch-lint Build done. Details

buildbot/vexp-code-patch-linux-x86_64 Build done. Details

buildbot/vexp-code-patch-darwin-x86_64 Build done. Details

buildbot/vexp-code-patch-windows-amd64 Build done. Details

buildbot/vexp-code-patch-darwin-arm64 Build done. Details

buildbot/vexp-code-patch-coordinator Build done. Details

0ac48647bb ffmpeg: cache swscale contexts instead of re-creating them

ffmpeg's libswscale is used to do RGB<->YUV conversions on movie reading
and writing. The "context" for the scale operation was being created
and destroyed for each movie clip / animation. Now, maintain a cache
of the scale contexts instead.

E.g. in Gold edit, it only ever needs two contexts (one for reading
source movie clips since they are all exactly the same resolution
and format; and one for rendering the resulting movie).

During playback, on some of the "slow" frames (camera cuts) this
saves 10-20ms (Windows, Ryzen 5950X). Rendering whole movie goes
from 390sec to 376sec.

Aras Pranckevicius commented

2024-02-12 10:35:43 +01:00

@blender-bot build

Aras Pranckevicius requested review from Sergey Sharybin 2024-02-12 10:53:19 +01:00

Aras Pranckevicius requested review from Richard Antalik 2024-02-12 10:53:30 +01:00

Aras Pranckevicius added this to the Video Sequencer project 2024-02-12 11:33:44 +01:00

Sergey Sharybin reviewed 2024-02-12 11:36:45 +01:00

Sergey Sharybin left a comment

This seems quite nice speedup for a simple change :)

Are these contexts "heavy"? As in, if there is a lot of media with mismatched resolutions, are we fine having all contexts in-memory, or do we want to eventually free unused ones, to give renderer more RAM?

This seems quite nice speedup for a simple change :) Are these contexts "heavy"? As in, if there is a lot of media with mismatched resolutions, are we fine having all contexts in-memory, or do we want to eventually free unused ones, to give renderer more RAM?

Aras Pranckevicius commented

2024-02-12 12:01:15 +01:00

@Sergey yeah I was thinking about that. They do internally create some threads, some lookup tables and possibly some buffers. Haven't measured how "heavy" they are exactly. I was thinking what should be the "proper logic" for that though, I would imagine:

Make the cache only store "up to N" contexts, like say 16. Number pulled out of a hat, any number larger than 1 is better than no cache at all. Oldest contexts are removed once cache is full.
Additionally (or instead?), at "some points" the cache is just flushed. A good question is, when to do that. Entering/Stopping the playback? Starting/finishing rendering? Loading a file? something else?

@Sergey yeah I was thinking about that. They do internally create some threads, some lookup tables and possibly some buffers. Haven't measured how "heavy" they are exactly. I was thinking what should be the "proper logic" for that though, I would imagine: - Make the cache only store "up to N" contexts, like say 16. Number pulled out of a hat, any number larger than 1 is better than no cache at all. Oldest contexts are removed once cache is full. - Additionally (or instead?), at "some points" the cache is just flushed. A good question is, when to do that. Entering/Stopping the playback? Starting/finishing rendering? Loading a file? something else?

Sergey Sharybin commented

2024-02-12 12:06:04 +01:00

Having some LRU mechanic of N contexts seems like a good thing.

I wouldn't worry too much about flushing it entirely on playback stop. Odds that you'll resume playback very soon, and having lower latency will be welcome.

Having some LRU mechanic of N contexts seems like a good thing. I wouldn't worry too much about flushing it entirely on playback stop. Odds that you'll resume playback very soon, and having lower latency will be welcome.

👍 1

Richard Antalik commented

2024-02-12 15:07:08 +01:00

I have experimented a bit yesterday and created 2 builds to compare delays during cuts: #118112 and #118114.
From the results, it seems that all ffmpeg data other than AVCodecContext uses only 14MB. So I wouldn't be conservative with how much of swscale contexts would you keep around. IMO it would be fine to not care about that at all especially if these are re-used.

I have an idea how to combine those 2 PR's to significantly improve this decoding delay by keeping around some data ready to go. If AVCodecContext could be shared across anims (technically it can be) it would improve this much much more, but that may be bit more tricky.

This PR seems to be well aligned with what I would want to do.

I have experimented a bit yesterday and created 2 builds to compare delays during cuts: #118112 and #118114. From the results, it seems that all ffmpeg data other than `AVCodecContext` uses only 14MB. So I wouldn't be conservative with how much of swscale contexts would you keep around. IMO it would be fine to not care about that at all especially if these are re-used. I have an idea how to combine those 2 PR's to significantly improve this decoding delay by keeping around some data ready to go. If `AVCodecContext` could be shared across anims (technically it can be) it would improve this much much more, but that may be bit more tricky. This PR seems to be well aligned with what I would want to do.

❤️ 1

Aras Pranckevicius commented

2024-02-12 15:50:39 +01:00

@iss not directly related to this, but I've been looking at better "culling" of strips (#117790) which particularly on Gold seem to bring in quite some benefits (none of the strips there cover "whole screen", but it often happens that there are three movies one atop another, with only topmost being visible).

I need to add more extensive testing to that before it's ready, but so far looking promising. Feels like that, plus faster ffmpeg "context things", might be good to avoid the worst playback hiccups.

@iss not directly related to this, but I've been looking at better "culling" of strips (#117790) which particularly on Gold seem to bring in quite some benefits (none of the strips there cover "whole screen", but it often happens that there are three movies one atop another, with only topmost being visible). I need to add more extensive testing to that before it's ready, but so far looking promising. Feels like that, plus faster ffmpeg "context things", might be good to avoid the worst playback hiccups.

Richard Antalik approved these changes 2024-02-12 15:56:27 +01:00

Richard Antalik commented

2024-02-12 16:06:13 +01:00

@iss not directly related to this, but I've been looking at better "culling" of strips (#117790) which particularly on Gold seem to bring in quite some benefits (none of the strips there cover "whole screen", but it often happens that there are three movies one atop another, with only topmost being visible).

Yes, I have noticed that. It's nice to have a way to detect such case.

I need to add more extensive testing to that before it's ready, but so far looking promising. Feels like that, plus faster ffmpeg "context things", might be good to avoid the worst playback hiccups.

The media used in Gold edit are quite optimized for fast decoding. I am not sure, but wouldn't be surprised if 4K movie with large GOP size would take much longer to create codec context. Also other users may be facing situations where they need to render 3 movies at once. So IMO this delay should be optimized out if possible.

> @iss not directly related to this, but I've been looking at better "culling" of strips (#117790) which particularly on Gold seem to bring in quite some benefits (none of the strips there cover "whole screen", but it often happens that there are three movies one atop another, with only topmost being visible). Yes, I have noticed that. It's nice to have a way to detect such case. > I need to add more extensive testing to that before it's ready, but so far looking promising. Feels like that, plus faster ffmpeg "context things", might be good to avoid the worst playback hiccups. The media used in Gold edit are quite optimized for fast decoding. I am not sure, but wouldn't be surprised if 4K movie with large GOP size would take much longer to create codec context. Also other users may be facing situations where they need to render 3 movies at once. So IMO this delay should be optimized out if possible.

👍 1

Aras Pranckevicius added 1 commit 2024-02-13 10:08:19 +01:00

buildbot/vexp-code-patch-lint Build done. Details

buildbot/vexp-code-patch-linux-x86_64 Build done. Details

buildbot/vexp-code-patch-darwin-arm64 Build done. Details

buildbot/vexp-code-patch-windows-amd64 Build done. Details

buildbot/vexp-code-patch-darwin-x86_64 Build done. Details

buildbot/vexp-code-patch-coordinator Build done. Details

89d5801d6b ffmpeg: ensure swscale context cache does not get too large

Do not go above 32 unused contexts being kept alive. When exceeding
that, remove the oldest used ones.

Aras Pranckevicius commented

2024-02-13 10:09:03 +01:00

Added the context scale cache max cap. Now no more than 32 spare swscale contexts are kept around.

Aras Pranckevicius added 2 commits 2024-02-13 10:23:12 +01:00

808be4c910 Format

buildbot/vexp-code-patch-windows-amd64 Build done. Details

buildbot/vexp-code-patch-lint Build done. Details

buildbot/vexp-code-patch-linux-x86_64 Build done. Details

buildbot/vexp-code-patch-darwin-x86_64 Build done. Details

buildbot/vexp-code-patch-darwin-arm64 Build done. Details

buildbot/vexp-code-patch-coordinator Build done. Details

7782025ea8 Merge branch 'main' into ffmpeg_swscale_cache

Aras Pranckevicius commented

2024-02-13 10:23:30 +01:00

@blender-bot build

Sergey Sharybin approved these changes 2024-02-15 10:30:05 +01:00

Aras Pranckevicius merged commit ffbc90874b into main

2024-02-15 10:35:13 +01:00

Aras Pranckevicius referenced this issue from a commit

2024-02-15 10:35:14 +01:00

ffmpeg: cache swscale contexts instead of re-creating them

Aras Pranckevicius deleted branch ffmpeg_swscale_cache

2024-02-15 10:35:15 +01:00

Sign in to join this conversation.

No reviewers

No Label

Download

What's New

Blender Studio

Manual

Developers Blog

Documentation

Benchmark

Blender Conference

Development Fund

One-time Donations

ffmpeg: cache swscale contexts instead of re-creating them #118130