ffmpeg: cache swscale contexts instead of re-creating them #118130

Merged
Aras Pranckevicius merged 4 commits from aras_p/blender:ffmpeg_swscale_cache into main 2024-02-15 10:35:13 +01:00

ffmpeg's libswscale is used to do RGB<->YUV conversions on movie reading
and writing. The "context" for the scale operation was being created
and destroyed for each movie clip / animation. Now, maintain a cache
of the scale contexts instead.

E.g. in Gold edit, it only ever needs two contexts (one for reading
source movie clips since they are all exactly the same resolution
and format; and one for rendering the resulting movie).

During playback, on some of the "slow" frames (camera cuts) this
saves 10-20ms (Windows, Ryzen 5950X). Rendering whole movie goes
from 390sec to 376sec.

ffmpeg's libswscale is used to do RGB<->YUV conversions on movie reading and writing. The "context" for the scale operation was being created and destroyed for each movie clip / animation. Now, maintain a cache of the scale contexts instead. E.g. in Gold edit, it only ever needs two contexts (one for reading source movie clips since they are all exactly the same resolution and format; and one for rendering the resulting movie). During playback, on some of the "slow" frames (camera cuts) this saves 10-20ms (Windows, Ryzen 5950X). Rendering whole movie goes from 390sec to 376sec.
Aras Pranckevicius added 1 commit 2024-02-12 10:35:29 +01:00
buildbot/vexp-code-patch-lint Build done. Details
buildbot/vexp-code-patch-linux-x86_64 Build done. Details
buildbot/vexp-code-patch-darwin-x86_64 Build done. Details
buildbot/vexp-code-patch-windows-amd64 Build done. Details
buildbot/vexp-code-patch-darwin-arm64 Build done. Details
buildbot/vexp-code-patch-coordinator Build done. Details
0ac48647bb
ffmpeg: cache swscale contexts instead of re-creating them
ffmpeg's libswscale is used to do RGB<->YUV conversions on movie reading
and writing. The "context" for the scale operation was being created
and destroyed for each movie clip / animation. Now, maintain a cache
of the scale contexts instead.

E.g. in Gold edit, it only ever needs two contexts (one for reading
source movie clips since they are all exactly the same resolution
and format; and one for rendering the resulting movie).

During playback, on some of the "slow" frames (camera cuts) this
saves 10-20ms (Windows, Ryzen 5950X). Rendering whole movie goes
from 390sec to 376sec.
Author
Member

@blender-bot build

@blender-bot build
Aras Pranckevicius requested review from Sergey Sharybin 2024-02-12 10:53:19 +01:00
Aras Pranckevicius requested review from Richard Antalik 2024-02-12 10:53:30 +01:00
Aras Pranckevicius added this to the Video Sequencer project 2024-02-12 11:33:44 +01:00
Sergey Sharybin reviewed 2024-02-12 11:36:45 +01:00
Sergey Sharybin left a comment
Owner

This seems quite nice speedup for a simple change :)

Are these contexts "heavy"? As in, if there is a lot of media with mismatched resolutions, are we fine having all contexts in-memory, or do we want to eventually free unused ones, to give renderer more RAM?

This seems quite nice speedup for a simple change :) Are these contexts "heavy"? As in, if there is a lot of media with mismatched resolutions, are we fine having all contexts in-memory, or do we want to eventually free unused ones, to give renderer more RAM?
Author
Member

@Sergey yeah I was thinking about that. They do internally create some threads, some lookup tables and possibly some buffers. Haven't measured how "heavy" they are exactly. I was thinking what should be the "proper logic" for that though, I would imagine:

  • Make the cache only store "up to N" contexts, like say 16. Number pulled out of a hat, any number larger than 1 is better than no cache at all. Oldest contexts are removed once cache is full.
  • Additionally (or instead?), at "some points" the cache is just flushed. A good question is, when to do that. Entering/Stopping the playback? Starting/finishing rendering? Loading a file? something else?
@Sergey yeah I was thinking about that. They do internally create some threads, some lookup tables and possibly some buffers. Haven't measured how "heavy" they are exactly. I was thinking what should be the "proper logic" for that though, I would imagine: - Make the cache only store "up to N" contexts, like say 16. Number pulled out of a hat, any number larger than 1 is better than no cache at all. Oldest contexts are removed once cache is full. - Additionally (or instead?), at "some points" the cache is just flushed. A good question is, when to do that. Entering/Stopping the playback? Starting/finishing rendering? Loading a file? something else?

Having some LRU mechanic of N contexts seems like a good thing.

I wouldn't worry too much about flushing it entirely on playback stop. Odds that you'll resume playback very soon, and having lower latency will be welcome.

Having some LRU mechanic of N contexts seems like a good thing. I wouldn't worry too much about flushing it entirely on playback stop. Odds that you'll resume playback very soon, and having lower latency will be welcome.

I have experimented a bit yesterday and created 2 builds to compare delays during cuts: #118112 and #118114.
From the results, it seems that all ffmpeg data other than AVCodecContext uses only 14MB. So I wouldn't be conservative with how much of swscale contexts would you keep around. IMO it would be fine to not care about that at all especially if these are re-used.

I have an idea how to combine those 2 PR's to significantly improve this decoding delay by keeping around some data ready to go. If AVCodecContext could be shared across anims (technically it can be) it would improve this much much more, but that may be bit more tricky.

This PR seems to be well aligned with what I would want to do.

I have experimented a bit yesterday and created 2 builds to compare delays during cuts: #118112 and #118114. From the results, it seems that all ffmpeg data other than `AVCodecContext` uses only 14MB. So I wouldn't be conservative with how much of swscale contexts would you keep around. IMO it would be fine to not care about that at all especially if these are re-used. I have an idea how to combine those 2 PR's to significantly improve this decoding delay by keeping around some data ready to go. If `AVCodecContext` could be shared across anims (technically it can be) it would improve this much much more, but that may be bit more tricky. This PR seems to be well aligned with what I would want to do.
Author
Member

@iss not directly related to this, but I've been looking at better "culling" of strips (#117790) which particularly on Gold seem to bring in quite some benefits (none of the strips there cover "whole screen", but it often happens that there are three movies one atop another, with only topmost being visible).

I need to add more extensive testing to that before it's ready, but so far looking promising. Feels like that, plus faster ffmpeg "context things", might be good to avoid the worst playback hiccups.

@iss not directly related to this, but I've been looking at better "culling" of strips (#117790) which particularly on Gold seem to bring in quite some benefits (none of the strips there cover "whole screen", but it often happens that there are three movies one atop another, with only topmost being visible). I need to add more extensive testing to that before it's ready, but so far looking promising. Feels like that, plus faster ffmpeg "context things", might be good to avoid the worst playback hiccups.
Richard Antalik approved these changes 2024-02-12 15:56:27 +01:00

@iss not directly related to this, but I've been looking at better "culling" of strips (#117790) which particularly on Gold seem to bring in quite some benefits (none of the strips there cover "whole screen", but it often happens that there are three movies one atop another, with only topmost being visible).

Yes, I have noticed that. It's nice to have a way to detect such case.

I need to add more extensive testing to that before it's ready, but so far looking promising. Feels like that, plus faster ffmpeg "context things", might be good to avoid the worst playback hiccups.

The media used in Gold edit are quite optimized for fast decoding. I am not sure, but wouldn't be surprised if 4K movie with large GOP size would take much longer to create codec context. Also other users may be facing situations where they need to render 3 movies at once. So IMO this delay should be optimized out if possible.

> @iss not directly related to this, but I've been looking at better "culling" of strips (#117790) which particularly on Gold seem to bring in quite some benefits (none of the strips there cover "whole screen", but it often happens that there are three movies one atop another, with only topmost being visible). Yes, I have noticed that. It's nice to have a way to detect such case. > I need to add more extensive testing to that before it's ready, but so far looking promising. Feels like that, plus faster ffmpeg "context things", might be good to avoid the worst playback hiccups. The media used in Gold edit are quite optimized for fast decoding. I am not sure, but wouldn't be surprised if 4K movie with large GOP size would take much longer to create codec context. Also other users may be facing situations where they need to render 3 movies at once. So IMO this delay should be optimized out if possible.
Aras Pranckevicius added 1 commit 2024-02-13 10:08:19 +01:00
buildbot/vexp-code-patch-lint Build done. Details
buildbot/vexp-code-patch-linux-x86_64 Build done. Details
buildbot/vexp-code-patch-darwin-arm64 Build done. Details
buildbot/vexp-code-patch-windows-amd64 Build done. Details
buildbot/vexp-code-patch-darwin-x86_64 Build done. Details
buildbot/vexp-code-patch-coordinator Build done. Details
89d5801d6b
ffmpeg: ensure swscale context cache does not get too large
Do not go above 32 unused contexts being kept alive. When exceeding
that, remove the oldest used ones.
Author
Member

Added the context scale cache max cap. Now no more than 32 spare swscale contexts are kept around.

Added the context scale cache max cap. Now no more than 32 spare swscale contexts are kept around.
Aras Pranckevicius added 2 commits 2024-02-13 10:23:12 +01:00
buildbot/vexp-code-patch-windows-amd64 Build done. Details
buildbot/vexp-code-patch-lint Build done. Details
buildbot/vexp-code-patch-linux-x86_64 Build done. Details
buildbot/vexp-code-patch-darwin-x86_64 Build done. Details
buildbot/vexp-code-patch-darwin-arm64 Build done. Details
buildbot/vexp-code-patch-coordinator Build done. Details
7782025ea8
Merge branch 'main' into ffmpeg_swscale_cache
Author
Member

@blender-bot build

@blender-bot build
Sergey Sharybin approved these changes 2024-02-15 10:30:05 +01:00
Aras Pranckevicius merged commit ffbc90874b into main 2024-02-15 10:35:13 +01:00
Aras Pranckevicius deleted branch ffmpeg_swscale_cache 2024-02-15 10:35:15 +01:00
Sign in to join this conversation.
No Label
Interest
Alembic
Interest
Animation & Rigging
Interest
Asset Browser Project Overview
Interest
Asset System
Interest
Audio
Interest
Automated Testing
Interest
Blender Asset Bundle
Interest
BlendFile
Interest
Collada
Interest
Compatibility
Interest
Compositing
Interest
Core
Interest
Cycles
Interest
Dependency Graph
Interest
Development Management
Interest
EEVEE
Interest
EEVEE & Viewport
Interest
Freestyle
Interest
Geometry Nodes
Interest
Grease Pencil
Interest
ID Management
Interest
Images & Movies
Interest
Import Export
Interest
Line Art
Interest
Masking
Interest
Metal
Interest
Modeling
Interest
Modifiers
Interest
Motion Tracking
Interest
Nodes & Physics
Interest
OpenGL
Interest
Overlay
Interest
Overrides
Interest
Performance
Interest
Physics
Interest
Pipeline, Assets & IO
Interest
Platforms, Builds & Tests
Interest
Python API
Interest
Render & Cycles
Interest
Render Pipeline
Interest
Sculpt, Paint & Texture
Interest
Text Editor
Interest
Translations
Interest
Triaging
Interest
Undo
Interest
USD
Interest
User Interface
Interest
UV Editing
Interest
VFX & Video
Interest
Video Sequencer
Interest
Virtual Reality
Interest
Vulkan
Interest
Wayland
Interest
Workbench
Interest: X11
Legacy
Blender 2.8 Project
Legacy
Milestone 1: Basic, Local Asset Browser
Legacy
OpenGL Error
Meta
Good First Issue
Meta
Papercut
Meta
Retrospective
Meta
Security
Module
Animation & Rigging
Module
Core
Module
Development Management
Module
EEVEE & Viewport
Module
Grease Pencil
Module
Modeling
Module
Nodes & Physics
Module
Pipeline, Assets & IO
Module
Platforms, Builds & Tests
Module
Python API
Module
Render & Cycles
Module
Sculpt, Paint & Texture
Module
Triaging
Module
User Interface
Module
VFX & Video
Platform
FreeBSD
Platform
Linux
Platform
macOS
Platform
Windows
Priority
High
Priority
Low
Priority
Normal
Priority
Unbreak Now!
Status
Archived
Status
Confirmed
Status
Duplicate
Status
Needs Info from Developers
Status
Needs Information from User
Status
Needs Triage
Status
Resolved
Type
Bug
Type
Design
Type
Known Issue
Type
Patch
Type
Report
Type
To Do
No Milestone
No Assignees
4 Participants
Notifications
Due Date
The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: blender/blender#118130
No description provided.