[WIP] VSE: share ImBufAnim resource between strips #118670

Draft
Richard Antalik wants to merge 23 commits from iss/blender:anim-sharing-2 into main

When changing the target branch, be careful to rebase the branch in your fork to match. See documentation.

This patch implements cache for anims, so that multiple strips can reuse this resource. This can significantly reduce memory usage. For example sample, if anims are not freed, Gold edit used 30GB memory in main, but only 6.5GB with this PR. So about 5x less memory.

It supports multiview setups by storing vector of anims. These are stored in order of how they are assigned to the strip. So setups where one of strips which use same input file can be set up as multiview while another is single view.
Since anim is allocated resource, it is tracking user count. The anim itself is freed when it has no users.


For now, this PR is functionally useless, since code in render.cc calls SEQ_relations_free_all_anim_ibufs() which frees anims for all strips apart from strips that are being viewed.

To fully take advantage of this PR we need anim prefetcher, so we can keep bunch of anims around. The tricky part about prefetcher is, that it has to lock rendering mutex.

  • So it has to be singleton
  • Initialize pool of anims
  • Only when anims are loaded, quickly add these to cache.
    The freeing part should be probably responsibility of manager, but perhaps asynchronous to prefetcher. This may get very tricky, will have to experiment with bunch of strategies.

There was idea about integrating this with image cache memory limiter, but only way I see this working is by allocating say 10% to anims, 90% to images. For now I would just hardcode number of anims kept around to... /looks out the window/ ~15. Will need to test a bit to get better number.

With #118112, only the prefetching performance would be sped up, so it is question, whether added complexity is worth this benefit. I expect, that prefetching thread would be stuck on IO operations and not use too much of CPU cycles.


With bit more refactoring StripAnim struct could be removed and code could use this cache directly. This can potentially simplify the codebase (but it could make it more complicated too). However this is not in scope of this PR.

Existing strip prefetch probably could use this cache, but thread safety may prove this tricky.

This patch implements cache for anims, so that multiple strips can reuse this resource. This can significantly reduce memory usage. For example sample, if anims are not freed, Gold edit used 30GB memory in main, but only 6.5GB with this PR. So about 5x less memory. It supports multiview setups by storing vector of anims. These are stored in order of how they are assigned to the strip. So setups where one of strips which use same input file can be set up as multiview while another is single view. Since anim is allocated resource, it is tracking user count. The anim itself is freed when it has no users. ------ For now, this PR is functionally useless, since code in render.cc calls `SEQ_relations_free_all_anim_ibufs()` which frees anims for all strips apart from strips that are being viewed. To fully take advantage of this PR we need anim prefetcher, so we can keep bunch of anims around. The tricky part about prefetcher is, that it has to lock rendering mutex. - So it has to be singleton - Initialize pool of anims - Only when anims are loaded, quickly add these to cache. The freeing part should be probably responsibility of manager, but perhaps asynchronous to prefetcher. This may get very tricky, will have to experiment with bunch of strategies. There was idea about integrating this with image cache memory limiter, but only way I see this working is by allocating say 10% to anims, 90% to images. For now I would just hardcode number of anims kept around to... /looks out the window/ ~15. Will need to test a bit to get better number. With #118112, only the prefetching performance would be sped up, so it is question, whether added complexity is worth this benefit. I expect, that prefetching thread would be stuck on IO operations and not use too much of CPU cycles. ------ With bit more refactoring `StripAnim` struct could be removed and code could use this cache directly. This can potentially simplify the codebase (but it could make it more complicated too). However this is not in scope of this PR. Existing strip prefetch probably could use this cache, but thread safety may prove this tricky.
Richard Antalik added 4 commits 2024-02-23 16:37:37 +01:00
5214fb0d34 VSE: Share ImBufAnims between strips
So far, refactoring has been done.
e7830a92c9 Fix: Crash when multiview movie right view is missing.
Ultimately this was due to missing nullptr check before trying to
process the image, but this should have been caught when loading
ImBufAnims. If any is missing, cancel multiview loading and load
movie as if it was single view only.
Richard Antalik added 1 commit 2024-02-23 16:40:44 +01:00
Richard Antalik added 2 commits 2024-02-23 21:49:17 +01:00
Richard Antalik added 1 commit 2024-02-23 22:08:36 +01:00
Richard Antalik added 1 commit 2024-02-24 19:29:20 +01:00
Richard Antalik added 1 commit 2024-02-24 20:07:39 +01:00
Richard Antalik added 1 commit 2024-02-24 20:44:19 +01:00
Richard Antalik changed title from WIP: VSE: share ImBufAnim resource between strips to VSE: share ImBufAnim resource between strips 2024-02-24 20:55:46 +01:00
Richard Antalik added 1 commit 2024-03-12 09:55:56 +01:00
466517ff87 Add anim prefetcher.
Details are TODO
Richard Antalik added 1 commit 2024-03-21 20:55:22 +01:00
Richard Antalik changed title from VSE: share ImBufAnim resource between strips to [WIP] VSE: share ImBufAnim resource between strips 2024-03-21 20:59:14 +01:00
Richard Antalik added 1 commit 2024-03-21 21:02:23 +01:00
Author
Member

Note to self: The manager has to only prefetch anims. Strips need to acquire anim in main thread, because they are freed by depshraph.

Note to self: The manager has to only prefetch anims. Strips need to acquire anim in main thread, because they are freed by depshraph.
Richard Antalik added 4 commits 2024-05-21 17:32:19 +02:00
Richard Antalik added 1 commit 2024-05-28 22:14:24 +02:00
Richard Antalik added 1 commit 2024-05-28 22:17:24 +02:00
Richard Antalik added 2 commits 2024-05-29 00:04:32 +02:00
Richard Antalik added 1 commit 2024-05-29 01:10:59 +02:00
This pull request has changes conflicting with the target branch.
  • release/datafiles/assets
  • source/blender/blenkernel/intern/scene.cc
  • source/blender/makesdna/DNA_sequence_types.h
  • source/blender/makesrna/intern/rna_color.cc
  • source/blender/sequencer/intern/strip_add.cc
  • source/blender/sequencer/intern/utils.cc

Checkout

From your project repository, check out a new branch and test the changes.
git fetch -u anim-sharing-2:iss-anim-sharing-2
git checkout iss-anim-sharing-2
Sign in to join this conversation.
No reviewers
No Label
Interest
Alembic
Interest
Animation & Rigging
Interest
Asset Browser
Interest
Asset Browser Project Overview
Interest
Audio
Interest
Automated Testing
Interest
Blender Asset Bundle
Interest
BlendFile
Interest
Collada
Interest
Compatibility
Interest
Compositing
Interest
Core
Interest
Cycles
Interest
Dependency Graph
Interest
Development Management
Interest
EEVEE
Interest
EEVEE & Viewport
Interest
Freestyle
Interest
Geometry Nodes
Interest
Grease Pencil
Interest
ID Management
Interest
Images & Movies
Interest
Import Export
Interest
Line Art
Interest
Masking
Interest
Metal
Interest
Modeling
Interest
Modifiers
Interest
Motion Tracking
Interest
Nodes & Physics
Interest
OpenGL
Interest
Overlay
Interest
Overrides
Interest
Performance
Interest
Physics
Interest
Pipeline, Assets & IO
Interest
Platforms, Builds & Tests
Interest
Python API
Interest
Render & Cycles
Interest
Render Pipeline
Interest
Sculpt, Paint & Texture
Interest
Text Editor
Interest
Translations
Interest
Triaging
Interest
Undo
Interest
USD
Interest
User Interface
Interest
UV Editing
Interest
VFX & Video
Interest
Video Sequencer
Interest
Virtual Reality
Interest
Vulkan
Interest
Wayland
Interest
Workbench
Interest: X11
Legacy
Blender 2.8 Project
Legacy
Milestone 1: Basic, Local Asset Browser
Legacy
OpenGL Error
Meta
Good First Issue
Meta
Papercut
Meta
Retrospective
Meta
Security
Module
Animation & Rigging
Module
Core
Module
Development Management
Module
EEVEE & Viewport
Module
Grease Pencil
Module
Modeling
Module
Nodes & Physics
Module
Pipeline, Assets & IO
Module
Platforms, Builds & Tests
Module
Python API
Module
Render & Cycles
Module
Sculpt, Paint & Texture
Module
Triaging
Module
User Interface
Module
VFX & Video
Platform
FreeBSD
Platform
Linux
Platform
macOS
Platform
Windows
Priority
High
Priority
Low
Priority
Normal
Priority
Unbreak Now!
Status
Archived
Status
Confirmed
Status
Duplicate
Status
Needs Info from Developers
Status
Needs Information from User
Status
Needs Triage
Status
Resolved
Type
Bug
Type
Design
Type
Known Issue
Type
Patch
Type
Report
Type
To Do
No Milestone
No project
No Assignees
1 Participants
Notifications
Due Date
The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: blender/blender#118670
No description provided.