[WIP] VSE: share ImBufAnim resource between strips #118670

Richard Antalik · 2024-02-23T16:37:36+01:00

Richard Antalik commented

2024-02-23 16:37:36 +01:00

This patch implements cache for anims, so that multiple strips can reuse this resource. This can significantly reduce memory usage. For example sample, if anims are not freed, Gold edit used 30GB memory in main, but only 6.5GB with this PR. So about 5x less memory.

It supports multiview setups by storing vector of anims. These are stored in order of how they are assigned to the strip. So setups where one of strips which use same input file can be set up as multiview while another is single view.
Since anim is allocated resource, it is tracking user count. The anim itself is freed when it has no users.

For now, this PR is functionally useless, since code in render.cc calls SEQ_relations_free_all_anim_ibufs() which frees anims for all strips apart from strips that are being viewed.

To fully take advantage of this PR we need anim prefetcher, so we can keep bunch of anims around. The tricky part about prefetcher is, that it has to lock rendering mutex.

So it has to be singleton
Initialize pool of anims
Only when anims are loaded, quickly add these to cache.
The freeing part should be probably responsibility of manager, but perhaps asynchronous to prefetcher. This may get very tricky, will have to experiment with bunch of strategies.

There was idea about integrating this with image cache memory limiter, but only way I see this working is by allocating say 10% to anims, 90% to images. For now I would just hardcode number of anims kept around to... /looks out the window/ ~15. Will need to test a bit to get better number.

With #118112, only the prefetching performance would be sped up, so it is question, whether added complexity is worth this benefit. I expect, that prefetching thread would be stuck on IO operations and not use too much of CPU cycles.

With bit more refactoring StripAnim struct could be removed and code could use this cache directly. This can potentially simplify the codebase (but it could make it more complicated too). However this is not in scope of this PR.

Existing strip prefetch probably could use this cache, but thread safety may prove this tricky.

This patch implements cache for anims, so that multiple strips can reuse this resource. This can significantly reduce memory usage. For example sample, if anims are not freed, Gold edit used 30GB memory in main, but only 6.5GB with this PR. So about 5x less memory. It supports multiview setups by storing vector of anims. These are stored in order of how they are assigned to the strip. So setups where one of strips which use same input file can be set up as multiview while another is single view. Since anim is allocated resource, it is tracking user count. The anim itself is freed when it has no users. ------ For now, this PR is functionally useless, since code in render.cc calls `SEQ_relations_free_all_anim_ibufs()` which frees anims for all strips apart from strips that are being viewed. To fully take advantage of this PR we need anim prefetcher, so we can keep bunch of anims around. The tricky part about prefetcher is, that it has to lock rendering mutex. - So it has to be singleton - Initialize pool of anims - Only when anims are loaded, quickly add these to cache. The freeing part should be probably responsibility of manager, but perhaps asynchronous to prefetcher. This may get very tricky, will have to experiment with bunch of strategies. There was idea about integrating this with image cache memory limiter, but only way I see this working is by allocating say 10% to anims, 90% to images. For now I would just hardcode number of anims kept around to... /looks out the window/ ~15. Will need to test a bit to get better number. With #118112, only the prefetching performance would be sped up, so it is question, whether added complexity is worth this benefit. I expect, that prefetching thread would be stuck on IO operations and not use too much of CPU cycles. ------ With bit more refactoring `StripAnim` struct could be removed and code could use this cache directly. This can potentially simplify the codebase (but it could make it more complicated too). However this is not in scope of this PR. Existing strip prefetch probably could use this cache, but thread safety may prove this tricky.

Richard Antalik added 4 commits 2024-02-23 16:37:37 +01:00

5214fb0d34 VSE: Share ImBufAnims between strips

So far, refactoring has been done.

e7830a92c9 Fix: Crash when multiview movie right view is missing.

Ultimately this was due to missing nullptr check before trying to
process the image, but this should have been caught when loading
ImBufAnims. If any is missing, cancel multiview loading and load
movie as if it was single view only.

3156e9c26c Merge branch 'main' into anim-sharing-2

3b29303e28 Fix bunch of stuff

Richard Antalik added 1 commit 2024-02-23 16:40:44 +01:00

675d22b817 Forgot to include bloody files again

Richard Antalik added 2 commits 2024-02-23 21:49:17 +01:00

a1c11af809 Integrate multiview in anims cache. At least in theory. In practice it does not work...

3c3a7814c2 Fix memory freeing issue

Richard Antalik added 1 commit 2024-02-23 22:08:36 +01:00

fa4d0b2648 Fix multiview not working

Richard Antalik added 1 commit 2024-02-24 19:29:20 +01:00

98065a137d Rename anim lookup to anim manager, since this will be babysitting anim usage in VSE.

Richard Antalik added 1 commit 2024-02-24 20:07:39 +01:00