VSE 2.0: Performance #78992

New Issue

Sergey Sharybin · 2020-07-16T14:49:54+02:00

Sergey Sharybin commented

2020-07-16 14:49:54 +02:00

This task is to keep track of performance-related topics which needs to be addressed in order to achieve pleasant video editing experience. If any of the topics needs any deeper consideration and discussion it is to become a sub-task of this one.

Flat assorted list of performance and memory-usage related topics:

- Seek in video files is rather slow, feels measurably (2x) slower compared to seek in VLC with the same video file.
- Proxy file generation is slower than realtime. Sometimes only time code (TC) is needed, so, probably, the solution is to allow generating TC without scaled-down video. Encoding of different resolution should be possible to do in separate threads.
- Color strip puts ever frame in cache. Simple to reproduce: crate new 4K project, add black color strip with the duration of entire 250 frames, playback. In theory, the final VSE frame cache should be able to re-use final strip frame.
- Image strip caches every frame. Similar to the color case: adding and stretching image to occupy multiple frames will lead to way higher memory usage. This makes memory usage way higher in projects like story board, where edit consists of handful of image strips, and every image is stretched to occupy a frame range.
- Image strips performs read-from-disk on every frame. Again, leads to worse performance for story-board like edit.
- Moving image and cached video strips while being under the playhead is very slow. In 2.79 VSE the strip cache was attempted to be re-used as much as possible. Image strips should not be doing re-reads at all, video strips should reuse frames which are already in the cache.
- Operate in preview resolution. Operating and caching on original 4K images just to make the result be 320x240px in preview is very wasteful. Operation and cache should use the preview resolution. This will lower memory usage for all types of projects, and will make VSE faster when a lot of effects are used.

For the caching related topics see #80278 as well.

This task is to keep track of performance-related topics which needs to be addressed in order to achieve pleasant video editing experience. If any of the topics needs any deeper consideration and discussion it is to become a sub-task of this one. Flat assorted list of performance and memory-usage related topics: * - [x] Seek in video files is rather slow, feels measurably (2x) slower compared to seek in VLC with the same video file. * - [x] Proxy file generation is slower than realtime. Sometimes only time code (TC) is needed, so, probably, the solution is to allow generating TC without scaled-down video. Encoding of different resolution should be possible to do in separate threads. * - [ ] Color strip puts ever frame in cache. Simple to reproduce: crate new 4K project, add black color strip with the duration of entire 250 frames, playback. In theory, the final VSE frame cache should be able to re-use final strip frame. * - [x] Image strip caches every frame. Similar to the color case: adding and stretching image to occupy multiple frames will lead to way higher memory usage. This makes memory usage way higher in projects like story board, where edit consists of handful of image strips, and every image is stretched to occupy a frame range. * - [x] Image strips performs read-from-disk on every frame. Again, leads to worse performance for story-board like edit. * - [x] Moving image and cached video strips while being under the playhead is very slow. In 2.79 VSE the strip cache was attempted to be re-used as much as possible. Image strips should not be doing re-reads at all, video strips should reuse frames which are already in the cache. * - [ ] Operate in preview resolution. Operating and caching on original 4K images just to make the result be 320x240px in preview is very wasteful. Operation and cache should use the preview resolution. This will lower memory usage for all types of projects, and will make VSE faster when a lot of effects are used. For the caching related topics see #80278 as well.

Sergey Sharybin commented

2020-07-16 14:49:54 +02:00

Changed status from 'Needs Triage' to: 'Confirmed'

Sergey Sharybin commented

2020-07-16 14:49:54 +02:00

Added subscribers: @Sergey, @fsiddi, @iss, @dfelinto

Domingo Riesco commented

2020-07-21 12:56:30 +02:00

Added subscriber: @domingo

tintwotin commented

2020-08-12 09:42:17 +02:00

Added subscriber: @tintwotin

tintwotin commented

2020-08-12 09:42:17 +02:00

Seek in video files is rather slow, feels measurably (2x) slower compared to seek in VLC with the same video file.
Proxy file generation is slower than realtime. Sometimes only time code (TC) is needed, so, probably, the solution is to allow generating TC without scaled-down video. Encoding of different resolution should be possible to do in separate threads.

If the codec of the proxies is changed from mjpeg to h264(the avi container would still work), all sorts of hardware accelerated options will be available.

Adding this to the h264 command line: -g 1 -preset ultrafast -tune fastdecode -crf 0 make the files behave like mjpeg(but really small).
-g 1 generate full images(like mjpeg).
-preset ultrafast fast encoding.
-tune fastdecode fast decoding.
-crf 0 (Should be a optional setting) 0=lossless
https://wiki.libav.org/Encoding/h264

Alternatively HDxHR doesn't have the resolution restrictions of DNxHD and could be used for proxies, but compared to h264 this is an i-frame heavy codec(like mjpeg) and will produce a huge files(which also could be used as intermediate files(improved seeking)). It comes in a variety of options including 10-bit, but I don't think avi will work as a container, so going this way will mean a new container format(which also could contain metadata).

For keeping opacity in proxies, png in mov container could be used for proxies.

For optimal quality for color grading, EXR format could be considered for proxies, however since 10-bit is not enabled in the VSE, this may currently be overkill?

On the playback of proxies, a mismatch between proxy-file resolution and project-resolution will cause loss in fps. https://developer.blender.org/T76726

And the additional scaling of the proxies in order to get text(text strips) scaled correct will also result in a loss of fps. https://developer.blender.org/T76724

On hardware enhanced encoding and decoding: https://developer.nvidia.com/blog/nvidia-ffmpeg-transcoding-guide/#disqus_thread

There is a slight pause in playhead movement when seeking(arrow keys), this should be removed, if possible?

The seeking backwards is poor when Prefetch Frames is on: https://developer.blender.org/T77209
(A fairly normal operation is to quickly seek backwards while playing forward, but the cached area is kept ahead of the playhead, so this operation will choke because of uncached frames before the playhead.)

To improve the seeking control maybe an industry-standard of seeking could be considered? Ex. tabbing forward(L key) will increase seek speed and tabbing backward(J key), will decrease seek speed(and seek reverse), (K for stop).
(These controls could also be used in modal-move/extend operations)

On Threads, as the libav seems to be running multithreaded, it's on the Blender side of things things aren't, which is why the various parallel render add-ons will render faster. Ex. https://github.com/elmopl/ktba/blob/master/scripts/addons/parallel_render.py So maybe it's better to deal with the image processing on the Blender side than separating generation of various proxy files into separate threads?

On the cluttered and incomprehensive Proxy UI:
https://devtalk.blender.org/t/design-proposal-proxy-use-in-the-video-sequence-editor/12065
https://developer.blender.org/D6077

Moving image and cached video strips while being under the playhead is very slow. In 2.79 VSE the strip cache was attempted to be re-used as much as possible.

https://developer.blender.org/T73500

The poor performance of Scene Strips(regression) should also be mentioned here, and the fact that proxy generation of Scene Strips currently is removed, so there is no workaround to improve the performance. If storyboards are drawn in Blender with GP, an improved performance of Scene Strips would mean that the boards wouldn't have to be rendered to file before adding them to the Sequencer.

Effect strips also slow down playback, not utilizing GPU and some of them are not even multi-threaded.

> Seek in video files is rather slow, feels measurably (2x) slower compared to seek in VLC with the same video file. > **Proxy** file generation is slower than realtime. Sometimes only time code (TC) is needed, so, probably, the solution is to allow generating TC without scaled-down video. Encoding of different resolution should be possible to do in separate threads. If the codec of the proxies is changed from mjpeg to h264(the avi container would still work), all sorts of hardware accelerated options will be available. Adding this to the h264 command line: `-g 1 -preset ultrafast -tune fastdecode -crf 0 ` make the files behave like mjpeg(but really small). `-g 1` generate full images(like mjpeg). `-preset ultrafast` fast encoding. `-tune fastdecode` fast decoding. `-crf 0 ` (Should be a optional setting) 0=lossless https://wiki.libav.org/Encoding/h264 Alternatively HDxHR doesn't have the resolution restrictions of DNxHD and could be used for proxies, but compared to h264 this is an i-frame heavy codec(like mjpeg) and will produce a huge files(which also could be used as intermediate files(improved seeking)). It comes in a variety of options including 10-bit, but I don't think avi will work as a container, so going this way will mean a new container format(which also could contain metadata). For keeping opacity in proxies, png in mov container could be used for proxies. For optimal quality for color grading, EXR format could be considered for proxies, however since 10-bit is not enabled in the VSE, this may currently be overkill? On the playback of proxies, a mismatch between proxy-file resolution and project-resolution will cause loss in fps. https://developer.blender.org/T76726 And the additional scaling of the proxies in order to get text(text strips) scaled correct will also result in a loss of fps. https://developer.blender.org/T76724 On hardware enhanced encoding and decoding: https://developer.nvidia.com/blog/nvidia-ffmpeg-transcoding-guide/#disqus_thread There is a slight pause in playhead movement when seeking(arrow keys), this should be removed, if possible? The seeking backwards is poor when Prefetch Frames is on: https://developer.blender.org/T77209 (A fairly normal operation is to quickly seek backwards while playing forward, but the cached area is kept ahead of the playhead, so this operation will choke because of uncached frames before the playhead.) To improve the seeking control maybe an industry-standard of seeking could be considered? Ex. tabbing forward(L key) will increase seek speed and tabbing backward(J key), will decrease seek speed(and seek reverse), (K for stop). (These controls could also be used in modal-move/extend operations) On Threads, as the libav seems to be running multithreaded, it's on the Blender side of things things aren't, which is why the various parallel render add-ons will render faster. Ex. https://github.com/elmopl/ktba/blob/master/scripts/addons/parallel_render.py So maybe it's better to deal with the image processing on the Blender side than separating generation of various proxy files into separate threads? On the cluttered and incomprehensive Proxy UI: https://devtalk.blender.org/t/design-proposal-proxy-use-in-the-video-sequence-editor/12065 https://developer.blender.org/D6077 > Moving image and cached video strips while being under the playhead is very slow. In 2.79 VSE the strip cache was attempted to be re-used as much as possible. https://developer.blender.org/T73500 The poor performance of Scene Strips(regression) should also be mentioned here, and the fact that proxy generation of Scene Strips currently is removed, so there is no workaround to improve the performance. If storyboards are drawn in Blender with GP, an improved performance of Scene Strips would mean that the boards wouldn't have to be rendered to file before adding them to the Sequencer. Effect strips also slow down playback, not utilizing GPU and some of them are not even multi-threaded.

Domingo Riesco commented

2020-08-15 01:30:14 +02:00

I wonder if it is part of this project to make available playback in realtime of H264 files (fHD without effects at least), without latency and without need of proxies. I"m reading a lot of mention to multithreaded generation of proxies on the net, but not much focus on multithreaded direct playback of source files. Like commercial non-linear editors have been able to do better and better throughout the years, some being able even to playback 200mbit fHD h264 files without any latency even in old machines with practically no GPU acceleration (HD4000 for example). I wonder how they manage to do so and where Blender's bottleneck to do so is. Even considering that standard playback softwares like vlc and so on can do it too.

Thanks for opening this thread! Constantly looking forward to its evolution and attentive to contribute in something if I can.

I wonder if it is part of this project to make available playback in realtime of H264 files (fHD without effects at least), without latency and without need of proxies. I"m reading a lot of mention to multithreaded generation of proxies on the net, but not much focus on multithreaded direct playback of source files. Like commercial non-linear editors have been able to do better and better throughout the years, some being able even to playback 200mbit fHD h264 files without any latency even in old machines with practically no GPU acceleration (HD4000 for example). I wonder how they manage to do so and where Blender's bottleneck to do so is. Even considering that standard playback softwares like vlc and so on can do it too. Thanks for opening this thread! Constantly looking forward to its evolution and attentive to contribute in something if I can.

hujin commented

2020-08-15 03:42:55 +02:00

Added subscriber: @2046411367

Sergey Sharybin commented

2020-08-17 09:48:04 +02:00

Please refer to the parent task, which states expected baseline for playback.

Marcos Perez commented

2020-08-17 10:59:34 +02:00

Added subscriber: @pistolario

Christian Clavet commented

2020-08-17 19:05:16 +02:00

Added subscriber: @christian-clavet

Nils-Erik commented

2020-08-19 02:59:34 +02:00

Added subscriber: @clayhands

blender-admin commented

2020-08-19 15:39:00 +02:00

This issue was referenced by 5b2bfb2fed

This issue was referenced by 5b2bfb2fed03274bd0bc2007c66118ca33c9f1ca

Andrea Monzini commented

2020-09-09 14:48:01 +02:00

Added subscriber: @AndreaMonzini

Andrea Monzini commented

2020-09-09 14:48:01 +02:00

Hello, i hope to write in the right section.
Could you please confirm that playback performance of the OpenEXR in the VSE is relatively slow ?
It's ~10/20 fps in my system for 1080P without the cache ( Blender 2.90, Ryzen 1700 @3700, 32 GB RAM, RX 580 8 GB, SSD )

As the OpenEXR is the way we ( should ) render in Blender, it would be important to not forget the OpenEXR playback optimization for the VSE 2.0 so we can edit easily what we render.
And as wrote by Peter, it would very interesting to consider OpenEXR for the proxy too.
Example to reproduce 1080P exported to OpenEXR ( The Daily Dweebs intro by Blender Animation Studio ) :

Example_EXR_Half_Float.zip

Anyway thank you also for the work on the VSE 2.0 !

Hello, i hope to write in the right section. Could you please confirm that playback performance of the OpenEXR in the VSE is relatively slow ? It's ~10/20 fps in my system for 1080P without the cache ( Blender 2.90, Ryzen 1700 @3700, 32 GB RAM, RX 580 8 GB, SSD ) As the OpenEXR is the way we ( should ) render in Blender, it would be important to not forget the OpenEXR playback optimization for the VSE 2.0 so we can edit easily what we render. And as wrote by Peter, it would very interesting to consider OpenEXR for the proxy too. Example to reproduce 1080P exported to OpenEXR ( The Daily Dweebs intro by Blender Animation Studio ) : [Example_EXR_Half_Float.zip](https://archive.blender.org/developer/F8857874/Example_EXR_Half_Float.zip) Anyway thank you also for the work on the VSE 2.0 !

Sergey Sharybin commented

2020-09-09 15:06:23 +02:00

Playback of EXR is way more involved than playback of PNG or movie files. Comes from the complexity of decoder, bigger amount of data, more involved color management.

There are ways to mitigate that, is something we keep in mind.

Playback of EXR is way more involved than playback of PNG or movie files. Comes from the complexity of decoder, bigger amount of data, more involved color management. There are ways to mitigate that, is something we keep in mind.

Andrea Monzini commented

2020-09-09 15:52:59 +02:00

Thank you for the answer, looking forward for the updates !
About file size and encode time there is a very interesting codec comparison work ( by Robert Gützkow ) :

https://blender.stackexchange.com/questions/148231/what-image-format-encodes-the-fastest-or-at-least-faster-png-is-too-slow/148466#148466

It seems that file size and encoding time of OpenEXR ( compressed lossy DWAA and loseless Zip for example ) are very good :)

Thank you for the answer, looking forward for the updates ! About file size and encode time there is a very interesting codec comparison work ( by Robert Gützkow ) : https://blender.stackexchange.com/questions/148231/what-image-format-encodes-the-fastest-or-at-least-faster-png-is-too-slow/148466#148466 It seems that file size and encoding time of OpenEXR ( compressed lossy DWAA and loseless Zip for example ) are very good :)

tintwotin commented

2020-09-09 23:58:13 +02:00

As long the main slowdowns are on the Blender side of things, codecs and ffmpeg settings properly will not make that much of a difference. I can't help thinking that there is no way around doing the profiling as LazyDodo started/tutorial'ed here: https://developer.blender.org/D8126

In his example(exporting video):

Encoding a movie sounds and feels expensive, however this profile tells us of that 13.97s spend saving, only 1.63s is spend in ffmpeg, so the cost of encoding is NOT the most expensive process at hand here.
IMB_colormanagement_imbuf_for_write however is soaking up a solid 12.08 seconds, that's not super great.

He finds a way to optimize it:

Alright we're down to 19.6 seconds, not as much as i'd hoped but 30% improvement in save time is still pretty good, and worth this small profiling exercise

But now the patch is abandoned, without any explanation....?

Imo, this needs to be done for all video handling processes including preview, import, export, playback, caching, proxy, effect strips, compositing etc. in order to locate the spots where the real performance improvements can be made(and maybe in that process, investigations can be made into what adding 10+ bit handling would entail).

And then, when those possible optimizations have been done, hopefully Blender is now that fast at processing with video-images, that it makes sense to optimize the ffmpeg (hardware acc.) settings and codec options.

As long the main slowdowns are on the Blender side of things, codecs and ffmpeg settings properly will not make that much of a difference. I can't help thinking that there is no way around doing the profiling as LazyDodo started/tutorial'ed here: https://developer.blender.org/D8126 In his example(exporting video): > Encoding a movie sounds and feels expensive, however this profile tells us of that 13.97s spend saving, only 1.63s is spend in ffmpeg, so the cost of encoding is *NOT* the most expensive process at hand here. > IMB_colormanagement_imbuf_for_write however is soaking up a solid 12.08 seconds, that's not super great. He finds a way to optimize it: > Alright we're down to 19.6 seconds, not as much as i'd hoped but 30% improvement in save time is still pretty good, and worth this small profiling exercise But now the patch is abandoned, without any explanation....? Imo, this needs to be done for all video handling processes including preview, import, export, playback, caching, proxy, effect strips, compositing etc. in order to locate the spots where the real performance improvements can be made(and maybe in that process, investigations can be made into what adding 10+ bit handling would entail). And then, when those possible optimizations have been done, hopefully Blender is now that fast at processing with video-images, that it makes sense to optimize the ffmpeg (hardware acc.) settings and codec options.

david mcsween commented

2020-09-10 04:05:26 +02:00

Added subscriber: @davidmcsween

david mcsween commented

2020-09-10 04:05:26 +02:00

I would certainly concur about the lack of handling for 10bit movie files. As it stands you must use standalone ffmpeg command-line or external converter to create a frame sequence from these files, otherwise Blender downsamples (sometimes badly with an old ffmpeg) and you loose the benefit of shooting higher quality media.
Not having access to external 10bit video is odd; when you consider that Blender has a compositor and material texture input that is being used in a high bit depth environment (Blender renders).

I would certainly concur about the lack of handling for 10bit movie files. As it stands you must use standalone ffmpeg command-line or external converter to create a frame sequence from these files, otherwise Blender downsamples (sometimes badly with an old ffmpeg) and you loose the benefit of shooting higher quality media. Not having access to external 10bit video is odd; when you consider that Blender has a compositor and material texture input that is being used in a high bit depth environment (Blender renders).

Andy Cuccaro commented

2020-09-10 15:59:29 +02:00

Added subscriber: @AndyCuccaro

Hugo Schouppe commented

2020-09-11 20:25:27 +02:00

Added subscriber: @HUSCH

Giumo commented

2020-11-01 07:58:12 +01:00

Added subscriber: @Giumo

solarlune commented

2020-11-05 07:27:35 +01:00

Added subscriber: @solarlune

Pipeliner commented

2020-12-08 11:53:18 +01:00

Added subscriber: @Pipeliner

RedMser commented

2021-01-03 01:39:39 +01:00

Added subscriber: @RedMser

Alexey commented

2021-02-11 09:32:30 +01:00

Added subscriber: @alexeygrigoriev

Alexey commented

2021-02-11 09:32:30 +01:00

Why GPU support for strips preview and final rendering is not here? Or CPU multi-threading atleast.. imao that is first need to do for performance in VSE.
Or it is meant by one of the points?

Why GPU support for strips preview and final rendering is not here? Or CPU multi-threading atleast.. imao that is first need to do for performance in VSE. Or it is meant by one of the points?

Russell commented

2021-03-30 17:24:39 +02:00

Added subscriber: @Russ1642

Philipp Oeser removed the

Download

What's New

Blender Studio

Manual

Developers Blog

Documentation

Benchmark

Blender Conference

Development Fund

One-time Donations

VSE 2.0: Performance #78992