Video Editor: Proxy clips will play with poor performance if used as proxies, but not when added to the sequencer-timeline #70415

New Issue

tintwotin · 2019-10-01T11:51:27+02:00

tintwotin commented

2019-10-01 11:51:27 +02:00

System Information
Operating system: Windows-10-10.0.17763 64 Bits
Graphics card: Intel(R) UHD Graphics 600 Intel 4.5.0 - Build 26.20.100.7158

Blender Version
Broken: version: 2.81 (sub 12), branch: master, commit date: 2019-09-30 22:00, hash: 60a827a2a9

Short description of error
Blender rendered proxy files will play with a much worse performance when played as proxies, as compared to adding them directly to the timeline. On my computer the difference is around 12 fps difference on a 100 % proxy file and 10 fps on a 75% proxy file.

Exact steps for others to reproduce the error
(For faster preview performance:
Uncheck View > Show Cache
Set Sidebar > Blendmode: Cross
Sidebar uncheck Cache rendering)

Add a clip to the Sequencer.
Strip > Movie > Set Render Size(if there is a mismatch in resolution, performance will suffer)
Select the strip.
Open the Proxy Settings panel in the Sidebar.
Click the 'Set Selected Strip Proxies'
Check 100% and Overwrite.
Click 'Rebuild Proxies'
Wait for the proxies to be encoded.
In the Sequencer Preview in the 'View Settings' change Proxy Render Size to 100%
Note the frame rate of the clip playing.
On my computer the rate is 19 fps for a 29.97 fps HD clip.

Now open the BL_proxy folder(located in the same folder as your original clip), then open the clip named folder and find the proxy_100.avi file.
Add this file to the sequencer(above the first one)
In the Sequencer Preview in the 'View Settings' change Proxy Render Size to 'No proxy, full render'.
Note the frame rate of the clip playing.
On my computer the rate is the correct 30 fps.

If you in both cases get the correct fps then try with a 4k clip instead.
Free clips can be found here: https://pixabay.com/videos/search/?order=ec

Another odd thing, but maybe related, is changing the Proxy Render Size in the Preview sidebar will also lower the playrate, even though there are no proxies generated.

Comparing the playback rate of a proxyfile used as a proxy file and the same proxy file added to the timeline(gif):

**System Information** Operating system: Windows-10-10.0.17763 64 Bits Graphics card: Intel(R) UHD Graphics 600 Intel 4.5.0 - Build 26.20.100.7158 **Blender Version** Broken: version: 2.81 (sub 12), branch: master, commit date: 2019-09-30 22:00, hash: `60a827a2a9` **Short description of error** Blender rendered proxy files will play with a much worse performance when played as proxies, as compared to adding them directly to the timeline. On my computer the difference is around 12 fps difference on a 100 % proxy file and 10 fps on a 75% proxy file. **Exact steps for others to reproduce the error** (For faster preview performance: Uncheck View > Show Cache Set Sidebar > Blendmode: Cross Sidebar uncheck Cache rendering) Add a clip to the Sequencer. Strip > Movie > Set Render Size(if there is a mismatch in resolution, performance will suffer) Select the strip. Open the Proxy Settings panel in the Sidebar. Click the 'Set Selected Strip Proxies' Check 100% and Overwrite. Click 'Rebuild Proxies' Wait for the proxies to be encoded. In the Sequencer Preview in the 'View Settings' change Proxy Render Size to 100% Note the frame rate of the clip playing. On my computer the rate is 19 fps for a 29.97 fps HD clip. Now open the BL_proxy folder(located in the same folder as your original clip), then open the clip named folder and find the proxy_100.avi file. Add this file to the sequencer(above the first one) In the Sequencer Preview in the 'View Settings' change Proxy Render Size to 'No proxy, full render'. Note the frame rate of the clip playing. On my computer the rate is the correct 30 fps. If you in both cases get the correct fps then try with a 4k clip instead. Free clips can be found here: https://pixabay.com/videos/search/?order=ec Another odd thing, but maybe related, is changing the Proxy Render Size in the Preview sidebar will also lower the playrate, even though there are no proxies generated. Comparing the playback rate of a proxyfile used as a proxy file and the same proxy file added to the timeline(gif): ![Poor_Proxy_Performance.gif](https://archive.blender.org/developer/F7783348/Poor_Proxy_Performance.gif)

tintwotin commented

2019-10-01 11:51:27 +02:00

Added subscriber: @tintwotin

tintwotin commented

2019-10-01 22:22:40 +02:00

Just guessing here, but it seems to me changing the Set Proxy Size, when no proxies are generated, is still changing the resolution, properly mirroring the Resolution % in the Output options:

However values above 50% will actually lower the play-rate performance(properly because of scaling):

That could be some of the reason for a file is playing much slower when it is used as a proxy file.

If this is the case, the question is, if the swapping of generated proxies merged with Resolution % is a good idea, or if it is better to separate the proxy file swapping from the live-rescaling(Resolution %)? Or in other words, the Enum for selecting should be preserved file swapping and the Preview Resolution % should have a separate widget(like in Output options)?

This way users might avoid some of the performance loss when using the current 'Set Proxy Size' function in the 50%, 75% and 100% proxy resolutions. As mentioned in the first post about this problem a HD 29.97 proxy file played with the accurate framerate, when played from the Sequencer Timeline(so there is no need for the Resolution % on top of the file swapping).

Or alternatively check if the image resizing for the VSE could be optimized? Resizing is lowering the playrate in many areas of the VSE, and for sure causing poor playback.

But as I wrote, I just guessing here, so could this be some the reason for the performance drop?

Just guessing here, but it seems to me changing the Set Proxy Size, when no proxies are generated, is still changing the resolution, properly mirroring the Resolution % in the Output options: ![ResolutionProxy.gif](https://archive.blender.org/developer/F7784161/ResolutionProxy.gif) However values above 50% will actually lower the play-rate performance(properly because of scaling): ![ResolutionP.gif](https://archive.blender.org/developer/F7784156/ResolutionP.gif) That could be some of the reason for a file is playing much slower when it is used as a proxy file. If this is the case, the question is, if the swapping of generated proxies merged with Resolution % is a good idea, or if it is better to separate the proxy file swapping from the live-rescaling(Resolution %)? Or in other words, the Enum for selecting should be preserved file swapping and the Preview Resolution % should have a separate widget(like in Output options)? This way users might avoid some of the performance loss when using the current 'Set Proxy Size' function in the 50%, 75% and 100% proxy resolutions. As mentioned in the first post about this problem a HD 29.97 proxy file played with the accurate framerate, when played from the Sequencer Timeline(so there is no need for the Resolution % on top of the file swapping). Or alternatively check if the image resizing for the VSE could be optimized? Resizing is lowering the playrate in many areas of the VSE, and for sure causing poor playback. But as I wrote, I just guessing here, so could this be some the reason for the performance drop?

tintwotin commented

2019-10-20 12:05:01 +02:00

Added subscriber: @Sergey

tintwotin commented

2019-10-20 12:05:01 +02:00

It looks like you @Sergey back in the day, did some work on the (vse?) proxy system?

If you did, do you know if there are any logic explanations for playing a clip as proxy at 100%(unscaled) delivers a much worse playback rate compared to playing the very same clip directly in the VSE timeline?

It looks like you @Sergey back in the day, did some work on the (vse?) proxy system? If you did, do you know if there are any logic explanations for playing a clip as proxy at 100%(unscaled) delivers a much worse playback rate compared to playing the very same clip directly in the VSE timeline?

Eitan Traurig self-assigned this 2019-11-27 12:58:16 +01:00

Eitan Traurig commented

2019-11-27 14:38:50 +01:00

I haven't been able to reproduce this bug in blender 2.81 can you please confirm that it still exists.

tintwotin commented

2019-11-27 16:20:41 +01:00

Changed status from 'Open' to: 'Resolved'

tintwotin closed this issue

2019-11-27 16:20:41 +01:00

tintwotin commented

2019-11-27 16:20:41 +01:00

I'm not at my computer right now, but iirc the main difference was with proxies rendered at 100℅. Did you try that?

Oh i didn't men to close this, can't reopen it now!?

I'm not at my computer right now, but iirc the main difference was with proxies rendered at 100℅. Did you try that? Oh i didn't men to close this, can't reopen it now!?

tintwotin commented

2019-11-27 16:37:05 +01:00

Wow when posting, I can only choose between Resolved, Invalid or Archived. That must surely be a bug on Phabricator?

Eitan Traurig commented

2019-11-27 16:56:07 +01:00

I tried 100% and there is no performance loss

tintwotin commented

2019-11-27 17:59:19 +01:00

Did you try with a clip or a frame rate(ex. 100 fps) that your hardware is not capable at playing at full speed?

Richard Antalik commented

2019-11-27 19:37:51 +01:00

Added subscriber: @iss

Richard Antalik commented

2019-11-27 19:37:51 +01:00

Changed status from 'Resolved' to: 'Open'

Richard Antalik reopened this issue

2019-11-27 19:37:51 +01:00

Richard Antalik commented

2019-11-27 19:37:51 +01:00

I could reproduce this, but I haven't opened this because I wanted to confirm cause of this.

But I can confirm that this is bug and the cause is in sequencer.c

  /* dirty hack to distinguish 100% render size from PROXY_100 */
  if (render_size == 99) {
    render_size = 100;
  }

There is another issue in indexer.c, but that I wouldn't consider to be a bug.

  /* JPEG requires this */
  width = round_up(width, 8);
  height = round_up(height, 8);

I could reproduce this, but I haven't opened this because I wanted to confirm cause of this. But I can confirm that this is bug and the cause is in `sequencer.c` ``` /* dirty hack to distinguish 100% render size from PROXY_100 */ if (render_size == 99) { render_size = 100; } ``` There is another issue in `indexer.c`, but that I wouldn't consider to be a bug. ``` /* JPEG requires this */ width = round_up(width, 8); height = round_up(height, 8); ```

tintwotin commented

2019-11-27 21:38:28 +01:00

Changed status from 'Open' to: 'Resolved'

tintwotin closed this issue

2019-11-27 21:38:28 +01:00

tintwotin commented

2019-11-27 21:38:28 +01:00

NB. In order to test this stuff it is quite important to run the clip in a custom frame rate your hardware is not able to max out, or else you won't see the difference.

On my computer, using 2.82 these are my numbers:
100 % proxy clip is playing 12 fps sloweras a proxy as compared to the same clip playing directly in the timeline.
A 75 % proxy is playing a few frames faster as a proxy as compared to the same clip playing directly in the timeline.
A 25 % proxy is playing 10-30 frames faster as a proxy as compared to the same clip playing directly in the timeline.

Since the playback rates are not the same, it seems to me some additional scaling might be going on.

Having the Proxy Render Size at Scene Render Size(btw. needs to be in title case in the Menu) and noting the playback rates changing the Scene Resolution % to 75%, 50% and 25% and comparing them with the same Scene Render Size settings at 75%, 50% and 25% of a clip with no encoded proxy files, the frame rates matches exactly. So the obvious conclusion is that a clip with no encoded proxy files is going through a identical process in the render % and in the proxy preview %.

When playing encoded proxy files as proxies the resolution is of course lowered by the smaller encoded proxy files, but it seems to me, they must be processed by additional proxy preview % down-scaling. Or in other words you only get to see 25% of the encoded 25% proxy file, which explains why the 25% file is playing much faster as a proxy file, as compared to playing the same clip directly in the timeline. And it also explains why a 100% proxy is playing slower as a proxy clip as compared to playing the same clip directly in the timeline, simply because it is a heavier process to scale to a lot of pixels as compared to scale to a few pixels.

The question is if there is additional proxy preview % scaling going on, and if there is, if it really is needed on top of the encoded clips? And if it is considered needed, then maybe the Proxy Preview Resolution % should be separated into a slider like the Render Resolution %. And Proxy Render Size should only deal with file swapping if there are any.

NB. In order to test this stuff it is quite important to run the clip in a custom frame rate your hardware is not able to max out, or else you won't see the difference. On my computer, using 2.82 these are my numbers: **100 % proxy clip is playing 12 fps slower**as a proxy as compared to the same clip playing directly in the timeline. A 75 % proxy is playing a few frames faster as a proxy as compared to the same clip playing directly in the timeline. **A 25 % proxy is playing 10-30 frames faster** as a proxy as compared to the same clip playing directly in the timeline. Since the playback rates are not the same, it seems to me **some additional scaling might be going on.** Having the Proxy Render Size at Scene Render Size(btw. needs to be in title case in the Menu) and noting the playback rates changing the Scene Resolution % to 75%, 50% and 25% and comparing them with the same Scene Render Size settings at 75%, 50% and 25% of a clip with **no encoded** proxy files, the frame rates matches exactly. **So the obvious conclusion is that a clip with no encoded proxy files is going through a identical process in the render % and in the proxy preview %.** When playing encoded proxy files as proxies the resolution is of course lowered by the smaller encoded proxy files, but it seems to me, **they must be processed by additional proxy preview % down-scaling. Or in other words you only get to see 25% of the encoded 25% proxy file**, which explains why the 25% file is playing much faster as a proxy file, as compared to playing the same clip directly in the timeline. And it also explains why a 100% proxy is playing slower as a proxy clip as compared to playing the same clip directly in the timeline, simply because it is a heavier process to scale to a lot of pixels as compared to scale to a few pixels. **The question is if there is additional proxy preview % scaling going on, and if there is, if it really is needed on top of the encoded clips?** And if it is considered needed, then maybe the Proxy Preview Resolution % should be separated into a slider like the Render Resolution %. And Proxy Render Size should only deal with file swapping if there are any.

Richard Antalik commented

2019-11-27 21:45:32 +01:00

Changed status from 'Resolved' to: 'Open'

Richard Antalik reopened this issue

2019-11-27 21:45:32 +01:00

Richard Antalik commented

2019-11-27 21:45:32 +01:00

Please don't close this task, it has to be worked on.

tintwotin commented

2019-11-27 21:46:39 +01:00

Changed status from 'Open' to: 'Resolved'

tintwotin closed this issue

2019-11-27 21:46:39 +01:00

tintwotin commented

2019-11-27 21:46:39 +01:00

Oh, man, now I closed it again, this new "thing" is super annoying, that replying auto closes the issue you're commenting... I can't open it again... :-(

Robert Guetzkow commented

2019-11-27 21:51:04 +01:00

Changed status from 'Resolved' to: 'Open'

Robert Guetzkow reopened this issue

2019-11-27 21:51:04 +01:00

Eitan Traurig commented

2019-12-03 20:12:47 +01:00

This comment was removed by @EitanSomething

*This comment was removed by @EitanSomething*

Eitan Traurig commented

2019-12-03 20:16:38 +01:00

In #70415#819679, @iss wrote:
I could reproduce this, but I haven't opened this because I wanted to confirm cause of this.

But I can confirm that this is bug and the cause is in sequencer.c
  /* dirty hack to distinguish 100% render size from PROXY_100 */
  if (render_size == 99) {
    render_size = 100;
  }

The preview_render_size should be replaced with enums and we wont need to use this hack.

> In #70415#819679, @iss wrote: > I could reproduce this, but I haven't opened this because I wanted to confirm cause of this. > > But I can confirm that this is bug and the cause is in `sequencer.c` > ``` > /* dirty hack to distinguish 100% render size from PROXY_100 */ > if (render_size == 99) { > render_size = 100; > } > ``` The preview_render_size should be replaced with enums and we wont need to use this hack.

Richard Antalik commented

2019-12-04 06:55:06 +01:00

Indeed. In best case we could re-use IMB_ functions and enums.

I have done this quickly in the past, haven't been too happy with solution though. I think, that in that time I wasn't aware of DNA versioning so that's why I dropped it.

Indeed. In best case we could re-use IMB_ functions and enums. I have done this quickly in the past, haven't been too happy with solution though. I think, that in that time I wasn't aware of DNA versioning so that's why I dropped it.

Eitan Traurig commented

2019-12-05 01:49:59 +01:00

Can you test this Fix-#70415.diff

Can you test this [Fix-#70415.diff](https://archive.blender.org/developer/F8190370/Fix-#70415.diff)

tintwotin commented

2019-12-05 08:39:16 +01:00

@EitanSomething I'm not entirely sure what you did here, and what you want me to test?
Throwing some files around, it seems that the scaling of files with no encoded proxy files has been removed eg. the quality and framerate is the same no matter what Proxy Render Size is set. (Is this preview scaling to be used in a separate enum?)

When there are encoded proxy files, and the 'Proxy Render Size' is set to 'No proxy, full render' or 'Scene render size' it is actually using the 25% proxies - and not the source file. This means that I'm currently not able to compare the play rates of the proxy files played as proxies and as clips played directly in the time line(No proxy, full render).

@EitanSomething I'm not entirely sure what you did here, and what you want me to test? Throwing some files around, it seems that the scaling of files with no encoded proxy files has been removed eg. the quality and framerate is the same no matter what Proxy Render Size is set. (Is this preview scaling to be used in a separate enum?) When there are encoded proxy files, and the 'Proxy Render Size' is set to 'No proxy, full render' or 'Scene render size' it is actually using the 25% proxies - and not the source file. This means that I'm currently not able to compare the play rates of the proxy files played as proxies and as clips played directly in the time line(No proxy, full render).

Eitan Traurig commented

2019-12-05 13:17:50 +01:00

I sent the wrong diff

Eitan Traurig commented

2019-12-05 14:40:25 +01:00

Here is the patch D6368

Here is the patch [D6368](https://archive.blender.org/developer/D6368)

Eitan Traurig commented

2019-12-06 04:20:06 +01:00

The reason why proxy files at 25% are slower when added to timeline is because the VSE thinks its original size and runs

  rectx = (render_size * (float)scene->r.xsch) / 100.0f + 0.5f;
  recty = (render_size * (float)scene->r.ysch) / 100.0f + 0.5f;

  BKE_sequencer_new_render_data(
      bmain, depsgraph, scene, rectx, recty, proxy_size, false, &context);
  context.view_id = BKE_scene_multiview_view_id_get(&scene->r, viewname);

with render_size and proxy_size set to 100 when it should be 25 that is where the slowdown is.

The reason why proxy files at 25% are slower when added to timeline is because the VSE thinks its original size and runs ``` rectx = (render_size * (float)scene->r.xsch) / 100.0f + 0.5f; recty = (render_size * (float)scene->r.ysch) / 100.0f + 0.5f; BKE_sequencer_new_render_data( bmain, depsgraph, scene, rectx, recty, proxy_size, false, &context); context.view_id = BKE_scene_multiview_view_id_get(&scene->r, viewname); ``` with render_size and proxy_size set to 100 when it should be 25 that is where the slowdown is.

Eitan Traurig commented

2019-12-06 16:04:50 +01:00

Test files to reproduce Speed difference between using 25% proxy and proxy file added to timeline.
Steps to reproduce

Add Blender.mp4
Set Frame rate to more than what your computer will be able to play at
Turn on proxy for the Movie
Build 25% proxy
Set proxy render size to 25%
Turn on prefetch frame
Play Video and take down average FPS
Delete the video
Add the Proxy_25.avi file
Set proxy render size to no proxy, full render
Set Frame rate to more than what your computer will be able to play at
Play Video and take down average FPS

Average FPS of Proxy_25.avi in the timeline should be lower than using render proxy 25% of Blender.mp4.
Blender.zip

Test files to reproduce Speed difference between using 25% proxy and proxy file added to timeline. **Steps to reproduce** - Add Blender.mp4 - Set Frame rate to more than what your computer will be able to play at - Turn on proxy for the Movie - Build 25% proxy - Set proxy render size to 25% - Turn on prefetch frame - Play Video and take down average FPS - Delete the video - Add the Proxy_25.avi file - Set proxy render size to no proxy, full render - Set Frame rate to more than what your computer will be able to play at - Play Video and take down average FPS Average FPS of Proxy_25.avi in the timeline should be lower than using render proxy 25% of Blender.mp4. [Blender.zip](https://archive.blender.org/developer/F8193626/Blender.zip)

Richard Antalik commented

2019-12-09 06:27:24 +01:00

@EitanSomething
File you provided played at 90 FPS without proxy and 280 FPS with 25% proxy
Proxy file itself plays at 300FPS

This is because proxy of 1920 x 1060 clip has size of 480 x 272
However 1920 * 0.25 = 480
and 1040 * 0.25 = 265

480 x 272 is different to 480 x 265, therefore clip has to be scaled. This is limitation of MJPEG format.

@EitanSomething File you provided played at 90 FPS without proxy and 280 FPS with 25% proxy Proxy file itself plays at 300FPS This is because proxy of 1920 x 1060 clip has size of 480 x 272 However 1920 * 0.25 = 480 and 1040 * 0.25 = 265 480 x 272 is different to 480 x 265, therefore clip has to be scaled. This is limitation of MJPEG format.

tintwotin commented

2019-12-09 19:15:44 +01:00

@iss @EitanSomething

That clip plays here at 18 fps when there is a resolution mismatch 1060 vs. 1080.
Using Movie Strip > Set Render Size(so render resolution fits source) the playpack rate jumps to 25 fps.
(Disabling Cache and Cache Drawing)

Playing an encoded proxy file at 25% as proxy plays at 65 fps.
Playing the same file directly in the timeline(No Proxy) it plays at 35 fps.

@iss @EitanSomething That clip plays here at 18 fps when there is a resolution mismatch 1060 vs. 1080. Using Movie Strip > Set Render Size(so render resolution fits source) the playpack rate jumps to 25 fps. (Disabling Cache and Cache Drawing) Playing an encoded proxy file at 25% as proxy plays at 65 fps. Playing the same file directly in the timeline(No Proxy) it plays at 35 fps.

Eitan Traurig commented

2019-12-12 16:50:26 +01:00

Does the reason for this hack still exist

  /* JPEG requires this */
  width = round_up(width, 8);
  height = round_up(height, 8);

Does the reason for this hack still exist > ``` > /* JPEG requires this */ > width = round_up(width, 8); > height = round_up(height, 8); > ```

Richard Antalik commented

2019-12-12 17:32:35 +01:00

@EitanSomething
I think I have read that somewhere in ffmpeg docs, but that would have to be ages ago, so I assumed this to be the case. But just to be sure, I re-tested this by simply rendering at 960x540 (and other arbitrary sizes, that do contradict such rule), and it worked correctly.

So this very well may be historic issue.

You can try to comment this out and try building proxy of any size. None should work if those lines are necessary.

@EitanSomething I think I have read that somewhere in ffmpeg docs, but that would have to be ages ago, so I assumed this to be the case. But just to be sure, I re-tested this by simply rendering at 960x540 (and other arbitrary sizes, that do contradict such rule), and it worked correctly. So this very well may be historic issue. You can try to comment this out and try building proxy of any size. None should work if those lines are necessary.

Eitan Traurig was unassigned by Dalai Felinto

2019-12-23 13:48:38 +01:00

Dalai Felinto commented

2019-12-23 13:48:38 +01:00

Added subscriber: @EitanSomething

Eitan Traurig self-assigned this 2020-01-01 05:56:17 +01:00

Brecht Van Lommel commented

2020-01-15 17:02:09 +01:00

Added subscriber: @brecht