Audio scrubbing sounds glitchy #97750

New Issue

zyc17 · 2022-05-01T07:21:59+02:00

zyc17 commented

2022-05-01 07:21:59 +02:00

System Information
Operating system: macOS-12.2.1-x86_64-i386-64bit 64 Bits
Graphics card: Intel(R) Iris(TM) Plus Graphics 655 Intel Inc. 4.1 INTEL-18.4.6

Blender Version
Broken: After bdbc7e12a0
Worked: After ded68fb102

Short description of error
Audio scrubbing sounds glitchy
Kapture 2022-05-02 at 11.09.23.mp4

Exact steps for others to reproduce the error

Open attached .blend file
Make sure the 'Scrubbing' option is enabled in the playback panel
Drag the timeline cursor to (for example) frame 429. (Don't drop it)

In version 3.0 this frame is muted (wrong), in 2.93 you can hear the sound.

You can see that when dragging the timeline cursor, the audio sounds scattered and glitchy, for some other files I've tested it's even worse and there's only fuzzy noise, but in 2.93 it sounds smoother and just like it's speed down normally

2.93.blend

**System Information** Operating system: macOS-12.2.1-x86_64-i386-64bit 64 Bits Graphics card: Intel(R) Iris(TM) Plus Graphics 655 Intel Inc. 4.1 INTEL-18.4.6 **Blender Version** Broken: After bdbc7e12a0 Worked: After ded68fb102 **Short description of error** Audio scrubbing sounds glitchy [Kapture 2022-05-02 at 11.09.23.mp4](https://archive.blender.org/developer/F13044814/Kapture_2022-05-02_at_11.09.23.mp4) **Exact steps for others to reproduce the error** - Open attached .blend file - Make sure the 'Scrubbing' option is enabled in the playback panel - Drag the timeline cursor to (for example) frame 429. (Don't drop it) In version 3.0 this frame is muted (wrong), in 2.93 you can hear the sound. You can see that when dragging the timeline cursor, the audio sounds scattered and glitchy, for some other files I've tested it's even worse and there's only fuzzy noise, but in 2.93 it sounds smoother and just like it's speed down normally [2.93.blend](https://archive.blender.org/developer/F13044821/2.93.blend)

zyc17 commented

2022-05-01 07:21:59 +02:00

Added subscriber: @389150029

Pratik Borhade commented

2022-05-02 14:09:01 +02:00

Added subscriber: @PratikPB2123

Germano Cavalcante commented

2022-05-03 19:11:31 +02:00

Added subscriber: @mano-wii

Germano Cavalcante commented

2022-05-03 19:11:31 +02:00

Changed status from 'Needs Triage' to: 'Confirmed'

Germano Cavalcante commented

2022-05-03 19:11:31 +02:00

Thanks for the report. I can confirm the problem. Tested on a macOS-12.2-arm64-arm-64bit 64 Bits

It's not a recent regression, the problem appeared before efd7c95bb2 (Aug 2 2021). (It will take a while to bisect).

I still need to check if this is a macOS specific issue.

I'll edit the description to make the steps to reproduce the problem clearer.

Thanks for the report. I can confirm the problem. Tested on a macOS-12.2-arm64-arm-64bit 64 Bits It's not a recent regression, the problem appeared before `efd7c95bb2` (Aug 2 2021). (It will take a while to bisect). I still need to check if this is a macOS specific issue. I'll edit the description to make the steps to reproduce the problem clearer.

Germano Cavalcante commented

2022-05-04 02:22:23 +02:00

Added subscriber: @neXyon

Germano Cavalcante commented

2022-05-04 02:22:23 +02:00

Bisecting, this problem had apparently been resolved in ded68fb102,
but was reintroduced in bdbc7e12a0.

So, @neXyon, is this something you can check? Can you reproduce the problem?

Bisecting, this problem had apparently been resolved in ded68fb102, but was reintroduced in bdbc7e12a0. So, @neXyon, is this something you can check? Can you reproduce the problem?

Sergey Sharybin commented

2022-05-04 10:56:11 +02:00

Added subscribers: @ZedDB, @Sergey

Sergey Sharybin commented

2022-05-04 10:56:11 +02:00

Not sure how active Joerg is.
@ZedDB, do you mind giving some help here? :)

Not sure how active Joerg is. @ZedDB, do you mind giving some help here? :)

Joerg Mueller commented

2022-05-04 19:08:10 +02:00

To me the video doesn't seem to have an issue for either, to be honest. The Blender 3.0 version just seems to have a little less performance, which could be caused by a variety of factors. I don't see how bdbc7e12a0 would cause it though. Could be a different ffmpeg version even or so, but I don't think that would have changed during your bisection @mano-wii?

In any case, you can improve performance of scrubbing considerably if you enable Caching which you can find if you open the Source drop down on the right side of the VSE. That should improve performance.

Just to clarify how scrubbing works: if you drag the mouse (button down) on the timeline after every mouse movement it plays the sound of that frame that you are over at the moment and then stops. If you continuously move the mouse it keeps playing small frame snippets, always stopping the one that has been playing before if there is one playing already. At the zoom level the file is in, a frame covers more than a pixel in width, so the same frame is restarted to play multiple times when you drag over it which may make it sound glitchy. This is probably not ideal behavior, but that is how it's been since like forever, so that cannot be the issue here.

What I think happens here is a performance issue, as I said, where it sounds a little bit more glitchy, because there are longer breaks, since seeking in compressed audio files is not actually a fast operation. Seeking for every tiny mouse movement as happens during scrubbing is super slow and kind of the worst case. If not fast enough, you get longer pauses in the playback which to me is what I can hear in the video. Seeking in an uncompressed audio buffer in RAM is no effort at all in comparison though and thus turning on Caching (which is actually uncompressing the audio in RAM) should get rid of the performance issue.

To me the video doesn't seem to have an issue for either, to be honest. The Blender 3.0 version just seems to have a little less performance, which could be caused by a variety of factors. I don't see how bdbc7e12a0 would cause it though. Could be a different ffmpeg version even or so, but I don't think that would have changed during your bisection @mano-wii? In any case, you can improve performance of scrubbing considerably if you enable `Caching` which you can find if you open the `Source` drop down on the right side of the VSE. That should improve performance. Just to clarify how scrubbing works: if you drag the mouse (button down) on the timeline after every mouse movement it plays the sound of that frame that you are over at the moment and then stops. If you continuously move the mouse it keeps playing small frame snippets, always stopping the one that has been playing before if there is one playing already. At the zoom level the file is in, a frame covers more than a pixel in width, so the same frame is restarted to play multiple times when you drag over it which may make it sound glitchy. This is probably not ideal behavior, but that is how it's been since like forever, so that cannot be the issue here. What I think happens here is a performance issue, as I said, where it sounds a little bit more glitchy, because there are longer breaks, since seeking in compressed audio files is not actually a fast operation. Seeking for every tiny mouse movement as happens during scrubbing is super slow and kind of the worst case. If not fast enough, you get longer pauses in the playback which to me is what I can hear in the video. Seeking in an uncompressed audio buffer in RAM is no effort at all in comparison though and thus turning on Caching (which is actually uncompressing the audio in RAM) should get rid of the performance issue.

Germano Cavalcante commented

2022-05-04 21:04:50 +02:00

For bisect I focused specifically on frame 429.

Before `ded68fb102` and after `bdbc7e12a0` this frame is muted (which I thought was related to the glitch):	But I can confirm that enabling caching apparently improves the situation:	Here what the result looks like in a previous version without caching:
rBbdbc7e12a02e_no_caching.mov	rBbdbc7e12a02e_caching.mov	rB970c928f2710_no_caching.mov

For bisect I focused specifically on frame 429. |Before ded68fb102 and after bdbc7e12a0 this frame is muted (which I thought was related to the glitch):|But I can confirm that enabling caching apparently improves the situation:|Here what the result looks like in a previous version without caching: | -- | -- | -- | |[rBbdbc7e12a02e_no_caching.mov](https://archive.blender.org/developer/F13051962/rBbdbc7e12a02e_no_caching.mov)|[rBbdbc7e12a02e_caching.mov](https://archive.blender.org/developer/F13051959/rBbdbc7e12a02e_caching.mov)|[rB970c928f2710_no_caching.mov](https://archive.blender.org/developer/F13051958/rB970c928f2710_no_caching.mov)

Joerg Mueller commented

2022-05-05 21:56:50 +02:00

Changed status from 'Confirmed' to: 'Archived'

Joerg Mueller closed this issue

2022-05-05 21:56:50 +02:00

Joerg Mueller self-assigned this 2022-05-05 21:56:50 +02:00

Joerg Mueller commented

2022-05-05 21:56:50 +02:00

Ok, so everything pretty much works as expected.

The only question is why the performance changed with this commit, especially since none of the code changes in bdbc7e12a0 should have any major influence on performance for scrubbing. If you really want to look into it, you could revert just the changes in the seeking and length estimation code of FFMPEGReader since those are the only changes that could potentially change performance. However, since those changes are bug fixes, even if reverting them improves performance, I would not do so.

In conclusion: turn on caching if the performance of ffmpeg is not good enough for scrubbing.

Ok, so everything pretty much works as expected. The only question is why the performance changed with this commit, especially since none of the code changes in bdbc7e12a0 should have any major influence on performance for scrubbing. If you really want to look into it, you could revert just the changes in the seeking and length estimation code of `FFMPEGReader` since those are the only changes that could potentially change performance. However, since those changes are bug fixes, even if reverting them improves performance, I would not do so. In conclusion: turn on caching if the performance of ffmpeg is not good enough for scrubbing.

tintwotin commented

2022-05-07 08:13:28 +02:00

Added subscriber: @tintwotin

tintwotin commented

2022-05-07 08:13:28 +02:00

In #97750#1352687, @neXyon wrote:
In conclusion: turn on caching if the performance of ffmpeg is not good enough for scrubbing.

The downside here is that users will have no chance of knowing that they'll have to dig deep into the Source panel to switch on audio caching in order to get the best possible scrubbing. As a consequence, the caching functionality should be added to the scrubbing operator. So, scrubbing will always use caching for all audio strips. If there aren't any additional uses of audio caching, it could even be removed from the UI entirely.

Also, in the VSE, it is inconsistent that catching of images is a tab of its own and a general or a strip setting whereas caching of audio is located as a source setting and is only for each strip without a general setting:

The same thing with audio packing, is should be integrated into the general packing function and not be a source setting:

> In #97750#1352687, @neXyon wrote: > In conclusion: turn on caching if the performance of ffmpeg is not good enough for scrubbing. The downside here is that users will have no chance of knowing that they'll have to dig deep into the Source panel to switch on audio caching in order to get the best possible scrubbing. As a consequence, the caching functionality should be added to the scrubbing operator. So, scrubbing will always use caching for all audio strips. If there aren't any additional uses of audio caching, it could even be removed from the UI entirely. Also, in the VSE, it is inconsistent that catching of images is a tab of its own and a general or a strip setting whereas caching of audio is located as a source setting and is only for each strip without a general setting: ![image.png](https://archive.blender.org/developer/F13055943/image.png) The same thing with audio packing, is should be integrated into the general packing function and not be a source setting: ![image.png](https://archive.blender.org/developer/F13055946/image.png)

Joerg Mueller commented

2022-05-07 18:55:19 +02:00

I disagree: caching consumes way too much memory to enable it by default when scrubbing is enabled. I instead suggest to suggest in the documentation to enable caching for audio strips when necessary for performance.

As for the UI: the issue here is that various strips can use the same source and it's the source that is packed/cached not the strip. So these settings have to stay with the source.

I disagree: caching consumes way too much memory to enable it by default when scrubbing is enabled. I instead suggest to suggest in the documentation to enable caching for audio strips when necessary for performance. As for the UI: the issue here is that various strips can use the same source and it's the source that is packed/cached not the strip. So these settings have to stay with the source.

Sergey Sharybin commented

2022-05-10 10:21:13 +02:00

@neXyon, Making sure the current state is documented is surely welcome. However, it is a bad design to task an editor to worry about identifying where and when to enable caching. The good design is when an editor is focused on the editing task, and the performance is ensured by the system.

Look into any recent open movie edit file: it will have 100s of strips. Requiring an artist to manually tweak settings in every one of them to keep the system usable is not something I'd consider acceptable.

@neXyon, Making sure the current state is documented is surely welcome. However, it is a bad design to task an editor to worry about identifying where and when to enable caching. The good design is when an editor is focused on the editing task, and the performance is ensured by the system. Look into any recent open movie edit file: it will have 100s of strips. Requiring an artist to manually tweak settings in every one of them to keep the system usable is not something I'd consider acceptable.

Joerg Mueller commented

2022-05-22 19:22:05 +02:00

Well, you could add a setting to automatically enable caching, when the audio files are shorter than a specific length. Another option instead of using caching is to use proxies like done for video. I.e., convert the audio streams of the video file to PCM/wav/uncompressed audio for immediate seeking without performance issues.

Sergey Sharybin commented

2022-05-24 16:55:50 +02:00

I am not sure how useful restriction of cache used for only short audio. It is not uncommon to work on a montage of multiple longer videos files: things like reels, tutorials, etc.

Covering audio with a proxy system sounds like a natural extension of what we were doing for video, so it should work fine.

I am not sure how useful restriction of cache used for only short audio. It is not uncommon to work on a montage of multiple longer videos files: things like reels, tutorials, etc. Covering audio with a proxy system sounds like a natural extension of what we were doing for video, so it should work fine.

Sign in to join this conversation.

No Label

Download

What's New

Blender Studio

Manual

Developers Blog

Documentation

Benchmark

Blender Conference

Development Fund

One-time Donations

Audio scrubbing sounds glitchy #97750