VSE - Bad interpolation in exported audio #68946

New Issue

hudson barkley · 2019-08-21T00:53:20+02:00

hudson barkley commented

2019-08-21 00:53:20 +02:00

Blender Version
Broken: 2.80

Short description of error
Exported audio from the VSE is not properly interpolated, creating a 'stair-stepped' pattern when it should be smooth.

Channel 1 is the original waveform with keyframe applied, channel 2 is the timeline exported to wav, and imported back in.

Also, there appears to be a delay on the animation by a frame - the fade in should be at full volume on frame 7, but it reaches full volume at frame 8 on the exported waveform.

**Blender Version** Broken: 2.80 **Short description of error** Exported audio from the VSE is not properly interpolated, creating a 'stair-stepped' pattern when it should be smooth. ![Untitled3.png](https://archive.blender.org/developer/F7676264/Untitled3.png) Channel 1 is the original waveform with keyframe applied, channel 2 is the timeline exported to wav, and imported back in. Also, there appears to be a delay on the animation by a frame - the fade in should be at full volume on frame 7, but it reaches full volume at frame 8 on the exported waveform.

hudson barkley commented

2019-08-21 00:53:20 +02:00

Added subscriber: @snuq

blender-admin commented

2019-08-21 00:53:20 +02:00

#70682 was marked as duplicate of this issue

Paul McManus commented

2019-08-21 08:00:41 +02:00

Added subscriber: @PaulMcManus

hudson barkley commented

2019-08-21 18:56:40 +02:00

side note: The frequency of the interpolation can be controlled with the 'accuracy' setting when exporting as audio only, but there appears to be now way to control this when exporting to video.
My suggestion: This accuracy setting should be changed to a render setting, and always displayed in the render panel under the other audio settings.

side note: The frequency of the interpolation can be controlled with the 'accuracy' setting when exporting as audio only, but there appears to be now way to control this when exporting to video. My suggestion: This accuracy setting should be changed to a render setting, and always displayed in the render panel under the other audio settings.

Richard Antalik commented

2019-08-28 16:28:29 +02:00

Added subscriber: @iss

Richard Antalik commented

2019-08-28 16:28:29 +02:00

I have a strong feeling, that this, #68945 (VSE - Improper audio on frame 1 when exporting to lossy-compressed audio) and #69167 (VSE - "Render Audio" exports only one audio strip in some cases) are caused by the same issue, or at least an issue in depsgraph.

Richard Antalik commented

2019-09-29 17:26:21 +02:00

I am testing this now, and I can reproduce this issue in 2.79 also.
I will have to ask more devs how/if this can be mitigated.

I am testing this now, and I can reproduce this issue in 2.79 also. I will have to ask more devs how/if this can be mitigated.

Richard Antalik commented

2019-10-01 06:06:47 +02:00

Added subscribers: @neXyon, @dr.sybren

Richard Antalik commented

2019-10-01 06:06:47 +02:00

Perhaps I never noticed this effect, because I render 60FPS and rarely use fade-ins, but I always thought, that property animation was done in audaspace internally. @neXyon, do you know how this was/should be implemented?

Not sure who else to ask, perhaps @dr.sybren may know something about this?

Perhaps I never noticed this effect, because I render 60FPS and rarely use fade-ins, but I always thought, that property animation was done in audaspace internally. @neXyon, do you know how this was/should be implemented? Not sure who else to ask, perhaps @dr.sybren may know something about this?

Sybren A. Stüvel commented

2019-10-01 12:05:42 +02:00

I haven't looked at audio handling in the VSE, so I wouldn't know by heart what could cause this.

Richard Antalik commented

2019-10-01 12:28:59 +02:00

In #68946#786927, @dr.sybren wrote:
I haven't looked at audio handling in the VSE, so I wouldn't know by heart what could cause this.

This task has little(nothing) to do with VSE, but rather actual implementation of rendering the audio and its animation.

> In #68946#786927, @dr.sybren wrote: > I haven't looked at audio handling in the VSE, so I wouldn't know by heart what could cause this. This task has little(nothing) to do with VSE, but rather actual implementation of rendering the audio and its animation.

Joerg Mueller commented

2019-10-01 13:18:44 +02:00

The interpolation is not a bug per se. The audio animation system is evaluated once for every read, so you basically get nearest neighbor interpolation for your buffer size. The buffer size can be adjusted during audio export as you correctly noticed. When rendering a video though, the audio system is called once per (image) frame so you get this issue here. I think having a general option in the render settings would be a good idea to mitigate the issue.

Regarding the one frame off error: does this also happen in 2.79?

The interpolation is not a bug per se. The audio animation system is evaluated once for every read, so you basically get nearest neighbor interpolation for your buffer size. The buffer size can be adjusted during audio export as you correctly noticed. When rendering a video though, the audio system is called once per (image) frame so you get this issue here. I think having a general option in the render settings would be a good idea to mitigate the issue. Regarding the one frame off error: does this also happen in 2.79?

Richard Antalik commented

2019-10-01 13:50:56 +02:00

In #68946#786957, @neXyon wrote:
I think having a general option in the render settings would be a good idea to mitigate the issue.

I think there is the same issue with exporting audio right now. Perhaps it's the depsgraph limited to evaluation only at whole frames, I really don't know.

Regarding the one frame off error: does this also happen in 2.79?

I will bring up the offset issue in #68945 (VSE - Improper audio on frame 1 when exporting to lossy-compressed audio).
My suspicion just by reading code is, that we set wrong frame by AUD_SequenceEntry_setAnimationData(). But I want to test this myself first before I comment.

Thanks for help!

I will keep this open, because this behavior is quite bad.

> In #68946#786957, @neXyon wrote: > I think having a general option in the render settings would be a good idea to mitigate the issue. I think there is the same issue with exporting audio right now. Perhaps it's the depsgraph limited to evaluation only at whole frames, I really don't know. > Regarding the one frame off error: does this also happen in 2.79? I will bring up the offset issue in #68945 (VSE - Improper audio on frame 1 when exporting to lossy-compressed audio). My suspicion just by reading code is, that we set wrong frame by `AUD_SequenceEntry_setAnimationData()`. But I want to test this myself first before I comment. Thanks for help! I will keep this open, because this behavior is quite bad.

Joerg Mueller commented

2019-10-03 10:48:27 +02:00

Let me revise what I wrote in my previous comment, my bad memory caused some wrong statements:

There is indeed linear volume interpolation during audio mixing. It interpolates from the last value it had (initially 0 now, thanks to #68945) to whatever value is set during one of those buffer mixes. This most likely causes the delay that you observe which is bigger the bigger the buffer size is, since it will reach the value that the whole buffer would have had in the case of no/nearest neighbor interpolation only at the end of the buffer.

Thanks to #68945 I digged a bit deeper into the issue and I think (though not certain) the odd resulting curve is caused by the depsgraph that keeps changing animation cache values during mixing/rendering.
I also modified your test file to animate volume from 0 to 1 from beginning to end of the animation. When I load this file after blender startup and immediately mixdown/render audio I get a silent sound file (0 volume all the way through). The Properties -> Scene Properties -> Audio -> Update Animation Cache button that should mitigate this issue has no effect.

[rant start]
I have the feeling that the audio animation system and the new depsgraph are very difficult to get to work together. The audio animation system was already a hack in the first place to get audio animation working with Blender's animation system that is certainly not designed for audio. Seems like the number of issues increased further with the depsgraph. I have neither time nor inclination to work on another hack to get the hack to work (better). As it is now I see basically two options: depsgraph core developers think through the issues of audio animation that I've been raising for a while and together we come up with a proper, working solution or we completely remove audio animation from Blender.
[rant end]

Let me revise what I wrote in my previous comment, my bad memory caused some wrong statements: There is indeed linear volume interpolation during audio mixing. It interpolates from the last value it had (initially 0 now, thanks to #68945) to whatever value is set during one of those buffer mixes. This most likely causes the delay that you observe which is bigger the bigger the buffer size is, since it will reach the value that the whole buffer would have had in the case of no/nearest neighbor interpolation only at the end of the buffer. Thanks to #68945 I digged a bit deeper into the issue and I think (though not certain) the odd resulting curve is caused by the depsgraph that keeps changing animation cache values during mixing/rendering. I also modified your test file to animate volume from 0 to 1 from beginning to end of the animation. When I load this file after blender startup and immediately mixdown/render audio I get a silent sound file (0 volume all the way through). The Properties -> Scene Properties -> Audio -> Update Animation Cache button that should mitigate this issue has no effect. [rant start] I have the feeling that the audio animation system and the new depsgraph are very difficult to get to work together. The audio animation system was already a hack in the first place to get audio animation working with Blender's animation system that is certainly not designed for audio. Seems like the number of issues increased further with the depsgraph. I have neither time nor inclination to work on another hack to get the hack to work (better). As it is now I see basically two options: depsgraph core developers think through the issues of audio animation that I've been raising for a while and together we come up with a proper, working solution or we completely remove audio animation from Blender. [rant end]

Richard Antalik commented

2019-10-03 11:14:02 +02:00

Thanks for fix.
The animation cache stuff may explain #69167 (VSE - "Render Audio" exports only one audio strip in some cases). I will look into this as well. I've got quite a few audio related bugs in tracker, so this may have to be resolved in some way.

I will look into those and see what can be done.

Couldn't this be resolved by providing simple callback interface?
I mean instead of AUD_SequenceEntry_setAnimationData() it could be AUD_SequenceEntry_setPropertyEvalFunction(). Depsgraph would register its function to call during mixdown, so you can have precise data, exactly when you need it.

Thanks for fix. The animation cache stuff may explain #69167 (VSE - "Render Audio" exports only one audio strip in some cases). I will look into this as well. I've got quite a few audio related bugs in tracker, so this may have to be resolved in some way. I will look into those and see what can be done. Couldn't this be resolved by providing simple callback interface? I mean instead of `AUD_SequenceEntry_setAnimationData()` it could be `AUD_SequenceEntry_setPropertyEvalFunction()`. Depsgraph would register its function to call during mixdown, so you can have precise data, exactly when you need it.

Joerg Mueller commented

2019-10-03 23:39:31 +02:00

As I wrote in #59540#588156 doing a callback for every sound sample is way too slow. We could consider it for every buffer if it is fast enough and it is possible. The issue here is that the depsgraph would need to provide the functionality to get the value of a single property for a specific time. I'm not sure if it can easily do that?

Richard Antalik commented

2019-10-04 03:38:47 +02:00

Added subscriber: @Sergey

Richard Antalik commented

2019-10-04 03:38:47 +02:00

In #68946#789089, @neXyon wrote:
As I wrote in #59540#588156 doing a callback for every sound sample is way too slow. We could consider it for every buffer if it is fast enough and it is possible.

That's what I thought, not every sample, but fairly regulary. Even if it was for each sample, and would be slow, audio rendering is done in separate thread, and it's not likely it would bottleneck rendering itself. It would be at least correct. During preview callback could be called say every 10ms only, if that is distinguishable from rendering.

We can also pre-build a waveform(float array) with good enough resolution that could be used to modulate property value. The later would correspond to animation cache idea, not sure how it's done now.

The issue here is that the depsgraph would need to provide the functionality to get the value of a single property for a specific time. I'm not sure if it can easily do that?

Assuming, we want to correctly render audio, it has to be able to do that for each sample.

Not sure if @Sergey would like to add anything here, or if there is already some TODO written for this issue?

> In #68946#789089, @neXyon wrote: > As I wrote in #59540#588156 doing a callback for every sound sample is way too slow. We could consider it for every buffer if it is fast enough and it is possible. That's what I thought, not every sample, but fairly regulary. Even if it was for each sample, and would be slow, audio rendering is done in separate thread, and it's not likely it would bottleneck rendering itself. It would be at least correct. During preview callback could be called say every 10ms only, if that is distinguishable from rendering. We can also pre-build a waveform(float array) with good enough resolution that could be used to modulate property value. The later would correspond to animation cache idea, not sure how it's done now. > The issue here is that the depsgraph would need to provide the functionality to get the value of a single property for a specific time. I'm not sure if it can easily do that? Assuming, we want to correctly render audio, it has to be able to do that for each sample. Not sure if @Sergey would like to add anything here, or if there is already some TODO written for this issue?

hudson barkley commented

2019-10-04 08:16:07 +02:00

Just wanted to chime in as a user who animates audio all the time.

Im thinking that the current system (and accuracy) is fine for live playback, and i dont think it would be a big deal if audio rendering were slower (tho, a progress meter might be good... an hour+ of audio does take some time to render already).

On the other hand, if making the linear interpolation behave the way it should is a reasonable possible fix, i dont know if implementing every-sample depsgraph is needed really, thats kinda what the audio render accuracy is for, right?

Regardless tho, id be more than happy with even just putting the accuracy setting in scene render settings so i can crank it up and set it in my startup file.

Just wanted to chime in as a user who animates audio all the time. Im thinking that the current system (and accuracy) is fine for live playback, and i dont think it would be a big deal if audio rendering were slower (tho, a progress meter might be good... an hour+ of audio does take some time to render already). On the other hand, if making the linear interpolation behave the way it should is a reasonable possible fix, i dont know if implementing every-sample depsgraph is needed really, thats kinda what the audio render accuracy is for, right? Regardless tho, id be more than happy with even just putting the accuracy setting in scene render settings so i can crank it up and set it in my startup file.

Richard Antalik commented

2019-10-13 02:52:04 +02:00

Added subscriber: @ronsn

Philipp Oeser removed the

Download

What's New

Blender Studio

Manual

Developers Blog

Documentation

Benchmark

Blender Conference

Development Fund

One-time Donations

VSE - Bad interpolation in exported audio #68946