Regression: animated scene audio volume not changing the audible volume on playback #120726

Open
opened 2024-04-17 03:10:34 +02:00 by Campbell Barton · 19 comments

System Information
Operating system: Linux-6.8.5-arch1-1-x86_64-with-glibc2.39 64 Bits, WAYLAND UI
Graphics card: AMD Radeon RX 5700 XT (radeonsi, navi10, LLVM 17.0.6, DRM 3.57, 6.8.5-arch1-1) AMD 4.6 (Core Profile) Mesa 24.0.5-arch1.1

Blender Version
Broken: version: 4.1.0, branch: makepkg (modified), commit date: 2024-03-25 20:42, hash: 40a5e739e270
Worked: 2.79b

Short description of error
Animating the volume of the scene audio doesn't change the audible volume during playback.

Exact steps for others to reproduce the error

  • Change the timeline view to the sequence editor.
  • Add a sound fine (attached noise.mp3).
  • Add a key-frame to the scenes audio volume (1.0)
  • Change the frame to match the end of the audio.
  • Change the volume to zero.
  • Add a key-frame to the scenes audio volume.
  • Space to play the animation.

Notice:

  • The audio volume (as you hear it through speakers) doesn't change.
  • When you move the mouse cursor over the volume button it does change (only on refreshing).
  • When you scrub-time, the audio playback does change at the point where the play-head is moved to - but doesn't animate after that.

Attached an example that has animated volume, where you can play immediately to see the error.


#70569 may be related, although this seems like a simpler case.

**System Information** Operating system: Linux-6.8.5-arch1-1-x86_64-with-glibc2.39 64 Bits, WAYLAND UI Graphics card: AMD Radeon RX 5700 XT (radeonsi, navi10, LLVM 17.0.6, DRM 3.57, 6.8.5-arch1-1) AMD 4.6 (Core Profile) Mesa 24.0.5-arch1.1 **Blender Version** Broken: version: 4.1.0, branch: makepkg (modified), commit date: 2024-03-25 20:42, hash: `40a5e739e270` Worked: 2.79b **Short description of error** Animating the volume of the scene audio doesn't change the audible volume during playback. **Exact steps for others to reproduce the error** - Change the timeline view to the sequence editor. - Add a sound fine (attached `noise.mp3`). - Add a key-frame to the scenes audio volume (1.0) - Change the frame to match the end of the audio. - Change the volume to zero. - Add a key-frame to the scenes audio volume. - Space to play the animation. Notice: - The audio volume (as you hear it through speakers) doesn't change. - When you move the mouse cursor over the volume button it does change (only on refreshing). - When you scrub-time, the audio playback does change at the point where the play-head is moved to - but doesn't animate after that. --- Attached an example that has animated volume, where you can play immediately to see the error. ---- #70569 may be related, although this seems like a simpler case.
Campbell Barton added the
Type
Report
Priority
Normal
Status
Needs Triage
labels 2024-04-17 03:10:34 +02:00
Campbell Barton changed title from Animated volume not working to Regression: animated volume doesn't change audible volume on playback 2024-04-17 03:14:47 +02:00
Campbell Barton changed title from Regression: animated volume doesn't change audible volume on playback to Regression: animated scene audio volume not changing the audible volume on playback 2024-04-17 03:15:22 +02:00
Member

Hi, can confirm. #70569 does look related
cc @iss @neXyon

Hi, can confirm. #70569 does look related cc @iss @neXyon
Member

This is a bug in the UI, not sure where it comes from. This value should not be animatable, since PROP_ANIMATABLE is cleared in rna_scene.cc.

This is a bug in the UI, not sure where it comes from. This value should not be animatable, since `PROP_ANIMATABLE` is cleared in `rna_scene.cc`.
Author
Owner

This is a bug in the UI, not sure where it comes from. This value should not be animatable, since PROP_ANIMATABLE is cleared in rna_scene.cc.

There is an audio_volume as part of FFmpegSettings which is cleared, double checked and Scene's audio_volume can be animated.

> This is a bug in the UI, not sure where it comes from. This value should not be animatable, since `PROP_ANIMATABLE` is cleared in `rna_scene.cc`. There is an `audio_volume` as part of `FFmpegSettings` which is cleared, double checked and Scene's audio_volume can be animated.
Member

You're right, my mistake, sorry! I investigated a little bit. It's apparently an issue of the depsgraph, since all the functions that should be called to update the volume are basically never called. First sound_update_animation_flags in sound_ops.cc is never called, so scene->audio.flag |= AUDIO_VOLUME_ANIMATED never happens. And even if I press the "Update Animation Cache" button, and this gets called, then BKE_scene_update_sound does not get called still. Maybe @Sergey could have a look?

You're right, my mistake, sorry! I investigated a little bit. It's apparently an issue of the depsgraph, since all the functions that should be called to update the volume are basically never called. First `sound_update_animation_flags` in `sound_ops.cc` is never called, so `scene->audio.flag |= AUDIO_VOLUME_ANIMATED` never happens. And even if I press the "Update Animation Cache" button, and this gets called, then `BKE_scene_update_sound` does not get called still. Maybe @Sergey could have a look?
Member

There was #75686 / 6adb254bb0 already for this?

There was #75686 / 6adb254bb046 already for this?
Member

File from #75686 seems to work fine in 4.2...

File from #75686 seems to work fine in 4.2...
Member
CC @dr.sybren
Member

For the file from #75686, BKE_scene_update_tag_audio_volume is constantly called, however for the file from this report, it is not...

For the file from #75686, `BKE_scene_update_tag_audio_volume` is constantly called, however for the file from this report, it is not...
Member

Hm, even SOUND_OT_update_animation_flags -- which actually reaches scene->audio.flag |= AUDIO_VOLUME_ANIMATED does not have the consequence of BKE_scene_update_tag_audio_volume being called constantly

(EDIT: ah, I see @neXyon already checked this...)

Hm, even `SOUND_OT_update_animation_flags` -- which actually reaches `scene->audio.flag |= AUDIO_VOLUME_ANIMATED` does not have the consequence of `BKE_scene_update_tag_audio_volume` being called constantly (EDIT: ah, I see @neXyon already checked this...)
Sybren A. Stüvel self-assigned this 2024-04-25 13:01:07 +02:00

I've done some digging, and these are my findings:

  • The culprit is indeed the scene flag AUDIO_VOLUME_ANIMATED. This is not actually set when creating an F-Curve that animates the scene volume.
  • The dependency graph relations builder uses this flag to figure out whether to call BKE_scene_update_tag_audio_volume() during evaluation or not. If AUDIO_VOLUME_ANIMATED is not set, the relation is not added, and thus the function doesn't get called.
  • VSE only calls sound_update_animation_flags(scene) (which sets that flag) when there is a scene strip in the VSE, which in this demo file is not the case.
  • You can force an update of the scene flag with the "Update Animation" (bpy.ops.sound.update_animation_flags) operator, but that does not trigger a rebuild of the depsgraph relations. As a result, it seems to have no effect until something else triggers that. Once those conditions are met, the flags are set correctly, and the volume is animated.

So basically we have to find a way to set that flag to the appropriate value when the scene.audio_volume RNA property is animated (or its animation is cleared). The depsgraph is already tagged for rebuilding relations, whenever something gets keyed (in a way that creates a new FCurve), so once the AUDIO_VOLUME_ANIMATED is set correctly before the depsgraph relations are actually rebuilt, we should be good to go.

The issue is now: where do we put that code, in a way that doesn't add weirdly specific code in otherwise generic functions? @neXyon do you have ideas?

I've done some digging, and these are my findings: - The culprit is indeed the scene flag `AUDIO_VOLUME_ANIMATED`. This is **not actually set when creating an F-Curve** that animates the scene volume. - The dependency graph relations builder uses this flag to figure out whether to call `BKE_scene_update_tag_audio_volume()` during evaluation or not. **If `AUDIO_VOLUME_ANIMATED` is not set, the relation is not added,** and thus the function doesn't get called. - VSE only calls `sound_update_animation_flags(scene)` (which sets that flag) **when there is a scene strip** in the VSE, which in this demo file is not the case. - You can force an update of the scene flag with the "Update Animation" (`bpy.ops.sound.update_animation_flags`) operator, but that **does _not_ trigger a rebuild of the depsgraph relations**. As a result, it seems to have no effect until something else triggers that. Once those conditions are met, the flags are set correctly, and the volume is animated. So basically we have to find a way to set that flag to the appropriate value when the `scene.audio_volume` RNA property is animated (or its animation is cleared). The depsgraph is already tagged for rebuilding relations, whenever something gets keyed (in a way that creates a new FCurve), so once the `AUDIO_VOLUME_ANIMATED` is set correctly before the depsgraph relations are actually rebuilt, we should be good to go. The issue is now: where do we put that code, in a way that doesn't add weirdly specific code in otherwise generic functions? @neXyon do you have ideas?

This does work as a proof of concept (albeit wrong and ugly as it always sets the flag and does so very specifically in otherwise generic code):

diff --git a/source/blender/animrig/intern/keyframing.cc b/source/blender/animrig/intern/keyframing.cc
index 2944d109571..c56dc01f5cb 100644
--- a/source/blender/animrig/intern/keyframing.cc
+++ b/source/blender/animrig/intern/keyframing.cc
@@ -754,6 +754,10 @@ CombinedKeyingResult insert_keyframe(Main *bmain,
     if (adt != nullptr && adt->action != nullptr && adt->action != act) {
       DEG_id_tag_update(&adt->action->id, ID_RECALC_ANIMATION_NO_FLUSH);
     }
+    if (GS(id.name) == ID_SCE) {
+      Scene *scene = (Scene *)(&id);
+      scene->audio.flag |= AUDIO_VOLUME_ANIMATED;
+    }
   }
 
   return combined_result;

Compile with this change, then set any key that creates a new FCurve (for example hover over the Gravity checkbox and press I). That'll trigger this code, set the flag, and when the depsgraph is rebuilt, relations are correct.

So basically the bug is that this flag is used to track a generic audio property on the Scene (which should be set correctly regardless of whether the VSE is used or not), but it's only set by the VSE.

This does work as a proof of concept (albeit wrong and ugly as it always sets the flag and does so very specifically in otherwise generic code): ```diff diff --git a/source/blender/animrig/intern/keyframing.cc b/source/blender/animrig/intern/keyframing.cc index 2944d109571..c56dc01f5cb 100644 --- a/source/blender/animrig/intern/keyframing.cc +++ b/source/blender/animrig/intern/keyframing.cc @@ -754,6 +754,10 @@ CombinedKeyingResult insert_keyframe(Main *bmain, if (adt != nullptr && adt->action != nullptr && adt->action != act) { DEG_id_tag_update(&adt->action->id, ID_RECALC_ANIMATION_NO_FLUSH); } + if (GS(id.name) == ID_SCE) { + Scene *scene = (Scene *)(&id); + scene->audio.flag |= AUDIO_VOLUME_ANIMATED; + } } return combined_result; ``` Compile with this change, then set any key that creates a new FCurve (for example hover over the Gravity checkbox and press `I`). That'll trigger this code, set the flag, and when the depsgraph is rebuilt, relations are correct. So basically the bug is that this flag is used to track a generic audio property on the Scene (which should be set correctly regardless of whether the VSE is used or not), but it's only set by the VSE.
Sybren A. Stüvel removed their assignment 2024-04-25 15:55:30 +02:00
Member

Thanks for the investigation @dr.sybren ! Unfortunately, I don't know what the ideal place would be to implement this. Maybe @Sergey or @iss ?

Thanks for the investigation @dr.sybren ! Unfortunately, I don't know what the ideal place would be to implement this. Maybe @Sergey or @iss ?

VSE only calls sound_update_animation_flags(scene) (which sets that flag) when there is a scene strip in the VSE, which in this demo file is not the case.

This function seems to be called only from bpy.ops.sound.update_animation_flags operator. Don't see it being called from VSE code.

Not sure if I have any good insight here. Dependency graph seems to know whether sound strip volume is animated and in such case it calls seq_update_sound_properties() for every frame. There is SEQ_AUDIO_VOLUME_ANIMATED flag, but it is never set (it is only set by bpy.ops.sound.update_animation_flags). This relation seems to be created by build_animdata_fcurve_target(), but other than that, the code seems arcane to me.

That said, if this works for strips without flag, I don't see reason why this shouldn't work for scene without this flag. As bonus, perhaps these flags can be removed. @Sergey may have better idea still, so I like to know his opinion

> VSE only calls `sound_update_animation_flags(scene)` (which sets that flag) **when there is a scene strip** in the VSE, which in this demo file is not the case. This function seems to be called only from `bpy.ops.sound.update_animation_flags` operator. Don't see it being called from VSE code. Not sure if I have any good insight here. Dependency graph seems to know whether sound strip volume is animated and in such case it calls `seq_update_sound_properties()` for every frame. There is `SEQ_AUDIO_VOLUME_ANIMATED` flag, but it is never set (it is only set by `bpy.ops.sound.update_animation_flags`). This relation seems to be created by `build_animdata_fcurve_target()`, but other than that, the code seems arcane to me. That said, if this works for strips without flag, I don't see reason why this shouldn't work for scene without this flag. As bonus, perhaps these flags can be removed. @Sergey may have better idea still, so I like to know his opinion

VSE only calls sound_update_animation_flags(scene) (which sets that flag) when there is a scene strip in the VSE, which in this demo file is not the case.

This function seems to be called only from bpy.ops.sound.update_animation_flags operator. Don't see it being called from VSE code.

You're right, I must have mixed up some things.

> > VSE only calls `sound_update_animation_flags(scene)` (which sets that flag) **when there is a scene strip** in the VSE, which in this demo file is not the case. > > This function seems to be called only from `bpy.ops.sound.update_animation_flags` operator. Don't see it being called from VSE code. You're right, I must have mixed up some things.

Do we even need this flag? Can we always assume volume/pitch is animated, and get rid of the complexity of maintaining this flag?

Do we even need this flag? Can we always assume volume/pitch is animated, and get rid of the complexity of maintaining this flag?
Member

Could be possible, might introduce some audible issues though: The audio library distinguishes between animated and non-animated properties (i.e., constant value). If it's animated, you have to set the values for all frames, not just a single one.

So if you change the actually non-animated volume in Blender (but since you removed the flag you always tell the audio library it's animated) you have to update all frames:

  1. Either this happens immediately, which means you would need to know that there is no animation, thus you better use the flag instead.
  2. Or you do it whenever the frame is changed with the potential issue that if you are playing back and it is updated because of that, you might hear the old volume, because the value hasn't been updated on time.
Could be possible, might introduce some audible issues though: The audio library distinguishes between animated and non-animated properties (i.e., constant value). If it's animated, you have to set the values for all frames, not just a single one. So if you change the actually non-animated volume in Blender (but since you removed the flag you always tell the audio library it's animated) you have to update all frames: 1. Either this happens immediately, which means you would need to know that there is no animation, thus you better use the flag instead. 2. Or you do it whenever the frame is changed with the potential issue that if you are playing back and it is updated because of that, you might hear the old volume, because the value hasn't been updated on time.

Could be possible, might introduce some audible issues though: The audio library distinguishes between animated and non-animated properties (i.e., constant value). If it's animated, you have to set the values for all frames, not just a single one.

So if you change the actually non-animated volume in Blender (but since you removed the flag you always tell the audio library it's animated) you have to update all frames:

  1. Either this happens immediately, which means you would need to know that there is no animation, thus you better use the flag instead.
  2. Or you do it whenever the frame is changed with the potential issue that if you are playing back and it is updated because of that, you might hear the old volume, because the value hasn't been updated on time.

If we assume that properties are animated, then this won't be an issue, as each frame will be updated before it is played back. I agree with that approach. We just need to ensure, that animation is evaluated before updating volume, but this should be set up correctly already.

> Could be possible, might introduce some audible issues though: The audio library distinguishes between animated and non-animated properties (i.e., constant value). If it's animated, you have to set the values for all frames, not just a single one. > > So if you change the actually non-animated volume in Blender (but since you removed the flag you always tell the audio library it's animated) you have to update all frames: > 1. Either this happens immediately, which means you would need to know that there is no animation, thus you better use the flag instead. > 2. Or you do it whenever the frame is changed with the potential issue that if you are playing back and it is updated because of that, you might hear the old volume, because the value hasn't been updated on time. If we assume that properties are animated, then this won't be an issue, as each frame will be updated before it is played back. I agree with that approach. We just need to ensure, that animation is evaluated before updating volume, but this should be set up correctly already.

A bit of analysis:

Since this is audio and not video, it runs at a way higher rate, so I can imagine that only updating the volume on the frame itself can cause issues (24 FPS volume changes instead of gradual ones).

Currently Blender also behaves a bit strange here. Look at the attached blend file (in which I already called the 'update animation' operator so the flags are set correctly for volume animation). The scene frame rate is 1 FPS to make sub-frame volume changes obvious.

image

If you start playback at frame 2, I'd expect it to start at a low volume and ramp up until frame 4. What actually happens is that the volume starts loud, and ramps down until frame 3. Continuing playback it becomes clear that the volume is lagging behind the animated value by one frame. Basically setting volume = X at frame N just fades the volume to that value over the duration of that frame.

So this is a great way to avoid stepping volume changes, at the cost of a one frame lag.

Since the volume could be changed not only by F-Curves but also by drivers, Python code, etc. there is no general way to look into the future and see what the volume should be at the next frame (except by paying the cost of evaluating the depsgraph ahead of time).


Could be possible, might introduce some audible issues though: The audio library distinguishes between animated and non-animated properties (i.e., constant value). If it's animated, you have to set the values for all frames, not just a single one.

I don't quite understand the implications of this. The example blend file shows that there is an off-by-one issue with the audio, which wouldn't exist if the correct volumes were given for all frames.

A bit of analysis: Since this is audio and not video, it runs at a way higher rate, so I can imagine that only updating the volume on the frame itself can cause issues (24 FPS volume changes instead of gradual ones). Currently Blender also behaves a bit strange here. Look at the attached blend file (in which I already called the 'update animation' operator so the flags are set correctly for volume animation). The scene frame rate is 1 FPS to make sub-frame volume changes obvious. ![image](/attachments/3e1c790d-f72b-49df-94c1-0161435004d4) If you start playback at frame 2, I'd expect it to start at a low volume and ramp up until frame 4. What actually happens is that the volume starts loud, and ramps down until frame 3. Continuing playback it becomes clear that the volume is lagging behind the animated value by one frame. Basically setting `volume = X` at frame `N` just fades the volume to that value over the duration of that frame. So this is a great way to avoid stepping volume changes, at the cost of a one frame lag. Since the volume could be changed not only by F-Curves but also by drivers, Python code, etc. there is no general way to look into the future and see what the volume should be at the next frame (except by paying the cost of evaluating the depsgraph ahead of time). -------------- > Could be possible, might introduce some audible issues though: The audio library distinguishes between animated and non-animated properties (i.e., constant value). If it's animated, you have to set the values for all frames, not just a single one. I don't quite understand the implications of this. The example blend file shows that there is an off-by-one issue with the audio, which wouldn't exist if the correct volumes were given for all frames.
Member

This is all related to the detailed discussion I made 3 years ago, where I mostly quote a long post from 6 years ago: https://devtalk.blender.org/t/better-audio-integration/17810

I would love to have a better integration of the animation system and audio. Maybe we should think about a better solution that could help with all the issues? Would it be possible to have a separate evaluation state of the dependency graph that the audio system controls in as far as setting the current evaluation time and then being able to query all those properties that might be animated, like volume, pan, position and orientation of the camera and speakers, etc? Then I could probably even drop the audio system's animation cache and the code in Blender could become less complex (i.e., these flags can be dropped).

This is all related to the detailed discussion I made 3 years ago, where I mostly quote a long post from 6 years ago: https://devtalk.blender.org/t/better-audio-integration/17810 I would love to have a better integration of the animation system and audio. Maybe we should think about a better solution that could help with all the issues? Would it be possible to have a separate evaluation state of the dependency graph that the audio system controls in as far as setting the current evaluation time and then being able to query all those properties that might be animated, like volume, pan, position and orientation of the camera and speakers, etc? Then I could probably even drop the audio system's animation cache and the code in Blender could become less complex (i.e., these flags can be dropped).
Sign in to join this conversation.
No Label
Interest
Alembic
Interest
Animation & Rigging
Interest
Asset Browser
Interest
Asset Browser Project Overview
Interest
Audio
Interest
Automated Testing
Interest
Blender Asset Bundle
Interest
BlendFile
Interest
Collada
Interest
Compatibility
Interest
Compositing
Interest
Core
Interest
Cycles
Interest
Dependency Graph
Interest
Development Management
Interest
EEVEE
Interest
EEVEE & Viewport
Interest
Freestyle
Interest
Geometry Nodes
Interest
Grease Pencil
Interest
ID Management
Interest
Images & Movies
Interest
Import Export
Interest
Line Art
Interest
Masking
Interest
Metal
Interest
Modeling
Interest
Modifiers
Interest
Motion Tracking
Interest
Nodes & Physics
Interest
OpenGL
Interest
Overlay
Interest
Overrides
Interest
Performance
Interest
Physics
Interest
Pipeline, Assets & IO
Interest
Platforms, Builds & Tests
Interest
Python API
Interest
Render & Cycles
Interest
Render Pipeline
Interest
Sculpt, Paint & Texture
Interest
Text Editor
Interest
Translations
Interest
Triaging
Interest
Undo
Interest
USD
Interest
User Interface
Interest
UV Editing
Interest
VFX & Video
Interest
Video Sequencer
Interest
Virtual Reality
Interest
Vulkan
Interest
Wayland
Interest
Workbench
Interest: X11
Legacy
Blender 2.8 Project
Legacy
Milestone 1: Basic, Local Asset Browser
Legacy
OpenGL Error
Meta
Good First Issue
Meta
Papercut
Meta
Retrospective
Meta
Security
Module
Animation & Rigging
Module
Core
Module
Development Management
Module
EEVEE & Viewport
Module
Grease Pencil
Module
Modeling
Module
Nodes & Physics
Module
Pipeline, Assets & IO
Module
Platforms, Builds & Tests
Module
Python API
Module
Render & Cycles
Module
Sculpt, Paint & Texture
Module
Triaging
Module
User Interface
Module
VFX & Video
Platform
FreeBSD
Platform
Linux
Platform
macOS
Platform
Windows
Priority
High
Priority
Low
Priority
Normal
Priority
Unbreak Now!
Status
Archived
Status
Confirmed
Status
Duplicate
Status
Needs Info from Developers
Status
Needs Information from User
Status
Needs Triage
Status
Resolved
Type
Bug
Type
Design
Type
Known Issue
Type
Patch
Type
Report
Type
To Do
No Milestone
No project
No Assignees
7 Participants
Notifications
Due Date
The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: blender/blender#120726
No description provided.