VSE waveform drawing in sound strip #115274

New Issue

Francesco Siddi · 2023-11-22T16:39:47+01:00

Francesco Siddi commented

2023-11-22 16:39:47 +01:00

Currently the rendering of waveforms in the VSE looks as follows:

In this proposal, we represent the wave as "vertical bars". The bars have fixed width, and increase or decrease in number depending on the horizontal zoom level. Once we reach a certain zoom level, we can optionally draw only the outline of the bars.

Furthermore, depending on the zoom level (or even a view setting) we represent the waveform as an actual oscillating wave, or we mirror the negative values in order to create a symmetric representation of the wave. In the case of symmetric representation, the wave can also be displayed as "half", to show more detail.

Currently the rendering of waveforms in the VSE looks as follows: ![image](/attachments/5c2a0245-46c2-4e67-8f66-102aec7ff8c5) In this proposal, we represent the wave as "vertical bars". The bars have fixed width, and increase or decrease in number depending on the horizontal zoom level. Once we reach a certain zoom level, we can optionally draw only the outline of the bars. Furthermore, depending on the zoom level (or even a view setting) we represent the waveform as an actual oscillating wave, or we mirror the negative values in order to create a symmetric representation of the wave. In the case of symmetric representation, the wave can also be displayed as "half", to show more detail. ![image](/attachments/6e984ecc-da7e-4aa2-98ed-64e1be147dce)

image.png

142 KiB

image.png

458 KiB

Francesco Siddi added the

Type

Design

label 2023-11-22 16:39:47 +01:00

Francesco Siddi added this to the Video Sequencer project 2023-11-22 16:39:49 +01:00

Aras Pranckevicius commented

2023-11-23 18:35:59 +01:00

Some notes so they don't get lost in the chat:

Blender only samples sounds at 250 Hz for display, current code literally can't display more detail. This is deemed acceptable for now, primarily because VSE does not allow sub-frame audio placement anyway, so sample-level precision for display is not really needed.
Blender does not only display the waveform, but also displays RMS level (whiter part; similar is done in Audacity just there is lighter blue part) and indicates "clipped" parts in vertical red line.

As part of trying to optimize VSE timeline drawing performance, I've looked at waveform drawing. Doing the "square" like proposed above is actually a bit faster compared to current code. Here's what I have now in a local change; current blender vs modified code that displays "square" shape:

You can notice that the squares do not reach as high peaks as the "current" code. That is actually a bug in current code; it often overshoots the actual sample values when trying to interpolate them (which is partially why when zoomed in the wave looks kinda weird). It also results in sometimes the waveform display showing red "clipped" parts when in fact there aren't any (again due to same issue; it interpolates the samples wrong).

Some notes so they don't get lost in the chat: - Blender only samples sounds at 250 Hz for display, current code literally can't display more detail. This is deemed acceptable for now, primarily because VSE does not allow sub-frame audio placement anyway, so sample-level precision for display is not really needed. - Blender does not only display the waveform, but also displays RMS level (whiter part; similar is done in Audacity just there is lighter blue part) and indicates "clipped" parts in vertical red line. As part of trying to optimize VSE timeline drawing performance, I've looked at waveform drawing. Doing the "square" like proposed above is actually a bit faster compared to current code. Here's what I have now in a local change; current blender vs modified code that displays "square" shape: ![wave1.png](/attachments/78f60794-d13d-4b55-8a52-7f1a5270ecac) ![wave2.png](/attachments/fa7558a1-be2a-47d9-b45c-4f70fd2761bc) You can notice that the squares do not reach as high peaks as the "current" code. That is actually _a bug_ in current code; it often overshoots the actual sample values when trying to interpolate them (which is partially why when zoomed in the wave looks kinda weird). It also results in sometimes the waveform display showing red "clipped" parts when in fact there aren't any (again due to same issue; it interpolates the samples wrong).

wave1.png

35 KiB

wave2.png

30 KiB

🚀 1

Aras Pranckevicius referenced this issue from a commit

2023-11-23 18:51:32 +01:00

VSE: speedup / improve waveform overlay drawing

Aras Pranckevicius referenced this issue from a commit

2023-11-23 19:11:44 +01:00

VSE: speedup / improve waveform overlay drawing

Aras Pranckevicius referenced this issue

2023-11-24 09:03:18 +01:00

VSE: speedup timeline drawing, and improve waveform display #115311

Aras Pranckevicius referenced this issue from a commit

2023-11-29 20:25:32 +01:00

VSE: speedup timeline drawing, and improve waveform display

Francesco Siddi commented

2023-12-18 23:24:10 +01:00

As a follow up to this implementation, here is a proposal for how handle waveforms display in the VSE Overlays popover.

This mockup addressed three topics:

use a more consistent radio selection for waveform display, not an enum-radio hybrid
enable waveform display (on) by default, as performance should now be good enough.
add a new "Waveform Style" radio

Each topic should be addressed in its own commit.
Informally reviewed with Sergey and @dfelinto.

As a follow up to this implementation, here is a proposal for how handle waveforms display in the VSE Overlays popover. ![image](/attachments/33476068-c341-4eb9-9d90-dfbf2b4a0bf2) This mockup addressed three topics: - use a more consistent radio selection for waveform display, not an enum-radio hybrid - enable waveform display (on) by default, as performance should now be good enough. - add a new "Waveform Style" radio Each topic should be addressed in its own commit. Informally reviewed with Sergey and @dfelinto.

image.png

41 KiB

👍 1

Aras Pranckevicius self-assigned this 2023-12-19 06:04:14 +01:00

Aras Pranckevicius referenced this issue

2023-12-19 12:34:21 +01:00

VSE: display audio waveforms by default #116334

Aras Pranckevicius referenced this issue

2023-12-19 12:34:42 +01:00

VSE: more intuitive UI control for waveform display mode #116336

Aras Pranckevicius referenced this issue

2023-12-19 15:11:38 +01:00