VSE: Implement partial waveform data loading #110837

Lucas Tadeu · 2023-08-05T18:38:34+02:00

Lucas Tadeu commented

2023-08-05 18:38:34 +02:00

The waveform drawing is not totally functional yet, but the loads work correctly.

How the implementation works

When we want to draw a sound's waveform, we need to first read the sound samples and prepare the data so that the "waves" can be drawn.

You can visualise the sound data as a contiguous array of sample data:

In the main branch, the waveform data is loaded all at once, even if it is not needed to draw the current frame. This is problematic with large files as it requires a lot of data to be read before anything is drawn to the screen.

To address this issue, we determine what portion of the sound strip is visible in the sequencer and read only the data needed for that portion.

To keep track of what data is loaded and/or needs to be loaded to render a given sound strip, we use a segment tree to keep track of each segment:

Each node in the keeps track of whether the data it is responsible for is already loaded.

Only leaf nodes (the ones without any children in the tree) are loaded.

This takes advantage of the multi-threading introduced in #108877. If more than one segment needs to be loaded in order to render the strip's waveform, the segments get loaded in parallel.

Since no two segments overlap, they do not need to coordinate when writing to the waveform data buffer.

Each segment is only processed once thanks to tags in each segment tree node. If a node is either loaded or loading, no task is created to load its data.

Note: Traversing the segment tree is not a big deal. Since it is parameterised on the total number of seconds in the audio source, even if the audio was 5 hours long we would only need to traverse ~17 (log2(5 * 3600) nodes (in the worst case) to figure out if a segment is loaded or not.

Example: Display a segment with only partial data loaded

We want to render the waveform data for the node marked in yellow:

Since the left subtree is already loaded, we only need load the right subtree:

Once the right subtree is loaded, the yellow node is marked as loaded and ready for display:

Notes

The previous AUD_readSound function normalised the data read by default.
That function was replaced with AUD_readSoundAt.

AUD_readSoundAt does not normalise the data.
Instead, whenever we compute the waveform drawing data, the data normalises the sample using the current maximum sample value across all the loaded samples.

I considered always normalising and then renormalising the data when a new. maximum was found, but decided against it due to potentially losing precision due to the various floating point divisions and multiplications.

Another option would have been to keep a shadow version of the audio data that would be overridden when a new maximum was found. But that would double the memory requirements.

Demo

Here's a demo showcasing how the partial waveform load behaves: https://youtu.be/Ujg-EvFpVgg

The waveform drawing is not totally functional yet, but the loads work correctly. ## How the implementation works When we want to draw a sound's waveform, we need to first read the sound samples and prepare the data so that the "waves" can be drawn. You can visualise the sound data as a contiguous array of sample data: ![image](/attachments/7d16c498-1541-4959-ad1e-f65c38d86dd8) In the main branch, the waveform data is loaded all at once, even if it is not needed to draw the current frame. This is problematic with large files as it requires a lot of data to be read before anything is drawn to the screen. To address this issue, we determine what portion of the sound strip is visible in the sequencer and read only the data needed for that portion. To keep track of what data is loaded and/or needs to be loaded to render a given sound strip, we use a segment tree to keep track of each segment: ![image](/attachments/91888469-ca1c-4c2d-9e55-0f4f42dcf957) Each node in the keeps track of whether the data it is responsible for is already loaded. Only leaf nodes (the ones without any children in the tree) are loaded. This takes advantage of the multi-threading introduced in https://projects.blender.org/blender/blender/pulls/108877. If more than one segment needs to be loaded in order to render the strip's waveform, the segments get loaded in parallel. Since no two segments overlap, they do not need to coordinate when writing to the waveform data buffer. Each segment is only processed once thanks to tags in each segment tree node. If a node is either loaded or loading, no task is created to load its data. **Note:** Traversing the segment tree is not a big deal. Since it is parameterised on the total number of seconds in the audio source, even if the audio was 5 hours long we would only need to traverse ~17 (log2(5 * 3600) nodes (in the worst case) to figure out if a segment is loaded or not. ## Example: Display a segment with only partial data loaded We want to render the waveform data for the node marked in yellow: ![image](/attachments/feca8375-b50f-465a-8cc0-217c0332bc53) Since the left subtree is already loaded, we only need load the right subtree: ![image](/attachments/ee72d628-1fde-4068-a8c0-e7cb9ebc5f0e) Once the right subtree is loaded, the yellow node is marked as loaded and ready for display: ![image](/attachments/0c7a3203-aab5-4316-a5f9-f8a19ddaa4af) ## Notes The previous AUD_readSound function normalised the data read by default. That function was replaced with AUD_readSoundAt. AUD_readSoundAt does not normalise the data. Instead, whenever we compute the waveform drawing data, the data normalises the sample using the current maximum sample value across all the loaded samples. I considered always normalising and then renormalising the data when a new. maximum was found, but decided against it due to potentially losing precision due to the various floating point divisions and multiplications. Another option would have been to keep a shadow version of the audio data that would be overridden when a new maximum was found. But that would double the memory requirements. ## Demo Here's a demo showcasing how the partial waveform load behaves: https://youtu.be/Ujg-EvFpVgg

image.png

189 KiB

image.png

28 KiB

image.png

152 KiB

image.png

180 KiB

image.png

157 KiB

image.png

176 KiB

Lucas Tadeu added 1 commit 2023-08-05 18:38:45 +02:00

6b69d589a1 VSE: Implement partial waveform data loading

Lucas Tadeu requested review from Richard Antalik 2023-08-05 18:39:47 +02:00

Lucas Tadeu added 1 commit 2023-08-19 15:41:17 +02:00

0739a99084 VSE: Remove upfront normalization and clean up

Lucas Tadeu added 2 commits 2023-08-19 16:02:04 +02:00

e69e523961 Remove unused Range2i

fdbbabc907 Remove unused code

Lucas Tadeu added 1 commit 2023-08-19 16:48:38 +02:00

69ced56c05 Merge branch 'main' into Yup_Lucas/vse-partial-waveform-loads

Lucas Tadeu changed title from ~~WIP: VSE: Implement partial waveform data loading~~ to VSE: Implement partial waveform data loading

2023-08-19 18:28:12 +02:00

Richard Antalik commented

2023-08-25 14:54:25 +02:00

So far I was only able to quickly check functionality, unfortunately with this patch it double locks sound->spinlock and blender hangs.
When I commentedout the lock where hang occurs - clear_sound_waveform_loading_tag() it does not load all waveforms.

I can do some debugging on monday.

So far I was only able to quickly check functionality, unfortunately with this patch it double locks `sound->spinlock` and blender hangs. When I commentedout the lock where hang occurs - `clear_sound_waveform_loading_tag()` it does not load all waveforms. I can do some debugging on monday.

Lucas Tadeu commented

2023-08-25 15:10:30 +02:00

hey @iss , can you send me the repro steps so I can figure out what is wrong with the patch? 🤔

Lucas Tadeu reviewed 2023-08-25 15:25:38 +02:00

source/blender/editors/space_sequencer/sequencer_draw.cc

						
				@ -285,0 +296,4 @@

				  if (!(segment->tags & loaded_or_loading)) {

				    if (!segment->left && !segment->right) {

				      segment->tags |= SOUND_TAGS_WAVEFORM_LOADING;

				      sequencer_preview_add_sound_segment(C, seq, segment);

Lucas Tadeu commented

2023-08-25 15:25:38 +02:00

I think the issue with the double locking that @iss reported is here. waveform_job_start_all_needed_segements is called inside the spin lock lock.

If in sequencer_preview_add_segment the PreviewJob is stopped, we'll get double locking because of the call to clear_sound_waveform_loading_tag (this function tries to acquire the lock).

Thinking about this now, we probably don't need that lock to be used.
We do an atomic load of the tags in the waveform_job_start_all_needed_segments, and in the clear_sound_waveform_loading_tag and BKE_sound_read_waveform_segment functions do an atomic write to set the tags correctly.

TODO for myself is to update clear_sound_waveform_loading_tag to clear the segment tag, not the sound itself. We are not loading entire sound files anymore.

I think the issue with the double locking that @iss reported is here. `waveform_job_start_all_needed_segements` is called inside the spin lock lock. If in sequencer_preview_add_segment the PreviewJob is stopped, we'll get double locking because of the call to `clear_sound_waveform_loading_tag` (this function tries to acquire the lock). Thinking about this now, we probably don't need that lock to be used. We do an atomic load of the tags in the `waveform_job_start_all_needed_segments`, and in the `clear_sound_waveform_loading_tag` and `BKE_sound_read_waveform_segment` functions do an atomic write to set the tags correctly. TODO for myself is to update `clear_sound_waveform_loading_tag` to clear the segment tag, not the sound itself. We are not loading entire sound files anymore.

Richard Antalik commented

2023-08-29 12:39:04 +02:00

Unfortunately I have used file, that I cannot share. It had large number (~100) of shorter sound clips, with some longer ones.
I will try to produce .blend file with sound files that you can copy, to reproduce.
I have tried to comment out locks, but it didn't work either and haven't had time to look into it in bit more detail.

Unfortunately I have used file, that I cannot share. It had large number (~100) of shorter sound clips, with some longer ones. I will try to produce .blend file with sound files that you can copy, to reproduce. I have tried to comment out locks, but it didn't work either and haven't had time to look into it in bit more detail.

Lucas Tadeu added 1 commit 2023-08-29 21:02:56 +02:00

2eafb34572 Eliminate spinlocks to update waveform loading flags

Lucas Tadeu commented

2023-08-29 21:18:06 +02:00

I was able to replicate the problem on my machine by putting a really long sleep in the preview_startjob function in sequencer_preview:

 /** \file
  * \ingroup spseq
  */
+#include <unistd.h>
 
 #include "DNA_sequence_types.h"
 #include "DNA_sound_types.h"
@@ -173,6 +174,7 @@ static void preview_startjob(void *data, bool *stop, bool *do_update, float *pro
     BLI_mutex_unlock(pj->mutex);
   }
 
+  sleep(10); // sleep for 10 seconds before shutting down the preview job
   BLI_task_pool_work_and_wait(task_pool);
   BLI_task_pool_free(task_pool);
 }

** Why did the deadlock happen? **

When a segment was submitted to the PreviewJob, the sound's spin lock was acquired.
If the PreviewJob was terminating and a new segment was submitted, we cleared the sound's loading bit.
The function that cleared the bit acquired the same spin lock and the code deadlocked.

** Why didn't the waveforms load after removing the locks? **

The problem was in the function that cleared the sound loading bit.
Since we now load segments instead of the entire sound, the loading bit is set in each segment node (in the segment tree).
When we hit the case where the PreviewJob was finishing, we cleared the wrong bit and bailed.

From the sequencer_draw.cc code's point of view, the sound segment was still tagged as loading and was never resubmitted.

I changed the code to get rid of the locking and updated the function that clears the sound segment bit.

I added the sleep(10) call to the code after I made these changes and now waveform loading works properly when we hit the case where the PreviewJob is finishing.

@iss if you get a chance, can you re-do your tests using the set up you have there?

I'll try to create my own test file with many strips to see how it goes.
I'll also run another round of tests on my windows PC to see if I find anything else :)

I was able to replicate the problem on my machine by putting a really long sleep in the `preview_startjob` function in `sequencer_preview`: ```diff /** \file * \ingroup spseq */ +#include <unistd.h> #include "DNA_sequence_types.h" #include "DNA_sound_types.h" @@ -173,6 +174,7 @@ static void preview_startjob(void *data, bool *stop, bool *do_update, float *pro BLI_mutex_unlock(pj->mutex); } + sleep(10); // sleep for 10 seconds before shutting down the preview job BLI_task_pool_work_and_wait(task_pool); BLI_task_pool_free(task_pool); } ``` ** Why did the deadlock happen? ** When a segment was submitted to the PreviewJob, the sound's spin lock was acquired. If the PreviewJob was terminating and a new segment was submitted, we cleared the **sound**'s loading bit. The function that cleared the bit acquired the same spin lock and the code deadlocked. ** Why didn't the waveforms load after removing the locks? ** The problem was in the function that cleared the sound loading bit. Since we now load segments instead of the entire sound, the loading bit is set in *each segment node* (in the segment tree). When we hit the case where the PreviewJob was finishing, we cleared the wrong bit and bailed. From the sequencer_draw.cc code's point of view, the sound segment was still tagged as loading and was never resubmitted. ---- I changed the code to get rid of the locking and updated the function that clears the sound segment bit. I added the `sleep(10)` call to the code after I made these changes and now waveform loading works properly when we hit the case where the PreviewJob is finishing. ---- @iss if you get a chance, can you re-do your tests using the set up you have there? I'll try to create my own test file with many strips to see how it goes. I'll also run another round of tests on my windows PC to see if I find anything else :)

Richard Antalik commented

2023-08-31 01:49:18 +02:00

I have tested latest patch and it works. It does load waveforms a bit faster, but it seems, that it loads whole strip anyway. So I am confused, why it does load things faster, when doing more work?

To load only visible part seems to be simple mistake, but some strips aren't drawn correctly. It is even faster though (up to 8x with some files).

diff --git a/source/blender/editors/space_sequencer/sequencer_draw.cc b/source/blender/editors/space_sequencer/sequencer_draw.cc
index e977a9b056b..80f3a6a6f51 100644
--- a/source/blender/editors/space_sequencer/sequencer_draw.cc
+++ b/source/blender/editors/space_sequencer/sequencer_draw.cc
@@ -487,9 +487,9 @@ static void draw_seq_waveform_overlay(
     return; /* Not much to draw, exit before running job. */
   }

-  const float strip_length = x2 - x1_aligned;
-  const float visible_range_start = (frame_start - x1_aligned) / strip_length;
-  const float visible_range_end = (frame_end - x1_aligned) / strip_length;
+  const float strip_content_length = SEQ_time_strip_length_get(scene, seq);
+  const float visible_range_start = (frame_start - x1_aligned) / strip_content_length;
+  const float visible_range_end = (frame_end - x1_aligned) / strip_content_length;

   waveform_job_start_if_needed(C, seq, visible_range_start, visible_range_end);

One thing that I observed is, that with this patch I get noticably longer delay when opening a .blend file. Will have to double check whether this is result of this patch or somehow behavior is different with buildbot. If the delay is caused by the patch, then waveform drawing wouldn't be that much better.

I have tested latest patch and it works. It does load waveforms a bit faster, but it seems, that it loads whole strip anyway. So I am confused, why it does load things faster, when doing more work? To load only visible part seems to be simple mistake, but some strips aren't drawn correctly. It is even faster though (up to 8x with some files). ``` diff --git a/source/blender/editors/space_sequencer/sequencer_draw.cc b/source/blender/editors/space_sequencer/sequencer_draw.cc index e977a9b056b..80f3a6a6f51 100644 --- a/source/blender/editors/space_sequencer/sequencer_draw.cc +++ b/source/blender/editors/space_sequencer/sequencer_draw.cc @@ -487,9 +487,9 @@ static void draw_seq_waveform_overlay( return; /* Not much to draw, exit before running job. */ } - const float strip_length = x2 - x1_aligned; - const float visible_range_start = (frame_start - x1_aligned) / strip_length; - const float visible_range_end = (frame_end - x1_aligned) / strip_length; + const float strip_content_length = SEQ_time_strip_length_get(scene, seq); + const float visible_range_start = (frame_start - x1_aligned) / strip_content_length; + const float visible_range_end = (frame_end - x1_aligned) / strip_content_length; waveform_job_start_if_needed(C, seq, visible_range_start, visible_range_end); ``` One thing that I observed is, that with this patch I get noticably longer delay when opening a .blend file. Will have to double check whether this is result of this patch or somehow behavior is different with buildbot. If the delay is caused by the patch, then waveform drawing wouldn't be that much better.

Lucas Tadeu commented

2023-08-31 07:23:58 +02:00

It does load waveforms a bit faster, but it seems, that it loads whole strip anyway

Are the strips completely visible when you enable waveform drawing?
If they are, the code loads all of the data since that's what it needs to do the drawing.
One of the "future work" points I added to the report is that we can make the loading adapt to the zoom levels.
If you're totally zoomed out and seeing the whole strip, we could sample the audio data instead of reading all of it. You're not gonna have much resolution around the audio anyways.

why it does load things faster, when doing more work?

Each segment is pushed individually to the PreviewJob queue.
They get loaded in parallel and that's why you see better performance there.

To load only visible part seems to be simple mistake, but some strips aren't drawn correctly

@iss can you send me a screenshot to see what's up with the loading?
Not sure what's going wrong there with the strip.

const float strip_content_length = SEQ_time_strip_length_get(scene, seq);

This doesn't look right though.

SEQ_time_strip_length_get returns the number of frames. So the unit here is frames
(frame_start - x1_aligned) and (frame_end - x1_aligned) are in pixels.

We need to convert the result of SEQ_time_strip_length_get from frames to pixels for the division to make sense.

Isn't the original strip_length variable the size in terms of pixels that specific strip would have?

> It does load waveforms a bit faster, but it seems, that it loads whole strip anyway Are the strips *completely* visible when you enable waveform drawing? If they are, the code loads all of the data since that's what it needs to do the drawing. One of the "future work" points I added to the report is that we can make the loading adapt to the zoom levels. If you're totally zoomed out and seeing the whole strip, we could sample the audio data instead of reading all of it. You're not gonna have much resolution around the audio anyways. > why it does load things faster, when doing more work? Each segment is pushed individually to the PreviewJob queue. They get loaded in parallel and that's why you see better performance there. > To load only visible part seems to be simple mistake, but some strips aren't drawn correctly @iss can you send me a screenshot to see what's up with the loading? Not sure what's going wrong there with the strip. > `const float strip_content_length = SEQ_time_strip_length_get(scene, seq);` This doesn't look right though. `SEQ_time_strip_length_get` returns the number of frames. So the unit here is *frames* `(frame_start - x1_aligned)` and `(frame_end - x1_aligned) ` are in *pixels*. We need to convert the result of `SEQ_time_strip_length_get` from frames to pixels for the division to make sense. Isn't the original `strip_length` variable the size in terms of pixels that specific strip would have?

Richard Antalik commented

2023-09-01 02:22:08 +02:00

It does load waveforms a bit faster, but it seems, that it loads whole strip anyway

Are the strips completely visible when you enable waveform drawing?
If they are, the code loads all of the data since that's what it needs to do the drawing.
One of the "future work" points I added to the report is that we can make the loading adapt to the zoom levels.
If you're totally zoomed out and seeing the whole strip, we could sample the audio data instead of reading all of it. You're not gonna have much resolution around the audio anyways.

Strips are completely visible, but they are cut up, so only smaller part of content is visible.

I wouldn't implement adaptive resolution, because this increases code complexity, when I don't think it is necessary. In addition it may cause UI to feel bit busy seeing waveforms constantly redrawing. This patch already feels very fast, so I don't even think it's necessary.

why it does load things faster, when doing more work?

Each segment is pushed individually to the PreviewJob queue.
They get loaded in parallel and that's why you see better performance there.

Sure, but if it loads waveform of entire content, it is same amount of data, that has to be processed, so now it is segmented in additions, which I would expect to be same at best and bit slower at worst. Could be some very low level stuff I don't understand (hardware caches / prefetching).

To load only visible part seems to be simple mistake, but some strips aren't drawn correctly

@iss can you send me a screenshot to see what's up with the loading?
Not sure what's going wrong there with the strip.

const float strip_content_length = SEQ_time_strip_length_get(scene, seq);

This doesn't look right though.

SEQ_time_strip_length_get returns the number of frames. So the unit here is frames
(frame_start - x1_aligned) and (frame_end - x1_aligned) are in pixels.

frame_start, frame_end and x1_aligned are in timeline frames. So you want length factor, but in strip time domain.

  const float strip_content_length = SEQ_time_strip_length_get(scene, seq);
  const float visible_range_start = (frame_start - SEQ_time_start_frame_get(seq)) /
                                    strip_content_length;
  const float visible_range_end = (frame_end - SEQ_time_start_frame_get(seq)) /
                                  strip_content_length;

Then it draws waveforms correctly, but it still seems to load waveform for whole content - if I resize the strip it doesn't run job again and shows waveform.

From memory, the purpose of x1_aligned is to align first sample that should be drawn with pixels, so when you pan timeline, it doesnt pick random start samples and doesn't feel glitchy. Not great naming there...

I need to profile BKE_sound_init_waveform next time and have a closer look at the file opening delay I have mentioned. That is something I wouldn't like to introduce.

> > It does load waveforms a bit faster, but it seems, that it loads whole strip anyway > > Are the strips *completely* visible when you enable waveform drawing? > If they are, the code loads all of the data since that's what it needs to do the drawing. > One of the "future work" points I added to the report is that we can make the loading adapt to the zoom levels. > If you're totally zoomed out and seeing the whole strip, we could sample the audio data instead of reading all of it. You're not gonna have much resolution around the audio anyways. Strips are completely visible, but they are cut up, so only smaller part of content is visible. I wouldn't implement adaptive resolution, because this increases code complexity, when I don't think it is necessary. In addition it may cause UI to feel bit busy seeing waveforms constantly redrawing. This patch already feels very fast, so I don't even think it's necessary. > > why it does load things faster, when doing more work? > > Each segment is pushed individually to the PreviewJob queue. > They get loaded in parallel and that's why you see better performance there. Sure, but if it loads waveform of entire content, it is same amount of data, that has to be processed, so now it is segmented in additions, which I would expect to be same at best and bit slower at worst. Could be some very low level stuff I don't understand (hardware caches / prefetching). > > To load only visible part seems to be simple mistake, but some strips aren't drawn correctly > > @iss can you send me a screenshot to see what's up with the loading? > Not sure what's going wrong there with the strip. > > > `const float strip_content_length = SEQ_time_strip_length_get(scene, seq);` > > This doesn't look right though. > > `SEQ_time_strip_length_get` returns the number of frames. So the unit here is *frames* > `(frame_start - x1_aligned)` and `(frame_end - x1_aligned) ` are in *pixels*. `frame_start`, `frame_end ` and `x1_aligned` are in timeline frames. So you want length factor, but in strip time domain. ``` const float strip_content_length = SEQ_time_strip_length_get(scene, seq); const float visible_range_start = (frame_start - SEQ_time_start_frame_get(seq)) / strip_content_length; const float visible_range_end = (frame_end - SEQ_time_start_frame_get(seq)) / strip_content_length; ``` Then it draws waveforms correctly, but it still seems to load waveform for whole content - if I resize the strip it doesn't run job again and shows waveform. From memory, the purpose of `x1_aligned` is to align first sample that should be drawn with pixels, so when you pan timeline, it doesnt pick random start samples and doesn't feel glitchy. Not great naming there... I need to profile `BKE_sound_init_waveform` next time and have a closer look at the file opening delay I have mentioned. That is something I wouldn't like to introduce.

Richard Antalik commented

2023-09-05 04:28:37 +02:00

I did look into BKE_sound_init_waveform() impact on performance. To do this with higher accuraccy, I have forced initialization for all strips in a loop.

diff --git a/source/blender/editors/space_sequencer/sequencer_draw.cc b/source/blender/editors/space_sequencer/sequencer_draw.cc
index e977a9b056b..8513a630edf 100644
--- a/source/blender/editors/space_sequencer/sequencer_draw.cc
+++ b/source/blender/editors/space_sequencer/sequencer_draw.cc
@@ -2393,6 +2393,8 @@ static void draw_seq_timeline_channels(View2D *v2d)
   immUnbindProgram();
 }

+#include <PIL_time.h>
+
 static void draw_seq_strips(const bContext *C, Editing *ed, ARegion *region)
 {
   Scene *scene = CTX_data_scene(C);
@@ -2402,6 +2404,15 @@ static void draw_seq_strips(const bContext *C, Editing *ed, ARegion *region)
   int sel = 0, j;
   float pixelx = BLI_rctf_size_x(&v2d->cur) / BLI_rcti_size_x(&v2d->mask);

+  double start = PIL_check_seconds_timer();
+  LISTBASE_FOREACH (Sequence *, seq, ed->seqbasep) {
+    bSound *sound = seq->sound;
+    if (sound && !sound->waveform) {
+      BKE_sound_init_waveform(CTX_data_main(C), sound);
+    }
+  }
+  printf("%f\n", PIL_check_seconds_timer() - start);
+
   /* Loop through twice, first unselected, then selected. */
   for (j = 0; j < 2; j++) {
     /* Loop through strips, checking for those that are visible. */

I have tested this with 2 files:

File with 122 shorter(< 3min) sound strips took 0.70s
File with with 73 sound strips, but total duration of 6h took 1.35s

So there is rather large delay when .blend file is opened, that effectively negates effects of this patch. Also I would rather loading to be slower than lock UI for such significant amount of time. That said, I don't see reason, why this initialization has to be done in main thread. DEFAULT_MAX_SEGMENT_SIZE seems to be pretty conservative already, so don't think changing this value would help a lot.

I did look into `BKE_sound_init_waveform()` impact on performance. To do this with higher accuraccy, I have forced initialization for all strips in a loop. ``` diff --git a/source/blender/editors/space_sequencer/sequencer_draw.cc b/source/blender/editors/space_sequencer/sequencer_draw.cc index e977a9b056b..8513a630edf 100644 --- a/source/blender/editors/space_sequencer/sequencer_draw.cc +++ b/source/blender/editors/space_sequencer/sequencer_draw.cc @@ -2393,6 +2393,8 @@ static void draw_seq_timeline_channels(View2D *v2d) immUnbindProgram(); } +#include <PIL_time.h> + static void draw_seq_strips(const bContext *C, Editing *ed, ARegion *region) { Scene *scene = CTX_data_scene(C); @@ -2402,6 +2404,15 @@ static void draw_seq_strips(const bContext *C, Editing *ed, ARegion *region) int sel = 0, j; float pixelx = BLI_rctf_size_x(&v2d->cur) / BLI_rcti_size_x(&v2d->mask); + double start = PIL_check_seconds_timer(); + LISTBASE_FOREACH (Sequence *, seq, ed->seqbasep) { + bSound *sound = seq->sound; + if (sound && !sound->waveform) { + BKE_sound_init_waveform(CTX_data_main(C), sound); + } + } + printf("%f\n", PIL_check_seconds_timer() - start); + /* Loop through twice, first unselected, then selected. */ for (j = 0; j < 2; j++) { /* Loop through strips, checking for those that are visible. */ ``` I have tested this with 2 files: - File with 122 shorter(< 3min) sound strips took 0.70s - File with with 73 sound strips, but total duration of 6h took 1.35s So there is rather large delay when .blend file is opened, that effectively negates effects of this patch. Also I would rather loading to be slower than lock UI for such significant amount of time. That said, I don't see reason, why this initialization has to be done in main thread. `DEFAULT_MAX_SEGMENT_SIZE` seems to be pretty conservative already, so don't think changing this value would help a lot.

Lucas Tadeu commented

2023-09-07 18:02:06 +02:00

@iss sound good, I'll move the initialisation to a background thread.

It does make sense to have different segments be initialised in the background

@iss sound good, I'll move the initialisation to a background thread. It does make sense to have different segments be initialised in the background

This pull request has changes conflicting with the target branch.

extern/audaspace/bindings/C/AUD_Special.cpp
source/blender/blenkernel/intern/sound.cc
source/blender/editors/space_sequencer/sequencer_draw.cc
source/blender/editors/space_sequencer/sequencer_intern.h
source/blender/editors/space_sequencer/sequencer_preview.cc

View command line instructions.

Checkout

From your project repository, check out a new branch and test the changes.

git fetch -u Yup_Lucas/vse-partial-waveform-loads:Yup_Lucas-Yup_Lucas/vse-partial-waveform-loads

git checkout Yup_Lucas-Yup_Lucas/vse-partial-waveform-loads

Sign in to join this conversation.

No reviewers

No Label

Download

What's New

Blender Studio

Manual

Developers Blog

Documentation

Benchmark

Blender Conference

Development Fund

One-time Donations

VSE: Implement partial waveform data loading #110837

How the implementation works

Example: Display a segment with only partial data loaded

Notes

Demo

Checkout