Problems working with extreamly long image sequences (30k+ frames) #117817

Open
opened 2024-02-04 23:12:03 +01:00 by Thomas Wilshaw · 11 comments
Contributor

System Information
Operating system: Windows-10-10.0.19045-SP0 64 Bits
Graphics card: NVIDIA GeForce GTX 1650 Ti/PCIe/SSE2 NVIDIA Corporation 4.6.0 NVIDIA 546.26

Blender Version
Broken: version: 4.0.2, branch: blender-v4.0-release, commit date: 2023-12-05 07:41, hash: 9be62e85b727
Worked: (newest version of Blender that worked as expected)

I've created this issue here as I couldn't find a better palce to put it, it's not a specific bug but a series of related issues. If there is a better place to discusses this plese do point me to it.

I'm working on a feature film that was shot on 16mm film and has a total raw footage count of about 750k frames split across about 40 sequences due to the way it was scanned. This means the sequences range from an average of 30k frames to a maximum of ~60k frames. There's various bits of processing we're doing in Blender before the edit and grade (stabilising the scans and compressing the frames to a manageable size).

I'm having three main issues at the moment that I think are all related to how Blender handles large sequences internally.

  1. When pointing Blenders file browser to one of these sequences it can take the file browser as 10 minutes to show any files in the folder. For comparison Windows' file browser manages to partialy load the files in list view in a few seconds (running dir in the console can print out 36k frames in about 3 seconds).

  2. After selecting the sequence and clicking "Open Clip" it then takes Blender several minutes to actually do that and Blender reports itself as not responding for that time

  3. When I then change the colour space of the sequence Blender also hangs for several minutes. I would have thought this was a simgle tag somehwere but is it in fact going through and tagging every frame individually?

All of these seem to scale fairly linearly with the sequence size but I've not confirmed that with exact timings. I appreciate none of these are exactly bugs but do make working with large sequences rather difficult. Also when rendering 30k odd frames out I've had a suprisingly large number of corrupt frames, this may be an issue with the specific format or again some kind of problem with dealing with this many frames in a single sequence. I guess during development and testing none of these areas are tried with such a large sequence size (although I could be wrong).

As I said if there is a better palce to discuss this please to point me to it.

Thanks

**System Information** Operating system: Windows-10-10.0.19045-SP0 64 Bits Graphics card: NVIDIA GeForce GTX 1650 Ti/PCIe/SSE2 NVIDIA Corporation 4.6.0 NVIDIA 546.26 **Blender Version** Broken: version: 4.0.2, branch: blender-v4.0-release, commit date: 2023-12-05 07:41, hash: `9be62e85b727` Worked: (newest version of Blender that worked as expected) I've created this issue here as I couldn't find a better palce to put it, it's not a specific bug but a series of related issues. If there is a better place to discusses this plese do point me to it. I'm working on a feature film that was shot on 16mm film and has a total raw footage count of about 750k frames split across about 40 sequences due to the way it was scanned. This means the sequences range from an average of 30k frames to a maximum of ~60k frames. There's various bits of processing we're doing in Blender before the edit and grade (stabilising the scans and compressing the frames to a manageable size). I'm having three main issues at the moment that I think are all related to how Blender handles large sequences internally. 1. When pointing Blenders file browser to one of these sequences it can take the file browser as 10 minutes to show any files in the folder. For comparison Windows' file browser manages to partialy load the files in list view in a few seconds (running `dir` in the console can print out 36k frames in about 3 seconds). 2. After selecting the sequence and clicking "Open Clip" it then takes Blender several minutes to actually do that and Blender reports itself as not responding for that time 3. When I then change the colour space of the sequence Blender also hangs for several minutes. I would have thought this was a simgle tag somehwere but is it in fact going through and tagging every frame individually? All of these seem to scale fairly linearly with the sequence size but I've not confirmed that with exact timings. I appreciate none of these are exactly bugs but do make working with large sequences rather difficult. Also when rendering 30k odd frames out I've had a suprisingly large number of corrupt frames, this may be an issue with the specific format or again some kind of problem with dealing with this many frames in a single sequence. I guess during development and testing none of these areas are tried with such a large sequence size (although I could be wrong). As I said if there is a better palce to discuss this please to point me to it. Thanks
Thomas Wilshaw added the
Type
Report
Priority
Normal
Status
Needs Triage
labels 2024-02-04 23:12:04 +01:00
Member

When pointing Blenders file browser to one of these sequences it can take the file browser as 10 minutes to show any files in the folder

Hi, thanks for the report. Possibly due to the files on network drive (if you have any)? or could it be due to large number of files in particular folder and sorting them takes a while. Does this happen when file browser window is opened or when you open some specific folder in the file browser?

After selecting the sequence and clicking "Open Clip" it then takes Blender several minutes to actually do that and Blender reports itself as not responding for that time

I don't really have that large image sequence to confirm the problem (guess I need to create it in blender: default cube render of 30k frames)

> When pointing Blenders file browser to one of these sequences it can take the file browser as 10 minutes to show any files in the folder Hi, thanks for the report. Possibly due to the files on network drive (if you have any)? or could it be due to large number of files in particular folder and sorting them takes a while. Does this happen when file browser window is opened or when you open some specific folder in the file browser? > After selecting the sequence and clicking "Open Clip" it then takes Blender several minutes to actually do that and Blender reports itself as not responding for that time I don't really have that large image sequence to confirm the problem (guess I need to create it in blender: default cube render of 30k frames)
Author
Contributor

Possibly due to the files on network drive (if you have any)? or could it be due to large number of files in particular folder and sorting them takes a while. Does this happen when file browser window is opened or when you open some specific folder in the file browser?

The files are stored on exteranl USB3 drives so it's not a network issue. They're not the fastest drives in the world so that may be part of the issue but I think given the terminal can list all the ifles in a few seconds it still points to a not perfect loading algorithm. Delay one only happens when I navigate to the specific folder within the file browser.

I don't really have that large image sequence to confirm the problem (guess I need to create it in blender: default cube render of 30k frames)

Haha yes I appreciate this might be a bit of an edge case. The files I'm working with are ~3K TIFFs with no compression if that helps. If there's any tests you want me to do or data you would like from me I'll be happy to provide what I can although I'll have to get permission to privately share any of the files. If I had more time for this I'd do some debugging myself.

Cheers

> Possibly due to the files on network drive (if you have any)? or could it be due to large number of files in particular folder and sorting them takes a while. Does this happen when file browser window is opened or when you open some specific folder in the file browser? The files are stored on exteranl USB3 drives so it's not a network issue. They're not the fastest drives in the world so that may be part of the issue but I think given the terminal can list all the ifles in a few seconds it still points to a not perfect loading algorithm. Delay one only happens when I navigate to the specific folder within the file browser. > I don't really have that large image sequence to confirm the problem (guess I need to create it in blender: default cube render of 30k frames) Haha yes I appreciate this might be a bit of an edge case. The files I'm working with are ~3K TIFFs with no compression if that helps. If there's any tests you want me to do or data you would like from me I'll be happy to provide what I can although I'll have to get permission to privately share any of the files. If I had more time for this I'd do some debugging myself. Cheers
Author
Contributor

I've just been working on a sequence and timed things properly:

47222 frames
display in file viewer: 6min 55s
load clip: 3min 40s
change colour space: 11min 5s

I can sort of understand step two taking as long as it does but step one and 3 (especially 3) seem way too long.

I should also add that for all three steps my CPU is working fairly hard (~50% utilisation)

I've just been working on a sequence and timed things properly: ``` 47222 frames display in file viewer: 6min 55s load clip: 3min 40s change colour space: 11min 5s ```` I can sort of understand step two taking as long as it does but step one and 3 (especially 3) seem way too long. I should also add that for all three steps my CPU is working fairly hard (~50% utilisation)
Author
Contributor

I timed one of the smallest sequences I have and got the follwoing results:

5643 frmaes
display in fiel viewer: 9s
load clip: 13s
change colour space: 13s

If I were to linearly scale those these times up to 47222 frames I should get the follwoing times:

display in file viewer: 1min 15
load clip: 1m 49s
change colour space: 1m 49s

So there's clearly some very none linear time scaling going on

I timed one of the smallest sequences I have and got the follwoing results: ``` 5643 frmaes display in fiel viewer: 9s load clip: 13s change colour space: 13s ``` If I were to linearly scale those these times up to 47222 frames I should get the follwoing times: ``` display in file viewer: 1min 15 load clip: 1m 49s change colour space: 1m 49s ``` So there's clearly some very none linear time scaling going on
Author
Contributor

I've done a bit more testing and plotted the results. The cell in red is, I think, an anomaly as I'd already opened that file the day before so it might have cached something somewhere.

image

I've done a bit more testing and plotted the results. The cell in red is, I think, an anomaly as I'd already opened that file the day before so it might have cached something somewhere. ![image](/attachments/c0b82693-c23a-4ca5-b9cf-dd2a3c1934b3)
Author
Contributor

I've updated the chart with some more data points and it's clear the change in CS at least is O(n^2)

image

I've updated the chart with some more data points and it's clear the change in CS at least is O(n^2) ![image](/attachments/6a2f5bda-37c5-4b75-a74d-9bf08f9919b5)
100 KiB
Iliya Katushenock added the
Interest
Images & Movies
label 2024-02-05 16:14:57 +01:00

Thanks for report, just to make sure, this is the bug report tracker. And performance is not a bug case, in general.

When pointing Blenders file browser to one of these sequences it can take the file browser as 10 minutes to show any files in the folder. For comparison Windows' file browser manages to partialy load the files in list view in a few seconds (running dir in the console can print out 36k frames in about 3 seconds).

UI window of file browser can be improved a lot to be faster. I am hdd user and some time browser might be pretty low.

After selecting the sequence and clicking "Open Clip" it then takes Blender several minutes to actually do that and Blender reports itself as not responding for that time

UI handling of operators might be improved to handle them.

When I then change the colour space of the sequence Blender also hangs for several minutes. I would have thought this was a simgle tag somehwere but is it in fact going through and tagging every frame individually?

Well... maybe.

It is much better to split all of this topic in separate... hmm, not the reports, that is not a bugs..
@Harley Hay, do you think first 2 topics can be handled as designs for UI team, or move them in RCS?

Thanks for report, just to make sure, this is the bug report tracker. And performance is not a bug case, in general. > When pointing Blenders file browser to one of these sequences it can take the file browser as 10 minutes to show any files in the folder. For comparison Windows' file browser manages to partialy load the files in list view in a few seconds (running dir in the console can print out 36k frames in about 3 seconds). UI window of file browser can be improved a lot to be faster. I am hdd user and some time browser might be pretty low. > After selecting the sequence and clicking "Open Clip" it then takes Blender several minutes to actually do that and Blender reports itself as not responding for that time UI handling of operators might be improved to handle them. > When I then change the colour space of the sequence Blender also hangs for several minutes. I would have thought this was a simgle tag somehwere but is it in fact going through and tagging every frame individually? Well... maybe. It is much better to split all of this topic in separate... hmm, not the reports, that is not a bugs.. @Harley Hay, do you think first 2 topics can be handled as designs for UI team, or move them in RCS?
Member

@TMW

First, I should say that I am not doubting that you are seeing unacceptable performance. And I am hoping we can improve things. So don't take any of the of the following questions as being defensive. It is just narrowing down the issues.

but I think given the terminal can list all the files in a few seconds

This isn't a good thing to compare with. If you were to put your files on the worst network connection imaginable, over a dial-up phone line, that listing would still be quite quick. In a nutshell you are asking the operating system to give you a list of what's in the folder, it compiles a listing and just sends you that text. It doesn't involve any transmission of file contents. We necessarily have to do more than just get a listing since we need file attributes, sizes, dates, etc for each file.

So I am hoping you can do couple tests for me.

First, just make a folder on your computer's local hard drive. Then copy the contents of your USB to that folder. Note the time it takes to do the copy, but I'm not too worried about it if it takes a long time. Then load up Blender and browse that folder in the way you normally do with the USB. I suspect it will be a little faster, but it could be anything in between the same speed and 10 times faster. But it would be nice to know.

In the mean time, with a job like that needing 30,000 files or more I would personally be looking to do as much as possible from the command line using scripting rather than interactively.

@TMW First, I should say that I am not doubting that you are seeing unacceptable performance. And I am hoping we can improve things. So don't take any of the of the following questions as being defensive. It is just narrowing down the issues. > but I think given the terminal can list all the files in a few seconds This isn't a good thing to compare with. If you were to put your files on the worst network connection imaginable, over a dial-up phone line, that listing would still be quite quick. In a nutshell you are asking the operating system to give you a list of what's in the folder, it compiles a listing and just sends you that text. It doesn't involve any transmission of file contents. We necessarily have to do more than just get a listing since we need file attributes, sizes, dates, etc for each file. So I am hoping you can do couple tests for me. First, just make a folder on your computer's **local** hard drive. Then copy the contents of your USB to that folder. Note the time it takes to do the copy, but I'm not too worried about it if it takes a long time. Then load up Blender and browse that folder in the way you normally do with the USB. I suspect it will be a little faster, but it could be anything in between the same speed and 10 times faster. But it would be nice to know. In the mean time, with a job like that needing 30,000 files or more I would personally be looking to do as much as possible from the command line using scripting rather than interactively.
Author
Contributor

First, I should say that I am not doubting that you are seeing unacceptable performance. And I am hoping we can improve things. So don't take any of the of the following questions as being defensive. It is just narrowing down the issues.

Not at all and thanks for following up.

This isn't a good thing to compare with.

I appreciate it's not apples for apples but dir on Windows prints out file name, file date and file size, the same as Blenders file viewer, Loading thumbnails is obviously very different but for list view (the default I think?) it should be pretty similar? I'm quite probably missing something though, I usually am :)

So I am hoping you can do couple tests for me.

I'll try and get that done this evening but I'll need to check if I have enough internal HDD space for a folder that large. I may have to do one of the smaller ones.

In the mean time, with a job like that needing 30,000 files or more I would personally be looking to do as much as possible from the command line using scripting rather than interactively.

Yes you're right, I had naively thought that given I just needed to tweak a couple of settings and then render out it would be quicker to do that then go into a scripting wormhole (I'm not a Blender power users by any stretch of the imagination).

> First, I should say that I am not doubting that you are seeing unacceptable performance. And I am hoping we can improve things. So don't take any of the of the following questions as being defensive. It is just narrowing down the issues. Not at all and thanks for following up. > This isn't a good thing to compare with. I appreciate it's not apples for apples but `dir` on Windows prints out file name, file date and file size, the same as Blenders file viewer, Loading thumbnails is obviously very different but for list view (the default I think?) it should be pretty similar? I'm quite probably missing something though, I usually am :) > So I am hoping you can do couple tests for me. I'll try and get that done this evening but I'll need to check if I have enough internal HDD space for a folder that large. I may have to do one of the smaller ones. > In the mean time, with a job like that needing 30,000 files or more I would personally be looking to do as much as possible from the command line using scripting rather than interactively. Yes you're right, I had naively thought that given I just needed to tweak a couple of settings and then render out it would be quicker to do that then go into a scripting wormhole (I'm not a Blender power users by any stretch of the imagination).
Member

If I understand correctly you are loading the sequences into the Movie Clip Editor.

When I then change the colour space of the sequence Blender also hangs for several minutes. I would have thought this was a simgle tag somehwere but is it in fact going through and tagging every frame individually

Unfortunately it is not, there are caches to be cleared and individual image buffers recreated. I wont comment on the need for this for now (there might actually be ways to skip this when "just" changing colorspace I think), lets just see if we can proceed with this report differently.

Changing the colorspace there (in the Movie Clip Editor) seems to reload the whole sequence twice (due to 2cce65de96)

As a comparison: Do have the same drastic slowdowns if you load the sequence as an Image Strip into the Video Sequence Editor as well? How does changing colorspace behave there?

note to myself: look at this code:

static void rna_ColorManagedColorspaceSettings_reload_update
- DEG_id_tag_update(&ima->id, ID_RECALC_SOURCE);
- BKE_image_signal(bmain, ima, nullptr, IMA_SIGNAL_COLORMANAGE);

@aras_p might also be interested

If I understand correctly you are loading the sequences into the `Movie Clip Editor`. >When I then change the colour space of the sequence Blender also hangs for several minutes. I would have thought this was a simgle tag somehwere but is it in fact going through and tagging every frame individually Unfortunately it is not, there are caches to be cleared and individual image buffers recreated. I wont comment on the need for this for now (there might actually be ways to skip this when "just" changing colorspace I think), lets just see if we can proceed with this report differently. Changing the colorspace there (in the `Movie Clip Editor`) seems to reload the whole sequence **twice** (due to 2cce65de9695943c5189b74d1c3a480d2b72889f) As a comparison: Do have the same drastic slowdowns if you load the sequence as an Image Strip into the Video Sequence Editor as well? How does changing colorspace behave there? note to myself: look at this code: ``` static void rna_ColorManagedColorspaceSettings_reload_update - DEG_id_tag_update(&ima->id, ID_RECALC_SOURCE); - BKE_image_signal(bmain, ima, nullptr, IMA_SIGNAL_COLORMANAGE); ``` @aras_p might also be interested
Author
Contributor

@lichtwerk I'm loading the image sequences into a Movie CLip node in the compositor.

As a comparison: Do have the same drastic slowdowns if you load the sequence as an Image Strip into the Video Sequence Editor as well? How does changing colorspace behave there?

Loading into an image node seems to take a roughly similar amount of time as loading into a Movie Clip Node. However changing the colour space only take about 3 seconds which is a huge improvement.

@lichtwerk I'm loading the image sequences into a Movie CLip node in the compositor. >As a comparison: Do have the same drastic slowdowns if you load the sequence as an Image Strip into the Video Sequence Editor as well? How does changing colorspace behave there? Loading into an image node seems to take a roughly similar amount of time as loading into a Movie Clip Node. However changing the colour space only take about 3 seconds which is a huge improvement.
Sign in to join this conversation.
No Label
Interest
Alembic
Interest
Animation & Rigging
Interest
Asset Browser
Interest
Asset Browser Project Overview
Interest
Audio
Interest
Automated Testing
Interest
Blender Asset Bundle
Interest
BlendFile
Interest
Collada
Interest
Compatibility
Interest
Compositing
Interest
Core
Interest
Cycles
Interest
Dependency Graph
Interest
Development Management
Interest
EEVEE
Interest
EEVEE & Viewport
Interest
Freestyle
Interest
Geometry Nodes
Interest
Grease Pencil
Interest
ID Management
Interest
Images & Movies
Interest
Import Export
Interest
Line Art
Interest
Masking
Interest
Metal
Interest
Modeling
Interest
Modifiers
Interest
Motion Tracking
Interest
Nodes & Physics
Interest
OpenGL
Interest
Overlay
Interest
Overrides
Interest
Performance
Interest
Physics
Interest
Pipeline, Assets & IO
Interest
Platforms, Builds & Tests
Interest
Python API
Interest
Render & Cycles
Interest
Render Pipeline
Interest
Sculpt, Paint & Texture
Interest
Text Editor
Interest
Translations
Interest
Triaging
Interest
Undo
Interest
USD
Interest
User Interface
Interest
UV Editing
Interest
VFX & Video
Interest
Video Sequencer
Interest
Virtual Reality
Interest
Vulkan
Interest
Wayland
Interest
Workbench
Interest: X11
Legacy
Blender 2.8 Project
Legacy
Milestone 1: Basic, Local Asset Browser
Legacy
OpenGL Error
Meta
Good First Issue
Meta
Papercut
Meta
Retrospective
Meta
Security
Module
Animation & Rigging
Module
Core
Module
Development Management
Module
EEVEE & Viewport
Module
Grease Pencil
Module
Modeling
Module
Nodes & Physics
Module
Pipeline, Assets & IO
Module
Platforms, Builds & Tests
Module
Python API
Module
Render & Cycles
Module
Sculpt, Paint & Texture
Module
Triaging
Module
User Interface
Module
VFX & Video
Platform
FreeBSD
Platform
Linux
Platform
macOS
Platform
Windows
Priority
High
Priority
Low
Priority
Normal
Priority
Unbreak Now!
Status
Archived
Status
Confirmed
Status
Duplicate
Status
Needs Info from Developers
Status
Needs Information from User
Status
Needs Triage
Status
Resolved
Type
Bug
Type
Design
Type
Known Issue
Type
Patch
Type
Report
Type
To Do
No Milestone
No project
No Assignees
5 Participants
Notifications
Due Date
The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: blender/blender#117817
No description provided.