Problems working with extreamly long image sequences (30k+ frames) #117817
Labels
No Label
Interest
Alembic
Interest
Animation & Rigging
Interest
Asset Browser
Interest
Asset Browser Project Overview
Interest
Audio
Interest
Automated Testing
Interest
Blender Asset Bundle
Interest
BlendFile
Interest
Collada
Interest
Compatibility
Interest
Compositing
Interest
Core
Interest
Cycles
Interest
Dependency Graph
Interest
Development Management
Interest
EEVEE
Interest
EEVEE & Viewport
Interest
Freestyle
Interest
Geometry Nodes
Interest
Grease Pencil
Interest
ID Management
Interest
Images & Movies
Interest
Import Export
Interest
Line Art
Interest
Masking
Interest
Metal
Interest
Modeling
Interest
Modifiers
Interest
Motion Tracking
Interest
Nodes & Physics
Interest
OpenGL
Interest
Overlay
Interest
Overrides
Interest
Performance
Interest
Physics
Interest
Pipeline, Assets & IO
Interest
Platforms, Builds & Tests
Interest
Python API
Interest
Render & Cycles
Interest
Render Pipeline
Interest
Sculpt, Paint & Texture
Interest
Text Editor
Interest
Translations
Interest
Triaging
Interest
Undo
Interest
USD
Interest
User Interface
Interest
UV Editing
Interest
VFX & Video
Interest
Video Sequencer
Interest
Virtual Reality
Interest
Vulkan
Interest
Wayland
Interest
Workbench
Interest: X11
Legacy
Blender 2.8 Project
Legacy
Milestone 1: Basic, Local Asset Browser
Legacy
OpenGL Error
Meta
Good First Issue
Meta
Papercut
Meta
Retrospective
Meta
Security
Module
Animation & Rigging
Module
Core
Module
Development Management
Module
EEVEE & Viewport
Module
Grease Pencil
Module
Modeling
Module
Nodes & Physics
Module
Pipeline, Assets & IO
Module
Platforms, Builds & Tests
Module
Python API
Module
Render & Cycles
Module
Sculpt, Paint & Texture
Module
Triaging
Module
User Interface
Module
VFX & Video
Platform
FreeBSD
Platform
Linux
Platform
macOS
Platform
Windows
Priority
High
Priority
Low
Priority
Normal
Priority
Unbreak Now!
Status
Archived
Status
Confirmed
Status
Duplicate
Status
Needs Info from Developers
Status
Needs Information from User
Status
Needs Triage
Status
Resolved
Type
Bug
Type
Design
Type
Known Issue
Type
Patch
Type
Report
Type
To Do
No Milestone
No project
No Assignees
5 Participants
Notifications
Due Date
No due date set.
Dependencies
No dependencies set.
Reference: blender/blender#117817
Loading…
Reference in New Issue
No description provided.
Delete Branch "%!s(<nil>)"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
System Information
Operating system: Windows-10-10.0.19045-SP0 64 Bits
Graphics card: NVIDIA GeForce GTX 1650 Ti/PCIe/SSE2 NVIDIA Corporation 4.6.0 NVIDIA 546.26
Blender Version
Broken: version: 4.0.2, branch: blender-v4.0-release, commit date: 2023-12-05 07:41, hash:
9be62e85b727
Worked: (newest version of Blender that worked as expected)
I've created this issue here as I couldn't find a better palce to put it, it's not a specific bug but a series of related issues. If there is a better place to discusses this plese do point me to it.
I'm working on a feature film that was shot on 16mm film and has a total raw footage count of about 750k frames split across about 40 sequences due to the way it was scanned. This means the sequences range from an average of 30k frames to a maximum of ~60k frames. There's various bits of processing we're doing in Blender before the edit and grade (stabilising the scans and compressing the frames to a manageable size).
I'm having three main issues at the moment that I think are all related to how Blender handles large sequences internally.
When pointing Blenders file browser to one of these sequences it can take the file browser as 10 minutes to show any files in the folder. For comparison Windows' file browser manages to partialy load the files in list view in a few seconds (running
dir
in the console can print out 36k frames in about 3 seconds).After selecting the sequence and clicking "Open Clip" it then takes Blender several minutes to actually do that and Blender reports itself as not responding for that time
When I then change the colour space of the sequence Blender also hangs for several minutes. I would have thought this was a simgle tag somehwere but is it in fact going through and tagging every frame individually?
All of these seem to scale fairly linearly with the sequence size but I've not confirmed that with exact timings. I appreciate none of these are exactly bugs but do make working with large sequences rather difficult. Also when rendering 30k odd frames out I've had a suprisingly large number of corrupt frames, this may be an issue with the specific format or again some kind of problem with dealing with this many frames in a single sequence. I guess during development and testing none of these areas are tried with such a large sequence size (although I could be wrong).
As I said if there is a better palce to discuss this please to point me to it.
Thanks
Hi, thanks for the report. Possibly due to the files on network drive (if you have any)? or could it be due to large number of files in particular folder and sorting them takes a while. Does this happen when file browser window is opened or when you open some specific folder in the file browser?
I don't really have that large image sequence to confirm the problem (guess I need to create it in blender: default cube render of 30k frames)
The files are stored on exteranl USB3 drives so it's not a network issue. They're not the fastest drives in the world so that may be part of the issue but I think given the terminal can list all the ifles in a few seconds it still points to a not perfect loading algorithm. Delay one only happens when I navigate to the specific folder within the file browser.
Haha yes I appreciate this might be a bit of an edge case. The files I'm working with are ~3K TIFFs with no compression if that helps. If there's any tests you want me to do or data you would like from me I'll be happy to provide what I can although I'll have to get permission to privately share any of the files. If I had more time for this I'd do some debugging myself.
Cheers
I've just been working on a sequence and timed things properly:
I can sort of understand step two taking as long as it does but step one and 3 (especially 3) seem way too long.
I should also add that for all three steps my CPU is working fairly hard (~50% utilisation)
I timed one of the smallest sequences I have and got the follwoing results:
If I were to linearly scale those these times up to 47222 frames I should get the follwoing times:
So there's clearly some very none linear time scaling going on
I've done a bit more testing and plotted the results. The cell in red is, I think, an anomaly as I'd already opened that file the day before so it might have cached something somewhere.
I've updated the chart with some more data points and it's clear the change in CS at least is O(n^2)
Thanks for report, just to make sure, this is the bug report tracker. And performance is not a bug case, in general.
UI window of file browser can be improved a lot to be faster. I am hdd user and some time browser might be pretty low.
UI handling of operators might be improved to handle them.
Well... maybe.
It is much better to split all of this topic in separate... hmm, not the reports, that is not a bugs..
@Harley Hay, do you think first 2 topics can be handled as designs for UI team, or move them in RCS?
@TMW
First, I should say that I am not doubting that you are seeing unacceptable performance. And I am hoping we can improve things. So don't take any of the of the following questions as being defensive. It is just narrowing down the issues.
This isn't a good thing to compare with. If you were to put your files on the worst network connection imaginable, over a dial-up phone line, that listing would still be quite quick. In a nutshell you are asking the operating system to give you a list of what's in the folder, it compiles a listing and just sends you that text. It doesn't involve any transmission of file contents. We necessarily have to do more than just get a listing since we need file attributes, sizes, dates, etc for each file.
So I am hoping you can do couple tests for me.
First, just make a folder on your computer's local hard drive. Then copy the contents of your USB to that folder. Note the time it takes to do the copy, but I'm not too worried about it if it takes a long time. Then load up Blender and browse that folder in the way you normally do with the USB. I suspect it will be a little faster, but it could be anything in between the same speed and 10 times faster. But it would be nice to know.
In the mean time, with a job like that needing 30,000 files or more I would personally be looking to do as much as possible from the command line using scripting rather than interactively.
Not at all and thanks for following up.
I appreciate it's not apples for apples but
dir
on Windows prints out file name, file date and file size, the same as Blenders file viewer, Loading thumbnails is obviously very different but for list view (the default I think?) it should be pretty similar? I'm quite probably missing something though, I usually am :)I'll try and get that done this evening but I'll need to check if I have enough internal HDD space for a folder that large. I may have to do one of the smaller ones.
Yes you're right, I had naively thought that given I just needed to tweak a couple of settings and then render out it would be quicker to do that then go into a scripting wormhole (I'm not a Blender power users by any stretch of the imagination).
If I understand correctly you are loading the sequences into the
Movie Clip Editor
.Unfortunately it is not, there are caches to be cleared and individual image buffers recreated. I wont comment on the need for this for now (there might actually be ways to skip this when "just" changing colorspace I think), lets just see if we can proceed with this report differently.
Changing the colorspace there (in the
Movie Clip Editor
) seems to reload the whole sequence twice (due to2cce65de96
)As a comparison: Do have the same drastic slowdowns if you load the sequence as an Image Strip into the Video Sequence Editor as well? How does changing colorspace behave there?
note to myself: look at this code:
@aras_p might also be interested
@lichtwerk I'm loading the image sequences into a Movie CLip node in the compositor.
Loading into an image node seems to take a roughly similar amount of time as loading into a Movie Clip Node. However changing the colour space only take about 3 seconds which is a huge improvement.