Blender editing performance with many datablocks
#73359
Open
opened
No Branch/Tag Specified
main
blender-v3.6-release
temp-sculpt-dyntopo-hive-alloc
temp-sculpt-dyntopo
asset-shelf
cycles-light-linking
tmp-usd-python-mtl
brush-assets-project
blender-v2.93-release
blender-v3.3-release
universal-scene-description
node-group-operators
asset-browser-frontend-split
temp-sculpt-attr-api
blender-v3.5-release
realtime-clock
sculpt-dev
gpencil-next
bevelv2
microfacet_hair
blender-projects-basics
principled-v2
v3.3.7
v2.93.18
v3.5.1
v3.3.6
v2.93.17
v3.5.0
v2.93.16
v3.3.5
v3.3.4
v2.93.15
v2.93.14
v3.3.3
v2.93.13
v2.93.12
v3.4.1
v3.3.2
v3.4.0
v3.3.1
v2.93.11
v3.3.0
v3.2.2
v2.93.10
v3.2.1
v3.2.0
v2.83.20
v2.93.9
v3.1.2
v3.1.1
v3.1.0
v2.83.19
v2.93.8
v3.0.1
v2.93.7
v3.0.0
v2.93.6
v2.93.5
v2.83.18
v2.93.4
v2.93.3
v2.83.17
v2.93.2
v2.93.1
v2.83.16
v2.93.0
v2.83.15
v2.83.14
v2.83.13
v2.92.0
v2.83.12
v2.91.2
v2.83.10
v2.91.0
v2.83.9
v2.83.8
v2.83.7
v2.90.1
v2.83.6.1
v2.83.6
v2.90.0
v2.83.5
v2.83.4
v2.83.3
v2.83.2
v2.83.1
v2.83
v2.82a
v2.82
v2.81a
v2.81
v2.80
v2.80-rc3
v2.80-rc2
v2.80-rc1
v2.79b
v2.79a
v2.79
v2.79-rc2
v2.79-rc1
v2.78c
v2.78b
v2.78a
v2.78
v2.78-rc2
v2.78-rc1
v2.77a
v2.77
v2.77-rc2
v2.77-rc1
v2.76b
v2.76a
v2.76
v2.76-rc3
v2.76-rc2
v2.76-rc1
v2.75a
v2.75
v2.75-rc2
v2.75-rc1
v2.74
v2.74-rc4
v2.74-rc3
v2.74-rc2
v2.74-rc1
v2.73a
v2.73
v2.73-rc1
v2.72b
2.72b
v2.72a
v2.72
v2.72-rc1
v2.71
v2.71-rc2
v2.71-rc1
v2.70a
v2.70
v2.70-rc2
v2.70-rc
v2.69
v2.68a
v2.68
v2.67b
v2.67a
v2.67
v2.66a
v2.66
v2.65a
v2.65
v2.64a
v2.64
v2.63a
v2.63
v2.61
v2.60a
v2.60
v2.59
v2.58a
v2.58
v2.57b
v2.57a
v2.57
v2.56a
v2.56
v2.55
v2.54
v2.53
v2.52
v2.51
v2.50
v2.49b
v2.49a
v2.49
v2.48a
v2.48
v2.47
v2.46
v2.45
v2.44
v2.43
v2.42a
v2.42
v2.41
v2.40
v2.37a
v2.37
v2.36
v2.35a
v2.35
v2.34
v2.33a
v2.33
v2.32
v2.31a
v2.31
v2.30
v2.28c
v2.28a
v2.28
v2.27
v2.26
v2.25
Labels
Clear labels
Issues relating to security: https://wiki.blender.org/wiki/Process/Vulnerability_Reports
Apply labels
Interest
Alembic
Interest
Animation & Rigging
Interest
Asset Browser
Interest
Asset Browser Project Overview
Interest
Audio
Interest
Automated Testing
Interest
Blender Asset Bundle
Interest
BlendFile
Interest
Collada
Interest
Compositing
Interest
Core
Interest
Cycles
Interest
Dependency Graph
Interest
Development Management
Interest
Eevee & Viewport
Interest
Freestyle
Interest
Geometry Nodes
Interest
Grease Pencil
Interest
ID Management
Interest
Images & Movies
Interest/Import
Export
Interest
Line Art
Interest
Masking
Interest
Metal
Interest
Modeling
Interest
Modifiers
Interest
Motion Tracking
Interest
Nodes & Physics
Interest
OpenGL
Interest
Overrides
Interest
Performance
Interest
Physics
Interest
Pipeline, Assets & IO
Interest
Platforms, Builds & Tests
Interest
Python API
Interest
Render & Cycles
Interest
Render Pipeline
Interest
Sculpt, Paint & Texture
Interest
Text Editor
Interest
Translations
Interest
Triaging
Interest
Undo
Interest
USD
Interest
User Interface
Interest
UV Editing
Interest
VFX & Video
Interest
Video Sequencer
Interest
Virtual Reality
Interest
Vulkan
Interest: Wayland
Legacy
Blender 2.8 Project
Legacy
Milestone 1: Basic, Local Asset Browser
Legacy
OpenGL Error
Meta
Good First Issue
Meta
Papercut
Meta
Retrospective
Meta
Security
Issues relating to security: https://wiki.blender.org/wiki/Process/Vulnerability_Reports
Module
Animation & Rigging
Module
Core
Module
Development Management
Module
Eevee & Viewport
Module
Grease Pencil
Module
Modeling
Module
Nodes & Physics
Module
Pipeline, Assets & IO
Module
Platforms, Builds & Tests
Module
Python API
Module
Render & Cycles
Module
Sculpt, Paint & Texture
Module
Triaging
Module
User Interface
Module
VFX & Video
Platform
FreeBSD
Platform
Linux
Platform
macOS
Platform
Windows
Priority
High
Priority
Low
Priority
Normal
Priority
Unbreak Now!
Status
Archived
Status
Confirmed
Status
Duplicate
Status
Needs Info from Developers
Status
Needs Information from User
Status
Needs Triage
Status
Resolved
Type
Bug
Type
Design
Type
Known Issue
Type
Patch
Type
Report
Type
To Do
No Label
Interest
Alembic
Interest
Animation & Rigging
Interest
Asset Browser
Interest
Asset Browser Project Overview
Interest
Audio
Interest
Automated Testing
Interest
Blender Asset Bundle
Interest
BlendFile
Interest
Collada
Interest
Compositing
Interest
Core
Interest
Cycles
Interest
Dependency Graph
Interest
Development Management
Interest
Eevee & Viewport
Interest
Freestyle
Interest
Geometry Nodes
Interest
Grease Pencil
Interest
ID Management
Interest
Images & Movies
Interest/Import
Export
Interest
Line Art
Interest
Masking
Interest
Metal
Interest
Modeling
Interest
Modifiers
Interest
Motion Tracking
Interest
Nodes & Physics
Interest
OpenGL
Interest
Overrides
Interest
Performance
Interest
Physics
Interest
Pipeline, Assets & IO
Interest
Platforms, Builds & Tests
Interest
Python API
Interest
Render & Cycles
Interest
Render Pipeline
Interest
Sculpt, Paint & Texture
Interest
Text Editor
Interest
Translations
Interest
Triaging
Interest
Undo
Interest
USD
Interest
User Interface
Interest
UV Editing
Interest
VFX & Video
Interest
Video Sequencer
Interest
Virtual Reality
Interest
Vulkan
Interest: Wayland
Legacy
Blender 2.8 Project
Legacy
Milestone 1: Basic, Local Asset Browser
Legacy
OpenGL Error
Meta
Good First Issue
Meta
Papercut
Meta
Retrospective
Meta
Security
Module
Animation & Rigging
Module
Core
Module
Development Management
Module
Eevee & Viewport
Module
Grease Pencil
Module
Modeling
Module
Nodes & Physics
Module
Pipeline, Assets & IO
Module
Platforms, Builds & Tests
Module
Python API
Module
Render & Cycles
Module
Sculpt, Paint & Texture
Module
Triaging
Module
User Interface
Module
VFX & Video
Platform
FreeBSD
Platform
Linux
Platform
macOS
Platform
Windows
Priority
High
Priority
Low
Priority
Normal
Priority
Unbreak Now!
Status
Archived
Status
Confirmed
Status
Duplicate
Status
Needs Info from Developers
Status
Needs Information from User
Status
Needs Triage
Status
Resolved
Type
Bug
Type
Design
Type
Known Issue
Type
Patch
Type
Report
Type
To Do
Milestone
Set milestone
Clear milestone
No items
No Milestone
Projects
Set Project
Clear projects
No project
Assignees
Assign users
Clear assignees
No Assignees
42 Participants
Notifications
Due Date
The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.
No due date set.
Dependencies
No dependencies set.
Reference: blender/blender#73359
Reference in New Issue
There is no content yet.
Delete Branch "%!s(<nil>)"
Deleting a branch is permanent. It CANNOT be undone. Continue?
Status: 1st milestone almost complete, need final optimizations.
Team
Commissioner: @brecht
Project leader: @mont29
Project members: @ZedDB
Big picture: In heavy scenes, new features in Blender 2.8x are making the inherently linear (
O(N)
), when not quadratic (O(N²)
), complexity of most operations over data-blocks more problematic than in 2.7x era. The general goal is to bring those operations back to a constant (O(1)
) or logarithmic (O(log(N))
) complexity.Description
Use cases:
Need specific test cases.
Note: There are two types of “heavyness” in scenes (that can be combined of course):
Design:
Dependency graph should not have to re-evaluate objects that did not change.
** This is mainly a problem currently during global undo, as all data-blocks are replaced by new ones on each undo step.
Handling naming of data-blocks should be
O(log(N))
(it is currently close toO(N²)
, orO(N log(N))
in best cases).This will require caching currently used names.* We will probably also have to handle smartly the numbering in that cache, if we really want to be efficient in typical “worst case” scenarii (addition of thousands of objects with same base name e.g.).
*** This approach can usually be easily managed in a mostly lock-less way in a threaded context (proper locking is only needed when actually rebuilding a dirty cache).
Kept valid/in sync all the time (code affecting related data should take care of updating it itself, code using the cache can always assume it is valid and up-to-date).
***Such cache should be easy to incrementally update (it should almost never have to be rebuilt from scratch).*This implies that the cache is highly local (changes on a data-block only ever affect that data-block and a well-defined, easy to reach small number of “neighbors”). In general keeping this kind of cache valid will always be harder and more error-prone than approach #A.
*** This approach often needs proper complete locking (mutexes & co) in a threaded context.
Note: The ViewLayer's collections/objects cache e.g. currently uses the worst mix of #A and #B - it is always assumed valid, but updating it requires a complete rebuild from scratch almost all the time.
Note: Approach #B is usually interesting only if a complete rebuild of the cache is very costly, and/or if the cache is invalidated very often while being used.
Engineer plan:
-
Work plan
Milestone 1 - Optimized per-datablock global undo
Time estimate:
6 months
** #60695 (Optimized per-datablock global undo)
** D6580: WIP/Demonstration patch about undo speedup project.
Milestone 2 - Lazy collection synchronizations
Time estimate:
1 month
** #73411 (Fix ViewLayers cache building)
Milestone 3 - Data-blocks management performances with many of them
Time estimate:
5 months
Investigate how to best handle naming conflicts issues.
Most obvious idea would be store all used base-names in a GHash (one per ID type), along with used indices. Needs further design though. #73412 (Improve name conflict handling in ID management)
Investigate the best way to cache bi-directional relationships info between data-blocks.
We already have a start of this with
BKE_main_relations_create()
& co, but it is heavily under-used and needs better design to be usable more widely. #73413 (Cache data-block relationships info )Fix/Improve handling of the Bone pointers (which are sub-data of
Armature
ID) stored in poses (which are sub-data ofObject
ID).Ideally such horrible pointers should not exist ever. They are a very solid recurrent source of bugs in Blender. Not sure how to best deal with them, at the very least we could add some generic ID API to ensure those kind of caches are up-to-date?
See also #68938 (Blender editing performance with many datablocks).
Later
O(1)
orO(log(N))
(it is alsoO(N)
at the moment). ???? would assume linear would already be nice performance for such a code?Notes: -
Added subscribers: @ZedDB, @mont29, @dfelinto
Added subscriber: @MACHIN3
Added subscriber: @Teds
Added subscriber: @Lumpengnom-3
Added subscriber: @MichaelWeisheim
Added subscriber: @rwman
Edited task, sorted stage 2 sup-steps in order I think is the most relevant, and added sub-tasks for all those three ones.
Please not that this is still very vague, and mostly there to keep track of random ideas at this point. Proper technical design is still required, stage 2 is not likely to happen in immediate coming months anyway.
Added subscriber: @slumber
Added subscriber: @Scaredyfish
Added subscriber: @AditiaA.Pratama
Scene editing in object modeto Blender editing performance with many datablocksAdded subscriber: @dcolli23
Added subscriber: @Pipeliner
Removed subscriber: @Pipeliner
Added subscriber: @Pipeliner
Added subscriber: @MizManFryinPan
Added subscriber: @DuarteRamos
Added subscriber: @ManuelAlbert
Added subscriber: @elven_inside
Changed status from 'Needs Triage' to: 'Confirmed'
Added subscriber: @VladimirKunyansky
Added subscriber: @1D_Inc
Of course, algorithmization is primarily a complex mathematical problem.
(One of my favorite explanations for the big O notations )
Added subscriber: @Moult
Added subscriber: @Yianni
Added subscriber: @JacquesLucke
Added subscriber: @brecht
Added subscriber: @filibis
Added subscriber: @MeshVoid
Added subscriber: @RedMser
Added subscriber: @Noto
Added subscriber: @MichaelHermann
Added subscriber: @bblanimation
Added subscriber: @pauanyu_blender
Added subscriber: @PhilippeCrassous
Added subscriber: @MiroHorvath
Added subscriber: @2046411367
Added subscriber: @Ricardo-Navarro
Added subscriber: @AlexeyAdamitsky
Added subscriber: @ckohl_art
Added subscriber: @aras_p
Added subscriber: @JanKaderabek
Any update on this?
We have projects with thousands of objects and to process them we are forced to rewrite many built-in methods just to avoid using incredibly slow bpy.ops :-/
Thanks!
Added subscriber: @Lorenzo-Clerici
Added subscriber: @Dangry
This issue was referenced by
7f8d05131a
Added subscriber: @ahmed.hindy96
Removed subscriber: @Ricardo-Navarro
Added subscriber: @sebastian_k