Node Editor gets unusably slow in scenes of average production complexity #98574

New Issue

Ludvik Koutny · 2022-06-03T12:25:09+02:00

Ludvik Koutny commented

2022-06-03 12:25:09 +02:00

System Information
Operating system: Windows-10-10.0.19044-SP0 64 Bits
Graphics card: NVIDIA GeForce RTX 2080 Ti/PCIe/SSE2 NVIDIA Corporation 4.5.0 NVIDIA 512.59

Blender Version
Broken: version: 3.1.2, branch: SelectThrough, commit date: 2022-04-27 08:19, hash: a3ae0cf3a7
Worked: Probably never

Short description of error
The node editor, as well as other parts of Blender get unusably slow when the scene complexity reaches level of average production scenes from about 5-10 years ago (About 30 million triangles on average). This even includes the operations, which should not require re-evaluation of the entire complex scene dependencies, such as moving unconnected nodes in the shader editor.

Exact steps for others to reproduce the error
SlowScene_Base.blend

Open the attached .blend file:
Select all the objects in the scene and convert them to meshes using Object->Apply->Visual Geometry to Mesh (this step was necessary to keep the repro file size small)
Attempt regular material editing operations in shader editor, such as adding nodes, moving nodes around, editing properties of nodes, such as color ramp sliders, and so on
Result: The constant UI blocking freezes make it impossible to work with the node editor when scene complexity exceeds trivial level.
Expected: The performance of the node editor remains acceptable in the scenes of production complexity.

Note: Feel free to duplicate the objects in the viewport to increase the severity of the lags and freezes.

Bit of research: this is because of UNDO.

If you turn OFF Global Undo in the Preferences (or set Undo Steps to zero) everything is back to being snappy fast.
Not sure if this can be smarter, but this does make sense in a way because Global Undo keeps a copy of the whole file in memory?

**System Information** Operating system: Windows-10-10.0.19044-SP0 64 Bits Graphics card: NVIDIA GeForce RTX 2080 Ti/PCIe/SSE2 NVIDIA Corporation 4.5.0 NVIDIA 512.59 **Blender Version** Broken: version: 3.1.2, branch: SelectThrough, commit date: 2022-04-27 08:19, hash: `a3ae0cf3a7` Worked: Probably never **Short description of error** The node editor, as well as other parts of Blender get unusably slow when the scene complexity reaches level of average production scenes from about 5-10 years ago (About 30 million triangles on average). This even includes the operations, which should not require re-evaluation of the entire complex scene dependencies, such as moving unconnected nodes in the shader editor. **Exact steps for others to reproduce the error** [SlowScene_Base.blend](https://archive.blender.org/developer/F13130479/SlowScene_Base.blend) 1. Open the attached .blend file: 2. Select all the objects in the scene and convert them to meshes using Object->Apply->Visual Geometry to Mesh (this step was necessary to keep the repro file size small) 3. Attempt regular material editing operations in shader editor, such as adding nodes, moving nodes around, editing properties of nodes, such as color ramp sliders, and so on Result: The constant UI blocking freezes make it impossible to work with the node editor when scene complexity exceeds trivial level. Expected: The performance of the node editor remains acceptable in the scenes of production complexity. Note: Feel free to duplicate the objects in the viewport to increase the severity of the lags and freezes. --- Bit of research: this is because of UNDO. If you turn OFF `Global Undo` in the Preferences (or set `Undo Steps` to zero) everything is back to being snappy fast. Not sure if this can be smarter, but this does make sense in a way because Global Undo keeps a copy of the whole file in memory?

Ludvik Koutny commented

2022-06-03 12:25:09 +02:00

Added subscriber: @Rawalanche

blender-admin commented

2022-06-03 12:25:09 +02:00

#99299 was marked as duplicate of this issue

Omar Emara commented

2022-06-06 11:54:52 +02:00

Added subscriber: @OmarEmaraDev

Omar Emara commented

2022-06-06 11:54:52 +02:00

Changed status from 'Needs Triage' to: 'Needs User Info'

Omar Emara commented

2022-06-06 11:54:52 +02:00

The computations seems to be spent just redrawing the viewport, can you confirm this by closing the 3D view editor and see if it is still slow?

Ludvik Koutny commented

2022-06-06 12:09:36 +02:00

In #98574#1369236, @OmarEmaraDev wrote:
The computations seems to be spent just redrawing the viewport, can you confirm this by closing the 3D view editor and see if it is still slow?

That makes no sense. The viewport navigation runs at 60+ FPS, so the drawing of the viewport does not take anywhere near as long. The delay between for example click-dragging the node to move it and actual move operation starting is far longer than 16 milliseconds.

And yes, it also happens when I close the viewport. I am quite confused, as I've provided a repro scene, so you could have tested it yourself. I see no reason it should behave differently on your system than mine.

How did you arrive at a conclusion that the computation is spent just redrawing the scene? Have you actually opened the file and followed the repro steps?

> In #98574#1369236, @OmarEmaraDev wrote: > The computations seems to be spent just redrawing the viewport, can you confirm this by closing the 3D view editor and see if it is still slow? That makes no sense. The viewport navigation runs at 60+ FPS, so the drawing of the viewport does not take anywhere near as long. The delay between for example click-dragging the node to move it and actual move operation starting is far longer than 16 milliseconds. And yes, it also happens when I close the viewport. I am quite confused, as I've provided a repro scene, so you could have tested it yourself. I see no reason it should behave differently on your system than mine. How did you arrive at a conclusion that the computation is spent just redrawing the scene? Have you actually opened the file and followed the repro steps?

Omar Emara commented

2022-06-06 13:04:46 +02:00

I did open the file and tested your reproduction case, but I couldn't replicate the freezing. The only thing I experienced was some stuttering due to redraws, which doesn't happen if I close the 3D view, hence my inquiry.
Perhaps you can attach a small video that demonstrates the issue?

I did open the file and tested your reproduction case, but I couldn't replicate the freezing. The only thing I experienced was some stuttering due to redraws, which doesn't happen if I close the 3D view, hence my inquiry. Perhaps you can attach a small video that demonstrates the issue?

Ludvik Koutny commented

2022-06-06 14:06:47 +02:00

In #98574#1369258, @OmarEmaraDev wrote:
I did open the file and tested your reproduction case, but I couldn't replicate the freezing. The only thing I experienced was some stuttering due to redraws, which doesn't happen if I close the 3D view, hence my inquiry.
Perhaps you can attach a small video that demonstrates the issue?

2022-06-06 14-04-04.mp4

> In #98574#1369258, @OmarEmaraDev wrote: > I did open the file and tested your reproduction case, but I couldn't replicate the freezing. The only thing I experienced was some stuttering due to redraws, which doesn't happen if I close the 3D view, hence my inquiry. > Perhaps you can attach a small video that demonstrates the issue? [2022-06-06 14-04-04.mp4](https://archive.blender.org/developer/F13134973/2022-06-06_14-04-04.mp4)

Omar Emara commented

2022-06-06 14:08:45 +02:00

Changed status from 'Needs User Info' to: 'Needs Triage'

Philipp Oeser commented

2022-06-07 12:30:09 +02:00

Added subscriber: @lichtwerk

Philipp Oeser commented

2022-06-07 12:30:09 +02:00

Changed status from 'Needs Triage' to: 'Needs User Info'

Philipp Oeser commented

2022-06-07 12:30:09 +02:00

Hm, also no drastic slowdowns on my side (7 year old laptop!)

do you experience the same in 2.93?
can we try this with Factory Defaults please?

Broken: version: 3.1.2, branch: SelectThrough, commit date: 2022-04-27 08:19, hash: a3ae0cf3a7

can we try this in plain builds from https://builder.blender.org/download/daily/ (you seem to be on a feature branch)

Hm, also no drastic slowdowns on my side (7 year old laptop!) - do you experience the same in 2.93? - can we try this with Factory Defaults please? > Broken: version: 3.1.2, branch: SelectThrough, commit date: 2022-04-27 08:19, hash: a3ae0cf3a7 - can we try this in plain builds from https://builder.blender.org/download/daily/ (you seem to be on a feature branch)

Ludvik Koutny commented

2022-06-07 14:28:33 +02:00

In #98574#1369814, @lichtwerk wrote:
Hm, also no drastic slowdowns on my side (7 year old laptop!)

do you experience the same in 2.93?

can we try this with Factory Defaults please?

Broken: version: 3.1.2, branch: SelectThrough, commit date: 2022-04-27 08:19, hash: a3ae0cf3a7

can we try this in plain builds from https://builder.blender.org/download/daily/ (you seem to be on a feature branch)

This is 3.2.0 RC in factory settings:

It's most prominent when click-dragging not yet selected nodes in the shader editor to move them.2022-06-07 14-25-16.mp4

You will not manage to repro it if you don't apply the visual geometry to mesh. The important part is to have many high poly meshes in the scene. Not just low poly meshes which generate high polycount through dynamic modifiers. The reason I included the scene with additional steps is that if I uploaded the geometry collapsed to polygons, the file would be giant.

It's so bad that the click-drag gesture often even doesn't register.

> In #98574#1369814, @lichtwerk wrote: > Hm, also no drastic slowdowns on my side (7 year old laptop!) > > - do you experience the same in 2.93? > - can we try this with Factory Defaults please? > >> Broken: version: 3.1.2, branch: SelectThrough, commit date: 2022-04-27 08:19, hash: a3ae0cf3a7 > - can we try this in plain builds from https://builder.blender.org/download/daily/ (you seem to be on a feature branch) This is 3.2.0 RC in factory settings: It's most prominent when click-dragging not yet selected nodes in the shader editor to move them.[2022-06-07 14-25-16.mp4](https://archive.blender.org/developer/F13136725/2022-06-07_14-25-16.mp4) You will not manage to repro it if you don't apply the visual geometry to mesh. The important part is to have many high poly meshes in the scene. Not just low poly meshes which generate high polycount through dynamic modifiers. The reason I included the scene with additional steps is that if I uploaded the geometry collapsed to polygons, the file would be giant. It's so bad that the click-drag gesture often even doesn't register.

tom k commented

2022-06-14 14:41:08 +02:00

Added subscriber: @tomjk

tom k commented

2022-06-14 14:41:08 +02:00

Can reproduce on just-released blender 3.2 with load factory defaults: click-dragging selected nodes has a slight pause at drag end, and for unselected nodes an additional slightly longer pause at drag start. By eye, maybe ~100ms and ~150ms respectively. Similar delay at start and end when click-dragging an empty selection box.

This remains true after removing all other editors; also with no objects selected and editing world shader nodes; and after deleting all objects. If I then purge the orphan mesh datablocks, the pauses go away.

system:
linux mint 20
i7-6700K
RTX 2060 SUPER [driver 510.47.03; a little outdated, latest being 510.73.05]

Can reproduce on just-released blender 3.2 with load factory defaults: click-dragging selected nodes has a slight pause at drag end, and for unselected nodes an additional slightly longer pause at drag start. By eye, maybe ~100ms and ~150ms respectively. Similar delay at start and end when click-dragging an empty selection box. This remains true after removing all other editors; also with no objects selected and editing world shader nodes; and after deleting all objects. If I then purge the orphan mesh datablocks, the pauses go away. system: linux mint 20 i7-6700K RTX 2060 SUPER [driver 510.47.03; a little outdated, latest being 510.73.05]

Philipp Oeser commented

2022-06-26 16:02:39 +02:00

Changed status from 'Needs User Info' to: 'Needs Triage'

Omar Emara commented

2022-07-06 08:19:46 +02:00

Added subscribers: @Olliver, @mano-wii, @iss

Pratik Borhade commented

2022-08-02 15:36:33 +02:00

Added subscriber: @PratikPB2123

Pratik Borhade commented

2022-08-02 15:36:33 +02:00

Changed status from 'Needs Triage' to: 'Confirmed'

Pratik Borhade commented

2022-08-02 15:36:33 +02:00

Can notice significant slowdown in node operations after Visual Geometry to Mesh operation

Blender version; 3.1.2, 3.4
Operating system: Windows-10-10.0.22000-SP0 64 Bits
Graphics card: NVIDIA GeForce RTX 3050 Laptop GPU/PCIe/SSE2 NVIDIA Corporation 4.5.0 NVIDIA 512.78```
- - -
Confirming for now. AFAIK bug reports of performance complaints are not accepted on tracker
> Canned response:
> While we do continue to work on improving performance in general, potential performance improvements are not handled as bug reports.
> To improve performance, consider using less complex geometry, simpler shaders and smaller textures.

Can notice significant slowdown in node operations after `Visual Geometry to Mesh` operation ```**System Information** Blender version; 3.1.2, 3.4 Operating system: Windows-10-10.0.22000-SP0 64 Bits Graphics card: NVIDIA GeForce RTX 3050 Laptop GPU/PCIe/SSE2 NVIDIA Corporation 4.5.0 NVIDIA 512.78``` - - - Confirming for now. AFAIK bug reports of performance complaints are not accepted on tracker > Canned response: > While we do continue to work on improving performance in general, potential performance improvements are not handled as bug reports. > To improve performance, consider using less complex geometry, simpler shaders and smaller textures.

😕 1

Ludvik Koutny commented

2022-08-02 19:29:30 +02:00

In #98574#1398655, @PratikPB2123 wrote:
Can notice significant slowdown in node operations after Visual Geometry to Mesh operation

Blender version; 3.1.2, 3.4
Operating system: Windows-10-10.0.22000-SP0 64 Bits
Graphics card: NVIDIA GeForce RTX 3050 Laptop GPU/PCIe/SSE2 NVIDIA Corporation 4.5.0 NVIDIA 512.78```
- - -
Confirming for now. AFAIK bug reports of performance complaints are not accepted on tracker
> Canned response:
> While we do continue to work on improving performance in general, potential performance improvements are not handled as bug reports.
> To improve performance, consider using less complex geometry, simpler shaders and smaller textures.

Come on, seriously? "consider using less complex geometry"

30M polygons was average for a production scene 10 years ago. This is really not acceptable. This is also not a general performance issue, as in not every part of Blender slows down evenly. This is just one subset of operations in a specific editor. By these standards, how come this one was considered a bug and solved? https://developer.blender.org/T94609 It's the exact same class of issue. Something was triggering complete scene recalculation when it was not supposed to.

> In #98574#1398655, @PratikPB2123 wrote: > Can notice significant slowdown in node operations after `Visual Geometry to Mesh` operation > ```**System Information** > Blender version; 3.1.2, 3.4 > Operating system: Windows-10-10.0.22000-SP0 64 Bits > Graphics card: NVIDIA GeForce RTX 3050 Laptop GPU/PCIe/SSE2 NVIDIA Corporation 4.5.0 NVIDIA 512.78``` > - - - > Confirming for now. AFAIK bug reports of performance complaints are not accepted on tracker >> Canned response: >> While we do continue to work on improving performance in general, potential performance improvements are not handled as bug reports. >> To improve performance, consider using less complex geometry, simpler shaders and smaller textures. Come on, seriously? "consider using less complex geometry" 30M polygons was average for a production scene 10 years ago. This is really not acceptable. This is also not a general performance issue, as in not every part of Blender slows down evenly. This is just one subset of operations in a specific editor. By these standards, how come this one was considered a bug and solved? https://developer.blender.org/T94609 It's the exact same class of issue. Something was triggering complete scene recalculation when it was not supposed to.

Jakub Jaszewski commented

2022-08-06 00:52:45 +02:00

Added subscriber: @silex

Jakub Jaszewski commented

2022-08-06 00:52:45 +02:00

Unfortunately I must confirm this. I'm not working on 'production' size scenes, yet with more complicated ones I find that I pause Cycles viewport, tweak some shader parameters and unpause the viewport to see the changes. It's not very efficient way of work,

Scene from the report gives me very noticeable lag when moving nodes, or adjusting their parameters after duplicating objects 4 times (without applying visual geometry to mesh). With applying modifiers I have lag just after one duplication.

Also a note - I had similar issues in scenes with much less polycount than 30 mil - just around 2,5 mil faces (~5 mil tris).

CPU - Threadripper 3970X

Unfortunately I must confirm this. I'm not working on 'production' size scenes, yet with more complicated ones I find that I pause Cycles viewport, tweak some shader parameters and unpause the viewport to see the changes. It's not very efficient way of work, Scene from the report gives me very noticeable lag when moving nodes, or adjusting their parameters after duplicating objects 4 times (without applying visual geometry to mesh). With applying modifiers I have lag just after one duplication. Also a note - I had similar issues in scenes with much less polycount than 30 mil - just around 2,5 mil faces (~5 mil tris). CPU - Threadripper 3970X

Ludvik Koutny commented

2022-08-27 12:06:59 +02:00

3.3 is almost released and this is still ignored. This is not a general performance insufficiency. This is a bug which causes whole scene to re-evaluate when nodes are selected and moved around. A change, which should not even need to read, let alone re-evaluate mesh geometry in the viewport does so.

3.3 is almost released and this is still ignored. This is not a general performance insufficiency. This is a **bug** which causes whole scene to re-evaluate when nodes are selected and moved around. A change, which should not even need to read, let alone re-evaluate mesh geometry in the viewport does so.

Philipp Oeser commented

2022-08-31 10:38:05 +02:00

Let me add that I can reproduce this now (not sure why that was not the case previously).

Let me add that I **can** reproduce this now (not sure why that was not the case previously).

Philipp Oeser commented

2022-08-31 14:39:44 +02:00

Added subscribers: @JacquesLucke, @mont29

Philipp Oeser commented

2022-08-31 14:39:44 +02:00

Looked at this a bit and this is because of UNDO.

If you turn OFF Global Undo in the Preferences (or set Undo Steps to zero) everything is back to being snappy fast.
Not sure if this can be smarter, but this does make sense in a way because Global Undo keeps a copy of the whole file in memory?

@mont29 or @JacquesLucke might know more...

Looked at this a bit and this is because of UNDO. If you turn OFF `Global Undo` in the Preferences (or set `Undo Steps` to zero) everything is back to being snappy fast. Not sure if this can be smarter, but this does make sense in a way because Global Undo keeps a copy of the whole file in memory? @mont29 or @JacquesLucke might know more...

Jacques Lucke commented

2022-08-31 14:56:53 +02:00

Don't know too much about undo and possible mitigation strategies, but it feels like #95845 could solve this long term if we go "all in".

Ludvik Koutny commented

2022-08-31 18:00:33 +02:00

In #98574#1410573, @lichtwerk wrote:
Looked at this a bit and this is because of UNDO.

If you turn OFF Global Undo in the Preferences (or set Undo Steps to zero) everything is back to being snappy fast.
Not sure if this can be smarter, but this does make sense in a way because Global Undo keeps a copy of the whole file in memory?

@mont29 or @JacquesLucke might know more...

I see. That explains why this slowness affects more parts of Blender, and this being only one of many manifestations of this. Not sure how to solve this, but it's certainly not good, since complexity of this scenes is nothing out of ordinary these days.

Also, I am a bit confused about what does the "global undo" toggle do? It seems to turn off undo altogether. When I uncheck global undo, undo action stops working completely, so that's definitely not a feasible workaround. If it's called "global undo", then once unchecked, what does the "local undo" do? In which scope the undo works then, because it's certainly not 3D view or the node editor.

It's really crazy to think that every move of any node in the node editor will duplicate all 30 million triangles worth of meshes in the memory every time I do that.

> In #98574#1410573, @lichtwerk wrote: > Looked at this a bit and this is because of UNDO. > > If you turn OFF `Global Undo` in the Preferences (or set `Undo Steps` to zero) everything is back to being snappy fast. > Not sure if this can be smarter, but this does make sense in a way because Global Undo keeps a copy of the whole file in memory? > > @mont29 or @JacquesLucke might know more... I see. That explains why this slowness affects more parts of Blender, and this being only one of many manifestations of this. Not sure how to solve this, but it's certainly not good, since complexity of this scenes is nothing out of ordinary these days. Also, I am a bit confused about what does the "global undo" toggle do? It seems to turn off undo altogether. When I uncheck global undo, undo action stops working completely, so that's definitely not a feasible workaround. If it's called "global undo", then once unchecked, what does the "local undo" do? In which scope the undo works then, because it's certainly not 3D view or the node editor. It's really crazy to think that every move of any node in the node editor will duplicate all 30 million triangles worth of meshes in the memory every time I do that.

Jon Henry M. Faltis 9077149 commented

2022-09-01 01:05:55 +02:00

Added subscriber: @geocentric_wage

Alexey Adamitsky commented

2022-09-01 07:36:04 +02:00

Added subscriber: @AlexeyAdamitsky

Bastien Montagne commented

2022-09-01 09:08:25 +02:00

Added subscriber: @HooglyBoogly

Bastien Montagne commented

2022-09-01 09:08:25 +02:00

@HooglyBoogly are you planing to work on this issue, to tag it as high priority?

@JacquesLucke @lichtwerk would be interesting to do a profiling here, the 'undo system' is a very vague and wide area that can be affected by a lot of 'unrelated' code in Blender.

@Rawalanche Blender certainly does not write 30M triangles every time you move a node, otherwise you would run out of RAM in a few undo steps. But it does 'fake-write' the whole .blend file to be able to generate its memory diff, and only effectively keep in memory the differences from the previous step.

Again, without proper profiling, it's impossible to go further.

@HooglyBoogly are you planing to work on this issue, to tag it as high priority? @JacquesLucke @lichtwerk would be interesting to do a profiling here, the 'undo system' is a very vague and wide area that can be affected by a lot of 'unrelated' code in Blender. @Rawalanche Blender certainly does not write 30M triangles every time you move a node, otherwise you would run out of RAM in a few undo steps. But it does 'fake-write' the whole .blend file to be able to generate its memory diff, and only effectively keep in memory the differences from the previous step. Again, without proper profiling, it's impossible to go further.

Jesse Yurkovich commented

2022-09-01 09:32:22 +02:00

Added subscriber: @deadpin

Jesse Yurkovich commented

2022-09-01 09:32:22 +02:00

I did a quick profile below. This won't lead to a solution directly but hopefully it'll kickstart the process.

Most of the time is spent inside BLO_memfile_chunk_add. In the provided scene, making any viewport selection, or shader node movement, results in the following:

About 60,900 calls to memcmp
Average size of each call is 32,489 bytes totaling 1.84gb of memory being compared
Of the 60,900 calls, only 5 mem chunks were found to be "not identical" (printf debugging)
This takes about ~210ms each time an undo step needs to be pushed

I did a quick profile below. This won't lead to a solution directly but hopefully it'll kickstart the process. Most of the time is spent inside `BLO_memfile_chunk_add`. In the provided scene, making any viewport selection, or shader node movement, results in the following: - About 60,900 calls to `memcmp` - Average size of each call is 32,489 bytes totaling 1.84gb of memory being compared - Of the 60,900 calls, only 5 mem chunks were found to be "not identical" (printf debugging) - This takes about ~210ms each time an undo step needs to be pushed ![BLO_memfile_chunk_add.png](https://archive.blender.org/developer/F13445050/BLO_memfile_chunk_add.png)

👀 2

Jacques Lucke commented

2022-09-01 09:53:01 +02:00

Yeah, that's also what I found, almost all time is spent in memcmp. It could probably be speedup a bit with parallelization, but the bottleneck will likely remain the same. The benefit of #95845 is that this memcmp would just not be necessary. To store an array for undo, one would just have to increase a reference count.

Yeah, that's also what I found, almost all time is spent in `memcmp`. It could probably be speedup a bit with parallelization, but the bottleneck will likely remain the same. The benefit of #95845 is that this `memcmp` would just not be necessary. To store an array for undo, one would just have to increase a reference count.

Bastien Montagne commented

2022-09-01 10:34:22 +02:00

Thanks. There is really no reason to set this to high then, this is simply 'known issue' due to how code works currently. Not saying it should not be improved obviously.

@JacquesLucke not really sure how #95845 could be applied to our mem undo, not without a complete rewrite of the whole system at least...

Thanks. There is really no reason to set this to high then, this is simply 'known issue' due to how code works currently. Not saying it should not be improved obviously. @JacquesLucke not really sure how #95845 could be applied to our mem undo, not without a complete rewrite of the whole system at least...

Bastien Montagne commented

2022-09-01 10:45:07 +02:00

Also, here on a relatively recent machine under linux, I only start seeing a (barely) noticeable lag when over 100M triangles (duplicating all objects 4 times after apply geometry), this is an 8GB blend file... So would not call that a critical issue by any mean.

Bastien Montagne commented

2022-09-01 12:00:27 +02:00

As a note, I did a quick check on making that memcmp part multi-threaded, while this looks possible it would actually imply duplicating all buffers memory (memcpy), since there is no guarantee those buffers are not generated just for filewrite purpose and therefore we can keep them around until threaded memcmp is processed... So that would end up being much worse.

As a note, I did a quick check on making that memcmp part multi-threaded, while this looks possible it would actually imply duplicating all buffers memory (`memcpy`), since there is no guarantee those buffers are not generated just for filewrite purpose and therefore we can keep them around until threaded memcmp is processed... So that would end up being much worse.

Jacques Lucke commented

2022-09-01 12:41:23 +02:00

As a note, I did a quick check on making that memcmp part multi-threaded, while this looks possible it would actually imply duplicating all buffers memory (memcpy), since there is no guarantee those buffers are not generated just for filewrite purpose and therefore we can keep them around until threaded memcmp is processed... So that would end up being much worse.

That shouldn't be a problem when the memcmp is just parallized "inplace"? I think what you tried is to postpone the memcmp and do it outside of the scope of BLO_memfile_chunk_add.

not really sure how #95845 could be applied to our mem undo, not without a complete rewrite of the whole system at least...

Well, it might require some rewriting, don't know too many details of the current undo system. This is also why I mentioned it as a possible long term goal.
That said, it doesn't feel extremely complicated. Here is an API addition that might work:

/**
* The undo system increases the user count of the #bCopyOnWrite, taking ownership of the data.
* Now the ownership is shared between the original dna data and potentially multiple undo steps
* (and potentially other places). Since the data is shared, it must not be modified by anyone.
*
* `free_cow_data` is called when the undo step is freed and this undo step was the last owner of
* the data.
*/
void BLO_write_cow(BlendWriter *writer,

             const struct bCopyOnWrite *cow,
             const void *cow_data,
             void (*free_cow_data)(void *));


/**
 * Returns true when `cow_data` is an address written using #BLO_write_cow.
 */
bool BLO_read_is_cow(BlendDataReader *reader, const void *cow_data);

It would be used as follows:

struct MyDnaData {
  int *array;
  int array_size;
  bCopyOnWrite *array_cow;
};

void blend_write_my_dna_data(BlendWriter *writer, MyDnaData *data)
{
  if (BLO_write_is_undo(writer) && data->array_cow != nullptr) {
    /* If the array can have shared ownership, just share the ownership with the undo system. */
    BLO_write_cow(writer, data->array_cow, data->array, MEM_freeN);
  }
  else {
    BLO_write_int32_array(writer, data->array_size, data->array);
  }
}

void blend_read_my_dna_data(BlendDataReader *reader, MyDnaData *data)
{
  if (BLO_read_is_cow(reader, data->array)) {
    /* `array` and `array_cow` already point to valid memory that is owned by the undo system (and
     * potentially others). Just increase the user count to take ownership here as well. */
    BLI_assert(data->array_cow != nullptr);
    BLI_cow_user_add(data->array_cow);
  }
  else {
    BLO_read_int32_array(reader, data->array_size, &data->array);
    data->array_cow = nullptr;
  }
}

I did not try if this actually works, but this is what I have in mind currently.

> As a note, I did a quick check on making that memcmp part multi-threaded, while this looks possible it would actually imply duplicating all buffers memory (memcpy), since there is no guarantee those buffers are not generated just for filewrite purpose and therefore we can keep them around until threaded memcmp is processed... So that would end up being much worse. That shouldn't be a problem when the `memcmp` is just parallized "inplace"? I think what you tried is to postpone the `memcmp` and do it outside of the scope of `BLO_memfile_chunk_add`. > not really sure how #95845 could be applied to our mem undo, not without a complete rewrite of the whole system at least... Well, it might require some rewriting, don't know too many details of the current undo system. This is also why I mentioned it as a possible long term goal. That said, it doesn't feel extremely complicated. Here is an API addition that might work: ```lang=c++ /** * The undo system increases the user count of the #bCopyOnWrite, taking ownership of the data. * Now the ownership is shared between the original dna data and potentially multiple undo steps * (and potentially other places). Since the data is shared, it must not be modified by anyone. * * `free_cow_data` is called when the undo step is freed and this undo step was the last owner of * the data. */ void BLO_write_cow(BlendWriter *writer, ``` const struct bCopyOnWrite *cow, const void *cow_data, void (*free_cow_data)(void *)); ``` /** * Returns true when `cow_data` is an address written using #BLO_write_cow. */ bool BLO_read_is_cow(BlendDataReader *reader, const void *cow_data); ``` It would be used as follows: ```lang=c++ struct MyDnaData { int *array; int array_size; bCopyOnWrite *array_cow; }; void blend_write_my_dna_data(BlendWriter *writer, MyDnaData *data) { if (BLO_write_is_undo(writer) && data->array_cow != nullptr) { /* If the array can have shared ownership, just share the ownership with the undo system. */ BLO_write_cow(writer, data->array_cow, data->array, MEM_freeN); } else { BLO_write_int32_array(writer, data->array_size, data->array); } } void blend_read_my_dna_data(BlendDataReader *reader, MyDnaData *data) { if (BLO_read_is_cow(reader, data->array)) { /* `array` and `array_cow` already point to valid memory that is owned by the undo system (and * potentially others). Just increase the user count to take ownership here as well. */ BLI_assert(data->array_cow != nullptr); BLI_cow_user_add(data->array_cow); } else { BLO_read_int32_array(reader, data->array_size, &data->array); data->array_cow = nullptr; } } ``` I did not try if this actually works, but this is what I have in mind currently.

Ludvik Koutny commented

2022-09-01 13:05:14 +02:00

In #98574#1410984, @mont29 wrote:
Also, here on a relatively recent machine under linux, I only start seeing a (barely) noticeable lag when over 100M triangles (duplicating all objects 4 times after apply geometry), this is an 8GB blend file... So would not call that a critical issue by any mean.

If it's platform specific, then it's still critical, as linux is negligible platform userbase-wise. I have quite high end machine, and the node editor becomes literally unusable just at about 30M triangles, which, as I wrote above, is just average production scene these days.

The point here is that the action to click and drag a node to move it will just straight up fail 3 out of 4 times. I don't think that can be dismissed as a known issue.

Here's an example with 60M triangles. The click-drag action has failed 9 out of 10 times. The one time it did not fail, it still failed to move the node to its final location:

2022-09-01 13-09-03.mp4

An issue where UI inputs simply stop working above certain scene complexity threshold, which is not uncommon, is serious, no matter how well known it is.

> In #98574#1410984, @mont29 wrote: > Also, here on a relatively recent machine under linux, I only start seeing a (barely) noticeable lag when over 100M triangles (duplicating all objects 4 times after apply geometry), this is an 8GB blend file... So would not call that a critical issue by any mean. If it's platform specific, then it's still critical, as linux is negligible platform userbase-wise. I have quite high end machine, and the node editor becomes literally unusable just at about 30M triangles, which, as I wrote above, is just average production scene these days. The point here is that the action to click and drag a node to move it will just straight up fail 3 out of 4 times. I don't think that can be dismissed as a known issue. Here's an example with 60M triangles. The click-drag action has failed 9 out of 10 times. The one time it did not fail, it still failed to move the node to its final location: [2022-09-01 13-09-03.mp4](https://archive.blender.org/developer/F13446189/2022-09-01_13-09-03.mp4) An issue where UI inputs simply stop working above certain scene complexity threshold, which is not uncommon, is serious, no matter how well known it is.

Ludvik Koutny commented

2022-10-18 11:24:39 +02:00

Any news on this? I have even nearly decade old production scenes which don't exceed any reasonable complexity, yet the node editor is unusable in them. For example:

A scene with mere 10M triangles and 600 objects should not be so slow that one can't use the node editor, even if the viewport is closed. This is not a limitation of current hardware or software architectures.

I'd understand if it was requested to have some state of the art performance optimization in Blender, but all that's requested is that the UI is usable with the scene complexity most of other software can handle easily with current day hardware.

Any news on this? I have even nearly decade old production scenes which don't exceed any reasonable complexity, yet the node editor is unusable in them. For example: ![image.png](https://archive.blender.org/developer/F13702999/image.png) A scene with mere 10M triangles and 600 objects should not be so slow that one can't use the node editor, even if the viewport is closed. This is not a limitation of current hardware or software architectures. I'd understand if it was requested to have some state of the art performance optimization in Blender, but all that's requested is that the UI is usable with the scene complexity most of other software can handle easily with current day hardware.

Ludvik Koutny commented

2022-12-10 12:31:59 +01:00

Is there any chance this ever gets fixed? I am now in an embarrassing scenario where I have just a single 12M poly mesh in my scene, and I can't even shade it because the node editor isn't reliably responding to mouse inputs.

Is there any chance this ever gets fixed? I am now in an embarrassing scenario where I have just a **single** 12M poly mesh in my scene, and I can't even shade it because the node editor isn't reliably responding to mouse inputs.

Nik-9 commented

2022-12-16 17:17:11 +01:00

Added subscriber: @Nik-9

Dorian commented

2022-12-19 16:41:34 +01:00

Added subscriber: @BD3D

Evan Wilson commented

2023-01-01 16:59:29 +01:00

Added subscriber: @EAW

Ludvik Koutny commented

2023-01-09 12:32:29 +01:00

Any news on this? I've updated to 3.4 and it's still not possible to work on scenes over 5M triangles without extremely laggy UI. Using Blender for not trivial lowpoly scenes is borderline impossible.

Jacques Lucke commented

2023-01-23 02:25:10 +01:00

I tested the refactor I mentioned in my comment above (#98574#1411027) as part of D14139. It's working as expected. The changes to the undo system are fairly small.
The time to execute BLO_write_file_mem drops from 150 ms to 0.4 ms, which is only slightly slower than the startup .blend file. For the initial undo step, for which we currently need a full copy in master, the improvement is from 590 ms to 0.4 ms.

I tested the refactor I mentioned in my comment above (#98574#1411027) as part of [D14139](https://archive.blender.org/developer/D14139). It's working as expected. The changes to the undo system are fairly small. The time to execute `BLO_write_file_mem` drops from `150 ms` to `0.4 ms`, which is only slightly slower than the startup .blend file. For the initial undo step, for which we currently need a full copy in `master`, the improvement is from `590 ms` to `0.4 ms`.

🚀 2

Ludvik Koutny commented

2023-01-23 10:09:28 +01:00

In #98574#1478249, @JacquesLucke wrote:
I tested the refactor I mentioned in my comment above (#98574#1411027) as part of D14139. It's working as expected. The changes to the undo system are fairly small.
The time to execute BLO_write_file_mem drops from 150 ms to 0.4 ms, which is only slightly slower than the startup .blend file. For the initial undo step, for which we currently need a full copy in master, the improvement is from 590 ms to 0.4 ms.

Thanks so much. Is there any estimate when this could end up in master?

> In #98574#1478249, @JacquesLucke wrote: > I tested the refactor I mentioned in my comment above (#98574#1411027) as part of [D14139](https://archive.blender.org/developer/D14139). It's working as expected. The changes to the undo system are fairly small. > The time to execute `BLO_write_file_mem` drops from `150 ms` to `0.4 ms`, which is only slightly slower than the startup .blend file. For the initial undo step, for which we currently need a full copy in `master`, the improvement is from `590 ms` to `0.4 ms`. Thanks so much. Is there any estimate when this could end up in master?

Jacques Lucke referenced a pull request that will close this issue

2023-02-07 19:48:49 +01:00

Curves: initial surface collision for curves sculpt mode #104427

Jacques Lucke referenced a pull request that will close this issue

2023-02-08 14:50:51 +01:00

WIP: use generic copy-on-write system to avoid unnecessary data copies #104470

Jacques Lucke referenced a pull request that will close this issue

2023-02-08 14:52:51 +01:00

WIP: Core: use generic copy-on-write system to avoid unnecessary data copies #104471

Bastien Montagne added this to the Core project 2023-02-09 15:42:54 +01:00

Bastien Montagne removed this from the Core project 2023-02-09 18:20:30 +01:00

Philipp Oeser removed the