Blender Library snailpaced slow on NAS loading .blend files. #105047

New Issue

Anon · 2023-02-21T20:31:35+01:00

Anon commented

2023-02-21 20:31:35 +01:00

System Information
Operating system: Windows-10-10.0.19044-SP0 64 Bits
Graphics card: NVIDIA GeForce RTX 2070 SUPER/PCIe/SSE2 NVIDIA Corporation 4.5.0 NVIDIA 528.24

Blender Version
Broken: version: 3.4.1, branch: blender-v3.4-release, commit date: 2022-12-19 17:00, hash: rB55485cb379f7
Worked: (newest version of Blender that worked as expected)

Short description of error
Blender Library is horribly slow.
Not really an error or a bug i wish to report but more something that has been bothering me for about a year now, since the release of the Blender Library.
It has just HORRIBLE TERRIBLE UNREPENTANTBLY CRAP! performance when you have your blend files stored on a NAS, like i do.
It takes me an HOUR to just load an essentially small library of things in, its ridiclious this feature was launched in this state. You guys need to really test this feature, place a bunch of Blend files on a NAS with 100mbit or 1Gbit connection, and see yourself how horribly slow it loads.
Exact steps for others to reproduce the error
Use a Raspberry Pi3 or and OdroidC4 as NAS with Openmediavault put on and some hard drives mounted through its USB.
Put some .blend files on the NAS and make blender load the files from there its horribly slow.

**System Information** Operating system: Windows-10-10.0.19044-SP0 64 Bits Graphics card: NVIDIA GeForce RTX 2070 SUPER/PCIe/SSE2 NVIDIA Corporation 4.5.0 NVIDIA 528.24 **Blender Version** Broken: version: 3.4.1, branch: blender-v3.4-release, commit date: 2022-12-19 17:00, hash: `rB55485cb379f7` Worked: (newest version of Blender that worked as expected) **Short description of error** Blender Library is horribly slow. Not really an error or a bug i wish to report but more something that has been bothering me for about a year now, since the release of the Blender Library. It has just HORRIBLE TERRIBLE UNREPENTANTBLY CRAP! performance when you have your blend files stored on a NAS, like i do. It takes me an HOUR to just load an essentially small library of things in, its ridiclious this feature was launched in this state. You guys need to really test this feature, place a bunch of Blend files on a NAS with 100mbit or 1Gbit connection, and see yourself how horribly slow it loads. **Exact steps for others to reproduce the error** Use a Raspberry Pi3 or and OdroidC4 as NAS with Openmediavault put on and some hard drives mounted through its USB. Put some .blend files on the NAS and make blender load the files from there its horribly slow.

Anon added the

labels 2023-02-21 20:31:36 +01:00

Harley Acheson commented

2023-03-14 00:22:35 +01:00

Hello,

Can you provide some details and measurements to help understand your situation? Perhaps include a blend file and also timings on how long it takes to do specific operations with it. That way we can better recreate what you see.

One odd thing I notice here though is that you say to "place a bunch of Blend files on a NAS with 100mbit or 1Gbit connection". Which is your exactly? That one is ten times as fast as the the other is significant.

You also mention that it is "mounted through its USB" but note that USB transfer rates vary from 1.5 Mbps to 10 Gbps. A typical transfer rate over USB is only 12 Mbps, which is significantly slower than the 100 Mpbs or 1Gbpbs you mention earlier.

What type of USB connection do you have and what is the transfer rate between the NAS and its locally-connected computer?

Hello, Can you provide some details and measurements to help understand your situation? Perhaps include a blend file and also timings on how long it takes to do *specific* operations with it. That way we can better recreate what you see. One odd thing I notice here though is that you say to "place a bunch of Blend files on a NAS with 100mbit or 1Gbit connection". Which is your exactly? That one is ten times as fast as the the other is significant. You also mention that it is "mounted through its USB" but note that USB transfer rates vary from 1.5 Mbps to 10 Gbps. A typical transfer rate over USB is only 12 Mbps, which is significantly slower than the 100 Mpbs or 1Gbpbs you mention earlier. What type of USB connection do you have and what is the transfer rate between the NAS and its locally-connected computer?

Anon commented

2023-03-14 17:38:57 +01:00

Greetings,

Thank you for your answer, my situation is as follows i have a 1Gbit connection to my Odroid C4, which acts as my NAS. The Odroid C4 has 4 USB3.0 slots that share 1 single root Hub.
While it does have a seperate controller for the 1Gbit connection. https://www.odroid.nl/ODROID-C4 See speccs here (i have the 4GB RAM model)

I have put Openmediavault on it as its OS to enable it to be a NAS.

A specific operation is simply to open up my library which has its file on the network shared drives. That are Located on the USB drives on the Odroid C4. I cant really tell which speeds the drives get locally sadly, as Openmediavault doesnt really tell me where to find that.
But it is a USB 3.0 main hub

This takes an hour or more to start to display all the items it contains, and then if i switch to any of the categories it takes again easily 15 to 30 minutes.

I hope to have provided enough data with this comment?

Greets
Jessy

Greetings, Thank you for your answer, my situation is as follows i have a 1Gbit connection to my Odroid C4, which acts as my NAS. The Odroid C4 has 4 USB3.0 slots that share 1 single root Hub. While it does have a seperate controller for the 1Gbit connection. https://www.odroid.nl/ODROID-C4 See speccs here (i have the 4GB RAM model) I have put Openmediavault on it as its OS to enable it to be a NAS. A specific operation is simply to open up my library which has its file on the network shared drives. That are Located on the USB drives on the Odroid C4. I cant really tell which speeds the drives get locally sadly, as Openmediavault doesnt really tell me where to find that. But it is a USB 3.0 main hub ![image](/attachments/423e6d2f-e325-4c06-b82b-ca0a2e78890b) This takes an hour or more to start to display all the items it contains, and then if i switch to any of the categories it takes again easily 15 to 30 minutes. I hope to have provided enough data with this comment? Greets Jessy

image.png

73 KiB

Harley Acheson commented

2023-03-14 18:23:04 +01:00

@JessyBerbers - Thanks! That's a great start.

First, I should say that what you are experiencing is very bad and I will not be giving excuses or trying to minimize this. But some background information is required, so don't read any of the following as patronizing; I'm just trying to explain it all as clearly as I can.

For the following lets assume that you are using the Blender File Browser to append from a blend file. These same things happen in the similar way from Asset Browser, but this makes it easier to follow.

When you browse an ordinary directory using File Browser we are generally just asking the operating system or device for a list of the files in it and we just show that list. The individual files themselves don't need to be read, except when in "Thumbnail" view and one isn't already available.

This means that a directory list is super fast no matter how slow your connection is to that computer or drive. It is very little data to move so it happens almost instantly whether the drive is on your local machine or if you connect to it via an old dial-up modem connection.

But when you browse inside of a blend file when appending, it might be shown in a similar manner, but it is much slower. This is because the entirely of that blend file must by read, by your computer, which means the entire thing has to be transmitted to you.

So the way to compare this performance is against a local connection. When the blend file is on your own computer it will probably seem that appending through it is about the same speed as browsing a filesystem directory. But as you move that blend file to place with a slower connection you will get an increasing difference between a directory listing and your appending performance.

This is easy to confirm. Just take the file that is taking forever to append from and move it to your own computer. It will all become instant. The difference is all in your connection speed. Specifically the speed of your slowest part of the path from you to the data.

Your storage device appears to have three methods of storing data, using USB or using its eMMC connector, or its Micro SD slot. The speed of all these things vary, with the slowest being a USB drive that is not USB 3.0.

For example if you plug in a USB 1.0 device you will get 1.5 Mbps. If you plug in one that is 1.1 it will be 12 Mbps maximum, etc. So you want to make sure that your USB drive plugged in is 3.0 "Superspeed".

Fastest would be to plug in a fast eMMC device into that slot on your drive. In the middle would be a Micro SD with HS mode UHS-I SDR104.

Of course it is possible to have everything in your device working as quickly as possible but then have it plugged into a slow hub, or to use a slow WIFI connection on your computer to connect to it.

It is the slowest part of the path that matters. And if you make that 10 times faster you will get a 10x speed up in this appending operation. And I would guess that you have opportunity to increase it 100x.

@JessyBerbers - Thanks! That's a great start. First, I should say that what you are experiencing is very bad and I will not be giving excuses or trying to minimize this. But some background information is required, so don't read any of the following as patronizing; I'm just trying to explain it all as clearly as I can. For the following lets assume that you are using the Blender File Browser to **append** from a blend file. These same things happen in the similar way from Asset Browser, but this makes it easier to follow. When you browse an **ordinary directory** using File Browser we are generally just asking the operating system or device for a list of the files in it and we just show that list. The individual files themselves don't need to be read, except when in "Thumbnail" view and one isn't already available. This means that a directory list is super fast no matter how slow your connection is to that computer or drive. It is very little data to move so it happens almost instantly whether the drive is on your local machine or if you connect to it via an old dial-up modem connection. But when you browse **inside** of a blend file when appending, it might be shown in a similar manner, but it is much slower. This is because the entirely of that blend file must by read, by your computer, which means the entire thing has to be transmitted to you. So the way to compare this performance is against a local connection. When the blend file is on your own computer it will probably seem that appending through it is about the same speed as browsing a filesystem directory. But as you move that blend file to place with a slower connection you will get an increasing difference between a directory listing and your appending performance. This is easy to confirm. Just take the file that is taking forever to append from and move it to your own computer. It will all become instant. The difference is all in your connection speed. Specifically the speed of your **slowest** part of the path from you to the data. Your storage device appears to have three methods of storing data, using USB or using its eMMC connector, or its Micro SD slot. The speed of all these things vary, with the slowest being a USB drive that is not USB 3.0. For example if you plug in a USB 1.0 device you will get 1.5 Mbps. If you plug in one that is 1.1 it will be 12 Mbps maximum, etc. So you want to make sure that your USB drive plugged in is 3.0 "**Superspeed**". Fastest would be to plug in a fast eMMC device into that slot on your drive. In the middle would be a Micro SD with HS mode UHS-I SDR104. Of course it is possible to have everything in your device working as quickly as possible but then have it plugged into a slow hub, or to use a slow WIFI connection on your computer to connect to it. It is the slowest part of the path that matters. And if you make that 10 times faster you will get a 10x speed up in this appending operation. And I would guess that you have opportunity to increase it 100x.

Anon commented

2023-03-14 18:59:54 +01:00

Thing is, when i do so from DAZ3D which also has a Library it loads far far faster then Blender does, because it doesnt Load the whole file just thumbnail, and its Directory path or something along those lines. And it only loads in the asset when you actually put it into the scene.
Not to mention my Daz3D Content library is vastly vastly! more vast then the tens of items maybe 100 at best of items i have in Blender its Library.
Like atleast 1000+ items in my Daz3D library.
Yet it pretty much loads and shows me thumbnails instantly and they are ALL located on that same drive as the Blender files. yet there is vastly more of them.

So i see it as a bad excuse to put my hardware as the problem its Blender that is the problem. Seeing as everything else including Streaming 4K video from the Odroid C4 works perfectly fine, as it does use USB3 harddrives, i have 2 WD USB3 My book drives and a rather old Toshiba drive but i dont keep my files on that apart from some Legacy Files.

My usual filetransfer speeds are easily 108Mb/s to any of those 2 USB3 WD My Book drives.
I've included 2 example GIFs. and in this case accessing the blender library even makes Blender crash after a while.
So you see my problem with how the Library works in Blender, its arduously slow, prone to crashing, and doesnt really function as advertised. Unless you seem to be on some Corpo 10000TrillionBit speed network or something. Or have your library Only locally saved.(preferable apparently on SSDs)

This is a ridiclious ask.

~Jessy

Thing is, when i do so from DAZ3D which also has a Library it loads far far faster then Blender does, because it doesnt Load the whole file just thumbnail, and its Directory path or something along those lines. And it only loads in the asset when you actually put it into the scene. Not to mention my Daz3D Content library is vastly vastly! more vast then the tens of items maybe 100 at best of items i have in Blender its Library. Like atleast 1000+ items in my Daz3D library. Yet it pretty much loads and shows me thumbnails instantly and they are ALL located on that same drive as the Blender files. yet there is vastly more of them. So i see it as a bad excuse to put my hardware as the problem its Blender that is the problem. Seeing as everything else including Streaming 4K video from the Odroid C4 works perfectly fine, as it does use USB3 harddrives, i have 2 WD USB3 My book drives and a rather old Toshiba drive but i dont keep my files on that apart from some Legacy Files. My usual filetransfer speeds are easily 108Mb/s to any of those 2 USB3 WD My Book drives. I've included 2 example GIFs. and in this case accessing the blender library even makes Blender crash after a while. So you see my problem with how the Library works in Blender, its arduously slow, prone to crashing, and doesnt really function as advertised. Unless you seem to be on some Corpo 10000TrillionBit speed network or something. Or have your library Only locally saved.(preferable apparently on SSDs) This is a ridiclious ask. ~Jessy

Blender no workies.gif

1.2 MiB

Fast Daz3D library.gif

4.4 MiB

Harley Acheson commented

2023-03-14 19:23:03 +01:00

Thing is, when i do so from DAZ3D which also has a Library it loads far far faster then Blender does, because it doesnt Load the whole file just thumbnail, and its Directory path or something along those lines. And it only loads in the asset when you actually put it into the scene.

For sure. The central thing here is that Blend files weren't made to be streamed. It is more like a memory snapshot of everything. But this isn't something that could just be "fixed" easily, or possibly at all.

We basically need the entire contents because what you require might be an object, plus mesh, plus textures, and materials, and collections, etc.

So i see it as a bad excuse to put my hardware as the problem its Blender that is the problem. Seeing as everything else including Streaming 4K video from the Odroid C4 works perfectly fine, as it does use USB3 harddrives, i have 2 WD USB3 My book drives and a rather old Toshiba drive but i dont keep my files on that apart from some Legacy Files.

Streaming videos is another good example. To do so you don't need to download the entire thing, just the first part then little bits while you play it. Appending from a Blend file is more like downloading the entire movie to your local temp drive and then playing the video from that.

So you see my problem with how the Library works in Blender, its arduously slow, prone to crashing, and doesnt really function as advertised. Unless you seem to be on some Corpo 10000TrillionBit speed network or something. Or have your library Only locally saved.(preferable apparently on SSDs)

This is a ridiclious ask.

It is frustrating. But this isn't about how "the library works". Blender has no idea where your files are located and doesn't do anything to make itself work slower depending on where the file is.

As mentioned, you can place that same file on your local machine and see how fast it works. The difference in speed between it being there and on your NAS is the difference in speed between reading the entirely of any file on your NAS and doing so locally.

I'm afraid that changing the nature of "blend" files so that they can be streamed isn't really an option. So the only thing under your control is to put them on a medium that is as fast as possible. We have no way of making access faster when they are put on a slow medium. This could only be done by changing the file format, creating some sort of blender server service that would work on your NAS, or to perhaps locally cache details of the remote files.

There just isn't anything about this that is a "bug" that can be acted upon. You have multiple drives connected to your computer (local and remote) with your remote ones giving you much slower access. That is consequence of the speed of your connection to that remote drive.

> Thing is, when i do so from DAZ3D which also has a Library it loads far far faster then Blender does, because it doesnt Load the whole file just thumbnail, and its Directory path or something along those lines. And it only loads in the asset when you actually put it into the scene. For sure. The central thing here is that Blend files weren't made to be streamed. It is more like a memory snapshot of everything. But this isn't something that could just be "fixed" easily, or possibly at all. We basically need the entire contents because what you require might be an object, plus mesh, plus textures, and materials, and collections, etc. > So i see it as a bad excuse to put my hardware as the problem its Blender that is the problem. Seeing as everything else including Streaming 4K video from the Odroid C4 works perfectly fine, as it does use USB3 harddrives, i have 2 WD USB3 My book drives and a rather old Toshiba drive but i dont keep my files on that apart from some Legacy Files. Streaming videos is another good example. To do so you don't need to download the entire thing, just the first part then little bits while you play it. Appending from a Blend file is more like downloading the entire movie to your local temp drive and then playing the video from that. > So you see my problem with how the Library works in Blender, its arduously slow, prone to crashing, and doesnt really function as advertised. Unless you seem to be on some Corpo 10000TrillionBit speed network or something. Or have your library Only locally saved.(preferable apparently on SSDs) > > This is a ridiclious ask. It is frustrating. But this isn't about how "the library works". Blender has no idea where your files are located and doesn't do anything to make itself work slower depending on where the file is. As mentioned, you can place that same file on your local machine and see how fast it works. The difference in speed between it being there and on your NAS is the difference in speed between reading the entirely of any file on your NAS and doing so locally. I'm afraid that changing the nature of "blend" files so that they can be streamed isn't really an option. So the only thing under your control is to put them on a medium that is as fast as possible. We have no way of making access faster when they are put on a slow medium. This could only be done by changing the file format, creating some sort of blender server service that would work on your NAS, or to perhaps locally cache details of the remote files. There just isn't anything about this that is a "bug" that can be acted upon. You have multiple drives connected to your computer (local and remote) with your remote ones giving you much slower access. That is consequence of the speed of your connection to that remote drive.

Anon commented

2023-03-14 19:31:36 +01:00

Geuss i'll just grab the drag from windows filebrowser to Blender addon then, and just completely shelf the library, cause it aint working very well unless your on hyper fast drives. https://blendermarket.com/products/drag-n-drop-import
Least that would show me proper fast thumbnails of the stuff i want to append/import into a scene, because the windows file explorer has no such issues either.
It also displays blender file thumbnails pretty much instantly.
¯_(ツ)_/¯

Geuss i'll just grab the drag from windows filebrowser to Blender addon then, and just completely shelf the library, cause it aint working very well unless your on hyper fast drives. https://blendermarket.com/products/drag-n-drop-import Least that would show me proper fast thumbnails of the stuff i want to append/import into a scene, because the windows file explorer has no such issues either. It also displays blender file thumbnails pretty much instantly. ![image](/attachments/89e07a6c-d238-4164-9949-4cec651c8d8e) ¯\_(ツ)_/¯

image.png

273 KiB

Harley Acheson commented

2023-03-14 19:57:49 +01:00

the windows file explorer has no such issues either.
It also displays blender file thumbnails pretty much instantly.

Yes, we talked of that previously in this thread. Listings of regular file directories are not at all comparable to looking inside of a file for its contents, for reasons explained earlier.

Closing for reasons mentioned, as there isn't a bug here that can be acted upon by developers to solve your issue.

> the windows file explorer has no such issues either. > It also displays blender file thumbnails pretty much instantly. Yes, we talked of that previously in this thread. Listings of regular file directories are not at all comparable to looking **inside** of a file for its contents, for reasons explained earlier. Closing for reasons mentioned, as there isn't a bug here that can be acted upon by developers to solve your issue.

Harley Acheson closed this issue

2023-03-14 19:57:50 +01:00

Blender Bot added

and removed

labels 2023-03-14 19:57:51 +01:00

Sign in to join this conversation.