Loading times of files that link in external data has become inacceptable #61855

New Issue

Rainer Trummer · 2019-02-22T16:39:54+01:00

Rainer Trummer commented

2019-02-22 16:39:54 +01:00

System Information
Operating system: Windows 10, latest SP
Graphics card: Quadro K4000

Blender Version
Broken: 358e07f447
Worked: f3e9dff03d

Short description of error
The mentioned commit tries to reduce the memory usage when loading linked data. While this may work fine, loading times of such files increased for us in the studio dramatically. As an example, loading time of a file that only links from a single library increased from 2 seconds to roughly two minutes. I am loading both the file and the linked data from a network drive which is write protected. Other files have not suffered that much, but also open way slower than before.

Exact steps for others to reproduce the error
Can't provide a blend as these projects are confidential. The only thing I can post so far is a screenshot of the outliner in 'blend file' mode:

**System Information** Operating system: Windows 10, latest SP Graphics card: Quadro K4000 **Blender Version** Broken: 358e07f447e9ed7f87f419e8d027e73c97c00077 Worked: f3e9dff03d9bde0d3002eecb1d51dd0baabbb285 **Short description of error** The mentioned commit tries to reduce the memory usage when loading linked data. While this may work fine, loading times of such files increased for us in the studio dramatically. As an example, loading time of a file that only links from a single library increased from 2 seconds to roughly two minutes. I am loading both the file and the linked data from a network drive which is write protected. Other files have not suffered that much, but also open way slower than before. **Exact steps for others to reproduce the error** Can't provide a blend as these projects are confidential. The only thing I can post so far is a screenshot of the outliner in 'blend file' mode: ![items.PNG](https://archive.blender.org/developer/F6687569/items.PNG)

Rainer Trummer commented

2019-02-22 16:39:54 +01:00

Added subscriber: @RainerTrummer

Campbell Barton was assigned by Rainer Trummer

2019-02-22 16:40:34 +01:00

Brecht Van Lommel commented

2019-02-22 19:46:10 +01:00

Added subscriber: @brecht

Brecht Van Lommel commented

2019-02-22 19:46:10 +01:00

I tested .blend load time on a Spring production file on my Linux desktop. Here the change actually improved load time significantly.

	Before	After
Local SSD	08.28s	5.04s
NFS Network	19.18s	14.04s

This problem may be Windows specific, I've run into extremely slow random access on network drivers there before. Using std::ifstream and seeking instead of mmap helped in that case, apparently due to memory mapping not always going through the system file cache. This might be the same kind of issue.

@ideasman42, maybe we can disable this change on Windows for now, and then investigate how to fix it later? The Linux file load time is actually a great improvement I didn't expect from this change.

I tested .blend load time on a Spring production file on my Linux desktop. Here the change actually improved load time significantly. | | Before | After | | -- | -- | -- | | Local SSD | 08.28s | 5.04s | | NFS Network | 19.18s | 14.04s | This problem may be Windows specific, I've run into extremely slow random access on network drivers there before. Using `std::ifstream` and seeking instead of `mmap` helped in that case, apparently due to memory mapping not always going through the system file cache. This might be the same kind of issue. @ideasman42, maybe we can disable this change on Windows for now, and then investigate how to fix it later? The Linux file load time is actually a great improvement I didn't expect from this change.

blender-admin commented

2019-02-22 23:21:12 +01:00

This issue was referenced by 779860d34e

This issue was referenced by 779860d34e26bedfb24904bc6261db5cbe9afdbf

Campbell Barton commented

2019-02-22 23:37:47 +01:00

Changed status from 'Open' to: 'Resolved'

Campbell Barton closed this issue

2019-02-22 23:37:47 +01:00

Campbell Barton commented

2019-02-22 23:37:47 +01:00

I'll disable this for windows - for now, this could be tuned by only applying to blocks over a certain size - however if seeking is very slow, this may still be slower.

In my own tests on a HDD (not an SSD) I saw speedups similar to Brecht's, so I didn't think it necessary to tweak.

I'll disable this for windows - for now, this could be tuned by only applying to blocks over a certain size - however if seeking is very slow, this may still be slower. In my own tests on a HDD (not an SSD) I saw speedups similar to Brecht's, so I didn't think it necessary to tweak.

Campbell Barton commented

2019-02-24 22:57:31 +01:00

Looks like this is a duplicate for #61880, re-enabling read on demand for WIN32.

Sign in to join this conversation.

No Label

Download

What's New

Blender Studio

Manual

Developers Blog

Documentation

Benchmark

Blender Conference

Development Fund

One-time Donations

Loading times of files that link in external data has become inacceptable #61855