FBX IO: Search for images on separate threads #105125

Thomas Barlow · 2024-01-15T04:19:01+01:00

Thomas Barlow commented

2024-01-15 04:19:01 +01:00

This offloads IO work to find image files into separate threads.

Texture loading is moved to the start of FBX import so that the rest of
the fbx import can continue while searching for images runs in the
background.

This can reduce import times when the Image Search option of the
importer is enabled (it is enabled by default) and an imported file has
images with paths that cause a search to begin in a directory with many
nested subdirectories, which can happen when a relative path goes up
many parent directories, the relative path starting with "../../../" or
similar.

While not as common, this can also reduce import times when an imported
file references many images with file paths that start with the same
drive letter as a slower drive, such as a CD drive, or when an imported
file references an image on a drive that has been powered off due to
inactivity.

Aside from these cases, import times are expected to be the same or
slightly slower, likely due to the time spent starting threads or due to
each thread contending for the GIL for its parts that don't release the
GIL.

The implementation uses ThreadPoolExecutor and Future objects from
concurrent.futures. While unlikely, to account for
concurrent.futures not being available, simple Executor and Future
implementations are added to fbx_utils_threading.py.

The attached .fbx contains reference to a texture with a RelativeFilename of "..\\..\\..\\..\\..\\..\\..\\512test.png". This PR won't do much to the import of this specific file because it's just a default cube, but the idea of the PR is that the rest of the import can continue while the image search continues in the background.

This offloads IO work to find image files into separate threads. Texture loading is moved to the start of FBX import so that the rest of the fbx import can continue while searching for images runs in the background. This can reduce import times when the Image Search option of the importer is enabled (it is enabled by default) and an imported file has images with paths that cause a search to begin in a directory with many nested subdirectories, which can happen when a relative path goes up many parent directories, the relative path starting with "../../../" or similar. While not as common, this can also reduce import times when an imported file references many images with file paths that start with the same drive letter as a slower drive, such as a CD drive, or when an imported file references an image on a drive that has been powered off due to inactivity. Aside from these cases, import times are expected to be the same or slightly slower, likely due to the time spent starting threads or due to each thread contending for the GIL for its parts that don't release the GIL. The implementation uses `ThreadPoolExecutor` and `Future` objects from `concurrent.futures`. While unlikely, to account for `concurrent.futures` not being available, simple `Executor` and `Future` implementations are added to fbx_utils_threading.py. --- The attached .fbx contains reference to a texture with a RelativeFilename of `"..\\..\\..\\..\\..\\..\\..\\512test.png"`. This PR won't do much to the import of this specific file because it's just a default cube, but the idea of the PR is that the rest of the import can continue while the image search continues in the background.

ManyParentRelativeTest.fbx

18 KiB

Thomas Barlow added 1 commit 2024-01-15 04:19:02 +01:00

FBX IO: Search for images on separate threads 513ef96e52

This offloads some IO work to find image files into separate threads.

Texture loading is moved to the start of FBX import so that the rest of
the fbx import can continue while searching for images runs in the
background.

This can significantly reduce import times when the Image Search option
of the importer is enabled (it is enabled by default) and an imported
file has images with paths that cause a search to begin in a directory
with many nested subdirectories, which can happen when a relative path
goes up many parent directories, the relative path starting with
"../../../" or similar.

While not as common, this can also significantly reduce import times
when an imported file references many images with file paths that start
with the same drive letter as a slower drive, such as a CD drive.

Aside from these cases, import times are expected to be the same or
slightly slower, likely due to the time spent starting threads or due to
each thread contending for the GIL for its parts that don't release the
GIL.

Thomas Barlow force-pushed fbx_import_image_search_threaded from 513ef96e52 to 6518667f71

2024-02-01 02:19:40 +01:00

Compare

Thomas Barlow changed title from ~~WIP: FBX IO: Search for images on separate threads~~ to FBX IO: Search for images on separate threads

2024-02-01 02:26:14 +01:00

Thomas Barlow added 1 commit 2024-02-01 03:25:31 +01:00

Remove default executor argument for import_fbx.load() f9f3507c6c

Add-ons shouldn't be calling `import_fbx` functions directly and this
helps separate the `executor` argument from the operator properties
which are passed to `import_fbx.load()` as keyword-arguments.

This pull request can be merged automatically.

This branch is out-of-date with the base branch

You are not authorized to merge this pull request.

View command line instructions.

Checkout

From your project repository, check out a new branch and test the changes.

git fetch -u fbx_import_image_search_threaded:Mysteryem-fbx_import_image_search_threaded

git checkout Mysteryem-fbx_import_image_search_threaded

Sign in to join this conversation.

No reviewers

No Label

Download

What's New

Blender Studio

Manual

Developers Blog

Documentation

Benchmark

Blender Conference

Development Fund

One-time Donations

FBX IO: Search for images on separate threads #105125

Checkout