FBX IO: Add utility to schedule tasks to run on separate threads #105017

Thomas Barlow · 2023-11-21T06:45:06+01:00

Thomas Barlow commented

2023-11-21 06:45:06 +01:00

Adds a class intended for processing lots of smaller CPU-bound tasks
that release the GIL. This class allows a function to be wrapped such
that calling the wrapper will schedule the wrapped function to be called
on another thread.

When multithreading is not available on the current system, the function
to wrap is returned instead, resulting in single-threaded execution, so
the calling code doesn't need to be aware of whether multithreading is
available.

The class starts its own threads as needed using a ThreadPoolExecutor
and a context manager is used to wait for its threads to shut down when
exiting the context.

Longer tasks, such as those which are IO related may be more suited to
using a ThreadPoolExecutor directly.

This patch on its own makes no changes to FBX import/export.

I intend to update #104739 to use this new functionality, but the main speedup will come from #105018 that multithreads array compression on export.

~~It's worth noting that fbx2json.py and json2fbx.py won't be able to make use of this new utility because they can't import fbx_utils due to fbx_utils importing bpy.~~
edit: The new utility has been moved into a new module fbx_utils_threading which doesn't require the bpy module.

Adds a class intended for processing lots of smaller CPU-bound tasks that release the GIL. This class allows a function to be wrapped such that calling the wrapper will schedule the wrapped function to be called on another thread. When multithreading is not available on the current system, the function to wrap is returned instead, resulting in single-threaded execution, so the calling code doesn't need to be aware of whether multithreading is available. The class starts its own threads as needed using a ThreadPoolExecutor and a context manager is used to wait for its threads to shut down when exiting the context. Longer tasks, such as those which are IO related may be more suited to using a ThreadPoolExecutor directly. This patch on its own makes no changes to FBX import/export. --- I intend to update https://projects.blender.org/blender/blender-addons/pulls/104739 to use this new functionality, but the main speedup will come from https://projects.blender.org/blender/blender-addons/pulls/105018 that multithreads array compression on export. ~~It's worth noting that `fbx2json.py` and `json2fbx.py` won't be able to make use of this new utility because they can't import `fbx_utils` due to `fbx_utils` importing `bpy`.~~ edit: The new utility has been moved into a new module `fbx_utils_threading` which doesn't require the `bpy` module.

Thomas Barlow added 1 commit 2023-11-21 06:45:09 +01:00

FBX IO: Add utility to schedule tasks to run on separate threads 8f977d1250

Adds a class intended for processing lots of smaller CPU-bound tasks
that release the GIL. This class allows a function to be wrapped such
that calling the wrapper will schedule the wrapped function to be called
on another thread.

When multithreading is not available on the current system, the function
to wrap is returned instead, resulting in single-threaded execution, so
the calling code doesn't need to be aware of whether multithreading is
available.

The class starts its own threads as needed using a ThreadPoolExecutor
and a context manager is used to wait for its threads to shut down when
exiting the context.

Longer tasks, such as those which are IO related may be more suited to
using a ThreadPoolExecutor directly.

This patch on its own makes no changes to FBX import/export.

Thomas Barlow referenced this pull request

2023-11-21 07:26:54 +01:00

FBX IO: Speed up export by multithreading array compression #105018

Thomas Barlow requested review from Bastien Montagne 2023-11-21 07:31:16 +01:00

Bastien Montagne approved these changes 2023-11-27 12:40:09 +01:00

Bastien Montagne left a comment

LGTM from quick overview of the code.

Think this could be in it's own module though (fbx_utils_thrading.py ?). That way it can also be used by non-Blender utils...

LGTM from quick overview of the code. Think this could be in it's own module though (`fbx_utils_thrading.py` ?). That way it can also be used by non-Blender utils...

Thomas Barlow added 1 commit 2023-12-03 01:03:29 +01:00

Move multithreading utils to fbx_utils_threading.py ae5f9bad40

This enables it to be used by the fbx2json.py and json2fbx.py scripts.

Thomas Barlow added 1 commit 2023-12-04 04:21:34 +01:00

Check if new threads can be created when determining if multithreading is enabled 26ce6e28f2

concurrent.futures may be available but some functionality may not be available.

To more accurately check whether multithreading is available, attempt to start a new thread using a ThreadPoolExecutor.

numpy.org's REPL (backed by Pyodide which is wasm32-emscripten) allows the module to be imported, but submitting a task (which should start a new thread) fails with a RuntimeError.

Bastien Montagne approved these changes 2023-12-04 16:40:09 +01:00

Thomas Barlow referenced this pull request

2023-12-18 03:06:31 +01:00

FBX IO: Speed up parsing by multithreading array decompression #104739

Thomas Barlow commented

2023-12-22 15:30:00 +01:00

Since this PR does nothing on its own, I'm intending to only merge it when one of the PRs that use it are also approved.

Thomas Barlow merged commit 5d4e397e3f into main

2024-01-12 21:29:22 +01:00

Thomas Barlow deleted branch fbx_multithread_utils_pr

2024-01-12 21:29:23 +01:00

Thomas Barlow referenced this issue from a commit