FBX IO: Speed up export by multithreading array compression #105018

Thomas Barlow · 2023-11-21T07:26:54+01:00

Thomas Barlow commented

2023-11-21 07:26:54 +01:00

zlib.compress() releases the GIL so can be multithreaded.

This patch adds a context manager to FBXElem that temporarily enables
multithreading of the compression of added arrays, by using the recently
added utility to schedule CPU-bound tasks to run on separate threads.

On my Ryzen 7 3800x, exporting non-animated rigged humanoid models
typically results in about a 1.2 to 1.4 times faster export.

Exporting only simple geometry, such as many subdivided default cubes,
can be about 2 to 3 times faster.

The multithreading is also enabled for the json2fbx.py script.

No changes are expected to the contents of exported files.

I'm open to ideas of better ways to temporarily enable the multithreading in FBXElem.

My only other idea was to set FBXElem._add_compressed_array_helper_multi to None in FBXElem's class definition and temporarily set it to the wrapped function that schedules the compression tasks when needed. _add_array_helper would then be updated to always check FBXElem._add_compressed_array_helper_multi is not None when it needs to compress an array. If the check passes it'll use FBXElem._add_compressed_array_helper_multi and if the check fails, it'll do the compression itself.

This PR depends on #105017 (its changes are not included in this PR)

zlib.compress() releases the GIL so can be multithreaded. This patch adds a context manager to FBXElem that temporarily enables multithreading of the compression of added arrays, by using the recently added utility to schedule CPU-bound tasks to run on separate threads. On my Ryzen 7 3800x, exporting non-animated rigged humanoid models typically results in about a 1.2 to 1.4 times faster export. Exporting only simple geometry, such as many subdivided default cubes, can be about 2 to 3 times faster. The multithreading is also enabled for the `json2fbx.py` script. No changes are expected to the contents of exported files. --- I'm open to ideas of better ways to temporarily enable the multithreading in FBXElem. My only other idea was to set `FBXElem._add_compressed_array_helper_multi` to `None` in `FBXElem`'s class definition and temporarily set it to the wrapped function that schedules the compression tasks when needed. `_add_array_helper` would then be updated to always check `FBXElem._add_compressed_array_helper_multi is not None` when it needs to compress an array. If the check passes it'll use `FBXElem._add_compressed_array_helper_multi` and if the check fails, it'll do the compression itself. This PR depends on https://projects.blender.org/blender/blender-addons/pulls/105017 (its changes are not included in this PR)

❤️ 1 🚀 1

Thomas Barlow added 2 commits 2023-11-21 07:26:57 +01:00

FBX IO: Multithread exported array compression 7864ccac7d

zlib.compress releases the GIL so can be multithreaded.

This patch adds a context manager to FBXElem that temporarily enables
multithreading of the compression of added arrays, by using the recently
added utility to schedule CPU-bound tasks to run on separate threads.

On my Ryzen 7 7800x, exporting non-animated rigged humanoid models
typically results in about a 1.2 to 1.4 times faster export.

Exporting only simple geometry, such as many subdivided default cubes,
can be about 2 to 3 times faster.

No changes are expected to the contents of exported files.

fix json2fbx.py 5aa031177f

Thomas Barlow referenced this pull request

2023-11-21 07:31:02 +01:00

FBX IO: Add utility to schedule tasks to run on separate threads #105017

Thomas Barlow requested review from Bastien Montagne 2023-11-21 07:31:34 +01:00

Thomas Barlow added 2 commits 2023-12-03 01:07:15 +01:00

Update for multithreading utils being moved to fbx_utils_threading.py d32879355d

Enable multithreaded array compression in json2fbx.py 5112fd91f5

Bastien Montagne approved these changes 2024-01-10 18:50:22 +01:00

Bastien Montagne left a comment

LGTM.

Actually, I think I prefer the context manager solution to control when the threaded compression is enabled.

Implementation might be a bit verbose and convoluted, but the usage of it looks much cleaner to me?

Unless I do not get exactly how your other idea would work.

LGTM. Actually, I think I prefer the context manager solution to control when the threaded compression is enabled. Implementation might be a bit verbose and convoluted, but the usage of it looks much cleaner to me? Unless I do not get exactly how your other idea would work.

Thomas Barlow commented

2024-01-10 19:05:58 +01:00

The other idea was to avoid entirely swapping out existing methods. Rather than swapping out the method, the existing method would be updated to always check for whether a _add_compressed_array_helper_multi attribute on the class is not None and then call that function instead of the existing non-threaded code. The context manager, in that case, would temporarily set the _add_compressed_array_helper_multi attribute on the class to the threaded function.

The other idea was to avoid entirely swapping out existing methods. Rather than swapping out the method, the existing method would be updated to always check for whether a `_add_compressed_array_helper_multi` attribute on the class is not `None` and then call that function instead of the existing non-threaded code. The context manager, in that case, would temporarily set the `_add_compressed_array_helper_multi` attribute on the class to the threaded function.

Thomas Barlow added 2 commits 2024-01-12 21:36:13 +01:00

Merge branch 'main' into fbx_multithread_array_compression_pr aca36c7ba0

Increase FBX IO version 1bf83e19ba

Thomas Barlow merged commit 2791f29806 into main

2024-01-12 21:39:20 +01:00

Thomas Barlow deleted branch fbx_multithread_array_compression_pr

2024-01-12 21:39:21 +01:00

Thomas Barlow referenced this issue from a commit

2024-01-12 21:39:22 +01:00

FBX IO: Speed up export by multithreading array compression

Sign in to join this conversation.

No reviewers

No Label

Download

What's New

Blender Studio

Manual

Developers Blog

Documentation

Benchmark

Blender Conference

Development Fund

One-time Donations

FBX IO: Speed up export by multithreading array compression #105018