FBX IO: Speed up animation export using NumPy #104884

Thomas Barlow · 2023-09-14T03:02:13+02:00

Thomas Barlow commented

2023-09-14 03:02:13 +02:00

The current intermediate data structure used to store baked animation
curve keyframes is a list of tuples where there is one tuple for each
frame and those tuples contain the frame time, a list of values and a
list of bools indicating which values on that frame should be written.
Represented through type hints it would be
list[tuple[float, list[float], list[bool]]].

Constructing this data structure and writing it out to arrays that can
be written to an FBX file can take a surprisingly large amount of the
export time.

This patch splits the data structure into 3 separate arrays, an array of
frame times (now immediately converted to FBX time), an array of values
and an array of bools.

An immediate benefit to this is that every animation curve created by a
single fbx_animations_do call has the same frame times, so they can
share the same array of times.

The values for every frame are iterated into one large array in a fixed
order. Using this fixed order, NumPy is used to create views into that
array for the values of each animation curve.

By having entire arrays of values, NumPy can then be used to efficiently
convert from Blender values to FBX values, such as converting from
radians to degrees.

Writing out the final data is also much faster because NumPy can
efficiently combine the arrays of times, values and bools to produce
final arrays of times and values for each curve, which are immediately
ready to be written to the FBX file.

An export of a 1523 frame animation of a 65 bone armature animating 47
of the bones (almost entirely rotation animation) goes from a 4.1s
export to a 2.9s export for me with this patch.

An export of a 10000 frame animation of a shape key activating on a
default cube, with "Simplify" set to 0.0, "NLA Strips" disabled and "All
Actions" disabled, goes from a 0.79s export to a 0.56s export for me
with this patch.

There are no expected changes to the contents of exported files with
this patch.

This was originally supposed to be a patch to speed up the simplification process of exported animation curves, which needed the intermediate data structure to be separate arrays. Profiling revealed that the change to the data structure increased performance far more than the change to the simplification process, so the data structure change is now its own patch (this patch) and the simplification process patch will come later. I have thus attempted to keep AnimationCurveNodeWrapper.simplify as similar to before as possible while still accounting for the change to the intermediate data structure.

The current intermediate data structure used to store baked animation curve keyframes is a list of tuples where there is one tuple for each frame and those tuples contain the frame time, a list of values and a list of bools indicating which values on that frame should be written. Represented through type hints it would be `list[tuple[float, list[float], list[bool]]]`. Constructing this data structure and writing it out to arrays that can be written to an FBX file can take a surprisingly large amount of the export time. This patch splits the data structure into 3 separate arrays, an array of frame times (now immediately converted to FBX time), an array of values and an array of bools. An immediate benefit to this is that every animation curve created by a single `fbx_animations_do` call has the same frame times, so they can share the same array of times. The values for every frame are iterated into one large array in a fixed order. Using this fixed order, NumPy is used to create views into that array for the values of each animation curve. By having entire arrays of values, NumPy can then be used to efficiently convert from Blender values to FBX values, such as converting from radians to degrees. Writing out the final data is also much faster because NumPy can efficiently combine the arrays of times, values and bools to produce final arrays of times and values for each curve, which are immediately ready to be written to the FBX file. An export of a 1523 frame animation of a 65 bone armature animating 47 of the bones (almost entirely rotation animation) goes from a 4.1s export to a 2.9s export for me with this patch. An export of a 10000 frame animation of a shape key activating on a default cube, with "Simplify" set to 0.0, "NLA Strips" disabled and "All Actions" disabled, goes from a 0.79s export to a 0.56s export for me with this patch. There are no expected changes to the contents of exported files with this patch. --- This was originally supposed to be a patch to speed up the simplification process of exported animation curves, which needed the intermediate data structure to be separate arrays. Profiling revealed that the change to the data structure increased performance far more than the change to the simplification process, so the data structure change is now its own patch (this patch) and the simplification process patch will come later. I have thus attempted to keep `AnimationCurveNodeWrapper.simplify` as similar to before as possible while still accounting for the change to the intermediate data structure.

Thomas Barlow added 10 commits 2023-09-14 03:02:17 +02:00

Initial working version 51ba7d6227

Revert to using bool(self) in force_keep check 3f9b11c9cb

Comment updates and rename variables for better consistency 4232afbc6a

Simplify iterables and p_rots in frame_values_gen 7c7ab21973

Make curves unpacked iteration variable names consistent 892a192b6b

fbx_animations_do comments and non-functional refactoring b3d4aa8d64

Restore original simplify code 5af95962e0

This will be slower, but means the patch can be split into two parts.

Remove unused real_currframes iteration from frame_values_gen 67b97a59a4

Convert from Blender to FBX time once per fbx_animations_do instead of for every exported curve 46062a745f

Refactor: Deduplicate animation simplifying and final data creation d3febb7602

Thomas Barlow requested review from Bastien Montagne 2023-09-14 03:03:54 +02:00

Bastien Montagne approved these changes 2023-09-18 15:54:00 +02:00

Bastien Montagne left a comment

General described changes make sense to me, and give nice performance improvements!

Will trust implementation details as usual now.

General described changes make sense to me, and give nice performance improvements! Will trust implementation details as usual now.

Thomas Barlow added 2 commits 2023-09-19 01:06:25 +02:00

Merge branch 'main' into fbx_anim_export_numpy_intermediate 4181656c76

Increase FBX IO version a634c1b7fb

Thomas Barlow merged commit 73c65b9a44 into main

2023-09-19 01:13:25 +02:00

Thomas Barlow deleted branch fbx_anim_export_numpy_intermediate

2023-09-19 01:13:25 +02:00

Thomas Barlow referenced this issue from a commit

2023-09-19 01:13:26 +02:00

FBX IO: Speed up animation export using NumPy

Sign in to join this conversation.

No reviewers

No Label

Download

What's New

Blender Studio

Manual

Developers Blog

Documentation

Benchmark

Blender Conference

Development Fund

One-time Donations

FBX IO: Speed up animation export using NumPy #104884