FBX Import: Speed up parsing by reading entire subtrees into BytesIO #104822

Merged
Bastien Montagne merged 1 commits from Mysteryem/blender-addons:fbx_subtrees_bytesio_pr into main 2023-08-09 10:43:56 +02:00

1 Commits

Author SHA1 Message Date
5e45e449cd FBX Import: Speed up parsing by reading entire subtrees into BytesIO
Python's default BufferedReader used with open() has to get the raw
stream position from the OS on every tell() call. This is about 10 times
slower than the tell() function of BytesIO objects. Because the number
of bytes in each FBX element's subtree is known ahead of time, entire
subtrees of bytes can be read at once and exposed through IO API as
BytesIO for a performance increase.

The "Objects" element's subtree is an exception and is not read all at
once because it usually makes up the vast majority of the file size.
Reading what could be a huge number of bytes into memory at once could
impact performance on systems with low free memory, especially because
those bytes won't be freed until they have been entirely parsed.

A small amount of refactoring has also been done to reduce the number of
required tell() calls.

This reduces the time taken to parse fbx files to about 88% to 90% of
the original time in most cases. Array decompression makes up about half
of the time taken currently, but this could be multithreaded, in which
case, this patch would reduce the time to about 75% to 80% instead.

Imported files containing lots of very small or non-nested subtrees
within the "Objects" element's subtree won't see much, if any effect
from this patch, which can happen with bone animations with many
animated bones.
2023-08-09 02:36:24 +01:00