FBX Import: Speed up parsing by reading entire subtrees into BytesIO #104822

Thomas Barlow · 2023-08-06T18:33:20+02:00

Thomas Barlow commented

2023-08-06 18:33:20 +02:00

Python's default BufferedReader used with open() has to get the raw
stream position from the OS on every tell() call. This is about 10 times
slower than the tell() function of BytesIO objects. Because the number
of bytes in each FBX element's subtree is known ahead of time, entire
subtrees of bytes can be read at once and exposed through IO API as
BytesIO for a performance increase.

The "Objects" element's subtree is an exception and is not read all at
once because it usually makes up the vast majority of the file size.
Reading what could be a huge number of bytes into memory at once could
impact performance on systems with low free memory, especially because
those bytes won't be freed until they have been entirely parsed.

A small amount of refactoring has also been done to reduce the number of
required tell() calls.

This reduces the time taken to parse fbx files to about 88% to 90% of
the original time in most cases. Array decompression makes up about half
of the time taken currently, but this could be multithreaded, in which
case, this patch would reduce the time to about 75% to 80% instead.

Imported files containing lots of very small or non-nested subtrees
within the "Objects" element's subtree won't see much, if any effect
from this patch, which can happen with bone animations with many
animated bones.

Python's default BufferedReader used with open() has to get the raw stream position from the OS on every tell() call. This is about 10 times slower than the tell() function of BytesIO objects. Because the number of bytes in each FBX element's subtree is known ahead of time, entire subtrees of bytes can be read at once and exposed through IO API as BytesIO for a performance increase. The "Objects" element's subtree is an exception and is not read all at once because it usually makes up the vast majority of the file size. Reading what could be a huge number of bytes into memory at once could impact performance on systems with low free memory, especially because those bytes won't be freed until they have been entirely parsed. A small amount of refactoring has also been done to reduce the number of required tell() calls. This reduces the time taken to parse fbx files to about 88% to 90% of the original time in most cases. Array decompression makes up about half of the time taken currently, but this could be multithreaded, in which case, this patch would reduce the time to about 75% to 80% instead. Imported files containing lots of very small or non-nested subtrees within the "Objects" element's subtree won't see much, if any effect from this patch, which can happen with bone animations with many animated bones.

Thomas Barlow requested review from Bastien Montagne 2023-08-06 19:10:15 +02:00

Bastien Montagne requested changes 2023-08-08 16:52:23 +02:00

Bastien Montagne left a comment

besides minor typo in comment noted below, LGTM, thanks!

And as usual, thanks for the detailed explanations in PR and comments!

besides minor typo in comment noted below, LGTM, thanks! And as usual, thanks for the detailed explanations in PR and comments!

io_scene_fbx/parse_fbx.py Outdated

						
				@ -132,0 +132,4 @@

				    if pos < local_end_offset:

				        # The default BufferedReader used when `open()`-ing files in 'rb' mode has to get the raw stream position from

				        # the OS every time its tell() function is called. This is about 10 times slower than the tell() function of

				        # BytesIO objects, so reading chunks of bytes from the file into memory as once and exposing them through

Bastien Montagne commented

2023-08-08 16:42:42 +02:00

Typo: ...into memory _at_ once... ;)

Typo: `...into memory _at_ once...` ;)

Thomas Barlow force-pushed fbx_subtrees_bytesio_pr from 08476de4a4 to 5e45e449cd

2023-08-09 03:39:30 +02:00

Compare

Thomas Barlow requested review from Bastien Montagne 2023-08-09 03:46:25 +02:00

Thomas Barlow commented

2023-08-09 04:10:59 +02:00

The typo in the comment has been corrected.

Bastien Montagne approved these changes 2023-08-09 10:43:44 +02:00

Bastien Montagne referenced this issue from a commit

2023-08-09 10:43:56 +02:00

FBX Import: Speed up parsing by reading entire subtrees into BytesIO

Bastien Montagne merged commit 890e51e769 into main

2023-08-09 10:43:56 +02:00

Bastien Montagne deleted branch fbx_subtrees_bytesio_pr

2023-08-09 10:43:56 +02:00

Sign in to join this conversation.

No reviewers

No Label

Download

What's New

Blender Studio

Manual

Developers Blog

Documentation

Benchmark

Blender Conference

Development Fund

One-time Donations

FBX Import: Speed up parsing by reading entire subtrees into BytesIO #104822