FBX Import: Speed up geometry layers with foreach_set and numpy #104487

Thomas Barlow · 2023-03-16T21:42:09+01:00

Thomas Barlow commented

2023-03-16 21:42:09 +01:00

Replace per-element xform functions with numpy vectorized functions.

Setting a stride larger than item_size remains supported despite being unused.
Setting a stride smaller than item_size is not implemented.

The only specific change to a geometry layer is that vertex and polygon normals are now expanded to loop normals using numpy and foreach_set.

The overall speedup will vary depending on what layers are being imported, how many and which mapping is being used for each layer.
This can result in a large speedup for imports that have many layers, especially larger layers such as UVs and vertex colors.

Since this patch uses foreach_set it does mean that the new functions are limited to importing into blender data that is exposed to the Python API as bpy_prop_collection instances.
All existing imported geometry layers meet this criterion.

An .fbx, with 600,000 total loops and with 8 uv layers and 8 vertex color layers per mesh that I've been using to compare performance, goes from a 20 second import to a 3 second import with this patch (using the timings from the PerfMon utility in fbx_utils.py).

Replace per-element xform functions with numpy vectorized functions. Setting a stride larger than item_size remains supported despite being unused. Setting a stride smaller than item_size is not implemented. The only specific change to a geometry layer is that vertex and polygon normals are now expanded to loop normals using numpy and foreach_set. The overall speedup will vary depending on what layers are being imported, how many and which mapping is being used for each layer. This can result in a large speedup for imports that have many layers, especially larger layers such as UVs and vertex colors. Since this patch uses foreach_set it does mean that the new functions are limited to importing into blender data that is exposed to the Python API as bpy_prop_collection instances. All existing imported geometry layers meet this criterion. ----- An .fbx, with 600,000 total loops and with 8 uv layers and 8 vertex color layers per mesh that I've been using to compare performance, goes from a 20 second import to a 3 second import with this patch (using the timings from the PerfMon utility in fbx_utils.py).

🚀 1

Thomas Barlow requested review from Bastien Montagne 2023-03-16 21:42:55 +01:00

Bastien Montagne requested changes 2023-03-21 12:14:14 +01:00

Bastien Montagne left a comment

Patch look good, would rather have the 'utils' part of it (shared with some other patches) either put in a single one, or even in a separate PR to help with the merging process.

Also think that the imports 'where it's used' can be ignored here, this is supposed to be a startup time optimization (to avoid blender's python to import everything when starting Blender), but in practice doubt this is giving any real improvements here, since numpy is also imported in fbx_utils etc.

Patch look good, would rather have the 'utils' part of it (shared with some other patches) either put in a single one, or even in a separate PR to help with the merging process. Also think that the `import`s 'where it's used' can be ignored here, this is supposed to be a startup time optimization (to avoid blender's python to import everything when starting Blender), but in practice doubt this is giving any real improvements here, since numpy is also imported in `fbx_utils` etc.

Thomas Barlow force-pushed fbx_import_geom_layers_foreach_pr from 698a7ffc0b to f38b0c829a

2023-03-22 05:38:05 +01:00

Compare

Thomas Barlow commented

2023-03-22 05:55:55 +01:00

The imports and shared parray_as_ndarray utility function have been moved to a separate commit that has its own PR: #104499

Since there is now a numpy import at the top of import_fbx.py, I removed the extra C_SINGLE and C_UINT strings that were added to data_types.py and replaced their usages with np.single and np.uintc respectively.

The imports and shared `parray_as_ndarray` utility function have been moved to a separate commit that has its own PR: https://projects.blender.org/blender/blender-addons/pulls/104499 Since there is now a numpy import at the top of `import_fbx.py`, I removed the extra `C_SINGLE` and `C_UINT` strings that were added to `data_types.py` and replaced their usages with `np.single` and `np.uintc` respectively.

Thomas Barlow requested review from Bastien Montagne 2023-03-22 05:58:13 +01:00

Thomas Barlow force-pushed fbx_import_geom_layers_foreach_pr from f38b0c829a to 0fc186d55d

2023-03-27 01:02:50 +02:00

Compare

Thomas Barlow commented

2023-03-27 01:18:36 +02:00

Fixed a bug in blen_read_geom_check_fbx_data_length where it was returning the wrong bool value if there was either too much or too little fbx_data.

Added doscstrings to blen_read_geom_check_fbx_data_length and blen_read_geom_xform.

And, I renamed the fbx_data variables and arguments to fbx_data_np when they're expected to be a np.ndarray. The original name fbx_data is now only used when the variable is expected to be an array.array. This should make it clearer what types of arguments each function is expecting.

Fixed a bug in `blen_read_geom_check_fbx_data_length` where it was returning the wrong bool value if there was either too much or too little fbx_data. Added doscstrings to `blen_read_geom_check_fbx_data_length` and `blen_read_geom_xform`. And, I renamed the `fbx_data` variables and arguments to `fbx_data_np` when they're expected to be a `np.ndarray`. The original name `fbx_data` is now only used when the variable is expected to be an `array.array`. This should make it clearer what types of arguments each function is expecting.

Bastien Montagne force-pushed fbx_import_geom_layers_foreach_pr from 0fc186d55d to 9859e253b5

2023-04-03 14:50:17 +02:00

Compare

Bastien Montagne approved these changes 2023-04-03 14:51:47 +02:00

Bastien Montagne merged commit 9859e253b5 into main

2023-04-03 14:54:56 +02:00

Bastien Montagne deleted branch fbx_import_geom_layers_foreach_pr

2023-04-03 14:54:57 +02:00

Sign in to join this conversation.

No reviewers

No Label

Download

What's New

Blender Studio

Manual

Developers Blog

Documentation

Benchmark

Blender Conference

Development Fund

One-time Donations

FBX Import: Speed up geometry layers with foreach_set and numpy #104487