FBX Import: Speed up geometry with numpy #104482

Thomas Barlow · 2023-03-14T06:35:51+01:00

Thomas Barlow commented

2023-03-14 06:35:51 +01:00

Use buffers matching the C data types with foreach_set to avoid iterating and casting each element in foreach_set's C code.
Use numpy vectorized operations.
Replace _vcos_transformed_gen with numpy version, vcos_transformed.

Testing with verts/edges/polys arrays created from subdivided default cubes:
~8-45 times faster for about 1000 to 100_000 loops (len(fbx_polys)) then starts reducing at a reducing rate down to about 35 times faster after 6 million loops.

Changes made to the import of malformed .fbx files:
.fbx that have edges, but no polygon-vertex-indices will now ignore the edges, print an error and continue the import, resulting in a mesh containing only vertices. Beforehand, an IndexError would be raised when an attempt would be made to index the empty fbx_polys array.
.fbx that have a verts array with a number of elements that isn't divisible by 3 will now have the remainder stripped from the end and an error will be printed. Beforehand, a RuntimeError would be raised by mesh.vertices.foreach_set("co", fbx_verts) because fbx_verts would not contain the number of elements exactly 3 times the number of vertices.

The changes to the two listed edge cases of malformed .fbx could be changed to raise errors similar to the original code if preferred.

I kept the numpy import within the function like the other external module imports, I wasn't sure how important that was.

Use buffers matching the C data types with foreach_set to avoid iterating and casting each element in foreach_set's C code. Use numpy vectorized operations. Replace \_vcos_transformed_gen with numpy version, vcos_transformed. Testing with verts/edges/polys arrays created from subdivided default cubes: ~8-45 times faster for about 1000 to 100_000 loops (len(fbx_polys)) then starts reducing at a reducing rate down to about 35 times faster after 6 million loops. Changes made to the import of malformed .fbx files: .fbx that have edges, but no polygon-vertex-indices will now ignore the edges, print an error and continue the import, resulting in a mesh containing only vertices. Beforehand, an IndexError would be raised when an attempt would be made to index the empty fbx_polys array. .fbx that have a verts array with a number of elements that isn't divisible by 3 will now have the remainder stripped from the end and an error will be printed. Beforehand, a RuntimeError would be raised by mesh.vertices.foreach_set("co", fbx_verts) because fbx_verts would not contain the number of elements exactly 3 times the number of vertices. ----- The changes to the two listed edge cases of malformed .fbx could be changed to raise errors similar to the original code if preferred. I kept the numpy import within the function like the other external module imports, I wasn't sure how important that was.

Thomas Barlow force-pushed fbx_import_polys_verts_edges_np_pr from c1f572ad6e to c648645cfc

2023-03-14 06:37:55 +01:00

Compare

Thomas Barlow requested review from Bastien Montagne 2023-03-14 06:40:29 +01:00

Thomas Barlow force-pushed fbx_import_polys_verts_edges_np_pr from c648645cfc to 600d0d8888

2023-03-19 23:25:19 +01:00

Compare

Thomas Barlow changed title from ~~FBX Import: Speed up verts/edges/polys with numpy~~ to FBX Import: Speed up geometry with numpy

2023-03-19 23:25:40 +01:00

Thomas Barlow commented

2023-03-19 23:42:54 +01:00

I pushed some additional changes to setting smoothing and custom split normals, since they're fairly small changes and part of the same blen_read_geom function that this patch covers.

I tried a lot of different ways to see if I could make the call to Mesh.normals_split_custom_set faster, but the best I could do was tie the original implementation. For anything faster I think the only way would be if Mesh.normals_split_custom_set was updated to be able to take advantage of inputs that are buffers, similar to the foreach_get/foreach_set functions of bpy_prop_array and bpy_prop_collection.
I managed to make a custom Blender build with buffer support which then runs Mesh.normals_split_custom_set 2-3 times faster, I'll see if I can get interest from one of the Python API devs, especially since I don't have much C experience.

I pushed some additional changes to setting smoothing and custom split normals, since they're fairly small changes and part of the same `blen_read_geom` function that this patch covers. I tried a lot of different ways to see if I could make the call to `Mesh.normals_split_custom_set` faster, but the best I could do was tie the original implementation. For anything faster I think the only way would be if `Mesh.normals_split_custom_set` was updated to be able to take advantage of inputs that are buffers, similar to the `foreach_get`/`foreach_set` functions of `bpy_prop_array` and `bpy_prop_collection`. I managed to make a custom Blender build with buffer support which then runs `Mesh.normals_split_custom_set` 2-3 times faster, I'll see if I can get interest from one of the Python API devs, especially since I don't have much C experience.

Bastien Montagne requested changes 2023-03-21 12:14:28 +01:00

Bastien Montagne left a comment

Patch look good, would rather have the 'utils' part of it (shared with some other patches) either put in a single one, or even in a separate PR to help with the merging process.

Also think that the imports 'where it's used' can be ignored here, this is supposed to be a startup time optimization (to avoid blender's python to import everything when starting Blender), but in practice doubt this is giving any real improvements here, since numpy is also imported in fbx_utils etc.

Patch look good, would rather have the 'utils' part of it (shared with some other patches) either put in a single one, or even in a separate PR to help with the merging process. Also think that the `import`s 'where it's used' can be ignored here, this is supposed to be a startup time optimization (to avoid blender's python to import everything when starting Blender), but in practice doubt this is giving any real improvements here, since numpy is also imported in `fbx_utils` etc.

Thomas Barlow force-pushed fbx_import_polys_verts_edges_np_pr from 600d0d8888 to bd2d2e1cd8

2023-03-22 05:37:28 +01:00

Compare

Thomas Barlow force-pushed fbx_import_polys_verts_edges_np_pr from bd2d2e1cd8 to 07d0491966

2023-03-22 06:01:20 +01:00

Compare

Thomas Barlow commented

2023-03-22 06:03:12 +01:00

The imports and shared parray_as_ndarray utility function have been moved to a separate commit that has its own PR: #104499

I removed the reference to adding the parray_as_ndarray function from the commit message for this patch.

The imports and shared `parray_as_ndarray` utility function have been moved to a separate commit that has its own PR: https://projects.blender.org/blender/blender-addons/issues/104499 I removed the reference to adding the `parray_as_ndarray` function from the commit message for this patch.

Thomas Barlow requested review from Bastien Montagne 2023-03-22 06:03:40 +01:00

Bastien Montagne force-pushed fbx_import_polys_verts_edges_np_pr from 07d0491966 to 2438ae5c27

2023-04-03 14:55:12 +02:00

Compare

Bastien Montagne approved these changes 2023-04-03 15:07:40 +02:00

Bastien Montagne merged commit 2438ae5c27 into main

2023-04-03 15:07:49 +02:00

Bastien Montagne deleted branch fbx_import_polys_verts_edges_np_pr

2023-04-03 15:07:49 +02:00

Sign in to join this conversation.

No reviewers

No Label

Download

What's New

Blender Studio

Manual

Developers Blog

Documentation

Benchmark

Blender Conference

Development Fund

One-time Donations

FBX Import: Speed up geometry with numpy #104482