Speed up FBX export of UVs with numpy #104453

Thomas Barlow · 2023-02-28T00:59:29+01:00

Thomas Barlow commented

2023-02-28 00:59:29 +01:00

Use buffers matching the C types of the data in foreach_get to avoid having to iterate and cast every single element in the C foreach_getset function.

Uses sorting-based uniqueness in the same way as numpy.unique, but avoids creating an array of triplets by keeping the uvs and vertex indices arrays separate and splits up the work needed for both arrays, since the vertex indices are the same for every uv layer, so only need to be sorted once.

Adds a specific check and exception for NaN values since the original code would also raise an exception when NaN values were encountered in uvs.

About 10-20 times faster with 1 uv layer, with further speedup the more uv layers there are. It is possible to export uvs in a similar manner as the original code, but with only a single iteration rather than the original two iterations, in which case, this patch is about 6-12 times faster with 1 uv layer.

This does change the exported fbx because the uvs are sorted (albeit not in order because they are viewed as a different type for sorting performance), causing the "UV" array to be in a different order and the "UVIndex" array to match the new ordering, but the full array of uvs reconstructed by indexing the "UV" array with each "UVIndex" in sequence does remain the same, so while the exported file may be different, the result of importing the different file will still be the same.

This is easily the most complicated of the numpy patches due to the requirement of having to base the uniqueness on triplets of (uv, vertex_idx), but also because the vertex indices are the same for each uv layer so it would be wasteful to fully re-process the vertex indices for every uv layer.

Combining the uvs and vertex indices into one big structured array takes up time and space, and finding the unique triplets with numpy.unique rather than fbx_utils.fast_first_axis_unique actually ended up slower than the original code modified to only iterate through _uvtuples_gen once.

The t_lvidx array is data that is also used by the export of polygon indices and edges/vertex colors, under the name t_lvi. t_lvi could be therefore be re-used for exporting UVs, though for UVs it needs to exclude any extra loops added for loose edges, which can be done with t_lvidx = t_lvi[:len(me.loops)].
See #104451 and #104454

The error is raised when NaN UVs are encountered based on discussion in https://archive.blender.org/developer/D17058 as otherwise this patch would hide the existence of NaN values in UVs. The error message may want to be changed and/or be raised in a different way.

This patch depends on
#104447
I wasn't sure if I should include its commit in this PR, I have included it for now.

Use buffers matching the C types of the data in foreach_get to avoid having to iterate and cast every single element in the C foreach_getset function. Uses sorting-based uniqueness in the same way as numpy.unique, but avoids creating an array of triplets by keeping the uvs and vertex indices arrays separate and splits up the work needed for both arrays, since the vertex indices are the same for every uv layer, so only need to be sorted once. Adds a specific check and exception for NaN values since the original code would also raise an exception when NaN values were encountered in uvs. About 10-20 times faster with 1 uv layer, with further speedup the more uv layers there are. It is possible to export uvs in a similar manner as the original code, but with only a single iteration rather than the original two iterations, in which case, this patch is about 6-12 times faster with 1 uv layer. This does change the exported fbx because the uvs are sorted (albeit not in order because they are viewed as a different type for sorting performance), causing the "UV" array to be in a different order and the "UVIndex" array to match the new ordering, but the full array of uvs reconstructed by indexing the "UV" array with each "UVIndex" in sequence does remain the same, so while the exported file may be different, the result of importing the different file will still be the same. ----- This is easily the most complicated of the numpy patches due to the requirement of having to base the uniqueness on triplets of (uv, vertex_idx), but also because the vertex indices are the same for each uv layer so it would be wasteful to fully re-process the vertex indices for every uv layer. Combining the uvs and vertex indices into one big structured array takes up time and space, and finding the unique triplets with numpy.unique rather than fbx_utils.fast_first_axis_unique actually ended up slower than the original code modified to only iterate through `_uvtuples_gen` once. The `t_lvidx` array is data that is also used by the export of polygon indices and edges/vertex colors, under the name `t_lvi`. `t_lvi` could be therefore be re-used for exporting UVs, though for UVs it needs to exclude any extra loops added for loose edges, which can be done with `t_lvidx = t_lvi[:len(me.loops)]`. See https://projects.blender.org/blender/blender-addons/pulls/104451 and https://projects.blender.org/blender/blender-addons/pulls/104454 The error is raised when NaN UVs are encountered based on discussion in https://archive.blender.org/developer/D17058 as otherwise this patch would hide the existence of NaN values in UVs. The error message may want to be changed and/or be raised in a different way. This patch depends on https://projects.blender.org/blender/blender-addons/pulls/104447 I wasn't sure if I should include its commit in this PR, I have included it for now.

Thomas Barlow referenced this pull request

2023-02-28 01:18:43 +01:00

Speed up FBX export of polygon indices and edges with numpy #104451

Thomas Barlow referenced this pull request

2023-02-28 01:20:11 +01:00

Speed up FBX export of vertex colors with numpy #104454

Thomas Barlow requested review from Bastien Montagne 2023-02-28 01:22:13 +01:00

Bastien Montagne force-pushed fbx_numpy_uvs_pr_standalone from 469c54650f to 65a1a580e9

2023-02-28 18:07:41 +01:00

Compare

Bastien Montagne referenced this issue from a commit

2023-03-06 15:17:08 +01:00