FBX export of Objects in Edit Mode while Blender is not in Edit Mode can fail. #104576

New Issue

Thomas Barlow · 2023-04-28T00:07:58+02:00

Thomas Barlow commented

2023-04-28 00:07:58 +02:00

System Information
Operating system: Windows-10-10.0.19045-SP0 64 Bits
Graphics card: NVIDIA GeForce GTX 1070 Ti/PCIe/SSE2 NVIDIA Corporation 4.5.0 NVIDIA 531.41

Blender Version
Broken: version: 3.5.1, branch: blender-v3.5-release, commit date: 2023-04-24 18:11, hash: e1ccd9d4a1d3
Worked: (newest version of Blender that worked as expected)

Addon Information
Name: FBX format (4, 37, 5)
Author: Campbell Barton, Bastien Montagne, Jens Restemeier

Short description of error
It's possible for an Object to be in Edit Mode while Blender itself is not in Edit Mode, this can be achieved by opening said Object in Edit Mode and then changing the active Object of the current View Layer to a different Object that is in Object Mode. Addons may directly set the active Object when they want to run operators on a specific Object without using a context override, doing so from Edit Mode will cause this unusual state.

The FBX exporter already has code for setting the active Object into Object Mode, but it doesn't have similar checks for any of the other Objects being exported.

Attempting an FBX export of a mesh Object that is in Edit Mode, but where the active Object is different and not in Edit Mode will often raise an Error if the mesh has UVs or Vertex Colors (or assert in a debug build blender/blender#107416).

The issues come about because some of the mesh data is not available while the mesh Object is in Edit mode.

It appears that most of the data that always exists for a mesh and is available through the older API, Mesh.vertices/Mesh.edges etc., is updated for any changes that happened while the mesh was in Edit mode as if the mesh had actually exited Edit mode. The equivalent attributes for these data do not exist at all, e.g. "position" attribute or ".select_vert" attribute.

Optional extra data such as UV layers tend to be empty or in an undefined/inconsistent state.

The export typically fails when attempting to export UVs or vertex colors:

In Blender 3.4 and earlier, the number of loops of the mesh will be the expected value, but the number of UV elements will be zero and foreach_get will fail due to expecting a zero-length array/list, but actually getting a len(loops)-length array.

In Blender 3.5, the number of loops will be the expected value and while the number of UV elements will be zero, foreach_get will still expect a len(loops)-length array/list, but foreach_get will attempt to fill the array with None, which will raise an error because None is not an acceptable value for setting in an array.array.

In Blender 3.6, there is the same situation as 3.5 where foreach_get will attempt to fill the array with None, but now numpy.ndarray are used in the FBX addon instead of array.array, which interprets None as nan, thus filling the ndarray with nan and raising an error when the uvs are checked for containing any nan values.

On debug builds of Blender 3.5 and newer, instead of setting None into the array/list, an assertion will fail.

With my planned changes to update FBX IO to use the newer attributes such as "position", this issue gets much worse because most attributes don't exist while a mesh is in Edit Mode, so even a simple cube in this unusual Edit mode state, without any UV maps or vertex colors, would fail to export correctly.

Exact steps for others to reproduce the error
The attached .blend file (authored in 3.4.1) is ready to export as .fbx and has one Cube set into Edit mode while Blender itself is in Object mode. Two scripts are included to assist with changing the active Object while in Edit mode (thus exiting Edit mode while leaving the Object in Edit mode) and for checking that the Object's state matches this unusual state that the FBX exporter has issues with.

Open the file and export as FBX, it should fail due to the UVs.
The scripts are reproduced here:

import bpy

# The 'Edit Mode Cube' should be in Edit Mode but while not being
# the active object. This puts the Object in an unusual state
# that fails to export properly.

edit_mode_cube = bpy.data.objects['Edit Mode Cube']

assert edit_mode_cube.mode == 'EDIT'
assert bpy.context.view_layer.objects.active != edit_mode_cube
assert bpy.context.mode != 'EDIT_MESH'

import bpy

# This has already been done in this test .blend, this
# script is here for convenience

other_cube = bpy.data.objects['Active Object Mode Cube']
bpy.context.view_layer.objects.active = other_cube

**System Information** Operating system: Windows-10-10.0.19045-SP0 64 Bits Graphics card: NVIDIA GeForce GTX 1070 Ti/PCIe/SSE2 NVIDIA Corporation 4.5.0 NVIDIA 531.41 **Blender Version** Broken: version: 3.5.1, branch: blender-v3.5-release, commit date: 2023-04-24 18:11, hash: `e1ccd9d4a1d3` Worked: (newest version of Blender that worked as expected) **Addon Information** Name: FBX format (4, 37, 5) Author: Campbell Barton, Bastien Montagne, Jens Restemeier **Short description of error** It's possible for an Object to be in Edit Mode while Blender itself is not in Edit Mode, this can be achieved by opening said Object in Edit Mode and then changing the active Object of the current View Layer to a different Object that is in Object Mode. Addons may directly set the active Object when they want to run operators on a specific Object without using a context override, doing so from Edit Mode will cause this unusual state. The FBX exporter already has code for setting the active Object into Object Mode, but it doesn't have similar checks for any of the other Objects being exported. Attempting an FBX export of a mesh Object that is in Edit Mode, but where the active Object is different and not in Edit Mode will often raise an Error if the mesh has UVs or Vertex Colors (or assert in a debug build https://projects.blender.org/blender/blender/issues/107416). The issues come about because some of the mesh data is not available while the mesh Object is in Edit mode. It appears that most of the data that always exists for a mesh and is available through the older API, `Mesh.vertices`/`Mesh.edges` etc., is updated for any changes that happened while the mesh was in Edit mode as if the mesh had actually exited Edit mode. The equivalent attributes for these data do not exist at all, e.g. `"position"` attribute or `".select_vert"` attribute. Optional extra data such as UV layers tend to be empty or in an undefined/inconsistent state. The export typically fails when attempting to export UVs or vertex colors: In Blender 3.4 and earlier, the number of loops of the mesh will be the expected value, but the number of UV elements will be zero and `foreach_get` will fail due to expecting a zero-length array/list, but actually getting a `len(loops)`-length array. In Blender 3.5, the number of loops will be the expected value and while the number of UV elements will be zero, `foreach_get` will still expect a `len(loops)`-length array/list, but `foreach_get` will attempt to fill the array with `None`, which will raise an error because `None` is not an acceptable value for setting in an `array.array`. In Blender 3.6, there is the same situation as 3.5 where `foreach_get` will attempt to fill the array with `None`, but now `numpy.ndarray` are used in the FBX addon instead of `array.array`, which interprets `None` as `nan`, thus filling the ndarray with `nan` and raising an error when the uvs are checked for containing any `nan` values. On debug builds of Blender 3.5 and newer, instead of setting `None` into the array/list, an assertion will fail. With my planned changes to update FBX IO to use the newer attributes such as `"position"`, this issue gets much worse because most attributes don't exist while a mesh is in Edit Mode, so even a simple cube in this unusual Edit mode state, without any UV maps or vertex colors, would fail to export correctly. **Exact steps for others to reproduce the error** The attached .blend file (authored in 3.4.1) is ready to export as .fbx and has one Cube set into Edit mode while Blender itself is in Object mode. Two scripts are included to assist with changing the active Object while in Edit mode (thus exiting Edit mode while leaving the Object in Edit mode) and for checking that the Object's state matches this unusual state that the FBX exporter has issues with. 1. Open the file and export as FBX, it should fail due to the UVs. The scripts are reproduced here: ```py import bpy # The 'Edit Mode Cube' should be in Edit Mode but while not being # the active object. This puts the Object in an unusual state # that fails to export properly. edit_mode_cube = bpy.data.objects['Edit Mode Cube'] assert edit_mode_cube.mode == 'EDIT' assert bpy.context.view_layer.objects.active != edit_mode_cube assert bpy.context.mode != 'EDIT_MESH' ``` ```py import bpy # This has already been done in this test .blend, this # script is here for convenience other_cube = bpy.data.objects['Active Object Mode Cube'] bpy.context.view_layer.objects.active = other_cube ```

NonActiveEditModeCube3_4_1.blend

104 KiB

Thomas Barlow added the

labels 2023-04-28 00:07:58 +02:00

Thomas Barlow commented

2023-04-28 00:10:50 +02:00

I will attempt a fix in the coming days. Likely either ensuring all Objects to be exported are not in Edit Mode or if that's not possible, raising an error.