FBX In-between Blend Shapes support #104698

New Issue

Thomas Barlow · 2023-06-16T17:53:34+02:00

Thomas Barlow commented

2023-06-16 17:53:34 +02:00

This is a ticket to discuss ideas on how in-between Blend Shapes support could be added to the FBX IO Addon and could be seen as a continuation of #90382.

This is something that I would intend to work on.

Background info

Where each Blender Shape Key (FBXBlendShapeChannel) has only a single set of vertex coordinates (FBXShape), FBX allows for a single Shape Key to have multiple. This is known as 'In-between Blend Shapes' or 'Progressive Blend'.

Each Shape connected to the BlendShapeChannel is assigned a Deform Percentage from the BlendShapeChannel's FullWeights array. As the DeformPercent of the BlendShapeChannel (.value of a ShapeKey) changes from 0% to 100%, the mesh shape interpolates between each Shape according to its DeformPercent value. This is very similar to Blender's Absolute Shape Keys, where each Shape Key has a .frame and the Shape Keys are interpolated between based on an Evaluation Time, but it's entirely contained within a single Shape Key.

Current `FullWeights` support

Blender currently uses the FullWeights array to store the vertex weights of the Shape Key's Vertex Group and will create a Vertex Group from the FullWeights when importing if the FullWeights contain a value other than 100.0.

This does not seem to be correct usage of FullWeights.

Because the Blender exported BlendShapeChannels have a single Shape, Unity appears to read the first index of FullWeights and sets that as the value that fully activates the BlendShapeChannel. Aside from the FullWeights not being used like vertex weights at all, this breaks animation of that Blend Shape in Unity because an animation that would fully enable the Blend Shape would set the Blend Shape's value to 100, but the value that fully activates the Blend Shape has been set to the first value in FullWeights.

Running a Blender exported .fbx through FBX Converter 2013 (to FBX 2013) also removes the excess FullWeights, leaving only the first value of the array in the converted output.

The import of any .fbx with more than one Shape per BlendShapeChannel currently fails because the FullWeights array will be the same length as the number of Shapes and not the same length as the number of indices in the Shape: #84111

Maintaining Vertex Group support without using `FullWeights`

If we are to stop using FullWeights to store Vertex Weights, we'll need another way to store that data in the .fbx:

The vertex group weights could be set into an extra array that is directly added to each Shape alongside its Indexes, Normals and Vertices. This extra array would likely be discarded by any FBX importers other than Blender, but that wouldn't be much different from the current behaviour if most software is discarding the extra values in FullWeights.
It should be possible to add FbxLayerElementUserData layers to the Mesh that specify the vertex weights for individual shape keys. I don't currently know how these layers are set up, but it appears you can have as many custom arrays as you want per layer, but two extra arrays are required that provide the types and names of each custom array. A potential problem with this is that FBX shape keys are sparse (only the data for the vertex indices used are included), but the UserData layer I don't think supports it., It wouldn't be as good as sparse data, but IndexToDirect would keep the file size down compared to Direct when there are many vertices with the same weights.
Custom properties are usually a single Number or Vector or similar, but if an array can be added, then the array of vertex weights could be added to each Shape's Properties70. If it comes down to it, there's always the option to write the Vertex Weights to a Blob of arbitrary binary data, since that's a supported property type.
Modern-style Shapes added in FBX 2020 support modifying other data layers such as UVs and Vertex Colors. The documentation doesn't specify that FbxLayerElementUserData layers are supported, but that doesn't mean we couldn't try adding them. The main issue with this is that it's an FBX 2020+ feature and the FBX IO addon currently has no support for Modern-style Shapes.

To support importing older Blender exported FBX, the importer would have to fall back to importing vertex weights from FullWeights when the FullWeights array has the same number of elements as the Indexes array.

Simple partial import support of in-between shapes

With a bit of refactoring to group imported Shapes by their BlendShapeChannels, it's possible to modify the existing code to discard all but the last Shape when importing.
This would enable the importing of .fbx containing in-between shapes, at the cost of losing the other Shapes.

Full IO support of in-between shapes

Without many changes to Blender itself (there are a few feature requests such as https://blender.community/c/rightclickselect/wSfbbc), the only reasonable way I can see to support in-between shapes is to import them as separate shape keys and combine them back together at export time.

Even doing that, there's a hurdle to overcome because Shape Keys are not ID types and I don't see any way to add custom properties to them to store their DeformPercent.
Storing the DeformPercents in a CollectionProperty on their .id_data, the Key, isn't particularly useful either, because Shape Keys can be renamed or re-ordered, at which point, the data in the CollectionProperty would no longer match the Shape Keys.

The only ideas I have are to either re-use an existing property (value or slider_max) or store the data in drivers.

Using an existing Shape Key property

This idea would import all but the last Shape of the BlendShapeChannel with a specific prefix recognised by the FBX IO addon, e.g. _FBX_IB. These shapes would then be placed before the last Shape in the Shape Keys list.

To store the DeformPercent of each in-between shape, I have two ideas:

Mute all in-between shapes and use their .value as the DeformPercent. This has the advantage that the .value shows in the Shape Keys list, but also means the operator to clear shape key values can't be used without losing the DeformPercents.
Use their .slider_max as the DeformPercent. This is more hidden away, but survives the operator to clear shape key values.
Both options are unfortunately disruptive to viewing the shape key in Object mode, and a custom operator would be needed to preview the blending between the shapes (swapping to Absolute Shape Keys is close, but the blending time between each shape key would not match and the blend from the 'Basis' to the first in-between shape would not be present).

With either option, the next shape key in the list not prefixed by _FBX_IB would always be assumed to have a DeformPercent of 100%.

For the conversion from .value or .slider_max to FullWeights value, the value would be multiplied by 100 at export time and clamped between 0.0 and 100.0.

I put together a rough proof-of-concept using the idea of storing the DeformPercent in slider_max and using an operator to view/edit the in-between shapes. For an actual implementation I would spend more time on the UI and probably go with a UI List to display each of the shapes and re-order them according to the values of their DeformPercents.

Using drivers

When a BlendShapeChannel has multiple Shapes, import all of them as separate Shape Keys, then either add an extra Shape Key as a controller that does nothing on its own, but use its .value to drive the .value of each of the Shape Keys according to their FullWeights values, or add a Float Custom Property to the Mesh and use its value to drive each of the Shape Keys.

This results in a really nice way to see each of the in-between shapes blend at the correct percentages, but adding, modifying, viewing or removing the FullWeights values becomes more complicated because they're tied to the drivers. Some custom UI in the Driver Editor or Shape Key Specials Menu would be a necessity and I think the addon code would be a little more complicated because it would have to figure out which in-between shapes belong to which Shape Key by looking through the drivers and then extract the FullWeights values from the drivers.

The drivers themselves are fairly simple, requiring only two or three points with either linear interpolation or vector handles. Using a Limits modifier on each driver to clamp the driven values to [0-1] can help too.

These videos show some existing implementations of using drivers to achieve in-between shapes:
https://www.youtube.com/watch?v=De3fiWcQnpA (uses an extra shape key to drive the in-between shapes)
https://www.youtube.com/watch?v=A6DRDk_MDBY (uses a custom property to drive the in-between shapes)

This is a ticket to discuss ideas on how in-between Blend Shapes support could be added to the FBX IO Addon and could be seen as a continuation of https://projects.blender.org/blender/blender-addons/issues/90382. This is something that I would intend to work on. ## Background info Where each Blender Shape Key ([FBXBlendShapeChannel](https://help.autodesk.com/view/FBX/2020/ENU/?guid=FBX_Developer_Help_cpp_ref_class_fbx_blend_shape_channel_html)) has only a single set of vertex coordinates ([FBXShape](https://help.autodesk.com/view/FBX/2020/ENU/?guid=FBX_Developer_Help_cpp_ref_class_fbx_shape_html)), FBX allows for a single Shape Key to have multiple. This is known as 'In-between Blend Shapes' or 'Progressive Blend'. Each Shape connected to the BlendShapeChannel is assigned a Deform Percentage from the BlendShapeChannel's `FullWeights` array. As the DeformPercent of the BlendShapeChannel (`.value` of a `ShapeKey`) changes from 0% to 100%, the mesh shape interpolates between each Shape according to its DeformPercent value. This is very similar to Blender's Absolute Shape Keys, where each Shape Key has a `.frame` and the Shape Keys are interpolated between based on an Evaluation Time, but it's entirely contained within a single Shape Key. ## Current `FullWeights` support Blender currently uses the `FullWeights` array to store the vertex weights of the Shape Key's Vertex Group and will create a Vertex Group from the `FullWeights` when importing if the `FullWeights` contain a value other than 100.0. This does not seem to be correct usage of `FullWeights`. Because the Blender exported BlendShapeChannels have a single Shape, Unity appears to read the first index of `FullWeights` and sets that as the value that fully activates the BlendShapeChannel. Aside from the `FullWeights` not being used like vertex weights at all, this breaks animation of that Blend Shape in Unity because an animation that would fully enable the Blend Shape would set the Blend Shape's value to 100, but the value that fully activates the Blend Shape has been set to the first value in `FullWeights`. Running a Blender exported .fbx through FBX Converter 2013 (to FBX 2013) also removes the excess `FullWeights`, leaving only the first value of the array in the converted output. The import of any .fbx with more than one Shape per BlendShapeChannel currently fails because the `FullWeights` array will be the same length as the number of Shapes and not the same length as the number of indices in the Shape: https://projects.blender.org/blender/blender-addons/issues/84111 ## Maintaining Vertex Group support without using `FullWeights` If we are to stop using `FullWeights` to store Vertex Weights, we'll need another way to store that data in the .fbx: 1) The vertex group weights could be set into an extra array that is directly added to each Shape alongside its `Indexes`, `Normals` and `Vertices`. This extra array would likely be discarded by any FBX importers other than Blender, but that wouldn't be much different from the current behaviour if most software is discarding the extra values in `FullWeights`. 1) It should be possible to add [FbxLayerElementUserData](https://help.autodesk.com/view/FBX/2020/ENU/?guid=FBX_Developer_Help_cpp_ref_class_fbx_layer_element_user_data_html) layers to the Mesh that specify the vertex weights for individual shape keys. I don't currently know how these layers are set up, but it appears you can have as many custom arrays as you want per layer, but two extra arrays are required that provide the types and names of each custom array. A potential problem with this is that FBX shape keys are sparse (only the data for the vertex indices used are included), but the UserData layer I don't think supports it., It wouldn't be as good as sparse data, but `IndexToDirect` would keep the file size down compared to Direct when there are many vertices with the same weights. 1) Custom properties are usually a single `Number` or `Vector` or similar, but if an array can be added, then the array of vertex weights could be added to each Shape's Properties70. If it comes down to it, there's always the option to write the Vertex Weights to a Blob of arbitrary binary data, since that's a supported property type. 1) [Modern-style Shapes](https://help.autodesk.com/view/FBX/2020/ENU/?guid=FBX_Developer_Help_cpp_ref_class_fbx_shape_html) added in FBX 2020 support modifying other data layers such as UVs and Vertex Colors. The documentation doesn't specify that FbxLayerElementUserData layers are supported, but that doesn't mean we couldn't try adding them. The main issue with this is that it's an FBX 2020+ feature and the FBX IO addon currently has no support for Modern-style Shapes. To support importing older Blender exported FBX, the importer would have to fall back to importing vertex weights from `FullWeights` when the `FullWeights` array has the same number of elements as the `Indexes` array. ## Simple partial import support of in-between shapes With a bit of refactoring to group imported Shapes by their BlendShapeChannels, it's possible to modify the existing code to discard all but the last Shape when importing. This would enable the importing of .fbx containing in-between shapes, at the cost of losing the other Shapes. ## Full IO support of in-between shapes Without many changes to Blender itself (there are a few feature requests such as https://blender.community/c/rightclickselect/wSfbbc), the only reasonable way I can see to support in-between shapes is to import them as separate shape keys and combine them back together at export time. Even doing that, there's a hurdle to overcome because Shape Keys are not ID types and I don't see any way to add custom properties to them to store their DeformPercent. Storing the DeformPercents in a CollectionProperty on their `.id_data`, the `Key`, isn't particularly useful either, because Shape Keys can be renamed or re-ordered, at which point, the data in the CollectionProperty would no longer match the Shape Keys. The only ideas I have are to either re-use an existing property (`value` or `slider_max`) or store the data in drivers. ### Using an existing Shape Key property This idea would import all but the last Shape of the BlendShapeChannel with a specific prefix recognised by the FBX IO addon, e.g. `_FBX_IB`. These shapes would then be placed before the last Shape in the Shape Keys list. To store the DeformPercent of each in-between shape, I have two ideas: 1) Mute all in-between shapes and use their `.value` as the DeformPercent. This has the advantage that the `.value` shows in the Shape Keys list, but also means the operator to clear shape key values can't be used without losing the DeformPercents. 1) Use their `.slider_max` as the DeformPercent. This is more hidden away, but survives the operator to clear shape key values. Both options are unfortunately disruptive to viewing the shape key in Object mode, and a custom operator would be needed to preview the blending between the shapes (swapping to Absolute Shape Keys is close, but the blending time between each shape key would not match and the blend from the 'Basis' to the first in-between shape would not be present). With either option, the next shape key in the list not prefixed by `_FBX_IB` would always be assumed to have a DeformPercent of 100%. For the conversion from `.value` or `.slider_max` to FullWeights value, the value would be multiplied by 100 at export time and clamped between 0.0 and 100.0. I put together a rough proof-of-concept using the idea of storing the DeformPercent in `slider_max` and using an operator to view/edit the in-between shapes. For an actual implementation I would spend more time on the UI and probably go with a UI List to display each of the shapes and re-order them according to the values of their DeformPercents. <video src="/attachments/e561b9aa-4af1-4e43-90db-98c16ec7f576" title="FbxInBetweenShapesProofOfConcept.mp4" controls></video> ### Using drivers When a BlendShapeChannel has multiple Shapes, import all of them as separate Shape Keys, then either add an extra Shape Key as a controller that does nothing on its own, but use its `.value` to drive the `.value` of each of the Shape Keys according to their FullWeights values, or add a Float Custom Property to the Mesh and use its value to drive each of the Shape Keys. This results in a really nice way to see each of the in-between shapes blend at the correct percentages, but adding, modifying, viewing or removing the FullWeights values becomes more complicated because they're tied to the drivers. Some custom UI in the Driver Editor or Shape Key Specials Menu would be a necessity and I think the addon code would be a little more complicated because it would have to figure out which in-between shapes belong to which Shape Key by looking through the drivers and then extract the FullWeights values from the drivers. The drivers themselves are fairly simple, requiring only two or three points with either linear interpolation or vector handles. Using a Limits modifier on each driver to clamp the driven values to [0-1] can help too. ![image](/attachments/15e9ee15-f9ef-458b-91b5-72a4a96b664a) These videos show some existing implementations of using drivers to achieve in-between shapes: https://www.youtube.com/watch?v=De3fiWcQnpA (uses an extra shape key to drive the in-between shapes) https://www.youtube.com/watch?v=A6DRDk_MDBY (uses a custom property to drive the in-between shapes)

image.png

53 KiB

FbxInBetweenShapesProofOfConcept.mp4

6.3 MiB

Thomas Barlow referenced this issue

2023-09-23 22:30:47 +02:00

FBX not importing some shapekeys #104909

Jesse Yurkovich added the

Type

Design

label 2023-09-23 22:39:09 +02:00

Bastien Montagne referenced this issue

2023-10-05 14:51:43 +02:00

WIP: FBX: Fix #84111: AssertionError importing shapekeys #104910

Thomas Barlow referenced this issue

2023-10-05 15:46:10 +02:00

WIP: FBX: Fix #84111: AssertionError importing shapekeys #104910

Thomas Barlow commented

2023-10-13 00:20:25 +02:00

Another advantage of using drivers and an extra 'controller' Shape Key when importing in-between shapes is that it would simplify the import of animations and default values.

If the 'controller' Shape Key is considered the imported Shape Key for which all the animations and default value are tied to, then by adding separate Shape Keys for each in-between Shape that are driven by the 'controller''s .value, then animations and the default values of the shape keys should simply work straight away.

Without using drivers, each imported animation and default value would have to be separated into one animation and one default value for each in-between Shape, adjusting the values for each in-between Shape based on its value in FullWeights. Having separate animations for each in-between Shape sounds like it would be a nightmare for animating in Blender if the intent is to export them back as a single BlendShapeChannel again.

I did come up with another way of setting up the drivers and in-between Shapes, which would be to have the Shape Keys for each Shape be relative to the Shape Key of the previous Shape, with the first Shape Key being relative to the Reference Key ('Basis'). This way, as the 'controller' Shape Key's .value increases towards 1.0, the first in-between Shape will start to activate until it reaches 1.0 at its FullWeight value and then remains clamped at 1.0, then the next Shape starts to activate until it reaches 1.0 at its FullWeight and then remains clamped, this repeats for all the in-between Shapes in series until they each have .value of 1.0. It's possible this sort of setup could simplify the required driver setup because each FCurve only needs to define a start time with value 0.0 and end time with value 1.0, with a linear interpolation between those two times.

Another advantage of using drivers and an extra 'controller' Shape Key when importing in-between shapes is that it would simplify the import of animations and default values. If the 'controller' Shape Key is considered the imported Shape Key for which all the animations and default value are tied to, then by adding separate Shape Keys for each in-between Shape that are driven by the 'controller''s `.value`, then animations and the default values of the shape keys should simply work straight away. Without using drivers, each imported animation and default value would have to be separated into one animation and one default value for each in-between Shape, adjusting the values for each in-between Shape based on its value in `FullWeights`. Having separate animations for each in-between Shape sounds like it would be a nightmare for animating in Blender if the intent is to export them back as a single BlendShapeChannel again. --- I did come up with another way of setting up the drivers and in-between Shapes, which would be to have the Shape Keys for each Shape be relative to the Shape Key of the previous Shape, with the first Shape Key being relative to the Reference Key ('Basis'). This way, as the 'controller' Shape Key's `.value` increases towards `1.0`, the first in-between Shape will start to activate until it reaches `1.0` at its FullWeight value and then remains clamped at `1.0`, then the next Shape starts to activate until it reaches `1.0` at its FullWeight and then remains clamped, this repeats for all the in-between Shapes in series until they each have `.value` of `1.0`. It's possible this sort of setup could simplify the required driver setup because each FCurve only needs to define a start time with value `0.0` and end time with value `1.0`, with a linear interpolation between those two times. ![image](/attachments/508a9f27-4c9c-4fe0-9e8f-ec6a57ba1777)

image.png

84 KiB

Thomas Barlow commented

2023-10-18 19:05:56 +02:00

Another idea for using drivers, but keeping the driver curves as simple as possible and making the scripted expressions more complicated.

Each driver contains no curve points, instead a Generator modifier is used to produce a y=x curve.

The imported FullWeights were stored in a custom property on the Mesh in this example because there's no UI for Key custom properties, but they could be stored on the Key specifically to hide them from users.

It still remains difficult to add/remove/re-order the in-between shapes, so I suspect an operator for managing in-between shapes will be required with whatever solution we decide to go with.

Because FBX IO is enabled by default, I think an unobtrusive way to add an FBX-specific operator would be to add AddonPreferences to FBX IO that has the operator hidden by default. That way we avoiding adding an operator to the Shape Key Specials menu that almost nobody will use, unless the user specifically opts in through UI in the Add-ons list.

Another idea for using drivers, but keeping the driver curves as simple as possible and making the scripted expressions more complicated. Each driver contains no curve points, instead a Generator modifier is used to produce a y=x curve. The imported FullWeights were stored in a custom property on the `Mesh` in this example because there's no UI for `Key` custom properties, but they could be stored on the `Key` specifically to hide them from users. It still remains difficult to add/remove/re-order the in-between shapes, so I suspect an operator for managing in-between shapes will be required with whatever solution we decide to go with. Because FBX IO is enabled by default, I think an unobtrusive way to add an FBX-specific operator would be to add AddonPreferences to FBX IO that has the operator hidden by default. That way we avoiding adding an operator to the Shape Key Specials menu that almost nobody will use, unless the user specifically opts in through UI in the Add-ons list. ![image](/attachments/93c7a777-0b49-40d1-a996-89adbc35cc22)

image.png

172 KiB

Pratik Borhade referenced this issue from blender/blender

2024-02-05 05:41:00 +01:00

FBX Blend Shapes exporting NOT compatible with Unity #117811

Sign in to join this conversation.