Python API: expose the math mapping vertex positions to B-Bone segments #105419

Merged
Alexander Gavrilov merged 1 commits from angavrilov/blender:pr-pyapi-bbone-weight into main 2023-08-03 14:44:46 +02:00

Recently a user expressed interest in exporting baked animation
with B-Bone segments. This is technically possible, because B-Bones
are essentially similar to Spline IK chains embedded into a few
bone options for convenience. Thus an export script can generate
ordinary bones for the segments, produce baked animation for them,
and remap vertices accordingly by splitting the weight of the B-Bone
between the segment sub-bones to achieve the same deformation.

Currently the python API already exposes the segment matrices
necessary for sub-bone placement and animation via a PoseBone method,
but there is no access to the mapping of vertices to the segments.

Although currently the mapping math is simple and easy to re-implement,
forcing Python add-ons to do that would cause the exporters to break
and need rewriting if the mapping is ever changed later (it's quite
dumb, ignoring the rest pose curve, and there definitely is room for
improvement).

This patch extracts the relevant math into a BKE function, and
exposes it in the python API as a new PoseBone method.


An example that deforms a vertex point in Pose space using B-Bone segments:

def deform_vertex(pose_bone, point):
  index, factor = pose_bone.bbone_segment_index(point)  # <- the new method

  rest1 = pose_bone.bbone_segment_matrix(index, rest=True)
  pose1 = pose_bone.bbone_segment_matrix(index, rest=False)
  deform1 = pose1 @ rest1.inverted()

  rest2 = pose_bone.bbone_segment_matrix(index + 1, rest=True)
  pose2 = pose_bone.bbone_segment_matrix(index + 1, rest=False)
  deform2 = pose2 @ rest2.inverted()

  deform = deform1 * (1 - factor) + deform2 * factor

  return pose_bone.matrix @ deform @ pose_bone.bone.matrix_local.inverted() @ point
Recently a user expressed interest in exporting baked animation with B-Bone segments. This is technically possible, because B-Bones are essentially similar to Spline IK chains embedded into a few bone options for convenience. Thus an export script can generate ordinary bones for the segments, produce baked animation for them, and remap vertices accordingly by splitting the weight of the B-Bone between the segment sub-bones to achieve the same deformation. Currently the python API already exposes the segment matrices necessary for sub-bone placement and animation via a PoseBone method, but there is no access to the mapping of vertices to the segments. Although currently the mapping math is simple and easy to re-implement, forcing Python add-ons to do that would cause the exporters to break and need rewriting if the mapping is ever changed later (it's quite dumb, ignoring the rest pose curve, and there definitely is room for improvement). This patch extracts the relevant math into a BKE function, and exposes it in the python API as a new PoseBone method. ------- An example that deforms a vertex point in Pose space using B-Bone segments: ``` def deform_vertex(pose_bone, point): index, factor = pose_bone.bbone_segment_index(point) # <- the new method rest1 = pose_bone.bbone_segment_matrix(index, rest=True) pose1 = pose_bone.bbone_segment_matrix(index, rest=False) deform1 = pose1 @ rest1.inverted() rest2 = pose_bone.bbone_segment_matrix(index + 1, rest=True) pose2 = pose_bone.bbone_segment_matrix(index + 1, rest=False) deform2 = pose2 @ rest2.inverted() deform = deform1 * (1 - factor) + deform2 * factor return pose_bone.matrix @ deform @ pose_bone.bone.matrix_local.inverted() @ point ```
Alexander Gavrilov requested review from Sybren A. Stüvel 2023-03-03 23:02:46 +01:00
Hans Goudey changed title from Python API: expose the math mapping vertex positions to B-Bone segments. to Python API: expose the math mapping vertex positions to B-Bone segments 2023-03-03 23:19:21 +01:00
Alexander Gavrilov force-pushed pr-pyapi-bbone-weight from b46698542c to 5143ec558f 2023-03-10 16:00:04 +01:00 Compare
Alexander Gavrilov force-pushed pr-pyapi-bbone-weight from 5143ec558f to 617c47abc5 2023-03-11 11:21:17 +01:00 Compare
Sybren A. Stüvel removed the
Interest
Animation & Rigging
label 2023-03-30 12:33:33 +02:00
Sybren A. Stüvel added this to the Animation & Rigging project 2023-03-30 12:33:38 +02:00
First-time contributor

B-Bones can be a bit of a black box but have tremendous potential. Exposing more of their inner workings like this PR does is a welcome addition. Especially for us game folks where access to vertex mapping data can help convert b-bones to regular bones.

B-Bones can be a bit of a black box but have tremendous potential. Exposing more of their inner workings like this PR does is a welcome addition. Especially for us game folks where access to vertex mapping data can help convert b-bones to regular bones.
First-time contributor

Adding some details / breadcrumbs into this request.

As shared in the Blender Chat:
I had asked the question originally, as I've been reviewing this old bbones article (https://code.blender.org/2016/05/an-in-depth-look-at-how-b-bones-work-including-details-of-the-new-bendy-bones) and have been wondering how this could be converted over to a format the would be compatible with other DCCs and game engines. Can the math (in the article) be used to take a bbone, segment it into an X number of (dcc/game engine compatible) bones, then distribute the weights along a number bones as using the fancy bbone math.

Could be used for a number of uses, including muscle rigging.

Describing this another way, Brad Clark said:
"Basically: in order to export you need to convert the b-bone into simple bones. Every segment can become a bone, and their transformation matrices are already accessible. But you also need to split the weight painting assignment between these segment bones, and that's what the new api would be necessayr for.
Effectively B-Bones deform exactly like ordinary bones, except that instead of affecting the vertices directly, they are implicitly assigned to their segment sub-bones, choosing a blend of two segments for each vertex."

@arminhlc also provided another use case with:
"This would be a pretty cool feature. I often do the same, but manual, when weighting twist bones in limbs. First I weight them to one long bone and then split it up to the twist segments. This addition would make the process easily handled with a simple operator."

Adding some details / breadcrumbs into this request. As shared in the Blender Chat: I had asked the question originally, as I've been reviewing this old bbones article (https://code.blender.org/2016/05/an-in-depth-look-at-how-b-bones-work-including-details-of-the-new-bendy-bones) and have been wondering how this could be converted over to a format the would be compatible with other DCCs and game engines. Can the math (in the article) be used to take a bbone, segment it into an X number of (dcc/game engine compatible) bones, then distribute the weights along a number bones as using the fancy bbone math. Could be used for a number of uses, including muscle rigging. Describing this another way, Brad Clark said: _"Basically: in order to export you need to convert the b-bone into simple bones. Every segment can become a bone, and their transformation matrices are already accessible. But you also need to split the weight painting assignment between these segment bones, and that's what the new api would be necessayr for. Effectively B-Bones deform exactly like ordinary bones, except that instead of affecting the vertices directly, they are implicitly assigned to their segment sub-bones, choosing a blend of two segments for each vertex."_ _@arminhlc also provided another use case with: "This would be a pretty cool feature. I often do the same, but manual, when weighting twist bones in limbs. First I weight them to one long bone and then split it up to the twist segments. This addition would make the process easily handled with a simple operator."_
Sybren A. Stüvel requested changes 2023-05-15 14:30:32 +02:00
@ -0,0 +17,4 @@
deform = deform1 * (1 - blend_next) + deform2 * blend_next
return pose_bone.matrix @ deform @ pose_bone.bone.matrix_local.inverted() @ point

Could you expand this example so that it actually does something? Could be as simple as moving an empty along with the bbone.

Could you expand this example so that it actually does something? Could be as simple as moving an empty along with the bbone.
Author
Member

I added a couple of lines showing how to emulate an Armature modifier or constraint with a single bone by updating vertex coordinates or matrices. However they are still abstract, because the purpose of this example is to document how exactly the math works with the output of the API functions, not to provide some code that moves empties and such.

I added a couple of lines showing how to emulate an Armature modifier or constraint with a single bone by updating vertex coordinates or matrices. However they are still abstract, because the purpose of this example is to document how exactly the math works with the output of the API functions, not to provide some code that moves empties and such.
@ -547,1 +547,4 @@
/**
* Calculate index and blend factor for the two B-Bone segment nodes
* affecting the specified point in object (pose) space.

Document what the parameters mean, including what r_index actually indexes.

Document what the parameters mean, including what `r_index` actually indexes.
@ -548,0 +549,4 @@
* Calculate index and blend factor for the two B-Bone segment nodes
* affecting the specified point in object (pose) space.
*/
void BKE_pchan_bbone_deform_segment_index_pt(const struct bPoseChannel *pchan,

What does the _pt suffix mean? I think the name is not different enough from BKE_pchan_bbone_deform_segment_index to make it clear when to use which one.

What does the `_pt` suffix mean? I think the name is not different enough from `BKE_pchan_bbone_deform_segment_index` to make it clear when to use which one.
Author
Member

changed _pt to _from_point

changed `_pt` to `_from_point`
@ -1583,0 +1589,4 @@
const float(*mat)[4] = mats[0].mat;
/* Transform co to bone space and get its y component. */
float y = mat[0][1] * co[0] + mat[1][1] * co[1] + mat[2][1] * co[2] + mat[3][1];

const

`const`
angavrilov marked this conversation as resolved
Alexander Gavrilov force-pushed pr-pyapi-bbone-weight from 617c47abc5 to 5bcef8453d 2023-05-16 14:25:00 +02:00 Compare
Member

Hello,
As described by Brad, I confirm (as the main glTF I/O dev) that having access by python to segment matrices and weights will be very useful. We will be able to convert it on the fly to export BBones to glTF.

Hello, As described by Brad, I confirm (as the main glTF I/O dev) that having access by python to segment matrices and weights will be very useful. We will be able to convert it on the fly to export BBones to glTF.
First-time contributor

Hello,
As described by Brad, I confirm (as the main glTF I/O dev) that having access by python to segment matrices and weights will be very useful. We will be able to convert it on the fly to export BBones to glTF.

Awesome!!

Unfortunately I'm currently kinda stuck with FBX, but sounds like I will have some reference available to hack something together for my purposes.

> Hello, > As described by Brad, I confirm (as the main glTF I/O dev) that having access by python to segment matrices and weights will be very useful. We will be able to convert it on the fly to export BBones to glTF. Awesome!! Unfortunately I'm currently kinda stuck with FBX, but sounds like I will have some reference available to hack something together for my purposes.
Sybren A. Stüvel requested changes 2023-05-22 14:39:35 +02:00
@ -0,0 +20,4 @@
return pose_bone.matrix @ deform @ pose_bone.bone.matrix_local.inverted()
# Armature modifier deforming vertices:
for vertex in vertices:

This gives me a NameError: name 'vertices' is not defined. Can you give an example that can actually be used to see the code in action?

This gives me a `NameError: name 'vertices' is not defined`. Can you give an example that can actually be used to see the code in action?
angavrilov marked this conversation as resolved
@ -46,6 +46,21 @@ static float rna_PoseBone_do_envelope(bPoseChannel *chan, float vec[3])
bone->dist * scale);
}
static void rna_PoseBone_bbone_segment_index(

Consistency: the RNA function ..._index is now calling the BKE ..._index_from_point function. Rename to rna_PoseBone_bbone_segment_index_from_point.

Consistency: the RNA function `..._index` is now calling the BKE `..._index_from_point` function. Rename to `rna_PoseBone_bbone_segment_index_from_point`.
angavrilov marked this conversation as resolved
@ -49,0 +54,4 @@
return;
}
if (pchan->runtime.bbone_segments != pchan->bone->segments) {
BKE_reportf(reports, RPT_ERROR, "Bone '%s' has out of date B-Bone segment data!", pchan->name);

Is there a way to rephrase this so that the message explains what to do? Even I wouldn't know what this means exactly, or how to resolve the situation as a user.

Is there a way to rephrase this so that the message explains what to do? Even I wouldn't know what this means exactly, or how to resolve the situation as a user.
angavrilov marked this conversation as resolved
@ -271,2 +286,4 @@
RNA_def_function_return(func, parm);
/* B-Bone segment index from point */
func = RNA_def_function(srna, "bbone_segment_index", "rna_PoseBone_bbone_segment_index");

Same consistency issue as above. What's the reason to have one API in BKE and another in RNA/Python? If both BKE functions are generally useful, wouldn't they also be useful from Python?

If there's going to be a difference in naming between those two, I'd suggest picking a better name than bbone_segment_index, as it doesn't just return the index. How about bbone_segment_weighted_index?

Same consistency issue as above. What's the reason to have one API in BKE and another in RNA/Python? If both BKE functions are generally useful, wouldn't they also be useful from Python? If there's going to be a difference in naming between those two, I'd suggest picking a better name than `bbone_segment_index`, as it doesn't just return the index. How about `bbone_segment_weighted_index`?
angavrilov marked this conversation as resolved

This is the result I'm getting (before vs. after):

image

with the following code:

import bpy
from mathutils import Vector

def bbone_deform_matrix(pose_bone, point):
  index, blend_next = pose_bone.bbone_segment_index(point)

  rest1 = pose_bone.bbone_segment_matrix(index, rest=True)
  pose1 = pose_bone.bbone_segment_matrix(index, rest=False)
  deform1 = pose1 @ rest1.inverted()

  rest2 = pose_bone.bbone_segment_matrix(index + 1, rest=True)
  pose2 = pose_bone.bbone_segment_matrix(index + 1, rest=False)
  deform2 = pose2 @ rest2.inverted()

  deform = deform1 * (1 - blend_next) + deform2 * blend_next

  return pose_bone.matrix @ deform @ pose_bone.bone.matrix_local.inverted()

vertices = bpy.data.objects['NurbsPath'].data.splines[0].points
pose_bone = bpy.data.objects['Armature'].pose.bones['Bone']

# Armature modifier deforming vertices:
for vertex in vertices:
  co = Vector(vertex.co[:3])
  co = bbone_deform_matrix(pose_bone, co) @ co
  vertex.co[:3] = co

Is this the expected deformation? Any idea why it wouldn't follow the bone?

This is the blend file I used: 105419-test-bbone-segments-pyapi.blend

This is the result I'm getting (before vs. after): ![image](/attachments/26849e33-82bd-4b5c-aa57-8a71cbfa8263) with the following code: ```py import bpy from mathutils import Vector def bbone_deform_matrix(pose_bone, point): index, blend_next = pose_bone.bbone_segment_index(point) rest1 = pose_bone.bbone_segment_matrix(index, rest=True) pose1 = pose_bone.bbone_segment_matrix(index, rest=False) deform1 = pose1 @ rest1.inverted() rest2 = pose_bone.bbone_segment_matrix(index + 1, rest=True) pose2 = pose_bone.bbone_segment_matrix(index + 1, rest=False) deform2 = pose2 @ rest2.inverted() deform = deform1 * (1 - blend_next) + deform2 * blend_next return pose_bone.matrix @ deform @ pose_bone.bone.matrix_local.inverted() vertices = bpy.data.objects['NurbsPath'].data.splines[0].points pose_bone = bpy.data.objects['Armature'].pose.bones['Bone'] # Armature modifier deforming vertices: for vertex in vertices: co = Vector(vertex.co[:3]) co = bbone_deform_matrix(pose_bone, co) @ co vertex.co[:3] = co ``` Is this the expected deformation? Any idea why it wouldn't follow the bone? This is the blend file I used: [105419-test-bbone-segments-pyapi.blend](/attachments/25cef53d-b209-4c0e-a9ed-cc788da2390e)

Never mind, the path in edit mode shows that the points are placed correctly:

image

Never mind, the path in edit mode shows that the points are placed correctly: ![image](/attachments/c52029c5-79ec-4770-a2e9-f04f22c980e8)
Member

Just bumping this to see if it is something that can get moved forward @angavrilov

Just bumping this to see if it is something that can get moved forward @angavrilov
Alexander Gavrilov force-pushed pr-pyapi-bbone-weight from 5bcef8453d to 5e01844c11 2023-07-14 15:24:50 +02:00 Compare
Author
Member

Like discussed in chat a while ago I renamed the old function and removed _from_point instead, and added a hint to update depsgraph to the errors.

Also, the example script now can be run in the attached blend file.

Like discussed in chat a while ago I renamed the old function and removed `_from_point` instead, and added a hint to update depsgraph to the errors. Also, the example script now can be run in the attached blend file.
Alexander Gavrilov force-pushed pr-pyapi-bbone-weight from 5e01844c11 to b4500fe268 2023-07-24 16:07:12 +02:00 Compare
Sybren A. Stüvel requested changes 2023-08-03 11:57:21 +02:00
Sybren A. Stüvel left a comment
Member

LGTM! There's one bit of code that IMO can be pulled out into a function by itself in the BLI_math library, but that's not critical.

The PR doesn't merge cleanly with main though. After merging in main and resolving one small conflict, I'm getting build errors:

/build_linux/source/blender/makesrna/intern/rna_pose_gen.cc:2232:2: error: no matching function for call to 'rna_PoseBone_bbone_segment_index'
        rna_PoseBone_bbone_segment_index(_self, reports, point, index, blend_next);
        ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/blender/source/blender/makesrna/intern/rna_pose_api.cc:50:13: note: candidate function not viable: 3rd argument ('const float *') would lose const qualifier
static void rna_PoseBone_bbone_segment_index(
            ^
LGTM! There's one bit of code that IMO can be pulled out into a function by itself in the BLI_math library, but that's not critical. The PR doesn't merge cleanly with `main` though. After merging in main and resolving one small conflict, I'm getting build errors: ``` /build_linux/source/blender/makesrna/intern/rna_pose_gen.cc:2232:2: error: no matching function for call to 'rna_PoseBone_bbone_segment_index' rna_PoseBone_bbone_segment_index(_self, reports, point, index, blend_next); ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ /blender/source/blender/makesrna/intern/rna_pose_api.cc:50:13: note: candidate function not viable: 3rd argument ('const float *') would lose const qualifier static void rna_PoseBone_bbone_segment_index( ^ ```
@ -1593,0 +1599,4 @@
const float(*mat)[4] = mats[0].mat;
/* Transform co to bone space and get its y component. */
const float y = mat[0][1] * co[0] + mat[1][1] * co[1] + mat[2][1] * co[2] + mat[3][1];

The line should IMO be a function in the math library, and not something that's inlined here. Getting a single element of a matrix-vector multiplication could be useful in other situations as well. It would also make the abstraction level of this function more uniform.

The line should IMO be a function in the math library, and not something that's inlined here. Getting a single element of a matrix-vector multiplication could be useful in other situations as well. It would also make the abstraction level of this function more uniform.
Author
Member

This is the only match for \[0\]\[1\].*\[1\]\[1\].*\[2\]\[1\].*\[3\]\[1\] that is not in the context of a complete matrix*vector multiplication, so I don't think introducing a single use function would be warranted.

This is the only match for `\[0\]\[1\].*\[1\]\[1\].*\[2\]\[1\].*\[3\]\[1\]` that is not in the context of a complete matrix*vector multiplication, so I don't think introducing a single use function would be warranted.
Alexander Gavrilov force-pushed pr-pyapi-bbone-weight from b4500fe268 to 755dcee650 2023-08-03 12:53:41 +02:00 Compare
Sybren A. Stüvel approved these changes 2023-08-03 14:09:43 +02:00
Sybren A. Stüvel left a comment
Member

👍

:+1:
Alexander Gavrilov merged commit 36c6bcca1a into main 2023-08-03 14:44:46 +02:00
Alexander Gavrilov deleted branch pr-pyapi-bbone-weight 2023-08-03 14:44:47 +02:00
Sybren A. Stüvel removed this from the Animation & Rigging project 2023-09-14 14:25:55 +02:00
Sign in to join this conversation.
No reviewers
No Label
Interest
Alembic
Interest
Animation & Rigging
Interest
Asset Browser
Interest
Asset Browser Project Overview
Interest
Audio
Interest
Automated Testing
Interest
Blender Asset Bundle
Interest
BlendFile
Interest
Collada
Interest
Compatibility
Interest
Compositing
Interest
Core
Interest
Cycles
Interest
Dependency Graph
Interest
Development Management
Interest
EEVEE
Interest
EEVEE & Viewport
Interest
Freestyle
Interest
Geometry Nodes
Interest
Grease Pencil
Interest
ID Management
Interest
Images & Movies
Interest
Import Export
Interest
Line Art
Interest
Masking
Interest
Metal
Interest
Modeling
Interest
Modifiers
Interest
Motion Tracking
Interest
Nodes & Physics
Interest
OpenGL
Interest
Overlay
Interest
Overrides
Interest
Performance
Interest
Physics
Interest
Pipeline, Assets & IO
Interest
Platforms, Builds & Tests
Interest
Python API
Interest
Render & Cycles
Interest
Render Pipeline
Interest
Sculpt, Paint & Texture
Interest
Text Editor
Interest
Translations
Interest
Triaging
Interest
Undo
Interest
USD
Interest
User Interface
Interest
UV Editing
Interest
VFX & Video
Interest
Video Sequencer
Interest
Virtual Reality
Interest
Vulkan
Interest
Wayland
Interest
Workbench
Interest: X11
Legacy
Blender 2.8 Project
Legacy
Milestone 1: Basic, Local Asset Browser
Legacy
OpenGL Error
Meta
Good First Issue
Meta
Papercut
Meta
Retrospective
Meta
Security
Module
Animation & Rigging
Module
Core
Module
Development Management
Module
EEVEE & Viewport
Module
Grease Pencil
Module
Modeling
Module
Nodes & Physics
Module
Pipeline, Assets & IO
Module
Platforms, Builds & Tests
Module
Python API
Module
Render & Cycles
Module
Sculpt, Paint & Texture
Module
Triaging
Module
User Interface
Module
VFX & Video
Platform
FreeBSD
Platform
Linux
Platform
macOS
Platform
Windows
Priority
High
Priority
Low
Priority
Normal
Priority
Unbreak Now!
Status
Archived
Status
Confirmed
Status
Duplicate
Status
Needs Info from Developers
Status
Needs Information from User
Status
Needs Triage
Status
Resolved
Type
Bug
Type
Design
Type
Known Issue
Type
Patch
Type
Report
Type
To Do
No Milestone
No project
No Assignees
6 Participants
Notifications
Due Date
The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: blender/blender#105419
No description provided.