Fix #101877, rigidbodies & constraints causing frequent crashes. #108399

himisa · 2023-05-29T21:00:56+02:00

himisa commented

2023-05-29 21:00:56 +02:00

Problem: See #101877 .

Rigidbody and constraint memory management is very likely to cause dereferencing of invalid (freed) pointers and crash.

Senario 1:

If rigidbody object A and B is constrained by C, press space to play animation, then delete one of the objects A/B, the btTypedConstraint C will still remember the pointer of both A and B, and tries to constrain them when animation is continued, but one of the objects is now freed, then blender crashes.
Steps to reproduce: 1. open test1a.blend; 2. press Space to play animation; 3. press Space again to pause; 4. delete one of the two cubes; 5. press Space to continue animation, and wait Blender to crash. Note: If you are using release version of Blender, there're chances that freed memory are still accessible and the memory content is still valid, so blender may survive this. You may need to try multiple times to produce a crash.
This is fixed by removing all the constraints related to a rigidbody when rigidbody is deleted (see change in btDiscreteDynamicsWorld.cpp).

Senario 2:

When performing undo after animation is played, blender will free all rigid bodies & constraints (before rebuilding them to a previous state). If a constraint is freed before its related rigid bodies, the rigid bodies will remember the freed pointer to the constraint. This will cause crash when freeing rigid bodies afterwards, since the code tries to access the (already freed) constraint to remove it from m_constraintRefs of the rigid body.
Steps to reproduce: 1. open test1a.blend; 2. press Space to play animation; 3. press Space again to pause; 4. delete one of the two cubes; 5. press ctrl+Z twice, and wait Blender to crash. This is because after first undo, the deleted cube is back at the end of object list, now its guaranteed to be deleted after the constraint object when traversing the object list to delete them.
This is fixed by removing the constraint from m_constraintRefs of its related rigid bodies when constraint is getting freed (see change in rb_bullet_api.cpp).

Notice:

Note that we need a new flag (m_isValid ) to remember whether the m_rbA and m_rbB is pointing to valid objects or deleted objects, since when a rigid body is deleted, its related constraint will be removed from btDiscreteDynamicsWorld, but not deleted (owned by the constraint objects, so no delete here). When constraint object wants to remove the constraint again, it should have ability to tell whether m_rbA and m_rbB are still present.

I don't know will it have side effects (in serializing/undo/...) to add an bool m_isValid; member to btTypedConstraint. It will not change sizeof(btTypedConstraint), because of struct padding. Invalidated constraints will also not be serialized, since it will be removed from m_constraints of btDiscreteDynamicsWorld. If there're any other effects that I didn't recognize, please let me know. Tests and comments are welcome.

This should also fix #91369, #91959, #92915, #95603, #100613.

Problem: See #101877 . Rigidbody and constraint memory management is very likely to cause dereferencing of invalid (freed) pointers and crash. Senario 1: - If rigidbody object A and B is constrained by C, press space to play animation, then delete one of the objects A/B, the btTypedConstraint C will still remember the pointer of both A and B, and tries to constrain them when animation is continued, but one of the objects is now freed, then blender crashes. - Steps to reproduce: 1. open test1a.blend; 2. press Space to play animation; 3. press Space again to pause; 4. delete one of the two cubes; 5. press Space to continue animation, and wait Blender to crash. Note: If you are using release version of Blender, there're chances that freed memory are still accessible and the memory content is still valid, so blender may survive this. You may need to try multiple times to produce a crash. - This is fixed by removing all the constraints related to a rigidbody when rigidbody is deleted (see change in btDiscreteDynamicsWorld.cpp). Senario 2: - When performing undo after animation is played, blender will free all rigid bodies & constraints (before rebuilding them to a previous state). If a constraint is freed before its related rigid bodies, the rigid bodies will remember the freed pointer to the constraint. This will cause crash when freeing rigid bodies afterwards, since the code tries to access the (already freed) constraint to remove it from m_constraintRefs of the rigid body. - Steps to reproduce: 1. open test1a.blend; 2. press Space to play animation; 3. press Space again to pause; 4. delete one of the two cubes; 5. press ctrl+Z twice, and wait Blender to crash. This is because after first undo, the deleted cube is back at the end of object list, now its guaranteed to be deleted after the constraint object when traversing the object list to delete them. - This is fixed by removing the constraint from m_constraintRefs of its related rigid bodies when constraint is getting freed (see change in rb_bullet_api.cpp). Notice: Note that we need a new flag (`m_isValid` ) to remember whether the `m_rbA` and `m_rbB` is pointing to valid objects or deleted objects, since when a rigid body is deleted, its related constraint will be removed from `btDiscreteDynamicsWorld`, but not deleted (owned by the constraint objects, so no delete here). When constraint object wants to remove the constraint again, it should have ability to tell whether `m_rbA` and `m_rbB` are still present. I don't know will it have side effects (in serializing/undo/...) to add an `bool m_isValid;` member to `btTypedConstraint`. It will not change `sizeof(btTypedConstraint)`, because of struct padding. Invalidated constraints will also not be serialized, since it will be removed from `m_constraints` of `btDiscreteDynamicsWorld`. If there're any other effects that I didn't recognize, please let me know. Tests and comments are welcome. This should also fix #91369, #91959, #92915, #95603, #100613.

Test1a.blend

792 KiB

himisa changed title from ~~WIP: Trying to fix #101877~~ to WIP: Trying to fix #101877, rigidbody & constraints causing frequent crashes

2023-05-29 22:19:43 +02:00

himisa changed title from ~~WIP: Trying to fix #101877, rigidbody & constraints causing frequent crashes~~ to Fix #101877, rigidbodies & constraints causing frequent crashes.

2023-05-30 09:03:44 +02:00

himisa requested review from Sonny Campbell 2023-06-03 11:34:44 +02:00

himisa requested review from YimingWu 2023-08-16 10:58:21 +02:00

YimingWu commented

2023-08-16 14:58:48 +02:00

Hi @himisa , did you added me to the reviewer due to #109108? I'm actually not that familiar with the physics module, but I can test this patch.

himisa commented

2023-08-16 15:58:06 +02:00

@ChengduLittleA

Thanks!

I browsed recent issues and feel like you have handled some rigidbody related issues and are recently active... I am amature and not familier with blender developers. I hope someone can test this or notify the relevant member.

This issue was getting in my way when I was adjusting my character with both clothes and rigid bodies. Adjusting clothes requires frequent undo and playback, and rigidbody crashes blender a lot.

I compiled a version of blender 3.6 with this fix and it worked fine for me now, so there's no urgency of merging this PR. However I still hope this can be fixed in 4.0 release, if possible.

@ChengduLittleA Thanks! I browsed recent issues and feel like you have handled some rigidbody related issues and are recently active... I am amature and not familier with blender developers. I hope someone can test this or notify the relevant member. This issue was getting in my way when I was adjusting my character with both clothes and rigid bodies. Adjusting clothes requires frequent undo and playback, and rigidbody crashes blender a lot. I compiled a version of blender 3.6 with this fix and it worked fine for me now, so there's no urgency of merging this PR. However I still hope this can be fixed in 4.0 release, if possible.

👍 1

Iliya Katushenock added the

Download

What's New

Blender Studio

Manual

Developers Blog

Documentation

Benchmark

Blender Conference

Development Fund

One-time Donations

Fix #101877, rigidbodies & constraints causing frequent crashes. #108399

Checkout