It turns out that after the fix to T83196 (rB814b2787cadd) the matrix
to quaternion conversion can produce noncanonical results in large
areas of the rotation space, when previously this was limited to
way smaller areas. This in turn causes Swing+Twist math to produce
angles beyond 180 degrees, e.g. outputting a -120..240 range.
This fixes both issues, ensuring that conversion outputs a canonical
result, and decomposition canonifies its input.
This was reported in chat by @jpbouza.
Adjust the threshold for switching from the base case to trace > 0,
based on very similar example code from www.euclideanspace.com to
avoid float precision issues when trace is close to -1.
Also, remove conversions to and from double, because using double
here doesn't really have benefit, especially with the new threshold.
Finally, add quaternion-matrix-quaternion round trip tests with
full coverage for all 4 branches.
Differential Revision: https://developer.blender.org/D9675
