This was caused by an incorrect assumption in the solver:
It tries to solve both collision and length constraints simultaneously,
using the projected movement of a point as a slide direction along the surface.
This only works if the distance of the previous curve point to the surface
is less than the allowed segment length. Otherwise the segment will
exceed the allowed length even with zero slide and NaN values are computed.
The case of larger surface distance can occur if the previous segment
solve was already stretching the current segment and then the point
moves further away. In this case we can simply clamp the segment length
without violating the contact constraint.
Pull Request #105499
During hair grooming in curves sculpt mode, it is very useful when hair strands
are prevented from intersecting with the surface mesh. Unfortunately, it also
decreases performance significantly so we don't want it to be turned on all the time.
The surface collision is used by the Comb, Pinch and Puff brushes currently.
It can be turned on or off on a per-geometry basis.
The intersection prevention quality of this patch is not perfect yet. This can
be improved over time using a better solver. Overall, perfect collision detection
at the cost of bad performance is not necessary for interactive sculpting,
because the user can fix small mistakes very quickly. Nevertheless, the quality
can probably still be improved significantly without too big slow-downs depending
on the use case. This can be done separately from this patch.
Pull Request #104469
