This was caused by an incorrect assumption in the solver:
It tries to solve both collision and length constraints simultaneously,
using the projected movement of a point as a slide direction along the surface.
This only works if the distance of the previous curve point to the surface
is less than the allowed segment length. Otherwise the segment will
exceed the allowed length even with zero slide and NaN values are computed.
The case of larger surface distance can occur if the previous segment
solve was already stretching the current segment and then the point
moves further away. In this case we can simply clamp the segment length
without violating the contact constraint.
Lukas Tönne