This crash can only happen in case faces in same 'smooth fan' have reversed normals. To support this, we have to always keep a way to get real values in loop_to_poly, even when loop itself is tagged as done, it might be needed in computation of one of its neighbor's split normal later.