It was only happening on 32bit platforms because of alignment differences when allocating class. Now got rid of copy of eigen matricies stored by value in the residual block which solves aligment issues and should also give some unmeasurable speedup.