The root of the issue was caused by the PredictMarkerPosition() always returning false when tracking backwards. This was making it so tracker always had to run brute initialization, which is an expensive operation. From own timing here: - Tracking forward takes 0.667637 seconds - Tracking backward used to take 2.591856 seconds - Tracking backward now takes 0.827724 seconds This is a very nice speedup, although the tracking backwards is still somewhat slower. Will be investigated further as part of a regular development.