Compositor tried to be too much smart and avoid unneeded re-calculations of the distortion model, but the way it was implemented is by falling back to the nearest interpolation first. We can't really cheat here, better to just look into faster models estimation.