The issue was caused by uninitialized ray used for composite and AO evaluation. Can;t really think of "proper" ray configuration here, it's all a bit arbitrary but think initializing the ray in a way so we look at the surface in a negative normal direction is much better alternative to uninitialized ray. Open for alternative suggestions tho.