The issue was that the `GeometrySet` that comes from the modifier does not have full ownership of the mesh for legacy reasons to avoid copies. Calling `ensure_owns_direct_data` makes sure that the geometry set actually owns the geometry so that it can be instanced.