VR: Controller Based Picking #77128

New Issue

Julian Eisel · 2020-05-28T16:02:17+02:00

Julian Eisel commented

2020-05-28 16:02:17 +02:00

VR: Controller Based Picking

Picking in this context means, finding which item (object, bone, vertex, etc) is being pointed at by the controller.

How to implement picking? OpenGL selection or ray casting?

(To perform OpenGL picking we would draw a controller's perspective to an offscreen buffer select based on that result.)

Few things to consider:

Blender has an own OpenGL selection implementation that does not use the legacy GL_SELECT.
Picking is not only needed for objects, but also vertices/edges/faces, bones, bezier control points, etc. Also: Gizmos.
Gizmos do currently only support OpenGL selection.
Empties, cameras, lights and other selectable items have no real geometry that can be used for ray casting. Unless that changes, OpenGL gives a more WYSIWYG selection.
We could in theory perform different selection methods in different modes. E.g. in Edit Mode we could do BVH accelerated selection, we have utilities ready for this.
Should we ever get acceleration structures for other modes that we could rely on for selection, ray casting would be much preferable, because of better scalability. Nothing like this is planned short term though.
Ray casting could be performed in a separate thread, especially to avoid blocking viewport drawing.
It's probably too expensive to perform picking on every frame. It should only be performed when needed (e.g. on trigger press). It seems like we can do some drawing tricks to visualize where the virtual controller ray intersects objects though. Needs more investigation.

How is picking executed?

What happens when pushing the trigger to select, or even select and tweak an object?
Some options:

Add an own operator for VR controller based selection.
Add a invoke_3d() callback for operators. It can wrap the existing logic, but directly hand it the 3D direction vector to use, rather than projecting the cursor from 2D to 3D space first.
Have a function to trigger picking, e.g. bpy.types.XrSessionState.pick() in Python. The result can be cached for multiple calls per frame.

# VR: Controller Based Picking Picking in this context means, finding which item (object, bone, vertex, etc) is being pointed at by the controller. **How to implement picking? OpenGL selection or ray casting?** (To perform OpenGL picking we would draw a controller's perspective to an offscreen buffer select based on that result.) Few things to consider: * Blender has an own OpenGL selection implementation that does not use the legacy `GL_SELECT`. * Picking is not only needed for objects, but also vertices/edges/faces, bones, bezier control points, etc. Also: Gizmos. * Gizmos do currently only support OpenGL selection. * Empties, cameras, lights and other selectable items have no real geometry that can be used for ray casting. Unless that changes, OpenGL gives a more WYSIWYG selection. * We could in theory perform different selection methods in different modes. E.g. in Edit Mode we **could** do BVH accelerated selection, we have utilities ready for this. * Should we ever get acceleration structures for other modes that we could rely on for selection, ray casting would be much preferable, because of better scalability. Nothing like this is planned short term though. * Ray casting could be performed in a separate thread, especially to avoid blocking viewport drawing. * It's probably too expensive to perform picking on every frame. It should only be performed when needed (e.g. on trigger press). It seems like we can do some drawing tricks to visualize where the virtual controller ray intersects objects though. Needs more investigation. **How is picking executed?** What happens when pushing the trigger to select, or even select and tweak an object? Some options: * Add an own operator for VR controller based selection. * Add a `invoke_3d()` callback for operators. It can wrap the existing logic, but directly hand it the 3D direction vector to use, rather than projecting the cursor from 2D to 3D space first. * Have a function to trigger picking, e.g. `bpy.types.XrSessionState.pick()` in Python. The result can be cached for multiple calls per frame.

Julian Eisel commented

2020-05-28 16:02:17 +02:00

Added subscriber: @JulianEisel

Julian Eisel commented

2020-05-28 16:04:11 +02:00

Added subscriber: @PeterK-4

Julian Eisel commented

2020-05-28 16:04:11 +02:00

Changed status from 'Needs Triage' to: 'Confirmed'

Jacob Merrill commented

2020-05-28 16:11:13 +02:00

Added subscriber: @JacobMerrill-1

Nicolas Fauvet commented

2020-05-29 09:52:55 +02:00

Added subscriber: @nfauvet

RedMser commented

2020-08-16 16:20:16 +02:00

Added subscriber: @RedMser

Satish Goda commented

2021-11-23 06:31:02 +01:00

Added subscriber: @satishgoda1

Kent Hinson commented

2022-09-25 07:57:47 +02:00

Added subscriber: @kenthinson

Kent Hinson commented

2022-09-25 08:04:13 +02:00

Does anyone currently own this task? Or is currently working on it? My first intuition is to use the mentioned selection code already built into Blender in case it's doing any magic. Unless something prevents doing so. I would be interested in looking into it if this is not currently being owned by someone.

Peter Kim commented

2022-09-25 09:39:12 +02:00

Added subscriber: @muxed-reality

Peter Kim commented

2022-09-25 09:39:12 +02:00

@kenthinson Currently no one owns this task so you are free to look into it.
I submitted a patch a while ago (D13153: XR: Mouse Position Simulation) to add viewplane projection-based selection but haven't had time to update it.
You might find it helpful as a reference though, and feel free to take over the patch if you want.

@kenthinson Currently no one owns this task so you are free to look into it. I submitted a patch a while ago ([D13153: XR: Mouse Position Simulation](https://archive.blender.org/developer/D13153)) to add viewplane projection-based selection but haven't had time to update it. You might find it helpful as a reference though, and feel free to take over the patch if you want.

Philipp Oeser removed the

Download

What's New

Blender Studio

Manual

Developers Blog

Documentation

Benchmark

Blender Conference

Development Fund

One-time Donations

VR: Controller Based Picking #77128

VR: Controller Based Picking