Virtual Reality (XR/VR/AR/MR) #68998

New Issue

Dalai Felinto · 2019-08-21T16:30:51+02:00

Dalai Felinto commented

2019-08-21 16:30:51 +02:00

NOTE: This section will is WIP and will be updated shortly.

Status:

Team
Commissioner: ?
Project leader:
Project members:
Big picture:

Description
Use cases:

Design:

Engineer plan: -

Work plan

Milestone 1 - Scene Inspection #71347
Time estimate: Mostly done, 2-3 weeks of work left (state: mid-February 2020)

Milestone 2 - Continuous Immersive Drawing #71348
Time estimate: ?

Later

Notes: -

The description here and in sub-tasks is based on a design document by @JulianEisel: VR/XR Big Picture Kickoff . Please don't take it as official document.

Goals and Overall Vision

XR enables an entirely new way to work with computer generated 3D worlds. Instead of trying to take the UIs we’ve already implemented for traditional monitor use and make them work in XR, we should establish an experience on entirely new grounds. Otherwise, what we’ll end up with is just a different, more difficult way to use Blender.
So to enter the new immersive world, we should think about which workflows could benefit the most from XR, and carefully craft a new, improved experience for them. Don’t try to enable huge amounts of features for XR, risking bad outcome - or even total failure - but find out which features matter the most and focus on them first.

Long term, the experience can then be enriched further, with more features added as smaller, more concise projects. Eventually, XR would allow creating 3D content with unseen interactivity. The entirety of Blender’s content creation capabilities available in an immersive 3D experience. However, this better be the long term vision for XR, not the goal for this initial project.

OpenXR

All VR support in Blender will be based on OpenXR , the brand new specification for VR, AR, MR, etc - or short: XR. It is created by the Khronos group and has a huge number of supporters from all over the VR industry. It’s likely going to be the standard for VR/AR/MR/… (XR).

OpenXR splits the workload between the application (Blender) and the OpenXR runtime. The runtime handles all the device specific stuff, provides many common features (e.g. time wrapping) and can add extensions to the OpenXR API. So the devices Blender will support are defined by the available OpenXR (compatible) runtimes. Currently:

Windows Mixed Reality
Oculus (though, not officially released)
Monado (FOSS Linux OpenXR runtime)

For information on how to test our current OpenXR driven implementation, see https://wiki.blender.org/wiki/User:Severin/GSoC-2019/How_to_Test

State of the Project

While there are multiple initiatives related to Blender and VR (e.g [MPX ]] and [ https:*www.marui-plugin.com/blender-xr/ | Marui’s Blender XR ), the first important milestone for XR support in main Blender is being finished with an ongoing Google Summer of Code project. It aims to bring stable and well performing OpenXR based VR rendering support into the core of Blender. It’s not supposed to enable any interaction, it’s solely focused on the foundation of HMD support. For more info, refer to #71365. Further work is being done to complete the first milestone: #71347 (Virtual Reality - Milestone 1 - Scene Inspection).

The one important part that’s missing is controller support, or more precisely, input and haptics support. Although we can do some experiments, it would be helpful to first get a better idea of how VR interaction should work eventually, before much effort is put into supporting this.

This is the foundation that we can build on. It should allow people with experience in the field to join efforts to bring rich immersive experiences to Blender.

Development Process

Proposal is to do a use-case driven development process. That means, we define a number (say 10-20) of specific use cases. These should be chosen wisely, based on what we think has most potential with XR.
First, just vaguely describe what requirements the system has for a use-case, a bit later on, define what tools and options are needed to get the use case done (possibly by a specific persona ).

On top of that, the process should be done iteratively. That means, we define short term goals for an iteration, work on them and evaluate afterwards before we get into the next iteration. The iterations may include implementation work. So we don’t do the waterfall-like “design first”, but there should be enough design work done before first implementations are made.

Any implementation should be driven by a use-case (or multiple) that the team agrees is drafted out well enough.

NOTE: This section will is WIP and will be updated shortly. **Status:** **Team** **Commissioner:** `?` **Project leader:** **Project members:** **Big picture:** **Description** **Use cases:** **Design:** **Engineer plan:** `-` **Work plan** **Milestone 1 - Scene Inspection #71347** Time estimate: Mostly done, 2-3 weeks of work left (state: mid-February 2020) **Milestone 2 - Continuous Immersive Drawing #71348** Time estimate: `?` **Later** **Notes:** - ---- The description here and in sub-tasks is based on a design document by @JulianEisel: [VR/XR Big Picture Kickoff ](https://dev-files.blender.org/file/data/nqh5c22ahvruutmdna5l/PHID-FILE-4nynef6hmrfcwaimvf3h/XR_Big_Picture_Kickoff_%28V2%29). Please don't take it as official document. ## Goals and Overall Vision XR enables an entirely new way to work with computer generated 3D worlds. Instead of trying to take the UIs we’ve already implemented for traditional monitor use and make them work in XR, we should establish an experience on entirely new grounds. Otherwise, what we’ll end up with is just a different, more difficult way to use Blender. So to enter the new immersive world, we should think about which workflows could benefit the most from XR, and carefully craft a new, improved experience for them. Don’t try to enable huge amounts of features for XR, risking bad outcome - or even total failure - but find out which features matter the most and focus on them first. Long term, the experience can then be enriched further, with more features added as smaller, more concise projects. Eventually, XR would allow creating 3D content with unseen interactivity. The entirety of Blender’s content creation capabilities available in an immersive 3D experience. However, this better be the long term vision for XR, not the goal for this initial project. ## OpenXR All VR support in Blender will be based on [OpenXR ](https://www.khronos.org/openxr), the brand new specification for VR, AR, MR, etc - or short: XR. It is created by the Khronos group and has a huge number of supporters from all over the VR industry. It’s likely going to be the standard for VR/AR/MR/… (XR). OpenXR splits the workload between the application (Blender) and the OpenXR runtime. The runtime handles all the device specific stuff, provides many common features (e.g. time wrapping) and can add extensions to the OpenXR API. So the devices Blender will support are defined by the available OpenXR (compatible) runtimes. Currently: * Windows Mixed Reality * Oculus (though, not officially released) * Monado (FOSS Linux OpenXR runtime) For information on how to test our current OpenXR driven implementation, see https://wiki.blender.org/wiki/User:Severin/GSoC-2019/How_to_Test ## State of the Project While there are multiple initiatives related to Blender and VR (e.g [MPX ]] and [[ https:*www.marui-plugin.com/blender-xr/ | Marui’s Blender XR ](https:*vimeo.com/channels/mpx)), the first important milestone for XR support in main Blender is being finished with an ongoing Google Summer of Code project. It aims to bring stable and well performing OpenXR based VR rendering support into the core of Blender. It’s not supposed to enable any interaction, it’s solely focused on the foundation of HMD support. For more info, refer to #71365. Further work is being done to complete the first milestone: #71347 (Virtual Reality - Milestone 1 - Scene Inspection). The one important part that’s missing is controller support, or more precisely, input and haptics support. Although we can do some experiments, it would be helpful to first get a better idea of how VR interaction should work eventually, before much effort is put into supporting this. This is the foundation that we can build on. It should allow people with experience in the field to join efforts to bring rich immersive experiences to Blender. ## Development Process Proposal is to do a use-case driven development process. That means, we define a number (say 10-20) of specific use cases. These should be chosen wisely, based on what we think has most potential with XR. First, just vaguely describe what requirements the system has for a use-case, a bit later on, define what tools and options are needed to get the use case done (possibly by a specific [persona ](https://www.interaction-design.org/literature/article/personas-why-and-how-you-should-use-them)). On top of that, the process should be done iteratively. That means, we define short term goals for an iteration, work on them and evaluate afterwards before we get into the next iteration. The iterations may include implementation work. So we don’t do the waterfall-like “design first”, but there should be enough design work done before first implementations are made. Any implementation should be driven by a use-case (or multiple) that the team agrees is drafted out well enough.

Julian Eisel was assigned by Dalai Felinto

2019-08-21 16:30:51 +02:00

Dalai Felinto commented

2019-08-21 16:30:51 +02:00

Added subscriber: @dfelinto

blender-admin commented

2019-08-21 16:30:51 +02:00

#47899 was marked as duplicate of this issue

Julian Eisel commented

2019-08-21 16:49:01 +02:00

Added subscribers: @JulianEisel, @TheOnlyJoey, @esummers, @Olm-e-9, @StirlingGoetz, @LeeP, @Ozeki, @jherico, @hoxolotl, @ppicazo, @EjnerFergo, @JoelGerlach, @david.rysk, @NicholasBenge, @rpavlik, @godbyk, @VRguy, @LazyDodo

Julian Eisel commented

2019-08-21 16:51:15 +02:00

Dalai and I agreed on merging #47899 into this. The old task can still be referred to, but development took new paths that are better discussed here and in the sub-tasks. We'd also like to discuss things on a broader level - the "big picture" for VR in Blender.
Will publish more info soon.

Dalai and I agreed on merging #47899 into this. The old task can still be referred to, but development took new paths that are better discussed here and in the sub-tasks. We'd also like to discuss things on a broader level - the "big picture" for VR in Blender. Will publish more info soon.

Emir Sinan Gürlek commented

2019-08-23 06:29:39 +02:00

Added subscriber: @filibis

Sam Green commented

2019-08-24 21:50:03 +02:00

Added subscriber: @SamGreen

Janne Aliu commented

2019-08-25 17:40:28 +02:00

Added subscriber: @Jaydead

Sebastian Brachi commented

2019-08-28 06:15:22 +02:00

Added subscriber: @Brachi

JUAN FRANCISCO PAEZ commented

2019-08-28 18:33:43 +02:00

Added subscriber: @JuanFranciscoPaez

Rainer Trummer commented

2019-08-30 09:57:57 +02:00

Added subscriber: @RainerTrummer

Peter Kim commented

2019-08-30 12:34:04 +02:00

Added subscriber: @muxed-reality

Simeon Conzendorf commented

2019-09-05 11:19:40 +02:00

Added subscriber: @SimeonConzendorf

Ben commented

2019-10-11 08:36:50 +02:00

Added subscriber: @Mr_SquarePeg

Francesco Sorrentino commented

2019-10-12 08:54:27 +02:00

Added subscriber: @Zeirus

MACHIN3 commented

2019-10-26 13:59:36 +02:00

Added subscriber: @MACHIN3

B.E. commented

2019-10-29 17:22:26 +01:00

Added subscriber: @jiku

Jacob Merrill commented

2019-11-05 00:28:43 +01:00

Added subscriber: @JacobMerrill-1

Jacob Merrill commented

2019-11-05 00:28:43 +01:00

I think having collapsible 'shelves' which hold icons or folders is the way to go

one could grab a 'icon' that is 2d thumbnail on a shelf -> pull it into vr where it becomes a 3d icon,
set it on a desk -> inspect a 'folder' inside it containing all it's attributes
(brush radius, feather, color etc)

further down the line, the actual python running the object will be visible inside one of these 'nests' or logic noodles

this way tools can be shared in a .blend
(some work will need to be done for infrastructure later to make more flexible 'chunks' used to make tools in py calling compiled code)

maybe also have 'active tool button bindings' in a folder*
(like when a tool is 'bound' to a VR handle the tool itself has the shortcuts in it for how to interact with it)

I think having collapsible 'shelves' which hold icons or folders is the way to go one could grab a 'icon' that is 2d thumbnail on a shelf -> pull it into vr where it becomes a 3d icon, set it on a desk -> inspect a 'folder' inside it containing all it's attributes (brush radius, feather, color etc) further down the line, the actual python running the object will be visible inside one of these 'nests' or logic noodles this way tools can be shared in a .blend (some work will need to be done for infrastructure later to make more flexible 'chunks' used to make tools in py calling compiled code) maybe also have 'active tool button bindings' in a folder* (like when a tool is 'bound' to a VR handle the tool itself has the shortcuts in it for how to interact with it)

Jacob Merrill commented

2019-11-05 01:11:44 +01:00

another important thing to consider is that gesture recognition has went open source.
sign language can become input gestures.

https://venturebeat.com/wp-content/uploads/2019/08/image2-3.gif?w=341&resize=341%2C400&strip=all

https://venturebeat.com/2019/08/19/google-open-sources-gesture-tracking-ai-for-mobile-devices/

another important thing to consider is that gesture recognition has went open source. sign language can become input gestures. https://venturebeat.com/wp-content/uploads/2019/08/image2-3.gif?w=341&resize=341%2C400&strip=all https://venturebeat.com/2019/08/19/google-open-sources-gesture-tracking-ai-for-mobile-devices/

Ben commented

2019-11-05 01:42:03 +01:00

On the Topic of UI.

This has been solved by tools such as [Tvori ]] and [ https:*nvrmind.io/#features | AnimVR . I would recommend looking at what they have done to solve this issue. Things that would be helpful would be the ability to pin, Scale uniformly, and move panels such as the timeline, Asset Management window, etc in 3D space for example.

Now on the topic of Animating rigged characters. It would be to be able to pose and Animate characters in VR using motion controllers.

Something like this example would be most welcome.

Hope this post has been helpful in some way.

On the Topic of UI. This has been solved by tools such as [Tvori ]] and [[ https:*nvrmind.io/#features | AnimVR ](https:*www.youtube.com/watch?v=gyGL6x0aPrc). I would recommend looking at what they have done to solve this issue. Things that would be helpful would be the ability to pin, Scale uniformly, and move panels such as the timeline, Asset Management window, etc in 3D space for example. Now on the topic of Animating rigged characters. It would be to be able to pose and Animate characters in VR using motion controllers. Something like [this example ](https://www.youtube.com/watch?v=_oxrQDBr6Mo) would be most welcome. Hope this post has been helpful in some way.

Dalai Felinto commented

2019-11-05 01:47:08 +01:00

@Mr_SquarePeg the idea is to support different use cases, and different UI/UX experiments via our Python Add-on API. This way interested developer teams can maintain complete solutions without us having to pick and choose a single definitive VR user interface.

Ben commented

2019-11-05 01:56:02 +01:00

@dfelinto That is understandable. I just want the default UI Solution for VR at least for animation and Scene Layout to be awesome! Which I have no doubt will be accomplished. Hence why I am making these suggestions early on.

Max Krichenbauer commented

2019-11-29 01:19:11 +01:00

Added subscriber: @makx

Max Krichenbauer commented

2019-11-29 01:19:11 +01:00

@JacobMerrill-1 regarding gesture input:
The link you posted is something different though: here "gesture" means estimating a hand shape from a picture (for example making a "peace" sign).
But in VR you usually don't have cameras tracking your hands and gestures are more like "drawing a 3D shape".
A more closely related example would be the gesture feature in Maya / MARUI: https://www.youtube.com/watch?v=Z2tI3CgG264

@JacobMerrill-1 regarding gesture input: The link you posted is something different though: here "gesture" means estimating a hand shape from a picture (for example making a "peace" sign). But in VR you usually don't have cameras tracking your hands and gestures are more like "drawing a 3D shape". A more closely related example would be the gesture feature in Maya / MARUI: https://www.youtube.com/watch?v=Z2tI3CgG264

Jacob Merrill commented

2019-11-29 03:56:29 +01:00

@makx - https://www.youtube.com/watch?v=2VkO-Kc3vks

Ben commented

2019-11-29 05:18:38 +01:00

In #68998#820748, @JacobMerrill-1 wrote:
@makx - https://www.youtube.com/watch?v=2VkO-Kc3vks

As far as I am aware the Oculus Quest does not have OpenXR support yet which is what Blender's VR support is based on. That said it would be better to use the controllers included with the Quest and Rift S because they are more accurate at this time, Use the leap motion, or use the Valve Index controllers when they release their OpenXR run time.

> In #68998#820748, @JacobMerrill-1 wrote: > @makx - https://www.youtube.com/watch?v=2VkO-Kc3vks As far as I am aware the Oculus Quest [does not have OpenXR support yet ](https://uploadvr.com/oculus-rift-basic-openxr-support/) which is what Blender's VR support is based on. That said it would be better to use the controllers included with the Quest and Rift S because they are more accurate at this time, Use the leap motion, or use the Valve Index controllers when they release their OpenXR run time.

Julian Eisel removed their assignment 2019-12-09 15:30:57 +01:00

Julian Eisel commented

2019-12-09 15:30:57 +01:00

This task is used as parent of all other VR tasks which are planned to be implemented by multiple teams. No reason to have this assigned to anybody.

Swann Martinez commented

2019-12-10 18:07:11 +01:00

Added subscriber: @slumber

Paul McManus commented

2020-01-16 13:27:35 +01:00

Added subscriber: @PaulMcManus

hujin commented

2020-02-04 11:59:33 +01:00

Added subscriber: @2046411367

Satish Goda commented

2020-02-08 01:54:44 +01:00

Added subscriber: @satishgoda1

Florian Riedelbauch commented

2020-02-09 11:16:21 +01:00

Added subscriber: @leFlo

Sky commented

2020-02-11 14:46:40 +01:00

Added subscriber: @Skyfish

huajian qiu commented

2020-02-26 00:50:32 +01:00

Added subscriber: @wakin

Berk Elyesa Yıldırım commented

2020-02-29 13:27:25 +01:00

Added subscriber: @Beryesa

Daniel Mecs commented

2020-04-05 01:07:49 +02:00

Added subscriber: @Sombrero

Christopher Underwood commented

2020-04-09 21:43:28 +02:00

Added subscriber: @Somnolent

junic yam commented

2020-05-29 23:37:01 +02:00

Added subscriber: @jamad

teh bucket commented

2020-06-12 00:12:32 +02:00

Added subscriber: @Teh_Bucket

teh bucket commented

2020-06-12 00:12:32 +02:00

Since I haven't seen it discussed, I should add that live capture of head, controllers, and additional trackers is very useful for animating bipeds. A simple IK setup and basic python script could be made by the user, so long as additional trackers are accessible within the blender XR system.

Osman Zeki commented

2020-06-12 03:19:03 +02:00

Removed subscriber: @Ozeki

Jacob Merrill commented

2020-06-12 07:07:36 +02:00

"Since I haven't seen it discussed, I should add that live capture of head, controllers, and additional trackers is very useful for animating bipeds. A simple IK setup and basic python script could be made by the user, so long as additional trackers are accessible within the blender XR system."

yeah animating agents in real time may be kinda neat too- like avatar.
Glycon is a software a friend of mine makes that does this using this same stuff ($paid$)

"Since I haven't seen it discussed, I should add that live capture of head, controllers, and additional trackers is very useful for animating bipeds. A simple IK setup and basic python script could be made by the user, so long as additional trackers are accessible within the blender XR system." - yeah animating agents in real time may be kinda neat too- like avatar. Glycon is a software a friend of mine makes that does this using this same stuff ($paid$)

Jeffrey Klug commented

2020-06-19 20:07:16 +02:00

Added subscriber: @JeffreyKlug