OpenXR: Virtual Monitor experiment #102182

New Issue

Jeroen Bakker · 2022-10-31T13:01:30+01:00

Jeroen Bakker commented

2022-10-31 13:01:30 +01:00

During Blender conference there was a discussion explaining a limitation for using Blender in a virtual production.
Most of the time was spend on explaining what should be done, what lacked in looking towards how blender is actually being used in such setup.

This task will gather some feedback from users to find out what is needed to integrate a virtual monitor inside a scene.
The provided solution was to render the virtual camera to an texture, and after that render the main scene camera that could include the texture from the virtual camera.

During the conference I thought of several ways how to do it (from a coding perspective)

Just wait until the Viewport Compositor can handle multiple scenes and "None blocking rendering".
Have a special shading node that could input the texture of the virtual camera. The material can than be put on any geometry.

Although both options are possible and would lead to similar performance I think that only the first option would be acceptable for blender in the longer run. The second option could be added to an experimental branch for testing purposes as it doesn't seem to be that complicated.

The main issue I still have is to have an overview of the exact problem. What I got out of it so far:

The virtual camera is controlled by a controller via openXR. It shows the visible frustum of the scene.
This part is rendered and displayed in the same scene on an image plane. This image plane will than act as a virtual monitor.
The main camera is rendered and displayed to the physical display(s).

Use cases for this:

View output of a virtual camera when still having an VR headset on. Eg. look at the camera controller and see what that camera is recording, when inside an XR session.

This is an experiment to see if this is possible and to demonstrate the consequences of this from user and developer perspective. After this experiment it should be possible to use it in an experimental branch. I don't think it should be in master yet as unless the consequences have been discussed and further evaluated.

For discussion of this task use #xr channel on blender chat.
Branch is called temp-xr-virtual-camera-experiment.

Technical details:

draw manager instance data is currently stored in 3d viewport, each virtual camera could have its own GPUOffscreen...
The virtual camera texture will only exists as GPUTexture. It will not be available on the CPU for performance reasons. GPUTexture is in linear color space.
Image planes are only visible in Eevee. as it uses a regular texture and material. To hide internal complexity a new Virtual Camera node will be available in Material shader graph.
When rendering the virtual camera, anu geometry using a Virtual Camera Node will not be drawn.

0001-1813.mp4

TODO:

Add Virtual Camera Node
Add runtime data structures to Camera.
Detect existence of virtual camera node and update its texture. We can do this outside the draw manager for clarity as it is a hack.
Hook generated GPU texture to the Virtual camera node.
Create temp view3d for correct view/win mats.
Enable this for OpenXR session.

Known limitations:

Performance.
Only Eevee is supported. Cycles could be supported and perhaps even without rendering twice.
Virtual camera texture is visible in reflections/refractions, shadow. Should be turned of by user.

During Blender conference there was a discussion explaining a limitation for using Blender in a virtual production. Most of the time was spend on explaining what should be done, what lacked in looking towards how blender is actually being used in such setup. This task will gather some feedback from users to find out what is needed to integrate a virtual monitor inside a scene. The provided solution was to render the virtual camera to an texture, and after that render the main scene camera that could include the texture from the virtual camera. During the conference I thought of several ways how to do it (from a coding perspective) 1. Just wait until the Viewport Compositor can handle multiple scenes and "None blocking rendering". 2. Have a special shading node that could input the texture of the virtual camera. The material can than be put on any geometry. Although both options are possible and would lead to similar performance I think that only the first option would be acceptable for blender in the longer run. The second option could be added to an experimental branch for testing purposes as it doesn't seem to be that complicated. The main issue I still have is to have an overview of the exact problem. What I got out of it so far: * The virtual camera is controlled by a controller via openXR. It shows the visible frustum of the scene. * This part is rendered and displayed in the same scene on an image plane. This image plane will than act as a virtual monitor. * The main camera is rendered and displayed to the physical display(s). Use cases for this: * View output of a virtual camera when still having an VR headset on. Eg. look at the camera controller and see what that camera is recording, when inside an XR session. This is an experiment to see if this is possible and to demonstrate the consequences of this from user and developer perspective. After this experiment it should be possible to use it in an experimental branch. I don't think it should be in master yet as unless the consequences have been discussed and further evaluated. For discussion of this task use #xr channel on blender chat. Branch is called `temp-xr-virtual-camera-experiment`. Technical details: * draw manager instance data is currently stored in 3d viewport, each virtual camera could have its own GPUOffscreen... * The virtual camera texture will only exists as GPUTexture. It will not be available on the CPU for performance reasons. GPUTexture is in linear color space. * Image planes are only visible in Eevee. as it uses a regular texture and material. To hide internal complexity a new Virtual Camera node will be available in Material shader graph. * When rendering the virtual camera, anu geometry using a Virtual Camera Node will not be drawn. [0001-1813.mp4](https://archive.blender.org/developer/F13891417/0001-1813.mp4) TODO: - [x] Add Virtual Camera Node - [x] Add runtime data structures to Camera. - [x] Detect existence of virtual camera node and update its texture. We can do this outside the draw manager for clarity as it is a hack. - [x] Hook generated GPU texture to the Virtual camera node. - [x] Create temp view3d for correct view/win mats. - [x] Enable this for OpenXR session. Known limitations: * Performance. * Only Eevee is supported. Cycles could be supported and perhaps even without rendering twice. * Virtual camera texture is visible in reflections/refractions, shadow. Should be turned of by user.

Jeroen Bakker commented

2022-10-31 13:01:30 +01:00

Changed status from 'Needs Triage' to: 'Confirmed'

Jeroen Bakker commented

2022-10-31 13:01:30 +01:00

Added subscriber: @Jeroen-Bakker

Jeroen Bakker changed title from ~~Virtual Studio~~ to OpenXR: Virtual Camera integration [Draft]

2022-10-31 13:06:00 +01:00

Sky commented

2022-10-31 14:05:30 +01:00

Added subscriber: @Skyfish

Sky commented

2022-10-31 14:05:30 +01:00

Could you add a link to where in the blender conference the VR was discussed?

What is meant by a virtual camera in this context?
I understand it to mean a camera object that you can grab and handle in VR, that displays what it sees in the VR space as though using a real life camera?
You can see examples of this in practice in other VR applications like Oculus/Adobe medium, where you can take a screenshot with a camera-like thing.

Could you add a link to where in the blender conference the VR was discussed? What is meant by a virtual camera in this context? I understand it to mean a camera object that you can grab and handle in VR, that displays what it sees in the VR space as though using a real life camera? You can see examples of this in practice in other VR applications like Oculus/Adobe medium, where you can take a screenshot with a camera-like thing.

Jeroen Bakker commented

2022-10-31 14:09:50 +01:00

Added subscriber: @sebastian_k

Jeroen Bakker commented

2022-10-31 14:11:32 +01:00

@Skyfish it was a discussion that was done outside the official agenda. I added this ticket as I was asked to come with some ideas how this could work. Currently I am looking into clearing up the design what everything means, therefore this is set as a draft design.

Jacob Merrill commented

2022-10-31 19:45:45 +01:00

Added subscriber: @JacobMerrill-1

Jacob Merrill commented

2022-10-31 19:46:36 +01:00

realtime composition in AR is also important to overlay the rendered scene on top of the incoming pass through video

Jeroen Bakker changed title from ~~OpenXR: Virtual Camera integration [Draft]~~ to OpenXR: Virtual Camera experiment

2022-11-02 08:05:21 +01:00

Sebastian Koenig commented

2022-11-02 13:55:53 +01:00

Added subscriber: @SimeonConzendorf

Pablo commented

2022-11-03 06:49:57 +01:00

Added subscriber: @OtherRealms

Kazashi Yoshioka commented

2022-11-03 09:12:39 +01:00

Added subscriber: @vnapdv

Brecht Van Lommel commented

2022-11-03 10:42:42 +01:00

Added subscriber: @brecht

Brecht Van Lommel commented

2022-11-03 10:42:42 +01:00

For reference here is an explanation of terminology used in virtual production:
https://www.vpglossary.com/

What's a bit confusing here is that the term "virtual camera" in this task is used both for the camera object and the camera feed. These may come from e.g. the same mobile device, but not necessarily, or there may be no camera feed at all. I suggest to use the term "camera feed" when referring to the image coming from the a real camera.

For reference here is an explanation of terminology used in virtual production: https://www.vpglossary.com/ What's a bit confusing here is that the term "virtual camera" in this task is used both for the camera object and the camera feed. These may come from e.g. the same mobile device, but not necessarily, or there may be no camera feed at all. I suggest to use the term "camera feed" when referring to the image coming from the a real camera.

RedMser commented

2022-11-03 13:00:36 +01:00

Added subscriber: @RedMser

Jacob Merrill commented

2022-11-03 17:13:50 +01:00

can the XR camera passthrough be exposed as well for AR as a image source?

meta quest 2 has black and white video, quest pro has pass through hd color.

can the XR camera passthrough be exposed as well for AR as a image source? meta quest 2 has black and white video, quest pro has pass through hd color.

Sebastian Koenig commented

2022-11-04 09:11:14 +01:00

In #102182#1440430, @brecht wrote:
For reference here is an explanation of terminology used in virtual production:
https://www.vpglossary.com/

What's a bit confusing here is that the term "virtual camera" in this task is used both for the camera object and the camera feed. These may come from e.g. the same mobile device, but not necessarily, or there may be no camera feed at all. I suggest to use the term "camera feed" when referring to the image coming from the a real camera.

I agree, more precise terminology is needed.
For example, as far as I know in the goal for this specific there wouldn't even be a "real camera", like no physical one. Not to say that this would not also be useful :)
Though according to that glossary the term "virtual camera" is actully accurate:

A camera in a real-time engine which behaves the same way a real-world camera would with respect to optics, aspect ratio, etc. A Vcam can be manipulated using a tracked device such as a mobile device, tablet, game controller, or a physical object with a tracking reference attached such as a real-world tripod, dolly, crane, drone, etc.

But which term do we use for the viewport camera, the one that is rendered to the VR headset?

> In #102182#1440430, @brecht wrote: > For reference here is an explanation of terminology used in virtual production: > https://www.vpglossary.com/ > > What's a bit confusing here is that the term "virtual camera" in this task is used both for the camera object and the camera feed. These may come from e.g. the same mobile device, but not necessarily, or there may be no camera feed at all. I suggest to use the term "camera feed" when referring to the image coming from the a real camera. I agree, more precise terminology is needed. For example, as far as I know in the goal for this specific there wouldn't even be a "real camera", like no physical one. Not to say that this would not also be useful :) Though according to that glossary the term "virtual camera" is actully accurate: > A camera in a real-time engine which behaves the same way a real-world camera would with respect to optics, aspect ratio, etc. A Vcam can be manipulated using a tracked device such as a mobile device, tablet, game controller, or a physical object with a tracking reference attached such as a real-world tripod, dolly, crane, drone, etc. But which term do we use for the viewport camera, the one that is rendered to the VR headset?

Brecht Van Lommel commented

2022-11-04 13:00:57 +01:00

My understanding is that "virtual camera" usually refers to the analog of a physical camera on set. Not a VR headset where someone arbitrarily walks and looks around, but a camera that is framing the shot the way the director wants it to be filmed.

And so if you're making a film for TV or cinema, the output of the virtual camera is usually displayed on a regular monitor. No VR needed.

You can involve a VR headset in this as well, but to me it seems mostly orthogonal to the concept of a virtual camera. You can position the virtual camera with various controller devices, and you can use the virtual camera to render for various display devices.

Re-reading the task, it sounds like the idea is to have some kind of "virtual monitor" that you can place in the scene, in case you want to work fully in a VR headset? That virtual monitor would display the output of a camera in the scene, but it does not need to make assumptions about how that camera is controlled. For example the camera motion could have been keyframed in advance, or the same virtual camera may be used for display on a virtual monitor in VR and a physical monitor at the same time.

What got me confused is that the first few sentences of the task seem to imply this is about solving fundamental issues for virtual production, whereas this seems to be about a very specific workflow that I'm guessing is not the norm.

My understanding is that "virtual camera" usually refers to the analog of a physical camera on set. Not a VR headset where someone arbitrarily walks and looks around, but a camera that is framing the shot the way the director wants it to be filmed. And so if you're making a film for TV or cinema, the output of the virtual camera is usually displayed on a regular monitor. No VR needed. You can involve a VR headset in this as well, but to me it seems mostly orthogonal to the concept of a virtual camera. You can position the virtual camera with various controller devices, and you can use the virtual camera to render for various display devices. Re-reading the task, it sounds like the idea is to have some kind of "virtual monitor" that you can place in the scene, in case you want to work fully in a VR headset? That virtual monitor would display the output of a camera in the scene, but it does not need to make assumptions about how that camera is controlled. For example the camera motion could have been keyframed in advance, or the same virtual camera may be used for display on a virtual monitor in VR and a physical monitor at the same time. What got me confused is that the first few sentences of the task seem to imply this is about solving fundamental issues for virtual production, whereas this seems to be about a very specific workflow that I'm guessing is not the norm.

Aaron Carlisle commented

2022-11-04 14:11:06 +01:00

Added subscriber: @Blendify

Satish Goda commented

2022-11-07 03:41:45 +01:00

Added subscriber: @satishgoda1

Jeroen Bakker commented

2022-11-09 13:32:13 +01:00

I agree with the term "virtual monitor" It took me some time to understand the case what is needed and agree that we should use consistent and correct names for these terms.

Jeroen Bakker changed title from ~~OpenXR: Virtual Camera experiment~~ to OpenXR: Virtual Monitor experiment

2022-11-09 14:58:46 +01:00

Sebastian Koenig commented

2022-11-10 08:40:03 +01:00

I agree with the term as well. It's a virtual monitor attached to a virtual camera. :)

Simeon Conzendorf commented

2022-11-17 10:50:37 +01:00

So cool, that you started working in this so fast. Because of the released windows build I could test it. I justed started a small scene "A virtual monitor showing the default cube :)" and it worked. But trying to do more, it crashed a lot.

Two spontaneous thoughts:
Should I report bugs/crashes here (or somewhere else) or is it just too early?
Important (and independent settings) for the virtual camera would be aspect and resolution. Changing the fov is one of the things, which crashes.

I'm looking forward to test and help, where I can.

So cool, that you started working in this so fast. Because of the released windows build I could test it. I justed started a small scene "A virtual monitor showing the default cube :)" and it worked. But trying to do more, it crashed a lot. Two spontaneous thoughts: Should I report bugs/crashes here (or somewhere else) or is it just too early? Important (and independent settings) for the virtual camera would be aspect and resolution. Changing the fov is one of the things, which crashes. I'm looking forward to test and help, where I can.

Simeon Conzendorf commented

2022-11-22 11:11:11 +01:00

Here's a compact crash report. If I can gather more useful information, tell me.
build: blender-3.5.0-alpha+temp-xr-virtual-camera-experiment.f7cdba506b0b-windows.amd64-release

It crashes when I...

browse the materials
want to show material properties
want to change the fov of the camera (and other settings too)
stop the vr session

Here's a video showing it. This all just start to crash after first usage of the virtual monitor node.
record_22-11-2022_10-55-07.mp4

Here's a compact crash report. If I can gather more useful information, tell me. build: blender-3.5.0-alpha+temp-xr-virtual-camera-experiment.f7cdba506b0b-windows.amd64-release It crashes when I... - browse the materials - want to show material properties - want to change the fov of the camera (and other settings too) - stop the vr session Here's a video showing it. This all just start to crash after first usage of the virtual monitor node. [record_22-11-2022_10-55-07.mp4](https://archive.blender.org/developer/F13959510/record_22-11-2022_10-55-07.mp4)

Peter Kim commented

2022-11-23 13:16:04 +01:00

Added subscriber: @muxed-reality

Peter Kim commented

2022-11-23 13:16:04 +01:00

In #102182#1449649, @SimeonConzendorf wrote:
It crashes when I...

browse the materials

want to show material properties

want to change the fov of the camera (and other settings too)

stop the vr session

The crash when stopping the VR session should be fixed now with db28a8b3d1.
The other crashes (browsing materials, showing material properties, changing camera settings) are due to a failed assert in gpu_framebuffer.cc L602, although I'm not sure why this occurs.

Here's a crash report using the xr-dev branch , which now includes the features from temp-xr-virtual-camera-experiment:
blender.crash.txt

> In #102182#1449649, @SimeonConzendorf wrote: > It crashes when I... > - browse the materials > - want to show material properties > - want to change the fov of the camera (and other settings too) > - stop the vr session The crash when stopping the VR session should be fixed now with db28a8b3d1. The other crashes (browsing materials, showing material properties, changing camera settings) are due to a failed assert in `gpu_framebuffer.cc` L602, although I'm not sure why this occurs. Here's a crash report using the `xr-dev` [branch ](https://builder.blender.org/download/experimental/xr-dev/), which now includes the features from `temp-xr-virtual-camera-experiment`: [blender.crash.txt](https://archive.blender.org/developer/F13963035/blender.crash.txt)

sebastien commented

2023-11-27 00:26:12 +01:00

Hi, i'm very interested by this experiment but for virtual production display
In this use case the problem is to get the projection of the main camera render onto arbitrary geometry (in general just planes or curved plane) and output the result unfolded (to display this image on the wall)
It's a bit like a realtime baking of the current render projected on simple geometry.

For this purpose i need 2 pass rendering like in this demo.

Where can i find the diff of this experiment to try to understand how it's made?
Thank you very much for all the stuff you did guys.

Hi, i'm very interested by this experiment but for virtual production display In this use case the problem is to get the projection of the main camera render onto arbitrary geometry (in general just planes or curved plane) and output the result unfolded (to display this image on the wall) It's a bit like a realtime baking of the current render projected on simple geometry. For this purpose i need 2 pass rendering like in this demo. Where can i find the diff of this experiment to try to understand how it's made? Thank you very much for all the stuff you did guys.

Peter Kim commented

2023-11-27 05:55:49 +01:00

@sebastientricoire Hi, you can use these diffs as a base but they may not apply cleanly as they were based on Blender 3.5-alpha and the xr-dev experimental branch:
3e725b55cf
786aad68b2

@sebastientricoire Hi, you can use these diffs as a base but they may not apply cleanly as they were based on Blender 3.5-alpha and the [xr-dev](https://projects.blender.org/blender/blender/src/branch/xr-dev) experimental branch: https://projects.blender.org/blender/blender/commit/3e725b55cf87287cdd65f99b2d654306b6766c9c https://projects.blender.org/blender/blender/commit/786aad68b2c210bcb2f56e2ae2bf5daa92b669eb

Sign in to join this conversation.

No Label

Download

What's New

Blender Studio

Manual

Developers Blog

Documentation

Benchmark

Blender Conference

Development Fund

One-time Donations

OpenXR: Virtual Monitor experiment #102182