XRManager: Add more integrated support for WebXR Layers.

[cabanier]

This PR will introduce deeper integration of XR Layers in XRManager.
Layers will be emulated on devices that don't support them.

There are still some issues (ie color mapping is wrong) and the layers are not using multisampling but those can be addressed in later PR's.

This contribution is funded by Meta

[github-actions]

📦 Bundle size

Full ESM build, minified and gzipped.

	Before	After	Diff
WebGL	336.02 78.26	336.02 78.26	+0 B +0 B
WebGPU	520.52 145.15	524.8 146.19	+4.27 kB +1.04 kB
WebGPU Nodes	519.99 145.04	524.26 146.19	+4.27 kB +1.15 kB

🌳 Bundle size after tree-shaking

Minimal build including a renderer, camera, empty scene, and dependencies.

	Before	After	Diff
WebGL	465.06 112.14	465.06 112.14	+0 B +0 B
WebGPU	591.01 160.79	597.65 162.47	+6.64 kB +1.67 kB
WebGPU Nodes	546.13 150.23	552.78 151.95	+6.64 kB +1.72 kB

[cabanier]

Can this be merged? I see some other issues in XRManager that I'd like to address.

[mrdoob]

I finally tried this example on my Quest 2.

When I go immersive the layers seem to be much darker than when outside immersive.
When I leave immersive the layers are upside down.

@cabanier Known issues? Should I take screenshots?

[cabanier]

I finally tried this example on my Quest 2.

When I go immersive the layers seem to be much darker than when outside immersive.

That wasn't happening before. Layers should be using the 2 pass rendering to do correct color managment.

When I leave immersive the layers are upside down.

@cabanier Known issues? Should I take screenshots?

These are both new. I will take a look.

[cabanier]

Things look very broken. It doesn't look like there's a depth buffer attached when rendering webxr...

[cabanier]

84b6feb7837d02cd445cfe2b17f2e5453c053a80 seems to be the cause of the breakage. I'll look into fixing it

[cabanier]

@mrdoob I don't see things rendered upside down when exiting XR, instead there are a bunch of GL errors. Are you on the dev branch?

[cabanier]

84b6feb7837d02cd445cfe2b17f2e5453c053a80 seems to be the cause of the breakage. I'll look into fixing it

#31099 will fix the errors introduced by that change and the problem were layers were upside down.
The darker layer rendering is still there. Even the scene itself looks darker to me. @Mugen87 , wasn't this supposed to be fixed with the 2 pass rendering?

[Mugen87]

If the colors of the entire beauty pass are wrong, something with the color space conversion isn't right. When colors are too dark, it's normally an indication that the conversion to sRGB is missing. I have to debug the layers example so I say more about the issue.

https://rawcdn.githack.com/mrdoob/three.js/980d7cab64e4a262c3bb0a78ad2b5f626f61e64d/examples/webgpu_xr_native_layers.html

[Mugen87]

BTW: I have noticed below WebGL warnings in webgpu_xr_native_layers and webgpu_xr_cubes that are logged while presenting. They are not present in production.

[.WebGL-0x720013cd00]GL ERROR :GL_INVALID_FRAMEBUFFER_OPERATION : glClearBufferfv: framebuffer
webgpu_xr_native_layers.html:1
[.WebGL-0x720013cd00]GL ERROR :GL_INVALID_FRAMEBUFFER_OPERATION : glClearBufferfi: framebuffer
[.WebGL-0x720013cd00]GL ERROR :GL_INVALID_FRAMEBUFFER_OPERATION : glDrawArrays: framebuffer
webgpu_xr_native_layers.html:1
[.WebGL-0x720013cd00]GL ERROR :GL_INVALID_FRAMEBUFFER_OPERATION : glDrawElements: framebuffer
webgpu_xr_native_layers.html:1
[.WebGL-0x720013cd00]GL ERROR :GL_INVALID_FRAMEBUFFER_OPERATION : glDrawElements: framebuffer
webgpu_xr_native_layers.html:1
[.WebGL-0x720013cd00]GL ERROR :GL_INVALID_FRAMEBUFFER_OPERATION : glDrawElements: framebuffer

[Mugen87]

Okay, so I think the issue is clear to me now.

1. When rendering webgpu_xr_native_layers without XR, the scenes of both layers are rendered into a render target and then used as a texture on a mesh for the actual scene. Hence, the (second) output pass will process the entire contents of the final image.

2. When rendering in XR, the material of the layer plane is switched to a non-textured version here:
   

   three.js/src/renderers/common/XRManager.js

   Lines 908 to 909 in 980d7ca

   	// change material so it "punches" out a hole to show the XR Layer.
   	layer.plane.material = new MeshBasicMaterial( { color: 0xffffff, side: layer.type === 'cylinder' ? BackSide : FrontSide } );

   
   Because of that, I believe Renderer.isOutputTarget must evaluate to true while rendering a layer in XR. That is currently not the case.

I think it should be possible to fix the issue by evaluating the wasPresenting flag in renderLayers() and if true, execute this code:

this._renderer.setOutputRenderTarget( layer.renderTarget );

And and the end of the method:

this._renderer.setOutputRenderTarget( null );

@cabanier Do you see any issues with that approach?

[cabanier]

BTW: I have noticed below WebGL warnings in webgpu_xr_native_layers and webgpu_xr_cubes that are logged while presenting. They are not present in production.

[.WebGL-0x720013cd00]GL ERROR :GL_INVALID_FRAMEBUFFER_OPERATION : glClearBufferfv: framebuffer
webgpu_xr_native_layers.html:1

I have noticed that as well. It looks like a frame executes at at the start but the rendertarget isn't set up correctly yet. I will see why that is

[cabanier]

Okay, so I think the issue is clear to me now.
...
@cabanier Do you see any issues with that approach?

I tried that but it didn't seem to work. Are you saying that in the non-xr case, the color of the layers is adjusted twice which makes it too bright?

[Mugen87]

The non-XR case looks right, imo. The layer colors during XR are too dark because the output pass is missing.

When I understand the code correctly, rendering the layer while presenting must follow the same pattern as the XR pass. So the layer is rendered into an intermediate render target and then the second output pass is performed. That does currently not happen since the layer pass is treated like a normal RTT pass. So no color space conversion happens right now when presenting.

[cabanier]

I was able to fix the runtime error. PR is #31108
I'm running into some issue with adding the output pass. That pass stores a new rendertarget under _frameBufferTarget but that should be different between layers and regular xr. I'm still investigating

[cabanier]

@Mugen87 I updated the PR and now everything renders as expected. It is very slow though... I'll look into that next

[cabanier]

I finally tried this example on my Quest 2.

When I go immersive the layers seem to be much darker than when outside immersive.

#31124 will fix this. I'm still investigating the performance problem.