WebXR Composition Layers — High-Performance Rendering

WebXR Hand Tracking — Von Gelenken zu Gesten

Ein praxisnaher Deep-Dive in die WebXR Hand Tracking API — Gelenkpositionen, Pinch-Erkennung, benutzerdefinierte Gestenerkennung und Interaktionsmuster für immersive VR-Anwendungen.

WebXR Hand Tracking — From Joints to Gestures

A practical deep-dive into the WebXR Hand Tracking API — joint positions, pinch detection, custom gesture recognition, and interaction patterns for immersive VR applications.

Standard WebXR rendering works by drawing everything to a WebGL framebuffer, which the runtime then copies to the headset display. This works, but it's inefficient for many common use cases — especially video, 2D UIs, and background environments.

WebXR Composition Layers let you bypass the app framebuffer for specific content, handing it directly to the GPU compositor. The result: lower power consumption, higher refresh rates, and better visual quality for layered content.

Why Layers?

Layers shine for:

• Video — 180°/360° video, 2D video panels, video call feeds
• 2D UI — Dashboards, menus, notifications (render once, display every frame)
• Backgrounds — 360° photos, environment maps, skyboxes
• Passthrough — In AR modes, layers composite on top of the real world

The trade-off: you cannot apply per-pixel Three.js effects to layer content. Layers are for content that doesn't need app-level shader processing.

Layer Types

WebXR defines four layer types, each serving a different purpose:

XRProjectionLayer

This is the default layer — the one that replaces the standard WebGL framebuffer. Create it during session setup:

const glBinding = new XRWebGLBinding(session, gl);
const projectionLayer = glBinding.createProjectionLayer({
  textureType: "texture", // 'texture' or 'texture-array'
  colorFormat: gl.RGBA8,
  depthFormat: gl.DEPTH_COMPONENT24,
  scaleFactor: 1.0, // Resolution scale (0.5 = half res)
});
session.updateRenderState({
  layers: [projectionLayer], // Must always be the first layer
});

The scaleFactor property is critical for performance:

• 1.0 = full device resolution
• 0.75 = good balance on Quest 3
• 0.5 = performance mode for complex scenes

XRQuadLayer

A flat, rectangular surface positioned in 3D space. Perfect for 2D content:

const quadLayer = glBinding.createQuadLayer({
  space: referenceSpace,
  layout: "mono",
  width: 2, // meters
  height: 1.5, // meters
  pixelWidth: 1024,
  pixelHeight: 768,
  blendTextureSourceAlpha: true,
});

// Position the quad
quadLayer.transform = new XRRigidTransform({
  x: 0,
  y: 1.5,
  z: -2, // 2m in front, 1.5m up
});

session.updateRenderState({
  layers: [projectionLayer, quadLayer],
});

Use cases: 2D dashboards in VR, video player UI, data visualization panels.

XRCylinderLayer

A curved display surface, useful for immersive dashboards and menus:

const cylinderLayer = glBinding.createCylinderLayer({
  space: referenceSpace,
  layout: "stereo-left-right",
  radius: 3, // meters
  centralAngle: Math.PI * 0.5, // 90 degree arc
  aspectRatio: 1.5, // height = radius * aspect
  pixelWidth: 2048,
  pixelHeight: 2048,
});

Use cases: Car dashboards, cockpit UIs, panoramic menus, monitor walls.

XREquirectangularLayer

A spherical (360°) layer for environment backgrounds and 360° media:

const equiLayer = glBinding.createEquirectangularLayer({
  space: referenceSpace,
  layout: "stereo-left-right",
  centralAngle: Math.PI * 2, // Full sphere
  radius: 0,
  pixelWidth: 4096,
  pixelHeight: 2048,
});

Use cases: 360° photos as environments, 360° video backgrounds, skyboxes.

Compositing Layers Together

Layers are composited in back-to-front order based on their position in the array:

// Background layer (equirectangular)
// ↓ composited behind
// Quad layer (dashboard)
// ↓ composited behind
// Projection layer (main 3D scene)
session.updateRenderState({
  layers: [projectionLayer, quadLayer, equiLayer],
});

Wait — this is actually front-to-back in the array order. The first layer is rendered first (closest to the viewer). The runtime composits them in the natural over-draw order: layers later in the array appear on top of earlier ones.

Important: Not all runtimes support more than 2-3 simultaneous layers. Check session.supportedLayers to determine the maximum.

Video Playback with XRMediaBinding

Video is where composition layers truly shine. Instead of decoding video to a canvas and uploading to a WebGL texture each frame, XRMediaBinding hands the video decoder output directly to the compositor.

Setup

const mediaBinding = new XRMediaBinding(session);

// Create a hidden video element
const video = document.createElement("video");
video.src = "/videos/my-360-video.mp4";
video.loop = true;
video.muted = true; // Required for autoplay policies
video.crossOrigin = "anonymous";
await video.play();

// Create a quad layer bound to the video
const videoLayer = mediaBinding.createQuadLayer(video, {
  space: referenceSpace,
  layout: "mono",
  width: 4,
  height: 2.25, // 16:9 aspect ratio
  invertStereo: false,
});

// Position the video screen
videoLayer.transform = new XRRigidTransform({
  x: 0,
  y: 1.5,
  z: -3,
  y: 0,
  z: 0,
  w: 1, // Quaternion (facing viewer)
});

session.updateRenderState({
  layers: [projectionLayer, videoLayer],
});

360° Video

For 360° video, use an equirectangular layer instead:

const videoLayer = mediaBinding.createEquirectangularLayer(video, {
  space: referenceSpace,
  layout: "stereo-left-right", // Most 360° video is stereoscopic
  centralAngle: Math.PI * 2,
  radius: 0,
  pixelWidth: 3840,
  pixelHeight: 1920,
});

Playback Controls in VR

Since the video element lives in the DOM, you can control it programmatically:

// Play/pause via gesture
function togglePlayback() {
  if (video.paused) {
    video.play();
  } else {
    video.pause();
  }
}

// Volume control
function setVolume(level: number) {
  video.volume = Math.max(0, Math.min(1, level));
}

The video layer automatically reflects changes to playback state, volume, and looping.

Performance Benefits

Compared to canvas-based video rendering:

Drawing Content to Layers

For non-video content, you need to render to the layer's color texture:

// Each frame, get the layer's sub-image
const layer = quadLayer;
const subImage = glBinding.getSubImage(layer, frame, eye);

// Bind the color texture
gl.bindTexture(gl.TEXTURE_2D, subImage.colorTexture);

// Bind the depth/stencil texture (optional)
gl.bindTexture(gl.TEXTURE_2D, subImage.depthStencilTexture);

// Render your content to the sub-image
gl.viewport(0, 0, subImage.viewport.width, subImage.viewport.height);
// ... draw calls ...

For quad layers, you typically render a fullscreen quad. For cylinder layers, you render a warped projection. For equirectangular, you render a panoramic image.

Fallback Strategy

Not all browsers or devices support composition layers. Always implement a fallback:

function createLayers(session: XRSession, gl: WebGL2RenderingContext) {
  if (session.supportedLayers?.length > 1) {
    // Use composition layers
    return createCompositionLayers(session, gl);
  } else {
    // Fall back to standard WebGL rendering
    session.updateRenderState({
      baseLayer: new XRWebGLLayer(session, gl),
    });
  }
}

Performance Tuning

• Layer count: Keep it minimal. Each layer adds GPU composition pass overhead.
• Texture resolution: Match pixel dimensions to what the content needs. A 4K video doesn't need an 8K quad layer.
• Blend modes: Use alpha-blend for transparent layers, opaque for solid content.
• Fixed foveated rendering (FFR): Available on some devices. Sets lower resolution in peripheral vision:

session.updateRenderState({
  layers: [projectionLayer],
  foveationLevel: 0.5, // 0.0 = no foveation, 1.0 = max
});

• Layer updates: Don't update layer transforms every frame unless necessary. Static layers can be set once and left alone.

Series Cross-References

• Building a WebXR App with Three.js — Integrate layers into your application
• WebXR Anchors & Plane Detection — Anchor layers in world space

References

• WebXR Composition Layers Spec (W3C)
• MDN: XRCompositionLayer
• Meta Quest: Layer Performance