Spatial UI & Interaction Design for WebXR

WebXR Hand Tracking: Building Touchless Interfaces

Learn to implement hand tracking in WebXR applications using the Hand Tracking API, including gesture recognition and accessibility considerations for immersive experiences.

Spatial UI & Interaction Design for WebXR

Spatial UI moves beyond flat screens into 3D environments where users interact with content using their hands, gaze, and controllers. Designing for this medium requires rethinking layout, typography, feedback, and accessibility from the ground up.

This guide covers proven patterns for building spatial interfaces in WebXR, with practical code examples and design principles drawn from production applications.

Why Spatial UI Is Different

The core rule: spatial UI should feel natural in the real world, not like a flat screen floating in VR.

Layout Fundamentals

Units and Scale

Use real-world units for spatial layout:

// 1 meter ≈ 1 unit in WebXR
const UI_DISTANCE = 1.5; // arm's reach
const BUTTON_SIZE = 0.08; // 8cm button
const TEXT_HEIGHT = 0.03; // 3cm text for readability

A button smaller than 5cm at arm's length is hard to tap. Text below 2cm becomes unreadable. Test your layouts at real-world distances.

The Viewport Funnel

Structure content in zones based on user attention:

1. Focal zone (0–30° center) — Primary content, confirmations
2. Peripheral zone (30–60°) — Secondary info, status
3. Background zone (60°+) — Ambient environment, not interactive

// Spatial zones helper
const ZONES = {
  focal: { x: 0, y: 0, z: -1.5 },
  peripheral: { x: 1.2, y: 0.3, z: -1.5 },
  background: { x: 0, y: 0, z: -5 },
};

Comfortable Reading

Use a curved billboard for text panels so every word is equidistant from the user:

const textPanel = new THREE.Mesh(
  new THREE.CylinderGeometry(0.8, 0.8, 0.5, 32, 1, true),
  textMaterial,
);
textPanel.rotation.x = -Math.PI / 2;

For dashboards and data-heavy UIs, consider a scrollable panel that the user reaches out to interact with, similar to a physical tablet.

Interaction Patterns

Direct Manipulation

Users should grab, tap, and drag objects as they would in the real world:

// Three.js drag interaction
controller.addEventListener("select", () => {
  const hit = raycast(intersectables);
  if (hit) {
    hit.object.userData.onSelect?.();
  }
});

controller.addEventListener("squeeze", () => {
  const grabbed = raycast(grabbables);
  if (grabbed) {
    attachToController(grabbed.object, controller);
  }
});

Gaze + Pinch

For hand tracking, use gaze for targeting and pinch for confirmation:

if (hand.pinchStrength > 0.8) {
  const target = gazeRaycast(scene.children);
  if (target) target.userData.onPinch?.();
}

This pattern is fatigue-free and works well on devices without controllers (Quest hand tracking, Vision Pro).

World-Locked vs Device-Locked

Visual Feedback

Spatial interactions need strong visual feedback since users can't feel buttons:

function InteractionFeedback({ hovered, selected, disabled }: FeedbackProps) {
  const scale = hovered ? 1.05 : 1;
  const color = selected ? "#4ade80" : disabled ? "#6b7280" : "#3b82f6";
  const opacity = disabled ? 0.4 : 1;

  return (
    <mesh scale={scale}>
      <planeGeometry args={[0.2, 0.08]} />
      <meshStandardMaterial color={color} transparent opacity={opacity} />
    </mesh>
  );
}

Always provide at least three feedback states: idle → hover → active.

Accessibility in Spatial UI

Audio Cues

Every interaction should have an audio response:

function playInteractionSound(type: "hover" | "select" | "error") {
  const buffers = {
    hover: audioBufferHover,
    select: audioBufferClick,
    error: audioBufferError,
  };
  const source = audioContext.createBufferSource();
  source.buffer = buffers[type];
  source.connect(audioContext.destination);
  source.start();
}

Visual Alternatives

Not all users can see clearly in VR. Support:

• High-contrast mode — Bold outlines, large text
• Voice labels — Speak element names on hover
• Alternate input — Keyboard/gamepad fallback for motor disabilities

Comfort Considerations

// Reduce motion for sensitive users
const userPrefersReducedMotion = window.matchMedia("(prefers-reduced-motion: reduce)").matches;

const ANIMATION_DURATION = userPrefersReducedMotion ? 0 : 300;

Performance Budget

Spatial UI must stay within the frame budget:

Use instanced meshes for repeated elements (buttons, labels):

const buttonInstances = new THREE.InstancedMesh(buttonGeometry, material, MAX_BUTTONS);

Putting It Together

A minimal spatial UI component:

export function SpatialButton({ position, label, onSelect, disabled }: SpatialButtonProps) {
  return (
    <group position={position}>
      <mesh onClick={onSelect} onPointerOver={() => playInteractionSound("hover")}>
        <planeGeometry args={[0.2, 0.08]} />
        <meshStandardMaterial
          color={disabled ? "#6b7280" : "#3b82f6"}
          transparent
          opacity={disabled ? 0.4 : 1}
        />
      </mesh>
      <SpatialText text={label} position={[0, 0, 0.001]} fontSize={0.025} />
    </group>
  );
}

Next Steps

• Explore cross-platform WebXR patterns for targeting Quest, Vision Pro, and mobile
• Read about AI-assisted XR tooling to accelerate spatial UI development
• Review the WebXR Hit Testing & Depth Sensing guide for real-world interaction

Designing spatial UI is still an emerging discipline. Test early, test often, and prioritize user comfort over visual polish.