XR Performance: Real-Time Inference Optimization | XR

Real-time XR demands relentless performance. VR headsets render at 90+ fps. Mixed reality overlays need to track your head with sub-millisecond latency. AI inference — for scene understanding, object recognition, or conversational NPCs — must complete in under 16ms to avoid frame drops.

This article is a practical guide to optimizing AI inference for XR workloads. We cover model quantization, NPU acceleration, hierarchical serving (device → edge → cloud), bandwidth optimization, and offloading frameworks that keep your XR app responsive.

The XR Latency Budget

Your total latency budget is 16.6ms per frame (60fps) or 11.1ms per frame (90fps). This budget must be shared among:

Component	Latency (typical)	Remaining budget (90fps)
6-DoF pose estimation	2-4ms	7-9ms
Scene graph traversal	1-2ms	5-8ms
AI inference	3-8ms (target)	0-5ms
Rendering (foveated)	4-6ms	-2ms to 1ms (over budget!)
Warp/compose	2-3ms	-5ms to -2ms

The AI inference budget is tight — and you can't steal from pose estimation or warping without causing nausea.

Rule of thumb: For VR, AI inference must complete in less than 10ms total per frame. For AR, you have slightly more headroom (up to 15ms) because the overlay doesn't cause motion sickness when delayed.

Model Quantization: The Free Speedup

Quantization reduces model precision from 32-bit floats to 8-bit integers, shrinking memory footprint 4x and accelerating inference 2-4x on CPUs, NPUs, and GPUs.

Supported Models

Model	FP16	INT8	Speedup (INT8 vs FP16)	Quality loss
LLaMA-7B	14GB	3.5GB	2-3x	less than 1% perplexity
Phi-3-mini	3GB	750MB	2-3x	Negligible
DistilBERT	420MB	105MB	3-4x	less than 0.5% F1
Florence-2 (VLM)	5GB	1.25GB	2-3x	less than 1% COCO mAP
MobileNetV3 (object det)	28MB	7MB	3-4x	less than 0.5% mAP

Quantization Workflow (PyTorch)

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "microsoft/Phi-3-mini-4k-instruct"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, torch_dtype=torch.float16)

# Static quantization (requires calibration data)
quantized_model = torch.quantization.quantize_dynamic(
    model,
    {torch.nn.Linear},
    dtype=torch.qint8
)

# Save for XR deployment
quantized_model.save_pretrained("./phi-3-int8")

Quantization for Web (TensorFlow.js)

import * as tf from "@tensorflow/tfjs";

async function quantizeModel(url) {
  // Load FP16 model
  const model = await tf.loadGraphModel(url);

  // Quantize to INT8
  const quantizedModel = tf.quantization.quantize(model, {
    quantizationBytes: 1, // 8-bit
    isSigned: true,
  });

  // Save for browser deployment
  await quantizedModel.save("indexeddb://quantized-xr-model");
}

Quality Considerations

Quantization is almost always safe for classification and detection tasks. For generative models (LLMs, diffusion), use:

Quantization-aware training (QAT): Train with simulated quantization for minimal loss
8-bit mixed precision: Keep attention heads in FP16, quantize MLP layers
GPTQ/AWQ: Advanced quantization that preserves LLM quality better than naive INT8

Avoid aggressive quantization (4-bit or lower) for generative models unless you're comfortable with degraded output quality.

NPU Acceleration: Hardware Without the Power Budget

Neural processing units (NPUs) are designed specifically for matrix operations — the core of AI inference. They offer 2-10x better performance per watt than CPUs or GPUs.

XR Device NPU Support

Device	NPU	TOPS (INT8)	Framework support
Meta Quest 3	Snapdragon 8 Gen 2 NPU	30+	Qualcomm QNN, ONNX Runtime
Apple Vision Pro	Apple Neural Engine	18+	Core ML, Metal
HoloLens 2	Snapdragon 850 NPU	4+	WinML, ONNX Runtime
Magic Leap 2	No dedicated NPU	—	CPU-only

Converting Models to ONNX for NPU

from transformers import AutoModel
import torch
import onnx
from onnxruntime.quantization import quantize_dynamic, QuantType

model_id = "microsoft/Phi-3-mini-4k-instruct"
model = AutoModel.from_pretrained(model_id, torch_dtype=torch.float16)

# Export to ONNX
dummy_input = torch.randint(0, 32000, (1, 512))
torch.onnx.export(
    model,
    dummy_input,
    "phi-3.onnx",
    input_names=["input_ids"],
    output_names=["logits"],
    dynamic_axes={"input_ids": {"0": "batch_size"}, "logits": {"0": "batch_size"}},
    opset_version=17
)

# Quantize for NPU
quantize_dynamic(
    "phi-3.onnx",
    "phi-3-quant.onnx",
    weight_type=QuantType.QUInt8
)

Running on Quest 3 (QNN SDK)

import qnn
from qnn import Model as QNNModel

# Load ONNX model
qnn_model = QNNModel.from_onnx("phi-3-quant.onnx")

# Compile for Snapdragon NPU
backend_config = qnn.BackendConfig(
    backend_type="HTP",  # Hexagon Tensor Processor
    performance_mode="high_performance"
)
compiled_model = qnn_model.compile(backend_config)

# Inference
input_ids = np.array([[1, 2, 3, 4, 5]], dtype=np.int64)
output = compiled_model.run(input_ids)

WebGPU NPU Access

// WebGPU exposes compute shaders for NPU acceleration
const adapter = await navigator.gpu.requestAdapter();
const device = await adapter.requestDevice();

// Load ONNX model via WebGPU backend
const session = await ort.InferenceSession.create("phi-3-quant.onnx", {
  executionProviders: ["webgpu"],
});

// Inference
const feeds = { input_ids: new ort.Tensor("int64", inputArray, ["1", "5"]) };
const results = await session.run(feeds);

Hierarchical Serving: Device → Edge → Cloud

Not all inference can run on-device. When models are too large or require external data, hierarchical serving distributes the workload.

Architecture

XR Device (NPU) → Edge Server (GPU) → Cloud (Multi-GPU)
      ↑                ↑                    ↑
    less than 10ms       less than 50ms        less than 200ms
   (object        (LLM, VLM,         (foundation
   detection)    scene analysis)      models)

When to Use Each Layer

Layer	Model types	Latency target	Use cases
Device (NPU)	Object detection, pose tracking, audio ASR	less than 10ms	Real-time feedback, offline mode
Edge (GPU)	LLMs (7-13B), VLMs, spatial mapping	less than 50ms	Conversational NPCs, scene understanding
Cloud (multi-GPU)	Foundation models (70B+), batch processing	less than 200ms	Offline training, complex queries

Edge Deployment with k3s

# xr-edge-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: xr-llm-server
  namespace: xr
spec:
  replicas: 2
  selector:
    matchLabels:
      app: xr-llm-server
  template:
    metadata:
      labels:
        app: xr-llm-server
    spec:
      nodeSelector:
        gpu: "true"
      containers:
        - name: vllm
          image: vllm/vllm-openai:latest
          resources:
            limits:
              nvidia.com/gpu: 1
          env:
            - name: MODEL
              value: "microsoft/Phi-3-mini-4k-instruct"
            - name: QUANTIZATION
              value: "awq"
            - name: MAX_TOKENS
              value: "128"
          ports:
            - containerPort: 8000
---
apiVersion: v1
kind: Service
metadata:
  name: xr-llm-service
  namespace: xr
spec:
  selector:
    app: xr-llm-server
  ports:
    - port: 8000
  type: LoadBalancer

Client-Side Offloading

async function inferenceWithFallback(input, type) {
  // Try device NPU first
  try {
    return await deviceInference(input, type);
  } catch (e) {
    console.warn("Device inference failed, falling back to edge");
  }

  // Fall back to edge
  const edgeLatency = await measureEdgeLatency();
  if (edgeLatency < 50) {
    return await edgeInference(input, type);
  }

  // Final fallback to cloud
  console.warn("Edge unavailable, using cloud");
  return await cloudInference(input, type);
}

async function deviceInference(input, type) {
  const session = await ort.InferenceSession.create(`${type}-quant.onnx`, {
    executionProviders: ["webgpu", "wasm"],
  });
  const results = await session.run({ input });
  return results;
}

Offloading Frameworks

When latency budgets are tight, intelligent offloading decides where inference runs.

Ada: Power-Aware Distributed Inference

Ada is an open-source framework that distributes inference across device, edge, and cloud based on power budgets and latency targets.

from ada import InferenceEngine

engine = InferenceEngine(
    device_model="mobilenetv3-int8.onnx",
    edge_model="faster-rcnn-quant.onnx",
    cloud_model="sam-2.7b.onnx",
    power_budget_watts=2.0,  // NPU power limit
    latency_budget_ms=10
)

# Ada automatically routes to the best target
result = engine.inference(input_frame)
# If device overheats or exceeds budget, Ada routes to edge

XRgo: 6-DoF Reprojection + Offloading

XRgo performs 6-DoF reprojection on-device while offloading computationally expensive tasks to the edge.

import xrgo from "xrgo";

// Initialize
const engine = await xrgo.Engine.init({
  mode: "6dof-reprojection",
  edgeServer: "wss://edge.xr.example.com/ws",
  deviceModel: "mobilenetv3-quant.onnx",
});

// Render loop
xrSession.requestAnimationFrame(async (time, frame) => {
  const pose = frame.getViewerPose(referenceSpace);

  // Reprojection on-device (1-2ms)
  const reprojected = await engine.reproject(pose);

  // Offload object detection to edge (5-10ms)
  const objects = await engine.detectObjects(reprojected.colorBuffer);

  // Render
  renderScene(objects, reprojected);
});

DRL-Based Offloading (Research)

Deep reinforcement learning (DRL) agents learn optimal offloading policies by observing network conditions, device power, and latency targets.

import gymnasium as gym
from stable_baselines3 import PPO

# Define offloading environment
env = gym.make("XROffloading-v0", config={
  "network_conditions": ["4g", "5g", "wifi"],
  "device_power_states": ["low", "medium", "high"],
  "latency_budget_ms": 10
})

# Train DRL agent
model = PPO("MlpPolicy", env, verbose=1)
model.learn(total_timesteps=100000)

# Agent learns to offload based on:
# - Current network bandwidth and latency
# - Device temperature and battery
# - Model complexity and queue depth

Bandwidth Optimization

When inference runs off-device, network bandwidth becomes a bottleneck. Three techniques help:

Model Compression

Technique	Compression ratio	Quality loss	When to use
Pruning	2-5x	less than 1% accuracy	Overparameterized models
Quantization	4x (FP16→INT8)	less than 0.5%	Almost always
Knowledge distillation	2-3x	1-3%	When you have a teacher model
LoRA adapters	10x (base + adapters)	Negligible	LLM fine-tuning

Pruning example:

import torch.nn.utils.prune as prune

# Prune 50% of weights in linear layers
for module in model.modules():
  if isinstance(module, torch.nn.Linear):
    prune.l1_unstructured(module, name="weight", amount=0.5)

# Fine-tune to recover accuracy
fine_tune(model, training_data)

# Remove pruning masks permanently
for module in model.modules():
  if isinstance(module, torch.nn.Linear):
    prune.remove(module, "weight")

Progressive Decoding

Send coarse results first, refine progressively. Critical for interactive XR.

async function streamDetectionResults(frame) {
  // Send request with progressive=enabled
  const response = await fetch("/api/detect", {
    method: "POST",
    headers: { "X-Progressive": "enabled" },
    body: frame,
  });

  const reader = response.body.getReader();
  let coarseResult = await reader.read(); // First chunk (50ms)

  // Render coarse result immediately
  renderOverlay(coarseResult);

  // Refine in background
  let refinedResult = await reader.read(); // Second chunk (100ms)
  updateOverlay(refinedResult);
}

Delta Updates

Only send changed data between frames, not full inference results.

class DeltaTracker {
  constructor() {
    this.lastFrame = null;
  }

  computeDelta(currentFrame) {
    if (!this.lastFrame) {
      this.lastFrame = currentFrame;
      return currentFrame;
    }

    const delta = {
      added: [],
      removed: [],
      modified: [],
    };

    // Object tracking by ID
    for (const obj of currentFrame.objects) {
      const lastObj = this.lastFrame.objects.find((o) => o.id === obj.id);

      if (!lastObj) {
        delta.added.push(obj);
      } else if (JSON.stringify(obj) !== JSON.stringify(lastObj)) {
        delta.modified.push(obj);
      }
    }

    delta.removed = this.lastFrame.objects.filter(
      (o) => !currentFrame.objects.find((c) => c.id === o.id),
    );

    this.lastFrame = currentFrame;
    return delta;
  }
}

Production Checklist

Before deploying AI inference in production XR, verify:

Latency measured: End-to-end latency less than 10ms (VR) or less than 15ms (AR) under 90th percentile load
Quantization validated: INT8 model quality within acceptable thresholds
NPU tested: Device NPU correctly utilized (not falling back to CPU)
Edge deployment tested: Edge server responds less than 50ms at peak load
Fallback defined: Offline mode when network unavailable
Power budgeted: Inference doesn't exceed device thermal limits
Bandwidth optimized: Progressive streaming or delta updates enabled
Cache strategy: Model weights cached locally, not re-downloaded
Monitoring in place: Latency, throughput, error rate, and battery drain tracked

Benchmarks: What's Actually Possible?

These benchmarks are from real devices and infrastructure:

Scenario	Device	Model	Latency (p95)	FPS
Object detection (MobileNetV3-INT8)	Quest 3 (NPU)	mobilenetv3-int8.onnx	6ms	166
LLM chat (Phi-3-mini-AWQ)	Edge RTX 4090	vllm	35ms	28
Scene description (Florence-2-INT8)	Edge RTX 4090	hf	45ms	22
Pose tracking (MediaPipe)	Quest 3 (NPU)	mediapipe.tasks	4ms	250
6-DoF reprojection (XRgo)	Quest 3 (GPU)	xrgo	8ms	125

Key insight: On-device inference for detection and tracking is viable today. LLMs and VLMs require edge deployment for real-time interaction.

The XR Latency Budget

Your total latency budget is 16.6ms per frame (60fps) or 11.1ms per frame (90fps). This budget must be shared among:

Component	Latency (typical)	Remaining budget (90fps)
6-DoF pose estimation	2-4ms	7-9ms
Scene graph traversal	1-2ms	5-8ms
AI inference	3-8ms (target)	0-5ms
Rendering (foveated)	4-6ms	-2ms to 1ms (over budget!)
Warp/compose	2-3ms	-5ms to -2ms

The AI inference budget is tight — and you can't steal from pose estimation or warping without causing nausea.

Model Quantization: The Free Speedup

Quantization reduces model precision from 32-bit floats to 8-bit integers, shrinking memory footprint 4x and accelerating inference 2-4x on CPUs, NPUs, and GPUs.

Supported Models

Model	FP16	INT8	Speedup (INT8 vs FP16)	Quality loss
LLaMA-7B	14GB	3.5GB	2-3x	less than 1% perplexity
Phi-3-mini	3GB	750MB	2-3x	Negligible
DistilBERT	420MB	105MB	3-4x	less than 0.5% F1
Florence-2 (VLM)	5GB	1.25GB	2-3x	less than 1% COCO mAP
MobileNetV3 (object det)	28MB	7MB	3-4x	less than 0.5% mAP

Quantization Workflow (PyTorch)

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "microsoft/Phi-3-mini-4k-instruct"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, torch_dtype=torch.float16)

# Static quantization (requires calibration data)
quantized_model = torch.quantization.quantize_dynamic(
    model,
    {torch.nn.Linear},
    dtype=torch.qint8
)

# Save for XR deployment
quantized_model.save_pretrained("./phi-3-int8")

Quantization for Web (TensorFlow.js)

import * as tf from "@tensorflow/tfjs";

async function quantizeModel(url) {
  // Load FP16 model
  const model = await tf.loadGraphModel(url);

  // Quantize to INT8
  const quantizedModel = tf.quantization.quantize(model, {
    quantizationBytes: 1, // 8-bit
    isSigned: true,
  });

  // Save for browser deployment
  await quantizedModel.save("indexeddb://quantized-xr-model");
}

Quality Considerations

Quantization is almost always safe for classification and detection tasks. For generative models (LLMs, diffusion), use:

Quantization-aware training (QAT): Train with simulated quantization for minimal loss
8-bit mixed precision: Keep attention heads in FP16, quantize MLP layers
GPTQ/AWQ: Advanced quantization that preserves LLM quality better than naive INT8

Avoid aggressive quantization (4-bit or lower) for generative models unless you're comfortable with degraded output quality.

NPU Acceleration: Hardware Without the Power Budget

Neural processing units (NPUs) are designed specifically for matrix operations — the core of AI inference. They offer 2-10x better performance per watt than CPUs or GPUs.

XR Device NPU Support

Device	NPU	TOPS (INT8)	Framework support
Meta Quest 3	Snapdragon 8 Gen 2 NPU	30+	Qualcomm QNN, ONNX Runtime
Apple Vision Pro	Apple Neural Engine	18+	Core ML, Metal
HoloLens 2	Snapdragon 850 NPU	4+	WinML, ONNX Runtime
Magic Leap 2	No dedicated NPU	—	CPU-only

Converting Models to ONNX for NPU

from transformers import AutoModel
import torch
import onnx
from onnxruntime.quantization import quantize_dynamic, QuantType

model_id = "microsoft/Phi-3-mini-4k-instruct"
model = AutoModel.from_pretrained(model_id, torch_dtype=torch.float16)

# Export to ONNX
dummy_input = torch.randint(0, 32000, (1, 512))
torch.onnx.export(
    model,
    dummy_input,
    "phi-3.onnx",
    input_names=["input_ids"],
    output_names=["logits"],
    dynamic_axes={"input_ids": {"0": "batch_size"}, "logits": {"0": "batch_size"}},
    opset_version=17
)

# Quantize for NPU
quantize_dynamic(
    "phi-3.onnx",
    "phi-3-quant.onnx",
    weight_type=QuantType.QUInt8
)

Running on Quest 3 (QNN SDK)

import qnn
from qnn import Model as QNNModel

# Load ONNX model
qnn_model = QNNModel.from_onnx("phi-3-quant.onnx")

# Compile for Snapdragon NPU
backend_config = qnn.BackendConfig(
    backend_type="HTP",  # Hexagon Tensor Processor
    performance_mode="high_performance"
)
compiled_model = qnn_model.compile(backend_config)

# Inference
input_ids = np.array([[1, 2, 3, 4, 5]], dtype=np.int64)
output = compiled_model.run(input_ids)

WebGPU NPU Access

// WebGPU exposes compute shaders for NPU acceleration
const adapter = await navigator.gpu.requestAdapter();
const device = await adapter.requestDevice();

// Load ONNX model via WebGPU backend
const session = await ort.InferenceSession.create("phi-3-quant.onnx", {
  executionProviders: ["webgpu"],
});

// Inference
const feeds = { input_ids: new ort.Tensor("int64", inputArray, ["1", "5"]) };
const results = await session.run(feeds);

Hierarchical Serving: Device → Edge → Cloud

Not all inference can run on-device. When models are too large or require external data, hierarchical serving distributes the workload.

Architecture

XR Device (NPU) → Edge Server (GPU) → Cloud (Multi-GPU)
      ↑                ↑                    ↑
    less than 10ms       less than 50ms        less than 200ms
   (object        (LLM, VLM,         (foundation
   detection)    scene analysis)      models)

When to Use Each Layer

Layer	Model types	Latency target	Use cases
Device (NPU)	Object detection, pose tracking, audio ASR	less than 10ms	Real-time feedback, offline mode
Edge (GPU)	LLMs (7-13B), VLMs, spatial mapping	less than 50ms	Conversational NPCs, scene understanding
Cloud (multi-GPU)	Foundation models (70B+), batch processing	less than 200ms	Offline training, complex queries

Edge Deployment with k3s

# xr-edge-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: xr-llm-server
  namespace: xr
spec:
  replicas: 2
  selector:
    matchLabels:
      app: xr-llm-server
  template:
    metadata:
      labels:
        app: xr-llm-server
    spec:
      nodeSelector:
        gpu: "true"
      containers:
        - name: vllm
          image: vllm/vllm-openai:latest
          resources:
            limits:
              nvidia.com/gpu: 1
          env:
            - name: MODEL
              value: "microsoft/Phi-3-mini-4k-instruct"
            - name: QUANTIZATION
              value: "awq"
            - name: MAX_TOKENS
              value: "128"
          ports:
            - containerPort: 8000
---
apiVersion: v1
kind: Service
metadata:
  name: xr-llm-service
  namespace: xr
spec:
  selector:
    app: xr-llm-server
  ports:
    - port: 8000
  type: LoadBalancer

Client-Side Offloading

async function inferenceWithFallback(input, type) {
  // Try device NPU first
  try {
    return await deviceInference(input, type);
  } catch (e) {
    console.warn("Device inference failed, falling back to edge");
  }

  // Fall back to edge
  const edgeLatency = await measureEdgeLatency();
  if (edgeLatency < 50) {
    return await edgeInference(input, type);
  }

  // Final fallback to cloud
  console.warn("Edge unavailable, using cloud");
  return await cloudInference(input, type);
}

async function deviceInference(input, type) {
  const session = await ort.InferenceSession.create(`${type}-quant.onnx`, {
    executionProviders: ["webgpu", "wasm"],
  });
  const results = await session.run({ input });
  return results;
}

Offloading Frameworks

When latency budgets are tight, intelligent offloading decides where inference runs.

Ada: Power-Aware Distributed Inference

Ada is an open-source framework that distributes inference across device, edge, and cloud based on power budgets and latency targets.

from ada import InferenceEngine

engine = InferenceEngine(
    device_model="mobilenetv3-int8.onnx",
    edge_model="faster-rcnn-quant.onnx",
    cloud_model="sam-2.7b.onnx",
    power_budget_watts=2.0,  // NPU power limit
    latency_budget_ms=10
)

# Ada automatically routes to the best target
result = engine.inference(input_frame)
# If device overheats or exceeds budget, Ada routes to edge

XRgo: 6-DoF Reprojection + Offloading

XRgo performs 6-DoF reprojection on-device while offloading computationally expensive tasks to the edge.

import xrgo from "xrgo";

// Initialize
const engine = await xrgo.Engine.init({
  mode: "6dof-reprojection",
  edgeServer: "wss://edge.xr.example.com/ws",
  deviceModel: "mobilenetv3-quant.onnx",
});

// Render loop
xrSession.requestAnimationFrame(async (time, frame) => {
  const pose = frame.getViewerPose(referenceSpace);

  // Reprojection on-device (1-2ms)
  const reprojected = await engine.reproject(pose);

  // Offload object detection to edge (5-10ms)
  const objects = await engine.detectObjects(reprojected.colorBuffer);

  // Render
  renderScene(objects, reprojected);
});

DRL-Based Offloading (Research)

Deep reinforcement learning (DRL) agents learn optimal offloading policies by observing network conditions, device power, and latency targets.

import gymnasium as gym
from stable_baselines3 import PPO

# Define offloading environment
env = gym.make("XROffloading-v0", config={
  "network_conditions": ["4g", "5g", "wifi"],
  "device_power_states": ["low", "medium", "high"],
  "latency_budget_ms": 10
})

# Train DRL agent
model = PPO("MlpPolicy", env, verbose=1)
model.learn(total_timesteps=100000)

# Agent learns to offload based on:
# - Current network bandwidth and latency
# - Device temperature and battery
# - Model complexity and queue depth

Bandwidth Optimization

When inference runs off-device, network bandwidth becomes a bottleneck. Three techniques help:

Model Compression

Technique	Compression ratio	Quality loss	When to use
Pruning	2-5x	less than 1% accuracy	Overparameterized models
Quantization	4x (FP16→INT8)	less than 0.5%	Almost always
Knowledge distillation	2-3x	1-3%	When you have a teacher model
LoRA adapters	10x (base + adapters)	Negligible	LLM fine-tuning

Pruning example:

import torch.nn.utils.prune as prune

# Prune 50% of weights in linear layers
for module in model.modules():
  if isinstance(module, torch.nn.Linear):
    prune.l1_unstructured(module, name="weight", amount=0.5)

# Fine-tune to recover accuracy
fine_tune(model, training_data)

# Remove pruning masks permanently
for module in model.modules():
  if isinstance(module, torch.nn.Linear):
    prune.remove(module, "weight")

Progressive Decoding

Send coarse results first, refine progressively. Critical for interactive XR.

async function streamDetectionResults(frame) {
  // Send request with progressive=enabled
  const response = await fetch("/api/detect", {
    method: "POST",
    headers: { "X-Progressive": "enabled" },
    body: frame,
  });

  const reader = response.body.getReader();
  let coarseResult = await reader.read(); // First chunk (50ms)

  // Render coarse result immediately
  renderOverlay(coarseResult);

  // Refine in background
  let refinedResult = await reader.read(); // Second chunk (100ms)
  updateOverlay(refinedResult);
}

Delta Updates

Only send changed data between frames, not full inference results.

class DeltaTracker {
  constructor() {
    this.lastFrame = null;
  }

  computeDelta(currentFrame) {
    if (!this.lastFrame) {
      this.lastFrame = currentFrame;
      return currentFrame;
    }

    const delta = {
      added: [],
      removed: [],
      modified: [],
    };

    // Object tracking by ID
    for (const obj of currentFrame.objects) {
      const lastObj = this.lastFrame.objects.find((o) => o.id === obj.id);

      if (!lastObj) {
        delta.added.push(obj);
      } else if (JSON.stringify(obj) !== JSON.stringify(lastObj)) {
        delta.modified.push(obj);
      }
    }

    delta.removed = this.lastFrame.objects.filter(
      (o) => !currentFrame.objects.find((c) => c.id === o.id),
    );

    this.lastFrame = currentFrame;
    return delta;
  }
}

Production Checklist

Before deploying AI inference in production XR, verify:

Latency measured: End-to-end latency less than 10ms (VR) or less than 15ms (AR) under 90th percentile load
Quantization validated: INT8 model quality within acceptable thresholds
NPU tested: Device NPU correctly utilized (not falling back to CPU)
Edge deployment tested: Edge server responds less than 50ms at peak load
Fallback defined: Offline mode when network unavailable
Power budgeted: Inference doesn't exceed device thermal limits
Bandwidth optimized: Progressive streaming or delta updates enabled
Cache strategy: Model weights cached locally, not re-downloaded
Monitoring in place: Latency, throughput, error rate, and battery drain tracked

Benchmarks: What's Actually Possible?

These benchmarks are from real devices and infrastructure:

Scenario	Device	Model	Latency (p95)	FPS
Object detection (MobileNetV3-INT8)	Quest 3 (NPU)	mobilenetv3-int8.onnx	6ms	166
LLM chat (Phi-3-mini-AWQ)	Edge RTX 4090	vllm	35ms	28
Scene description (Florence-2-INT8)	Edge RTX 4090	hf	45ms	22
Pose tracking (MediaPipe)	Quest 3 (NPU)	mediapipe.tasks	4ms	250
6-DoF reprojection (XRgo)	Quest 3 (GPU)	xrgo	8ms	125

Key insight: On-device inference for detection and tracking is viable today. LLMs and VLMs require edge deployment for real-time interaction.

The XR Latency Budget

Model Quantization: The Free Speedup

Supported Models

Quantization Workflow (PyTorch)

Quantization for Web (TensorFlow.js)

Quality Considerations

NPU Acceleration: Hardware Without the Power Budget

XR Device NPU Support

Converting Models to ONNX for NPU

Running on Quest 3 (QNN SDK)

WebGPU NPU Access

Hierarchical Serving: Device → Edge → Cloud

Architecture

When to Use Each Layer

Edge Deployment with k3s

Client-Side Offloading

Offloading Frameworks

Ada: Power-Aware Distributed Inference

XRgo: 6-DoF Reprojection + Offloading

DRL-Based Offloading (Research)

Bandwidth Optimization

Model Compression

Progressive Decoding

Delta Updates

Production Checklist

Benchmarks: What's Actually Possible?

Further Reading

Never miss a deep-dive

The XR Latency Budget

Model Quantization: The Free Speedup

Supported Models

Quantization Workflow (PyTorch)

Quantization for Web (TensorFlow.js)

Quality Considerations

NPU Acceleration: Hardware Without the Power Budget

XR Device NPU Support

Converting Models to ONNX for NPU

Running on Quest 3 (QNN SDK)

WebGPU NPU Access

Hierarchical Serving: Device → Edge → Cloud

Architecture

When to Use Each Layer

Edge Deployment with k3s

Client-Side Offloading

Offloading Frameworks

Ada: Power-Aware Distributed Inference

XRgo: 6-DoF Reprojection + Offloading

DRL-Based Offloading (Research)

Bandwidth Optimization

Model Compression

Progressive Decoding

Delta Updates

Production Checklist

Benchmarks: What's Actually Possible?

Further Reading

Never miss a deep-dive