LLM Observability and Eval-Driven Development

LLMs are probabilistic systems. Unlike a REST API that returns the same output for the same input every time, an LLM call can drift, degrade, or hallucinate without throwing an error. Traditional monitoring — CPU, memory, error rate — tells you almost nothing about whether your AI application is actually working.

Enter LLM observability: tracing every prompt-completion pair, evaluating outputs against expected behavior, and catching regressions before users do.

The Three Pillars of LLM Observability

1. Tracing

Every LLM call produces a trace:

• Input: raw prompt, system message, tools available, temperature/params
• Output: completion text, token counts, finish reason (stop / length / content_filter)
• Metadata: model name, latency, cost, request ID, user ID
• Chain context: if this call is part of a multi-step agent chain, link the traces

Langfuse (Self-Hosted)

import { Langfuse } from "langfuse";

const langfuse = new Langfuse({
  secretKey: process.env.LANGFUSE_SECRET_KEY,
  publicKey: process.env.LANGFUSE_PUBLIC_KEY,
  baseUrl: "https://observability.yourdomain.com", // self-hosted
});

async function generateResponse(userQuery: string) {
  const trace = langfuse.trace({
    name: "chat-completion",
    userId: userQuery.sessionId,
    metadata: { environment: "production" },
  });

  const generation = trace.generation({
    name: "llm-call",
    model: "gpt-4o",
    modelParameters: { temperature: 0.7, maxTokens: 2048 },
    input: userQuery,
  });

  const response = await callLLM(userQuery);
  generation.end({
    output: response,
    usage: {
      promptTokens: response.usage.prompt_tokens,
      completionTokens: response.usage.completion_tokens,
    },
  });

  return response;
}

Deploy with one line:

# docker-compose.observability.yml
services:
  langfuse:
    image: ghcr.io/langfuse/langfuse:latest
    ports:
      - "3000:3000"
    environment:
      DATABASE_URL: postgresql://langfuse:password@postgres/langfuse
      NEXTAUTH_SECRET: ${LANGFUSE_SECRET}

OpenTelemetry for LLMs

For teams already using OTel, the @opentelemetry/instrumentation-langchain or traceloop-sdk automatically instruments LLM calls without code changes:

pip install traceloop-sdk
export TRACELOOP_BASE_URL=https://otel.yourdomain.com
traceloop.init(app_name="my-llm-app")

2. Evaluation

Tracing tells you what happened. Evaluation tells you whether it was good.

Building an Eval Suite

interface EvalResult {
  name: string;
  passed: boolean;
  score: number; // 0-1
  details: string;
}

class EvalSuite {
  private tests: Array<{
    input: string;
    expected: string;
    criteria: EvalCriteria[];
  }> = [];

  addTestCase(input: string, expected: string, criteria: EvalCriteria[]) {
    this.tests.push({ input, expected, criteria });
  }

  async run(model: string): Promise<EvalSummary> {
    const results: EvalResult[] = [];

    for (const test of this.tests) {
      const output = await callModel(model, test.input);

      for (const criterion of test.criteria) {
        const result = await criterion.evaluate(output, test.expected);
        results.push({
          name: `${test.input.slice(0, 40)}... - ${criterion.name}`,
          ...result,
        });
      }
    }

    return this.summarize(results);
  }
}

Types of Evaluations

Category	What It Checks	Tooling
Exact match	Structured output matches expected JSON schema	JSON schema validation
Semantic similarity	Embedding cosine similarity ≥ threshold	text-embedding-3-small, all-MiniLM-L6-v2
LLM-as-judge	Another LLM rates output quality (1-5)	GPT-4o, Claude 3.5 Sonnet
Factual consistency	No hallucinated claims not in source	SelfCheckGPT, NLI models
Safety	No PII, toxic content, prompt injection	Guardrails, NeMo Guardrails
Latency	p95 response time ≤ threshold	Built into tracing
Cost	Total tokens per session	Built into tracing

Automating Eval as CI/CD

# .github/workflows/llm-eval.yml
name: LLM Evaluation

on:
  pull_request:
    paths:
      - "prompts/**"
      - "evals/**"

jobs:
  eval:
    runs-on: self-hosted
    steps:
      - uses: actions/checkout@v4
      - run: npm ci

      - name: Run eval suite
        run: npm run eval
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}

      - name: Compare against baseline
        run: |
          npm run eval:compare --baseline=main --current=HEAD
          # Gate on regressions > 5%
          if [ $? -ne 0 ]; then
            echo "❌ Eval regressions detected. Review before merging."
            exit 1
          fi

      - name: Update eval baseline
        if: github.ref == 'refs/heads/main'
        run: npm run eval:store-baseline

Cost-Aware Evaluation

Running eval suites costs real money — frontier models charge per token, and a comprehensive eval run can easily burn $50–200 per test cycle. Cost-blind evaluation leads to either insufficient coverage (to save money) or runaway eval bills. The solution: budgeted evaluation.

Budget Allocation Strategy

Total eval budget: $200 / week

┌────────────────────────────────────────────┐
│ Smoke (15%)   $30     Every PR             │
│   - 10 critical test cases                 │
│   - 1 high-quality judge (GPT-4o)          │
│   - < 2 min runtime                        │
├────────────────────────────────────────────┤
│ Full (50%)    $100    Nightly / per deploy │
│   - 200 test cases across all categories   │
│   - Mixture of LLM-as-judge + embedding    │
│   - Automated regression comparison        │
├────────────────────────────────────────────┤
│ Exploratory (35%) $70  Weekly              │
│   - 50 edge-case scenarios                 │
│   - Multiple judge models for calibration  │
│   - Human review of low-confidence results │
└────────────────────────────────────────────┘

Cost Optimization Techniques

Technique	Savings	Trade-off
Use cheaper judges	10-20x	Slightly lower correlation
(Llama 3.3 70B instead of GPT-4o)
Sampling	50-90%	Statistical uncertainty
(eval random 20% of traffic)
Caching	30-70%	Stale results if model drifts
(deduplicate identical inputs)
Progressive eval	40-60%	More complex orchestration
(cheap evals gate expensive)
Tiered model routing	20-40%	Requires confidence threshold
(easy cases skip frontier)

Cost Tracking Integration

// Track eval cost alongside quality metrics
interface EvalRun {
  timestamp: Date;
  model: string;
  testCount: number;
  passRate: number;
  totalTokens: number;
  cost: number; // USD
  costPerTest: number; // USD
  avgScore: number;
  regressionDelta: number; // vs baseline
}

async function budgetCheck(weekSpend: number, budget: number): Promise<boolean> {
  if (weekSpend >= budget) {
    await alert({
      severity: "warning",
      message: `Weekly eval budget exhausted ($${weekSpend.toFixed(2)})`,
    });
    return false; // halt expensive eval runs
  }
  return true;
}

Key insight: The best eval is one you can afford to run consistently. A medium-quality eval that runs on every PR is more valuable than a perfect eval that runs monthly.

4. Monitoring & Alerting

Once you have traces and eval scores flowing, build dashboards and alerts.

Key Metrics

LLM Application Dashboard

Latency:
  - p50, p95, p99 response time per model
  - p95 time-to-first-token (streaming)

Quality:
  - Eval pass rate (7-day rolling)
  - User feedback score (thumbs up/down ratio)

Cost:
  - Cost per day per model
  - Cost per user session
  - Token waste (cancelled generations, retries)

Volume:
  - Requests per minute
  - Active users
  - Concurrent sessions

Errors:
  - Rate limit hits
  - Context length exceeded
  - Content filter triggered (safety)
  - Timeout rate

Alerting Rules

# PrometheusRules
groups:
  - name: llm-alerts
    rules:
      - alert: HighLatency
        expr: llm_latency_p95 > 5000
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "LLM p95 latency > 5s"

      - alert: EvalRegression
        expr: llm_eval_pass_rate < 0.85
        for: 10m
        labels:
          severity: critical
        annotations:
          summary: "Eval pass rate dropped below 85%"

      - alert: CostSpike
        expr: rate(llm_cost_total[1h]) > 10
        labels:
          severity: warning

Prompt Versioning and Management

Treat prompts as code. Same PR, review, and CI gates.

# Directory structure
prompts/
├── chat/
│   ├── system-v1.md
│   ├── system-v2.md        # +few-shot examples
│   └── system-v2.test.md   # test cases for this prompt
├── classification/
│   ├── intent-v3.md
│   └── sentiment-v1.md
└── index.json               # which prompt version is active per model

// prompt-index.json
{
  "chat-system": {
    "active": "v2",
    "models": {
      "gpt-4o": "v2",
      "claude-3.5-sonnet": "v2",
      "llama-3.3-70b": "v1" // smaller model needs simpler prompt
    },
    "deployed_at": "2026-07-10T14:00:00Z",
    "eval_pass_rate": 0.92
  }
}

Production Runbook

Detecting Model Drift

Models change silently. GPT-4o today is not the same model it was last month. Track:

// Weekly: run your eval suite against each deployed model
// Compare results against the baseline stored in your DB
// Alert on any metric that regresses by > 5%

async function detectDrift() {
  const baseline = await loadBaseline();
  const current = await runEvalSuite();

  for (const metric of Object.keys(baseline)) {
    const diff = current[metric] - baseline[metric];
    if (diff < -0.05) {
      await alert({
        severity: "warning",
        message: `Model drift detected: ${metric} dropped ${Math.abs(diff) * 100}%`,
      });
    }
  }
}

Rolling Back a Prompt

# 1. Revert the prompt to previous version
cp prompt-index.json.bak prompt-index.json

# 2. Restart services (or hot-reload if supported)
docker compose restart llm-service

# 3. Verify eval scores recover
npm run eval:smoke

# 4. Document the incident
echo "$(date): Rolled back chat-system prompt to v1 due to eval regression" >> CHANGELOG.md

Stack Decisions

Tool	Self-Host	Cost	Best For
Langfuse	✅ Yes	Free (self-host)	Full tracing + eval + playground
Braintrust	❌ Cloud	Per-seat + usage	Enterprise, built-in eval
Arize Phoenix	✅ Yes	Free	Notebook-first, experiment tracking
Helicone	✅ Yes	Free tier	Simple proxy-based observability
OpenTelemetry	✅ Yes	Free	Teams already on OTel

For most self-hosted setups, Langfuse or Arize Phoenix are the best starting points.

Cross-Links

• Self-Hosted AI Stack — serve models that your observability pipeline monitors
• vLLM Self-Hosted Inference — production inference with metrics export (prometheus metrics built in)
• vLLM vs SGLang — how inference framework choice affects latency and cost observability
• Prompt Engineering Best Practices — prompts as versioned artifacts that evals validate
• LLM API Value for Money — cost-per-token analysis across providers
• Agent Skill Management — eval-driven skill iteration for multi-agent systems
• Production Observability Stack — infrastructure for the metrics and alerting layer
• Disaster Recovery — because losing your eval baseline is a disaster too
• MCP Servers — how MCP ecosystems integrate with tracing and eval

Summary

Capability	Tool / Practice
Trace every LLM call	Langfuse, OpenTelemetry
Evaluate output quality	Eval suite + LLM-as-judge
Gate deploys on evals	CI/CD eval step
Monitor for drift	Weekly auto-eval against baseline
Track cost per session	Langfuse cost tracking
Version prompts	Git + prompt-index.json
Alert on regression	Prometheus + Grafana

The difference between a demo and a production LLM application is observability. Without it, you're flying blind — and probabilistic systems will find ways to crash that deterministic systems never could.