Agent Observability with OpenTelemetry: Tracing LLM Calls and Agent Loops in Production

Stripe Testing Best Practices: From Test Cards to Production Confidence

Comprehensive guide to testing Stripe payment integrations — test cards, webhook simulation, checkout flows, edge cases, and CI/CD strategies for bulletproof payment systems.

Incident Response and SRE Practices for Production Systems

How to implement SLIs, SLOs, error budgets, incident management, and postmortem culture for self-hosted and distributed infrastructure.

Why LLM Calls Are Black Boxes

To ordinary tracing, an LLM call is a black box. You see a slow HTTP request — but not which model was called, how many tokens it burned, whether it looped on a tool, or why it stopped. For classic infrastructure, one slow request is rarely a mystery. For an AI agent that reasons, calls tools, and iterates, the same request can be the difference between a finished task and a runaway loop that costs you tokens for hours.

The observability gap becomes obvious the moment an agent misbehaves in production:

• What did the agent see? Which system instructions, conversation history, and tool results went into the prompt?
• What did it decide? Why did it call get_weather three times instead of once?
• Why did it stop? Did it finish, hit a tool-call limit, or error out?

Log grepping answers none of these questions. Traces do — if the agent emits them. This is where OpenTelemetry's GenAI semantic conventions come in: a standardized way to record LLM operations, agent structure, and tool calls as spans, so agents are observable with the same tooling as the rest of your stack.

The OpenTelemetry GenAI Semantic Conventions

OpenTelemetry graduated from the CNCF in May 2026, cementing OTLP as the common protocol for telemetry. The GenAI conventions extend it with a vocabulary for generative-AI operations: the gen_ai.* attribute namespace plus standardized span names.

Key Attributes

Span Names

Span names follow the pattern {operation} {model-or-name}, which gives you names like chat gpt-4o-mini, execute_tool get_weather, and invoke_agent travel-concierge. Three span shapes matter for agents:

• invoke_agent — the agent run itself, wrapping the whole loop
• chat — a single model call (auto-instrumented by most SDKs)
• execute_tool — a tool invocation, bridging the agent and the real world

Stability Status: Honest Disclaimer

The GenAI conventions are still Development status — no GenAI-specific span, event, metric, or attribute is marked Stable as of July 2026. The conventions moved to their own repository (open-telemetry/semantic-conventions-genai) in June 2026 with no versioned release there yet. Also, gen_ai.system was renamed to gen_ai.provider.name in v1.37.0 (August 2025), and frameworks emit both generations during the transition.

Practically: pin your instrumentation versions, set OTEL_SEMCONV_STABILITY_OPT_IN=gen_ai_latest_experimental if you want the latest shapes (set it at process start — instrumentations read it at import time), and be prepared to coalesce attribute generations in your analytics.

The Span Hierarchy of an Agent Run

Auto-instrumentation alone gives you a flat list of chat spans. It cannot know your agent's structure: which calls belong to which task, and when a tool ran. You add that structure by wrapping the run loop in an invoke_agent span and each tool call in an execute_tool span — the auto-instrumented chat spans then nest underneath:

This single trace answers the questions from the introduction: you can replay the exact tool sequence, see where each model call ended, and spot the loop before it costs you a night of tokens.

Instrumenting an Agent Stack

Auto-Instrumentation

For OpenAI, use opentelemetry-instrumentation-openai-v2 — the official OTel implementation (the older opentelemetry-instrumentation-openai from OpenLLMetry is community-maintained). For Anthropic, Bedrock, Vertex, and others you have a choice: OpenAI-compatible endpoints, community or framework instrumentors (OpenLLMetry, LangChain, LlamaIndex, OpenAI Agents SDK, CrewAI), or dedicated agents instrumentation. All emit the same gen_ai.* attributes, so the rest of this guide is provider-agnostic.

Manual Spans for Structure

import { trace } from "@opentelemetry/api";

const tracer = trace.getTracer("agent-runtime");

async function runAgent(conversationId: string, task: string) {
  const agentSpan = tracer.startSpan("invoke_agent travel-concierge", {
    kind: SpanKind.INTERNAL,
    attributes: {
      "gen_ai.agent.name": "travel-concierge",
      "gen_ai.agent.id": "asst_5j66UpCpwteGg4YSxUnt7lPY",
      "gen_ai.conversation.id": conversationId,
    },
  });

  return trace
    .getTracer("agent-runtime")
    .withActiveSpan(agentSpan, async () => {
      try {
        return await runLoop(task);
      } finally {
        agentSpan.end();
      }
    });
}

async function callTool(name: string, args: unknown) {
  const toolSpan = trace.getTracer("agent-runtime").startSpan(`execute_tool ${name}`, {
    kind: SpanKind.INTERNAL,
    attributes: {
      "gen_ai.tool.name": name,
      "gen_ai.tool.type": "function",
      "gen_ai.tool.call.id": crypto.randomUUID(),
    },
  });
  try {
    return await invoke(name, args);
  } finally {
    toolSpan.end();
  }
}

Key insight: gen_ai.agent.id and gen_ai.tool.call.id are worth setting even if your backend doesn't display them yet — typed columns populate only from emitted attributes.

Grouping Multi-Turn Conversations

gen_ai.conversation.id is a span attribute, not a resource attribute, and span attributes do not inherit. An id set only on the invoke_agent span leaves every auto-instrumented chat span without it — so you cannot group the conversation.

The pattern that covers every span is a span processor whose on_start hook reads the active conversation id and stamps it on each span as it opens:

import { Span, SpanProcessor } from "@opentelemetry/sdk-trace-node";
import { AsyncLocalStorage } from "node:async_hooks";

export const conversationStore = new AsyncLocalStorage<string>();

export class ConversationIdProcessor implements SpanProcessor {
  onStart(span: Span): void {
    const conversationId = conversationStore.getStore();
    if (conversationId) {
      span.setAttribute("gen_ai.conversation.id", conversationId);
    }
  }
  onEnd(span: Span): void {}
  forceFlush() {
    return Promise.resolve();
  }
  shutdown() {
    return Promise.resolve();
  }
}

Register it as the outermost processor on the TracerProvider, and wrap each user session in conversationStore.run(conversationId, handler). Because on_start runs while the span is still live, span.set_attribute() here is ordinary public API — no private SDK internals. Coverage becomes uniform: every span the process creates, whether your code or an auto-instrumentation created it, gets the conversation id.

Metrics: Token Usage and Cost

Two client metrics matter for agents:

• gen_ai.client.operation.duration — a latency histogram (seconds), showing how long operations take
• gen_ai.client.token.usage — a token histogram, split by gen_ai.token.type (input/output)

Every chat span carries gen_ai.usage.input_tokens and gen_ai.usage.output_tokens, so per-request cost is those counts multiplied by your model's price per token. The token histogram lets you spot token-hungry prompts before they reach production.

Detecting Runaway Tool Loops

gen_ai.response.finish_reasons is your loop detector. An agent stuck repeating ["tool_calls"] — call after call, never ["stop"] — is burning tokens. Alert on the pattern:

count by (gen_ai.agent.name) (
  gen_ai.client.token.usage
) > 100000
and
count by (span_name) (execute_tool) > 20

Or, in words: a single agent that emitted more than 20 execute_tool spans in one conversation is looping. Kill it, inspect the trace, and find the prompt or tool-availability bug.

The Self-Hosted Pipeline

The payoff of standard gen_ai.* telemetry: you instrument once and point it at any OTLP backend. For a sovereign stack, that backend is yours:

No SaaS vendor, no telemetry leaving your infrastructure. Swapping backends later is a one-line OTLP exporter change instead of a re-instrumentation project. For local debugging, a Jaeger or Grafana Tempo container suffices; the same data feeds production dashboards.

War Story: The 40-Token Tool Loop

A self-hosted agent processed support tickets overnight and burned through the night's budget. Logs showed nothing unusual — dozens of successful tool calls, no errors.

The trace told the story in seconds. An invoke_agent span with 40 nested execute_tool get_ticket_status calls, each followed by a chat span with finish_reasons: ["tool_calls"]. The agent was asking for a status, receiving it, and deciding it needed to ask again — the tool result lacked the confidence field the prompt demanded, so the agent kept re-querying instead of proceeding.

The fix was a prompt change (acknowledge the status value) plus a guardrail: the agent now aborts after 5 consecutive identical tool calls. Neither fix was findable from logs — only from the span hierarchy and finish-reason telemetry.

Caveats

• Pin everything. The conventions are Development status and live in a dedicated repo without a versioned release yet. Pin instrumentation versions and record what you tested.
• Query both attribute generations. During the transition, frameworks emit gen_ai.system alongside gen_ai.provider.name, and token usage under both prompt_tokens/completion_tokens and input_tokens/output_tokens. Coalesce with precedence — never sum the pairs, because compatibility-duplicated values are identical.
• Keep custom attributes out of gen_ai.*. A future revision can claim any name in that reserved namespace. Stamp your domain attributes with your own prefix (e.g., brew.*, agentx.*).
• Own your internal representation. A Development-status external schema should never be your database contract. Map framework telemetry into your own versioned model, and map outward to the current convention at export time.
• Set the opt-in before import. OTEL_SEMCONV_STABILITY_OPT_IN=gen_ai_latest_experimental is read at instrumentation import time; setting it later silently does nothing.

Further Reading

• Production Observability Stack with Grafana, Prometheus, and Loki — building the base stack this guide feeds into
• Grafana for Infrastructure Observability — dashboards and alerting patterns
• Incident Response and SRE Practices — runbooks for the failures your traces will surface
• Self-Hosted AI Stack with Ollama, Open WebUI, and LiteLLM — running the models these agents call, on your own hardware