Incident Response and SRE Practices for Production Systems

Agent Observability with OpenTelemetry: Tracing LLM Calls and Agent Loops in Production

How to trace AI agents in production with OpenTelemetry — gen_ai.* semantic conventions, invoke_agent/execute_tool spans, conversation grouping, and token-usage metrics on your own stack.

Site Reliability Engineering (SRE) is what happens when you treat operations as a software engineering problem. In 2026, as self-hosted infrastructure becomes more common and distributed systems grow more complex, SRE practices are no longer optional — they are the foundation of production reliability.

This guide covers the core SRE practices that apply whether you are running a three-node Kubernetes cluster or a multi-region fleet of 500 servers.

SLIs, SLOs, and Error Budgets

The fundamental contract of SRE is the Service Level Objective (SLO), measured by Service Level Indicators (SLIs).

Choosing SLIs

Good SLIs measure what users experience, not internal infrastructure metrics:

Setting SLOs

An SLO is the target value for an SLI over a rolling window (typically 30 days):

slo:
  name: api-latency
  sli: http_request_duration_seconds
  aggregation: p99
  window: 30d
  target: 0.5 # 500ms
  compliance: 99.9

The error budget is 100% minus the SLO. With a 99.9% SLO, you have 0.1% budget for errors — approximately 43 minutes of downtime per 30-day window.

Error Budget Policy

The error budget determines release velocity. When the budget is exhausted, all feature releases stop until reliability is restored:

error_budget_policy:
  exhaustion_action: freeze-deployments
  exhaustion_condition: budget < 10%
  notify:
    - on-call-engineer
    - team-leads
  top-level: require-approval

This creates a direct feedback loop: if reliability drops, engineering velocity drops. Teams are incentivized to both ship features and maintain reliability.

Incident Management

Incident Severity Levels

Define clear severity levels so the right people respond with the right urgency:

Incident Response Workflow

1. Detection: Automated alerts from Prometheus/Grafana, synthetic monitoring, or user reports
2. Triage: First responder acknowledges, assesses severity, declares incident if needed
3. Mitigation: Stop the bleeding — rollback, redirect traffic, scale up
4. Resolution: Apply permanent fix
5. Follow-up: Write postmortem, track action items

Communication During Incidents

Establish a communication channel per incident. Use a status page for external communication. Update every 30 minutes during ongoing incidents, even if there is no new information.

Postmortem Culture

A blameless postmortem is the most important reliability practice. Every SEV1 and SEV2 incident requires a postmortem within 5 business days:

## Postmortem: API Latency Spike on 2026-07-15

### Summary

p99 latency increased from 200ms to 4500ms for 23 minutes, affecting 12% of traffic.

### Timeline

- 14:23 — Alert fired (p99 > 1000ms)
- 14:25 — Engineer acknowledged
- 14:28 — Identified database connection pool exhaustion
- 14:32 — Scaled connection pool from 50 to 200
- 14:46 — Latency returned to baseline

### Root Cause

A deployment at 14:00 increased per-request database connections from 2 to 8 without corresponding pool adjustment.

### Action Items

- [ ] Add connection pool utilization dashboard to standard deploy checklist
- [ ] Implement connection pool saturation alert
- [ ] Add pool size assertion to deployment pipeline

The key principle: the root cause is never the person who pushed the button. The root cause is always a process gap, a missing test, or a design oversight.

On-Call Practices

Rotation

• Teams of 4-6 engineers minimum
• 1-week primary shifts
• Secondary for handoff and escalation
• No engineer on call more than one week out of four

Escalation

• Primary handles SEV3/4 within defined hours
• Secondary handles SEV1/2 or when primary is unavailable
• Engineering leads for prolonged or cross-team incidents

Quality of Life

• Incident response documentation accessible offline
• Runbooks for the top 20 failure modes
• Shadow shifts for new engineers
• Post-shift handoff documentation

Automation

The SRE team's primary investment should be automation that prevents incidents, shortens response time, and eliminates toil:

• Automated rollback: Deployments that automatically roll back on alert threshold breaches
• Auto-scaling: Horizontal pod autoscaling with HPA and cluster autoscaler
• Health checks: Liveness, readiness, and startup probes with proper failure thresholds
• Remediation runbooks: Automated responses for common failure patterns (restart, scale, redirect)

Measuring SRE Success

The ultimate SRE metric is not uptime. It is whether the team can sleep at night knowing their systems are observable, remediable, and understood.