Before you evaluate an AI SRE, give your existing agent context

If you're evaluating AI SRE products, you probably own most of one already. The coding agent your engineers use — Claude Code, Codex, or Cursor — can already investigate a Kubernetes cluster. What it usually lacks is context: what's failing right now, and what changed before it broke.

We measured how much that gap matters. The same agent and model worked through 50 broken clusters twice: once with a shell and raw kubectl, once with read-only MCP tools that return that context.

The numbers

Metric	Shell + kubectl	Read-only cluster context
Correct diagnoses (of 50)	45	46
Mean judge score (0–1)	0.905	0.936
Correct within 2 minutes	60%	86%
Correct within 5 minutes	78%	90%
Mean time to submit a diagnosis (right or wrong)	191 s	64 s
Median	87 s	44 s
90th percentile	632 s	94 s
Mean tool calls per fault	16.3	9.8

The accuracy gain was modest — 46 against 45, with both arms right on 44 of the same faults. The big difference was time, which is what counts during an active outage: nobody wants to wait fifteen minutes for an AI investigation while users are getting errors.

With context, the agent had a correct diagnosis within two minutes on 86% of faults, versus 60% with a shell. With a shell, one fault in ten took more than ten minutes. The gap was widest on the 14 faults where every pod reported Ready but the application was broken: 326 seconds on average with a shell, 79 with context. Fewer tool calls also means fewer model turns — usually lower token spend.

How it ran

50 faults from SREGym (the open Kubernetes fault benchmark from Microsoft and UIUC) on a live 3-node Amazon EKS cluster. Both arms used Claude Code on Claude Sonnet 5 through Bedrock. One arm got a shell and raw kubectl; the other got read-only MCP tools from Radar (open-source Kubernetes UI + MCP server), with kubectl blocked so it couldn't fall back to a shell. SREGym's judge — Claude Opus 5 at temperature 0 — graded the first diagnosis each run submitted.

Honest caveats from the authors: context didn't fix the model's judgment. Both arms missed the same three faults (a mutating webhook rewriting memory limits at admission, a PriorityClass mis-scoped as global default, a real-but-unrelated misconfiguration reported instead of the DNS fault). The two arms also had different permissions — only the shell could write — so the run doesn't isolate exactly what did the work.

What "context" means here

Radar's read-only MCP tools add four things on top of what kubectl returns:

1. Current failures, already detected — a ranked list of what's broken right now, so the first call starts from what's failing instead of a full inventory.
2. A change timeline — spec changes and Helm release history, kept after Kubernetes events expire (events last an hour by default via --event-ttl, and many changes never produce one).
3. Resources joined to their relationships — owners, topology, related issues come back with the object, so the agent doesn't fetch them one by one.
4. Logs filtered to the errors — diagnostic lines first, with secrets redacted.

The takeaway

Before you spend on a dedicated AI SRE product, give the agent you already have that context and measure it. That gives you a baseline for what the product adds. Most of an SRE's value during an incident is not judgment — it's orientation: knowing what changed and what's failing right now. That's cheap to provide and it's the single biggest lever on time-to-diagnosis this data shows.