Guide: Containing AI Agents on Kubernetes with Defense in Depth

Guide: MinIO as an S3-Compatible Backup Target

Self-hosted S3-compatible object storage for backups: MinIO behind an Nginx proxy, with an rclone-based backup agent and versioned buckets. Your offsite backups without a cloud bill.

Guide: PostgreSQL + Redis as a Production Foundation

The two services almost every application needs, tuned instead of defaulted: production-tuned PostgreSQL (shared buffers, WAL compression), Redis with AOF + RDB persistence, and a scheduled backup job

An AI agent inside your cluster is a new kind of principal, and the model running it is not a security boundary. This guide is the reference architecture for containing one anyway.

What this is

LLM agents read telemetry, call tools, write and execute code, and change cluster state. That collapses a boundary cloud-native security relies on: the boundary between data and control. A log line written by an attacker-controlled workload, an annotation on a resource, a field in a ticket, or a tool description served by a third-party MCP server is content the agent merely reads — and it can become a command the agent then executes with its own credentials.

The useful consequence is a design rule: assume the agent is fully compromised by prompt injection, and build every guarantee so that it still holds. A deployment guide with working examples — not a downloadable product. The manifests below are the reference architecture, abridged to the parts that matter. The framework follows the arXiv paper "Containing the Autonomous Operator", which is worth reading in full for the threat model.

Deployment example

RBAC answers which verbs on which resources in which namespace. It cannot express constraints on object content — it cannot forbid scaling to zero while permitting scaling, or forbid edits in control-plane namespaces while permitting edits elsewhere. A ValidatingAdmissionPolicy fills that gap with in-process CEL, scoped to agent identities with matchConditions.

Because the API server enforces it, this policy holds even if the agent bypasses the gateway and calls the API directly with a stolen token. That property is exactly what tool-layer controls alone cannot give you.

apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicy
metadata:
  name: agent-guardrails
spec:
  failurePolicy: Fail
  matchConstraints:
    resourceRules:
      - apiGroups: ["apps"]
        apiVersions: ["v1"]
        operations: ["CREATE", "UPDATE", "DELETE"]
        resources: ["deployments", "deployments/scale", "statefulsets", "statefulsets/scale"]
  matchConditions:
    - name: only-agent-identities
      expression: "request.userInfo.username.startsWith('system:serviceaccount:agents:')"
  validations:
    - expression: "!request.namespace.startsWith('kube-') && request.namespace != 'agents'"
      message: "agents may not modify control-plane or agent namespaces"
    - expression: "request.operation != 'DELETE'"
      message: "agents may not delete workloads; open a GitOps PR instead"
    - expression: "request.operation == 'DELETE' || !has(object.spec.replicas) || object.spec.replicas > 0"
      message: "agents may not scale workloads to zero"
---
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicyBinding
metadata:
  name: agent-guardrails-binding
spec:
  policyName: agent-guardrails
  validationActions: ["Deny", "Audit"]

Essential configuration

Two more artefacts carry most of the remaining weight. Egress: agent namespaces start from default-deny, and because external services are addressed by name rather than address, an IP-based NetworkPolicy is not enough — you need FQDN-aware policy. Sandboxes: generated code runs in kernel-isolated pods with no token mounted and no egress at all.

apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
  name: agent-egress
  namespace: agents
spec:
  endpointSelector:
    matchLabels: { app.kubernetes.io/component: agent }
  egress:
    - toEndpoints:
        - matchLabels:
            k8s:io.kubernetes.pod.namespace: kube-system
            k8s-app: kube-dns
      toPorts:
        - ports: [{ port: "53", protocol: ANY }]
          rules:
            dns: [{ matchPattern: "*" }]
    - toFQDNs:
        - matchName: bedrock-runtime.us-west-2.amazonaws.com
      toPorts:
        - ports: [{ port: "443", protocol: TCP }]

apiVersion: agents.x-k8s.io/v1alpha1
kind: Sandbox
metadata:
  name: session-7f3a
  namespace: agent-sandboxes
spec:
  podTemplate:
    spec:
      runtimeClassName: gvisor
      automountServiceAccountToken: false
      securityContext:
        runAsNonRoot: true
        seccompProfile: { type: RuntimeDefault }
      containers:
        - name: runtime
          image: registry.internal/sandbox/python@sha256:...
          securityContext:
            allowPrivilegeEscalation: false
            readOnlyRootFilesystem: true
            capabilities: { drop: ["ALL"] }
          resources:
            limits: { cpu: "1", memory: 1Gi, ephemeral-storage: 512Mi }

Production checklist

• Every agent and every tool server runs under its own ServiceAccount — the default one is never used
• automountServiceAccountToken: false on every pod that does not call the Kubernetes API
• Agents get namespaced Roles, never ClusterRoles with write verbs
• pods/exec, secrets, serviceaccounts/token and RBAC objects are excluded from agent roles
• Agent and sandbox namespaces enforce the restricted Pod Security Standard
• Egress is default-deny, and the FQDN allowlist names the model endpoint and nothing else
• The gateway mediates every tool call — there is no unmediated path from agent to API server
• Secrets are fetched server-side by the tool servers and never returned in tool output
• A rehearsed kill switch exists: revoke identity, scale to zero, quarantine, invalidate sessions
• Per-agent budgets for tool calls, mutations, model tokens and spend

The seven layers

• L1 Identity — per-agent ServiceAccount, audience-bound short-lived tokens, cloud workload identity instead of static keys
• L2 Authorization and admission — scoped RBAC, ValidatingAdmissionPolicy, Pod Security Standards
• L3 Execution isolation — the SIG Apps Agent Sandbox project, gVisor or Kata via RuntimeClass, ephemeral per session
• L4 Network — default-deny ingress and egress, FQDN allowlist, mutual TLS between gateway, tool servers and agents
• L5 Tool mediation — the agent gateway, policy over tool and arguments, signed tool registry, rate limits
• L6 Data and secrets — server-side secret fetching, redaction at the source, memory provenance
• L7 Runtime detection and response — eBPF enforcement, audit-log analytics, kill switch

Layer G sits across all of them: human approval, graduated autonomy, budgets and circuit breakers. The ordering matters — the gateway can block earlier and with richer context, but only admission control and the CNI cannot be bypassed by a compromised agent.

The lethal combination

Serious harm needs three properties at once: exposure to untrusted input, access to sensitive data or privileged actions, and the ability to communicate externally or change state. Remove any one of them for a session and the damage from a successful injection is sharply bounded.

Kubernetes is unusually good at enforcing this structurally, because each property maps onto a controllable resource — content sources onto tool authorisation, sensitive access onto RBAC and secret placement, external communication onto egress policy. The strongest version separates the component that reads untrusted data from the component that plans privileged actions: a quarantined reader agent with read-only tools and no egress, and a planner agent that never ingests raw untrusted content.

Autonomy tiers

Tool tier            Dev        Staging    Production
Observe (read)       auto       auto       auto
Propose (PR, dry-run)auto       auto       auto (human merges)
Act (mutate)         auto       notify     approval
Destructive (delete) approval   approval   denied by L2

Approval is bound to one exact call, by hash of tool and arguments, and expires — so it cannot be replayed for a different action. Where a mutation is allowed, prefer routing it through a GitOps pull request: Argo CD applies it with history and rollback, and a human reads the diff.

FAQ

Why not just rely on the model to refuse harmful commands? Because alignment, system prompts and input classifiers reduce the probability of misbehaviour but offer no guarantee, and their failure modes are adversarially discoverable. The model is not a security boundary, so every control here keeps holding even when the agent is fully compromised by prompt injection. The model is hygiene; the layers are the boundary.

Do I need all seven layers, or can I start with admission control? Admission control at the API server is the single highest-value layer, because it holds even if the agent bypasses the gateway with a stolen token. But it only constrains object content. Start there, add workload identity so requests are attributable, then default-deny egress, then tool mediation. Skip nothing that decides what the agent can reach.

Why is an IP-based NetworkPolicy not enough for egress control? External services are addressed by name and their addresses change, so an IP allowlist breaks silently the first time a provider moves an endpoint. FQDN-aware policies are required: Cilium on self-managed clusters, FQDNNetworkPolicy on GKE with Dataplane V2, ACNS FQDN filtering on AKS with Azure CNI. On EKS the VPC CNI is not FQDN-aware, so you need Cilium or a forced egress proxy.

What breaks first when I let an agent remediate incidents in production? Runaway loops, not attacks. An agent that restarts a deployment whose failures come from a downstream database will keep restarting it and make the incident worse. That is why governance needs per-object mutation budgets, a circuit breaker that escalates after repeated failed remediations, and approval-gated mutations in production. A kill switch has to be rehearsed before you need it.

For the cluster this runs on, start with the k3s production stack and the observability stack — audit-log baselines in layer 7 need metrics in place first. If the agent also calls a model endpoint, the self-hosted AI stack is where the FQDN allowlist earns its keep.

A guide, not a product. This page is deployment documentation with working examples — there is no zip, no download, no support contract. You adapt the patterns to your own environment, and you own the result.

Want this running production-grade in your infrastructure instead? I teach this as the two-day Kubernetes in Production workshop — details on the consulting page.