Security-First IaC: Benchmarking LLMs for Terraform Generation

Infrastructure as Code (IaC) is powerful but error-prone. This article benchmarks Large Language Models on their ability to generate secure Terraform configurations, addressing the critical question: can AI generate production-ready infrastructure that doesn't compromise security?

ARGUS: MCP-Grounded Root Cause Analysis for Kubernetes Incidents

Bridging the gap between LLM reasoning and Kubernetes reality with the Model Context Protocol

The Diagnosis Problem

Kubernetes incident response starts with a single painful question: "What actually happened?"

Even with comprehensive observability — Prometheus metrics, Loki logs, distributed traces — diagnosing root causes in complex microservices running on Kubernetes is enormously challenging. The typical diagnosis workflow looks like this:

User report → Alert → On-call engineer → 
    [Check 5 dashboards] → 
    [Grep through logs] → 
    [Query metrics] → 
    [Ask in Slack: "Anyone seen this?"] → 
    [Form hypothesis] → 
    [Test hypothesis] → 
    [Maybe find cause, maybe not] → 
    [Attempt fix, pray, repeat]

Each step requires:

• Context switching — Different tools, different UIs, different query languages
• Tribulation — Mental effort to correlate disparate signals
• Institutional knowledge — "I remember something about this from 6 months ago..."
• Time — Often 30-60 minutes for complex incidents

Meanwhile, your customers are experiencing the outage, and the clock is ticking.

The Promise and Reality of AI for Diagnosis

Large Language Models excel at:

✅ Pattern recognition — "This error looks like the incident from March 15th"
✅ Hypothesis generation — "It might be the connection pool, or the Redis eviction policy, or..."
✅ Natural language understanding — "Users report '502 Bad Gateway' means the proxy can't reach the pod"
✅ Documentation knowledge — "According to our runbook for service X..."

But they struggle with:

❌ Live data access — LLMs have no native way to query your clusters
❌ Deterministic reasoning — "The LLM thinks, but I can't see how it got there"
❌ Action execution — "The LLM suggested restarting the pod, but I need to do it safely"
❌ True root cause vs. symptoms — Distinguishing the spark from the wildfire

This is where MCP (Model Context Protocol) enters the picture.

What is MCP?

The Model Context Protocol is an open standard that communicates contexts to LLM applications. It provides a structured way for LLMs to:

1. Discover available tools and resources
2. Read from external systems (databases, APIs, files)
3. Execute operations (with permission)
4. Stream templates for structured requests

Think of MCP as the missing link between your LLM's reasoning power and your actual infrastructure. Without it, LLMs are operating with blinders on.

MCP vs. Traditional approaches

ARGUS Architecture

ARGUS (Adaptive Root Cause Understanding System) is a reference architecture for MCP-powered Kubernetes diagnosis. It connects LLMs directly to your cluster's runtime state while maintaining security and provide full traceability.

┌─────────────────────────────────────────────────────────────────┐
│                         ARGUS SYSTEM                               │
├─────────────────────────────────────────────────────────────────┤
│                                                                     │
│  ┌──────────────────┐     ┌──────────────────┐     ┌──────────┐ │
│  │                  │     │                  │     │          │ │
│  │  LLM AGENT       │────▶│  MCP SERVER      │────▶│ K8S API  │ │
│  │  (Diagnoser)     │     │  (Bridge)        │     │          │ │
│  │                  │     │                  │     │          │ │
│  └──────────────────┘     └────────┬─────────┘     └──────────┘ │
│                                       │                         │
│                          ┌────────────────────┼──────────────┐ │
│                          │                    │              │ │
│                          ▼                    ▼              ▼ │
│                   ┌──────────┐       ┌──────────┐   ┌─────────┐│
│                   │ Prometheus│       │   Loki   │   │  etc.   ││
│                   │  Metrics  │       │   Logs   │   │ vagrant ││
│                   └──────────┘       └──────────┘   └─────────┘│
│                                                                     │
│   ┌─────────────────────────────────────────────────────────────┐ │
│   │                    INTEGRATION LAYER                         │ │
│   │  - Security policies (RBAC, resource limits)                 │ │
│   │  - Query planning & optimization                              │ │
│   │  - Result caching & deduplication                            │ │
│   │  - Full audit trail                                         │ │
│   └─────────────────────────────────────────────────────────────┘ │
│                                                                     │
└─────────────────────────────────────────────────────────────────┘

MCP Server: The Bridge

The MCP server is the central component that:

1. Exposes Kubernetes resources as MCP tools — "List pods", "Get pod logs", "Describe deployment"
2. Provides structured templates — Helps the LLM format correct API calls
3. Handles authentication and authorization — Integrates with your existing RBAC
4. Implements rate limiting and safety — Prevents abusive or dangerous queries
5. Streams results incrementally — Enables the LLM to interactively refine requests

Example MCP Server Implementation

from mcp.server import Server
from mcp.server.models import InitializationOptions
import kubernetes.client as k8s
from kubernetes import config

# Load in-cluster config
config.load_incluster_config()
v1 = k8s.CoreV1Api()

app = Server("k8s-bridge")

@app.get_tools()
async def get_tools():
    """List all Kubernetes tools exposed via MCP."""
    return {
        "tools": [
            {
                "name": "list_pods",
                "description": "List pods in a namespace",
                "inputSchema": {
                    "type": "object",
                    "properties": {
                        "namespace": {
                            "type": "string",
                            "description": "Namespace to list pods from"
                        },
                        "label_selector": {
                            "type": "string",
                            "description": "Label selector filter"
                        }
                    },
                    "required": ["namespace"]
                }
            },
            {
                "name": "get_pod_logs",
                "description": "Get logs from a specific pod",
                "inputSchema": {
                    "type": "object",
                    "properties": {
                        "namespace": {"type": "string"},
                        "pod_name": {"type": "string"},
                        "container": {"type": "string"},
                        "since_time": {"type": "string"},
                        "tail_lines": {"type": "integer"}
                    },
                    "required": ["namespace", "pod_name"]
                }
            },
            # ... more tools: describe, exec, get_events, etc.
        ]
    }

@app.call_tool()
async def call_tool(name: str, arguments: dict):
    """Execute a Kubernetes tool."""
    if name == "list_pods":
        namespace = arguments.get("namespace", "default")
        label_selector = arguments.get("label_selector")
        
        pods = v1.list_namespaced_pod(
            namespace=namespace,
            label_selector=label_selector
        )
        
        return {
            "content": [{
                "name": pod.metadata.name,
                "status": pod.status.phase,
                "node": pod.spec.node_name,
                "created": pod.metadata.creation_timestamp.isoformat()
            } for pod in pods.items]
        }
    
    elif name == "get_pod_logs":
        namespace = arguments["namespace"]
        pod_name = arguments["pod_name"]
        tail_lines = arguments.get("tail_lines", 100)
        
        logs = v1.read_namespaced_pod_log(
            name=pod_name,
            namespace=namespace,
            tail_lines=tail_lines
        )
        
        return {"content": logs}
    
    # ... other tool implementations

@app.notification()
async def notification(request: dict):
    """Handle server-initiated notifications (e.g., pod events)."""
    # Could push real-time cluster events to the client
    pass

if __name__ == "__main__":
    app.run()

The ARGUS Diagnostic Loop

When ARGUS receives an incident alert, it follows this MCP-grounded diagnostic loop:

Step 1: Alert Ingestion and Correlation

# Incoming alert from Prometheus
alert:
  name: "HighErrorRate"
  severity: "critical"
  service: "checkout-service"
  namespace: "production"
  metric: "http_request_duration_seconds"
  threshold: 5.0
  current_value: 12.4
  summary: "Checkout service has 95% error rate"

ARGUS:

1. Correlates this with other recent alerts from the same service/namespace
2. Queries historical incidents for similar patterns
3. Forms initial hypotheses: [api-rate-limiting, db-connection-failure, upstream-dependency-issue]

Step 2: MCP-Powered Investigation

For each hypothesis, ARGUS uses MCP to gather evidence:

# Hypothetical LLM conversation with MCP

LLM: "I need to investigate the checkout-service. Let me start by checking its health."

# MCP: list_pods available
LLM: [call list_pods with namespace="production", label_selector="app=checkout-service"]

MCP: Returns 3 pods:
  - checkout-service-6789d4f5b-qrst: "Running"
  - checkout-service-6789d4f5b-uvwx: "Running" 
  - checkout-service-6789d4f5b-yzab: "CrashLoopBackOff"

LLM: "One pod is in CrashLoopBackOff. Let me check its logs."
LLM: [call get_pod_logs with namespace="production", pod_name="checkout-service-6789d4f5b-yzab", tail_lines=50]

MCP: Returns logs showing:
  "ConnectionError: Connection to payments-db timed out after 5 seconds"
  "Retrying connection... (attempt 17)"

LLM: "Now checking database connectivity."
LLM: [call list_pods with namespace="production", label_selector="app=payments-db"]

MCP: Returns database pods: all Running

LLM: "Checking database service and endpoints."
LLM: [call list_services with namespace="production", label_selector="app=payments-db"]

MCP: Returns service with ClusterIP

LLM: "Let me test connectivity from the problematic pod."
LLM: [call exec_pod with namespace="production", pod_name="checkout-service-6789d4f5b-yzab", command=["sh", "-c", "nc -zv payments-db 5432"]]

MCP: Returns: nc: connect to payments-db port 5432 (tcp) failed: Connection timed out

Step 3: Root Cause Identification

Based on the MCP-gathered evidence, ARGUS synthesizes:

root_cause:
  type: "network_connectivity_issue"
  primary: "checkout-service cannot reach payments-db"
  secondary: []
  evidence:
    - pod: "checkout-service-6789d4f5b-yzab"
      status: "CrashLoopBackOff"
      logs: "ConnectionError: Connection to payments-db timed out"
    - connectivity_test: "nc -zv payments-db 5432 failed"
    - other_pods: "checkout-service-6789d4f5b-qrst and uvwx can connect"
  probable_cause: "NetworkPolicy blocking traffic from new pod IP"
  severity: "critical"

Step 4: Remediation Suggestion

ARGUS doesn't just identify the problem — it provides actionable, safe remediation:

recommended_actions:
  - action: "verify_network_policy"
    description: "Check if NetworkPolicy is blocking traffic to payments-db"
    command: "kubectl get networkpolicy -n production | grep payments-db"
    risk: "read-only"
    confidence: 0.9
  
  - action: "check_pod_network_policy"
    description: "Verify the NetworkPolicy applied to checkout-service pods"
    command: "kubectl describe pod checkout-service-6789d4f5b-yzab -n production"
    risk: "read-only"
    confidence: 0.9
  
  - action: "test_network_policy"
    description: "Test if new pods can access payments-db using network policy tool"
    command: "kubectl port-forward svc/payments-db 5432:5432 -n production"
    risk: "low"
    confidence: 0.8

Step 5: Full Diagnostics Report

ARGUS generates a comprehensive report with:

1. Executive Summary — What happened, when, business impact
2. Timeline — Sequence of events with precise timestamps
3. Evidence Chain — All MCP queries, results, and reasoning steps
4. Root Cause — The verified underlying issue
5. Remediation — Immediate and long-term fixes
6. Prevention Tips — How to prevent similar incidents

Why MCP Makes This Possible

Argus wouldn't be effective without MCP because MCP provides:

1. Structured Tool Discovery

The LLM doesn't need to know the exact Kubernetes API — it just needs to know what capabilities are available. MCP handles the translation.

# LLM doesn't write this:
v1.layers_namespaced_pod(...)

# LLM writes this:
call_tool("list_pods", {"namespace": "production"})

# MCP handles the rest

2. Permission and Safety

MCP servers enforce security policies before executing any action:

@app.call_tool()
async def call_tool(name: str, arguments: dict):
    # Check RBAC
    if not check_rbac(name, arguments, user=current_user):
        raise PermissionError("Unauthorized")
    
    # Check resource limits
    if exceeds_limits(name, arguments):
        raise ValueError("Resource limit exceeded")
    
    # Rate limiting
    if rate_limited(user=current_user):
        raise RateLimitError("Too many requests")
    
    # Execute
    return await execute_tool(name, arguments)

3. Result Structuring

MCP returns results in a predictable, structured format that the LLM can reliably parse, eliminating the need for fragile string manipulation of raw API responses.

4. Implementation

MCP allows you to gradually expose more system it

Adoption Patterns

Pattern 1: Read-Only Diagnostics First

Start with ARGUS in read-only mode:

• Dos
  • List resources
  • Read logs
  • Describe configurations
  • Query metrics
• Don'ts
  • No modifications
  • No deletions
  • No deployments

This allows your team to validate ARGUS's accuracy without risk.

Pattern 2: Approval Workflow

Implement a human-in-the-loop approval system:

ARGUS identifies issue → ARGUS proposes remediation → 
Human reviews → Human approves → ARGUS executes → 
Human verifies → Human closes incident

Pattern 3: Automated Tiered Response

Create rules for automatic action based on severity and confidence:

auto_action_rules:
  - if: severity == "warning" and confidence > 0.95
    then: auto_approve
  - if: severity == "critical" and risk_level == "low"
    then: require_approval_from_oncall
  - if: severity == "critical" and risk_level == "high"
    then: require_approval_from_manager
  - if: severity == "critical" and risk_level == "very_high"
    then: escalate_to_page

The MCP Advantage Summarized

MCP transforms root cause analysis from a manual, reactive process into an agent-driven, explainable one. The key advantages:

Getting Started with ARGUS

Step 1: Build the MCP Server

Start with a read-only MCP server exposing your most-used Kubernetes resources: pods, logs, events, services, deployments. Get the LLM comfortable diagnosing before you add any write capability.

Step 2: Add Tool Schema and Guardrails

Define strict input schemas for every tool (namespace, name, limits). Add rate limiting, RBAC integration, and audit logging from day one. Never expose destructive operations through the initial MCP surface.

Step 3: Wire Up Observability

Feed the MCP server with metrics and logs so the agent can correlate. Prometheus for metrics, Loki for logs, and a service catalog for topology give the agent the signals it needs for real diagnosis.

Step 4: Human-in-the-Loop Approval

Before any write action, route through an approval workflow. Start with commands like restart pod that require one-click approval, and only relax as trust builds.

Step 5: Measure and Iterate

Track MTTR, diagnostic accuracy, and time-to-hypothesis. Use every post-incident review to feed new context and patterns back into the agent's knowledge.

Conclusion

ARGUS and MCP-grounded diagnosis represent a major shift in how Kubernetes incidents get resolved. Instead of expecting a human to hold the entire cluster topology, log structure, and deployment history in their head, the LLM becomes a grounded, tool-using diagnostician that can:

• Query live cluster state in real time
• Follow evidence chains deterministically
• Produce explainable root-cause analyses
• Hypothesize, verify, and iterate safely

MCP is the missing bridge between LLM reasoning and infrastructure reality. For teams serious about cutting MTTR and ending the era of blind Kubernetes debugging, MCP-grounded agents are no longer experimental — they're the fastest path to reliable, explainable incident response.

Interested in better Observability? Check out our guides on eBPF vs Service Mesh, Self-Hosted Observability Stacks, and Container Orchestration.