Security-First IaC: Benchmarking LLMs for Terraform Generation

Infrastructure as Code (IaC) is powerful but error-prone. This article benchmarks Large Language Models on their ability to generate secure Terraform configurations, addressing the critical question: can AI generate production-ready infrastructure that doesn't compromise security?

Observability for Agentic Systems: The New DevOps Imperative

When your application can make its own decisions, traditional monitoring isn't enough. Welcome to the era of AI-native observability.

The Observability Gap

DevOps has always been about understanding system state. Traditional applications are relatively straightforward: request → process → response. The three pillars of observability — metrics, logs, traces — have served us well.

But agentic systems are fundamentally different. They:

• Make autonomous decisions — Not just following code paths, but choosing them
• Have memory and context — Decisions depend on past interactions
• Use external knowledge — Retrieving from vector DBs, knowledge graphs
• Orchestrate multi-step workflows — One action leads to another, leads to another
• Are non-deterministic — The same input can produce different outputs on different runs

This creates an observability gap — our traditional tools can't easily answer questions like:

• Why did the agent make that decision?
• What knowledge did it have access to?
• How confident was it in its answer?
• What was the chain of reasoning?
• When did it go wrong?

The Evolution of Observability

Agentic systems require an evolution of observability. We need to extend the three pillars and add new dimensions:

Traditional Observability        Agentic Observability
────────────────────────        ────────────────────────

┌─────────────────┐              ┌───────────────────────────┐
│  METRICS        │              │  METRICS                  │
│  - Request rate │              │  - Token usage           │
│  - Error rate   │              │  - Inference latency     │
│  - Latency      │              │  - Confidence scores     │
│                 │              │  - Decision metrics      │
└─────────────────┘              │  - Stateful resource      │
       ▲                       │    usage                  │
       │                       └───────────────────────────┘
       │                              ▲ 
       │                              │
       ├─── Traditional          ├─── Still important
       │                              │
┌─────────────────┐              ┌───────────────────────────┐
│   LOGS          │              │   LOGS                    │
│  - stdout       │◀─────────────│  - LLM prompts            │
│  - stderr       │              │  - LLM completions        │
│  - Application  │              │  - Agent decisions        │
│    events       │              │  - Tool calls             │
└─────────────────┘              │  - Context snapshots      │
       ▲                       │  - Knowledge retrievals   │
       │                       └───────────────────────────┘
       │                              ▲
       │                              │
       └─────── Traditional     ├─── CRITICAL for agents
              issues logs

┌─────────────────┐              ┌───────────────────────────┐
│  TRACES         │              │  TRACES + REASONING       │
│  - Request flow │◀─────────────│  - End-to-end agent       │
│  - Service      │              │    execution flow        │
│    dependencies │              │  - LLM reasoning chains   │
└─────────────────┘              │  - Tool call chains       │
                                │  - Decision trees         │
                                │  - Cross-agent traces     │
                                └───────────────────────────┘
       ▲                              ▲
       │                              │
       └──────────────────────────────┘
              STILL IMPORTANT

                               NEW DIMENSIONS
                        ┌───────────────────────────┐
                        │  CONTEXT                  │
                        │  - Current agent state   │
                        │  - Memory contents        │
                        │  - Knowledge graph        │
                        │    snapshots              │
                        └───────────────────────────┘
                        ┌───────────────────────────┐
                        │  EVALUATIONS             │
                        │  - Human feedback        │
                        │  - Automated quality     │
                        │    scoring               │
                        │  - Safety classifications │
                        └───────────────────────────┘

The Fourth Pillar: Context

For agentic systems, context is the fourth pillar of observability. Context includes:

• Agent state — Current step, current goal, accumulated information
• Memory contents — What the agent remembers from previous interactions
• Knowledge access — What knowledge sources were queried and with what results
• Environment state — External system state that influenced decisions
• User context — Who is the user, what are their permissions and preferences

Why Context Matters

Without context, you can't explain why an agent made a particular decision. Consider this scenario:

Question: "Why did the agent recommend Product A instead of Product B?"

With Context:

• Context: User is on a budget, Product B is out of stock, Product A has better reviews
• Reasoning: Agent weighed price, availability, and quality, determining A was optimal
• Answerable: "User is on budget; B is unavailable; A has better reviews"

Without Context:

• Agent output: "I recommend Product A"
• Unanswerable: We have no idea why

Implementing Context Observability

class AgentContextTracker:
    def __init__(self):
        self.event_log = []
        self.current_state = {}
        self.memory_snapshots = []
    
    def capture_state(self, agent):
        """Capture current agent state for observability."""
        snapshot = {
            "timestamp": time.time(),
            "step": agent.current_step,
            "goal": agent.current_goal,
            "context": agent.context,
            "memory_hash": hash(agent.memory),
            "active_tools": agent.active_tools,
        }
        self.memory_snapshots.append(snapshot)
        return snapshot
    
    def capture_event(self, event_type, data):
        """Capture a significant event."""
        event = {
            "timestamp": time.time(),
            "type": event_type,
            "data": data,
            "step": self.current_state.get("step"),
        }
        self.event_log.append(event)
        return event
    
    def capture_llm_call(self, prompt, completion, model, tokens, latency):
        """Capture LLM interaction details."""
        return self.capture_event("llm_call", {
            "prompt": prompt[:500] + "...",  # Truncate for storage
            "completion": completion[:500] + "...",
            "model": model,
            "input_tokens": tokens.get("input"),
            "output_tokens": tokens.get("output"),
            "latency_ms": latency,
            "temperature": model_config.get("temperature"),
        })
    
    def capture_tool_call(self, tool_name, arguments, result, duration):
        """Capture tool usage."""
        return self.capture_event("tool_call", {
            "tool": tool_name,
            "arguments": arguments,
            "result_summary": str(result)[:500] + "...",
            "duration_ms": duration,
            "status": "success" if result else "failure",
        })
    
    def capture_decision(self, decision, confidence, reasoning):
        """Capture reasoning process and decision."""
        return self.capture_event("decision", {
            "decision": decision,
            "confidence": confidence,
            "reasoning": reasoning,
        })

Rereasoning Traces: The Missing Link

Traditional distributed tracing follows requests through services. Reasoning traces follow LLM thinking through decisions.

What are Reasoning Traces?

Reasoning traces capture the chain of thought that leads to an agent's decision, including:

1. Initial prompt — What was the agent asked to do?
2. LLM thinking — The intermediate reasoning steps (if using chain-of-thought)
3. Tool calls — What actions did the agent take to gather information?
4. Context updates — How did the agent's understanding evolve?
5. Final decision — What conclusion did the agent reach?
6. Confidence scores — How confident was the agent?

Example Reasoning Trace

{
  "trace_id": "agent_abc123_20260831_102345",
  "agent_id": "customer-support-agent",
  "session_id": "session_xyz789",
  "user_id": "user_42",
  "start_time": "2026-08-31T10:23:45Z",
  "end_time": "2026-08-31T10:25:12Z",
  "total_tokens": 4520,
  "total_latency_ms": 87000,
  "final_decision": "Refund customer $50 for defective product",
  "confidence": 0.92,
  "steps": [
    {
      "step_index": 1,
      "type": "llm_thought",
      "content": "The user wants a refund. I need to check if they're eligible and for how much.",
      "tokens": 25,
      "latency_ms": 2000,
      "timestamp": "2026-08-31T10:23:47Z"
    },
    {
      "step_index": 2,
      "type": "tool_call",
      "tool": "get_order_history",
      "arguments": {"user_id": "user_42"},
      "result": {
        "order_id": "ORD-9876",
        "product": "Bluetooth Headphones",
        "purchase_date": "2026-07-15",
        "price": 149.99,
        "return_window": "30 days",
        "status": "returned_defective"
      },
      "duration_ms": 1500,
      "timestamp": "2026-08-31T10:23:50Z"
    },
    {
      "step_index": 3,
      "type": "llm_thought",
      "content": "User has a defective product returned within 30 days. Full refund applicable. Standard policy is full refund for defective items within return window.",
      "tokens": 40,
      "latency_ms": 3000,
      "timestamp": "2026-08-31T10:24:00Z"
    },
    {
      "step_index": 4,
      "type": "tool_call",
      "tool": "check_refund_policy",
      "arguments": {"product_category": "electronics", "reason": "defective"},
      "result": {"refund_percentage": 100, "processing_fee": 0, "notes": "Full refund for defective items"},
      "duration_ms": 800,
      "timestamp": "2026-08-31T10:24:05Z"
    },
    {
      "step_index": 5,
      "type": "tool_call",
      "tool": "calculate_refund",
      "arguments": {"original_price": 149.99, "refund_percentage": 100},
      "result": {"refund_amount": 149.99},
      "duration_ms": 500,
      "timestamp": "2026-08-31T10:24:10Z"
    },
    {
      "step_index": 6,
      "type": "decision",
      "decision": "Refund customer $149.99",
      "reasoning": "Customer returned defective Bluetooth Headphones (ORD-9876) within 30-day window. Policy allows full refund for defective items. Refund amount: $149.99.",
      "confidence": 0.92,
      "timestamp": "2026-08-31T10:25:12Z"
    }
  ],
  "metrics": {
    "steps_per_decision": 6,
    " avg_llm_tokens_per_step": 80,
    "avg_tool_latency_ms": 933,
    "total_cost_usd": 0.0452
  }
}

Open Standard: OpenReasoning

Several companies and open-source projects are working on OpenReasoning, a proposed standard for reasoning trace format. Similar to OpenTelemetry for traditional observability, OpenReasoning would provide:

• Standard schema for reasoning events
• Collection and storage protocols
• Query interfaces
• Visualization tools

This would enable interoperability between different agent frameworks and observability tools.

Metrics for Agentic Systems

Traditional metrics need to be reimagined for agentic systems:

Token Metrics

Latency Metrics

Quality Metrics

Decision Metrics

Resource Metrics

Logging for Agentic Systems

What to Log

1. LLM Interactions

   • Prompt (truncated for sensitive data)
   • Model used and parameters (temperature, max_tokens)
   • Completion/generation
   • Token counts
   • Latency
   • Cost

2. Agent Decisions

   • The decision made
   • Confidence score
   • Reasoning chain
   • Context used

3. Tool Calls

   • Tool name and arguments
   • Result (truncated)
   • Duration
   • Status (success/failure)

4. Memory Changes

   • Memory operations (read, write, update, delete)
   • Memory contents (hashed or summarized for privacy)
   • Memory size

5. Knowledge Access

   • Knowledge base queries
   • Retrieval results
   • Source references

6. User Interactions

   • User requests
   • User feedback
   • Satisfaction ratings

Log Levels for Agents

Traces for Agentic Systems

End-to-End Agent Traces

Traditional distributed traces follow requests through services. Agent traces follow decisions through reasoning:

User Request ──▶ Agent Entry Point
                        │
                        ├─▶ LLM Call 1 (Understand request)
                        │    │
                        │    └─▶ Tool Call 1 (Get user history)
                        │         │
                        │         └─▶ Database Query
                        │
                        ├─▶ LLM Call 2 (Formulate response strategy)
                        │    │
                        │    └─▶ Tool Call 2 (Check inventory)
                        │         │
                        │         └─▶ API Call
                        │              │
                        │              └─▶ LLM Call 3 (Process API response)
                        │
                        └─▶ Agent Response

Trace Context Propagation

Extend OpenTelemetry to propagate agent context:

# Traditional OpenTelemetry span
span = tracer.start_span("http.request")

# Agent context extensions
agent_span = tracer.start_span(
    "agent.reasoning",
    attributes={
        "agent.id": "customer-support-1",
        "agent.version": "2.1.0",
        "agent.session_id": "sess_123",
        "agent.current_step": 3,
        "agent.current_goal": "process_refund",
    }
)

Multi-Agent Traces

For systems with multiple collaborating agents, traces need to capture:

• Agent-to-agent communication — What messages were passed?
• Delegation chains — Which agent delegated to which?
• Shared context — What information was shared between agents?
• Coordinated decisions — Which agent made which decisions?

{
  "trace_id": "multi_agent_001",
  "agents": [
    {
      "agent_id": "router",
      "role": "request_router",
      "actions": ["Delegated to customer-support agent"]
    },
    {
      "agent_id": "customer-support",
      "role": "customer_support",
      "actions": ["Analyzed refund request", "Called order_history tool", "Made refund decision"]
    },
    {
      "agent_id": "billing",
      "role": "billing_processor",
      "actions": ["Processed refund", "Updated accounting ledger"]
    }
  ],
  "message_flow": [
    {"from": "user", "to": "router", "content": "I want a refund"},
    {"from": "router", "to": "customer-support", "content": "Refund request forwarding"},
    {"from": "customer-support", "to": "billing", "content": "Refund approval: $149.99"},
    {"from": "billing", "to": "user", "content": "Refund processed"}
  ]
}

Visualization: The Agent Dashboard

Traditional dashboards show system metrics. Agent dashboards need to show agent activity:

Overview Dashboard

┌─────────────────────────────────────────────────────────────┐
│                    AI/CD DASHBOARD                          │
├──────────────┬──────────────┬──────────────┬──────────────┤
│  AGENT STATS │  TOKEN USAGE │  PERFORMANCE │  DECISIONS   │
├──────────────┼──────────────┼──────────────┼──────────────┤
│  Active: 42  │  Today:     │  Success:    │  Total:     │
│  Total: 128  │    2.4M     │    94%       │   1,247     │
│  Failures: 3 │  This hour: │  Avg conf:   │  High risk: │
│              │    180K     │    0.87      │      42     │
├──────────────┴──────────────┴──────────────┴──────────────┤
│                                                             │
│  ┌────────────────────────────────────────────────────────┐ │
│  │                    REAL-TIME ACTIVITY                  │ │
│  │                                                          │ │
│  │  customer-support-1 Contents processing refund for J   │ │
│  │  inventory-agent-2    ═ Check stock levels             │ │
│  │  deployment-agent-3   ● Deploying build #456           │ │
│  │                          +--------------------------------+ │
│  │  [████████░░░░░░░░]       Embedding search               │ │
│  │  LLM call                 Processing    2.4s          │ │
│  └────────────────────────────────────────────────────────┘ │
│                                                             │
└─────────────────────────────────────────────────────────────┘

Reasoning Trace Visualization

┌─────────────────────────────────────────────────────────────┐
│                    REASONING TRACE                           │
├─────────────────────────────────────────────────────────────┤
│                                                             │
│  User Query: "Why was my order delayed?"                    │
│                                                             │
│  ┌──────────┐     ┌──────────┐     ┌──────────┐            │
│  │  LLM     │────▶│  Tool    │────▶│  LLM     │            │
│  │  Thought │     │  Call:   │     │  Thought │            │
│  │          │     │  order   │     │          │            │
│  │ "Need to │     │  history │     │ "Order   │            │
│  │  check   │     │          │     │  was     │            │
│  │  order   │     │  Result: │     │  delayed │            │
│  │  status"│       │    3    │     │  but     │            │
│  └──────────┘     │   days  │     │  maybe   │            │
│         │          └────┬─────┘         │         │            │
│         │               │               ▼         │            │
│         │          ┌────▼─────┐     ┌──────────┐            │
│         │          │   Tool   │────▶│ Decision │            │
│         │          │  Call:   │     │          │            │
│         └──────────┘│ shipping │     │ "Order   │            │
│                      │  status  │     │  delay   │            │
│                      └──────────┘     │  due to  │            │
│                                           │  shipper │            │
│                                           │  delay"  │            │
│                                           └──────────┘            │
│                                                                      │
│  Total time: 8.2s   Total tokens: 4,230   Confidence: 0.93       │
│                                                             │
└─────────────────────────────────────────────────────────────┘

Agent Health Dashboard

┌─────────────────────────────────────────────────────────────┐
│                    AGENT HEALTH                              │
├─────────────────────────────────────────────────────────────┤
│                                                             │
│  AGENT: customer-support-v2        STATUS: ✅ Healthy        │
│                                                             │
│  ┌─────────────┬─────────────┬─────────────┬─────────────┐ │
│  │  RESOURCE   │   CURRENT   │   TREND     │    LIMIT    │ │
│  ├─────────────┼─────────────┼─────────────┼─────────────┤ │
│  │ Memory      │   2.4 GB    │   ↑ 5%      │    8 GB     │ │
│  │ Tokens/day │ 1.8M        │   ↗ 12%     │   10M/day   │ │
│  │ Latency p95 │   2.8s      │   ↘ 8%      │    10s      │ │
│  │ Error rate  │   0.3%      │   → 0%      │     1%      │ │
│  └─────────────┴─────────────┴─────────────┴─────────────┘ │
│                                                             │
│  RECENT INCIDENTS:                                         │
│  ├─ █ 15m ago: Low-confidence response (confidence: 0.62)   │
│  ├─ █ 2h ago: Tool call timeout (retry succeeded)           │
│  └─ █ 1d ago: Memory growth spike (resolved with restart)   │
│                                                             │
│  DEPENDENCIES:                                             │
│  ├─ LLM: gpt-4o ✅ Healthy (latency: 1.2s)                   │
│  ├─ Vector DB: Qdrant ✅ Healthy (latency: 45ms)             │
│  └─ Tools: 12/12 ✅ All operational                           │
│                                                             │
└─────────────────────────────────────────────────────────────┘

Implementation Patterns

Pattern 1: Instrument Everything at the Source

Add observability directly to your agent framework:

from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import ConsoleSpanExporter, BatchSpanProcessor

# Set up tracing
trace.set_tracer_provider(TracerProvider())
trace.get_tracer_provider().add_span_processor(
    BatchSpanProcessor(ConsoleSpanExporter())
)

tracer = trace.get_tracer(__name__)

class ObservableAgent:
    def __init__(self, name, version):
        self.name = name
        self.version = version
        self.tracer = tracer
    
    def run(self, user_query):
        with self.tracer.start_as_current_span(
            f"agent.{self.name}.run",
            attributes={
                "agent.name": self.name,
                "agent.version": self.version,
                "user.query": user_query[:100],  # Truncate
            }
        ) as span:
            # Agent logic
            try:
                result = self._execute_workflow(user_query)
                span.set_attribute("agent.result", str(result)[:100])
                span.set_status(trace.Status[trace.StatusCode.OK])
                return result
            except Exception as e:
                span.record_exception(e)
                span.set_status(trace.Status[trace.StatusCode.ERROR])
                raise
    
    def _execute_workflow(self, query):
        reasoning_steps = []
        
        with self.tracer.start_as_current_span("agent.reasoning_chain") as chain_span:
            # Step 1: Understand query
            llm_response = self._call_llm(f"Understand this query: {query}")
            reasoning_steps.append({"type": "llm", "content": llm_response})
            
            # Step 2: Call tools
            tool_result = self._call_tool("get_relevant_info", query)
            reasoning_steps.append({"type": "tool", "result": tool_result})
            
            # Step 3: Generate response
            final_response = self._call_llm(
                f"Based on: {llm_response}\nTool result: {tool_result}\nGenerate final answer:"
            )
            reasoning_steps.append({"type": "llm", "content": final_response})
            
            chain_span.set_attribute("reasoning.steps", len(reasoning_steps))
        
        return reasoning_steps
    
    def _call_llm(self, prompt):
        with self.tracer.start_as_current_span("llm.call") as span:
            span.set_attribute("llm.prompt", prompt[:100])
            
            # Actual LLM call
            response = llm_client.generate(prompt)
            
            span.set_attribute("llm.response", response[:100])
            span.set_attribute("llm.tokens.input", response.usage.input_tokens)
            span.set_attribute("llm.tokens.output", response.usage.output_tokens)
            span.set_attribute("llm.latency_ms", response.latency_ms)
            span.set_attribute("llm.model", response.model)
            
            return response.text
    
    def _call_tool(self, tool_name, arguments):
        with self.tracer.start_as_current_span(f"tool.{tool_name}") as span:
            span.set_attribute("tool.name", tool_name)
            span.set_attribute("tool.arguments", str(arguments)[:100])
            
            result = tool_registry[tool_name](**arguments)
            
            span.set_attribute("tool.result", str(result)[:100])
            span.set_attribute("tool.duration_ms", getattr(result, 'duration_ms', 0))
            
            return result

Pattern 2: Centralized Observability Pipeline

┌──────────────────┐
│                  │
│  Agent Framework │
│  (LangGraph,     │
│   CrewAI, etc.)  │
│                  │
└────────┬─────────┘
         │
         │ (OpenTelemetry traces/spans)
         ▼
┌──────────────────┐
│                  │
│  OpenTelemetry   │
│  Collector        │
│                  │
└────────┬─────────┘
         │
         ├───▶ Prometheus (metrics)
         │
         ├───▶ Jaeger/Tempo (traces)
         │
         ├───▶ Loki (logs)
         │
         └───▶ Neo4j (reasoning traces + recommendations)
               │
               ▼
        ┌──────────────────┐
        │                  │
        │  Observability   │
        │  Backend         │
        │  (Grafana, etc.) │
        │                  │
        └──────────────────┘

Pattern 3: Agent-Specific Storage

Store agent-specific data for deep analysis:

# Separate storage for different observability needs
observability_stores = {
    "metrics": PrometheusClient(),
    "traces": JaegerClient(),
    "logs": LokiClient(),
    "reasoning_traces": Neo4jGraph(),  # For reasoning chains
    "context_snapshots": VectorDB(),    # For agent state
    "evaluations": PostgreSQL(),       # For quality metrics
}

Pattern 4: Sampling for Cost Control

Observability can be expensive for high-volume agent systems:

class SmartSampler:
    def __init__(self, sample_rate=0.1, always_sample_patterns=None):
        self.sample_rate = sample_rate
        self.always_sample_patterns = always_sample_patterns or []
    
    def should_sample(self, context):
        # Always sample these
        for pattern in self.always_sample_patterns:
            if re.match(pattern, json.dumps(context)):
                return True
        
        # Sample a portion
        return random.random() < self.sample_rate
    
    def should_sample_full_prompt(self, context):
        # Only sample full prompts for 1% of requests (privacy)
        return random.random() < 0.01

# Usage
sampler = SmartSampler(
    sample_rate=0.5,  # Sample 50% of agent sessions
    always_sample_patterns=[
        ".*error.*",
        ".*failure.*", 
        ".*critical.*",
        ".*low.*confidence.*"
    ]
)

if sampler.should_sample(context):
    # Full observability
    capture_full_trace(agent_execution)
else:
    # Minimal observability
    capture_basic_metrics(agent_execution)

Tooling for Agent Observability

Open Source Tools

Commercial Tools

The Observability Maturity Model

Level 1: Basic Logging

• ⚠️ Logs to stdout
• ⚠️ No structured data
• ⚠️ Manual analysis
• Require traditional

Level 2: Metrics and Traces

• ✅ Standard three pillars
• ✅ Alerting on errors
• ✅ Basic dashboards
• Good for traditional apps

Level 3: Agent-Aware

• ✅ Agent-specific metrics
• ✅ Reasoning traces
• ✅ Decision tracking
• ✅ Tool call monitoring
• Minimum for agentic systems

Level 4: Intelligent Observability

• ✅ Adaptive sampling
• ✅ Anomaly detection for agents
• ✅ Reason chains with explanation
• ✅ Business impact tracking
• Emerging standard

Level 5: Self-Optimizing

• ✅ Observability-driven improvements
• ✅ Feedback loops between observability and agents
• ✅ Self-diagnosing agents
• ✅ Predictive issue prevention
• Future state

Challenges

1. Data Volume

Agentic systems generate massive amounts of data:

• Large LLM prompts and responses
• Detailed reasoning chains
• High-frequency tool calls
• Context snapshots

Mitigations:

• Smart sampling (as shown above)
• Data retention policies
• Efficient storage (columnar formats, compression)
• Archiving to cold storage

2. Privacy and Security

Observability data often contains:

• User queries and personal data
• Internal system information
• Business-sensitive decisions
• API keys and secrets (accidentally logged)

Mitigations:

• Data redaction and masking
• Tokenization of sensitive data
• Encryption at rest and in transit
• Strict access controls
• Regular security audits

3. Cost

Storing and querying observability data at scale is expensive.

Mitigations:

• Tiered storage (hot/warm/cold)
• Smart sampling
• Cost-aware storage choices
• Resource limits and quotas

4. Complexity

Agent observability adds significant complexity to already-complex systems.

Mitigations:

• Start with basic observability
• Gradually add agent-specific features
• Use managed services where possible
• Invest in training and documentation

Strategies

For Organization

1. Start small — Add basic agent observability alongside traditional
2. Use existing tools — Leverage OpenTelemetry, Prometheus, etc.
3. Standardize early — Establish patterns and standards before scaling
4. Invest in training — DevOps teams need new skills
5. Plan for growth — Observability needs will evolve rapidly

For Development Teams

1. Instrument first — Add observability from day one
2. Make it optional — Allow opt-out for sensitive contexts
3. Document the data — Clear schema and ownership
4. Test observability — Validate it works before deploying
5. Monitor observability — Ensure your observability system is observable

Conclusion

Observability for agentic systems isn't just an evolution of traditional observability — it's a revolution. As AI agents increasingly drive critical business decisions and operations, we need fundamentally new ways to understand and track their behavior.

The traditional three pillars (metrics, logs, traces) remain important, but they're no longer sufficient. We need:

• Context as the fourth pillar — Understanding agent state and knowledge
• Reasoning traces — Visibility into the decision-making process
• Agent-specific metrics — Measuring what matters for AI systems
• New visualization and analysis tools — Purpose-built for agentic complexity

The DevOps teams that embrace this new observability paradigm will be the ones that can safely deploy, manage, and scale agentic systems at production scale. Those that don't will find themselves flying blind in the age of AI.

The observability gap is real. But with the right approach, it's also bridgeable.

Want to learn more about modern DevOps practices? Check out our articles on eBPF vs Service Mesh for Observability and Self-Hosted Observability Stacks.

For AI-native monitoring, explore LLM Runtime Monitoring with OpenTelemetry.