Terraform on AWS: Building Production Infrastructure from Scratch

Complete guide to setting up AWS infrastructure with Terraform, including VPC design, EC2 instances, RDS databases, and best practices for state management.

Terraform Drift: Detect, Classify, Remediate — A Severity-Based Playbook

Stop treating every drift like a P1 — and stop letting the security-critical ones drown in tag noise.

Across 150+ AWS Terraform workspaces, severity-based filtering removed 73% of drift alerts while retaining 94% of security-relevant changes (tfdrift, arXiv:2608.18173, open source at github.com/sudarshan8417/tfdrift). That number captures the real problem with drift management: detection is a solved problem, triage is not. Teams either ignore drift entirely or page someone at 3 a.m. because a tag changed. This is a playbook for the middle ground — detect continuously, classify by risk, and remediate with intent.

What Drift Actually Is

"Drift" bundles three different diseases that need different detectors and different fixes:

1. State drift — real infrastructure differs from the state file. Someone changed a security group in the console, or an out-of-band automation did.
2. Code drift — state file and real infrastructure match, but neither matches your main branch. Typical cause: someone applied code from a feature branch or a laptop and never merged it.
3. Attribute drift — provider version changes or API defaults silently alter attributes your code never pinned.

Conflating these is why most "we have drift problems" conversations go nowhere. The first needs an ownership decision, the second a workflow fix, the third a version pin and a plan review.

Detection: Plan in CI, Not on Laptops

The core primitive is one flag:

terraform plan -detailed-exitcode
# exit 0 = no changes, 1 = error, 2 = drift/change detected

For pure drift inspection without change proposals, the refresh-only plan is the right tool — it compares real infrastructure against the state file and reports what would be reconciled:

terraform plan -refresh-only -detailed-exitcode

What matters is that this runs on a schedule, not only before applies. Drift that appears at 14:00 should be visible by 14:30, not at the next deployment. A minimal GitHub Actions workflow:

name: drift-detection
on:
  schedule:
    - cron: "*/30 * * * *"
  workflow_dispatch:
jobs:
  drift:
    runs-on: ubuntu-latest
    permissions:
      contents: read
      id-token: write   # OIDC, no long-lived keys
    steps:
      - uses: actions/checkout@v4
      - uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: arn:aws:iam::123456789012:role/terraform-readonly
          aws-region: eu-central-1
      - uses: hashicorp/setup-terraform@v3
      - name: Terraform init
        run: terraform init -input=false
      - name: Terraform drift check
        id: plan
        run: terraform plan -refresh-only -detailed-exitcode -no-color
        continue-on-error: true
      - name: Notify on drift
        if: steps.plan.outputs.exitcode == 2
        run: echo "Drift detected — open issue / send to Slack"

Note the read-only role in both the AWS credentials step and the backend — detection needs no write permissions anywhere. (The setup-terraform wrapper exposes exitcode as a step output; without it you would parse the exit code by hand.)

Two hygiene rules that get skipped too often:

• Treat state as sensitive — it contains secrets in plain text. Encrypted remote backend, no state in artefacts, and never on public runners without OIDC-scoped credentials.
• Pin the provider and backend versions in the workflow; a surprise provider upgrade during a drift check turns a detection run into a change run.

Classify: Four Risk Tiers

The tfdrift framework (open source, 60+ configurable rules covering AWS, Azure, and GCP patterns) classifies drift into four risk tiers based on resource type and attribute-level impact. The taxonomy is easy to approximate in policy code against the plan JSON:

A pragmatic approximation over the plan JSON, no ML required:

terraform plan -refresh-only -json > plan.json

# any new world-open INGRESS is critical — egress 0.0.0.0/0 is the default
# and harmless, so filter on the rule type
jq '.resource_changes[] | select(.type == "aws_security_group_rule")
    | select(.change.after.type == "ingress")
    | select(.change.after.cidr_blocks // [] | index("0.0.0.0/0"))' plan.json

The tfdrift evaluation backs the approach: tier-based filtering cuts alert volume by 73% while keeping 94% of security-relevant changes — comparable precision to ML-based filters at a fraction of the operational cost.

Remediate: Decide Who Is Right, Then Make It True

Every drift resolves to one of three outcomes:

Infrastructure is right. Someone fixed reality properly — adopt it instead of fighting it. Since Terraform 1.5 this is declarative, with import blocks and generated configuration:

import {
  to = aws_instance.web
  id = "i-0abc1234def5"
}

terraform plan -generate-config-out=generated.tf

Code is right. Reapply. But if the drift was manual and this is the second time, you have a permissions problem, not a Terraform problem — fix the console access (see prevention below).

Both are wrong. Fix the code first, then apply once. Never use terraform apply as a drift alarm clock without the code fix — that is how identical drift returns next week.

One anti-pattern deserves its own paragraph: manual state surgery as a first response. Reaching for terraform state rm or hand-editing state to "make the warning go away" converts an operations nuisance into a disaster recovery exercise. State surgery is for genuine state corruption, with a backup taken immediately beforehand and a rollback plan — not for a Tuesday afternoon.

Prevent: Shrink the Write Path

Prevention is boring and effective:

• GitOps as the single write path. All infrastructure changes go through PR → plan → apply. Atlantis, Terraform Cloud, or a plain CI pipeline with OIDC — the tool matters less than the constraint that nobody applies from a laptop.
• Read-only consoles for humans in production. SSO roles with view-only permissions; a break-glass role with expiry and alerting for genuine emergencies.
• Policy as code. OPA or Sentinel rules that reject any plan reintroducing critical-tier attributes (public ingress, disabled encryption, public buckets).
• Track drift as an SLO. Critical drift older than 24 hours is an incident, on the same dashboards as your availability metrics — see self-hosted observability for the stack side.

Where LLMs Fit (Carefully)

Two 2026 research results are worth knowing before anyone sells you "AI-powered drift remediation":

• RIVA (arXiv:2603.02345) shows that agentic drift verification breaks down when tools return wrong data — task accuracy falls to 27.3% for a naive ReAct agent. Its verifier/tool-generation agents with cross-validation recover accuracy to 50.0%. Lesson: LLM pipelines inherit every failure mode of the tools they call.
• The OpenSentinel thesis (RIT, 2026) outlines the right architecture for sovereignty-constrained teams: self-hosted LLMs receive structured, locality-aware plan contexts, propose remediation patches, Terraform itself stays the authoritative detector, every proposal is validated with a refresh-only plan, and humans merge the PR. The LLM proposes; Terraform disposes.

That maps directly onto the considerations in text-to-Terraform and security: generated infrastructure code is a draft for review, never an autonomous write path.

The Playbook, Condensed

• Scheduled refresh-only plan in CI, exit code 2 triggers triage
• Detection runs with read-only credentials, state in an encrypted backend
• Plan JSON piped through a tier classifier (critical → page, cosmetic → ignore)
• Import blocks and -generate-config-out for adopting intentional drift
• Code fix before every reconciliation apply — no blind reverts
• Single write path via GitOps; read-only consoles with break-glass
• Drift age metric on the same dashboard as availability

Drift never disappears — infrastructure is shared with colleagues, vendors, and cloud automation that do not read your repositories. The goal is not zero drift; it is critical drift that lives for hours, not weeks, and an alert channel nobody has learned to mute.