Terraform Drift: Detect, Classify, Remediate — A Severity-Based Playbook

Not all Terraform drift is equal. A hands-on playbook for detecting drift in CI, classifying it by risk tier, and remediating without drowning in alerts.

Air-Gapped Nix: The Self-Hosted Binary Cache Playbook

On the internet, a binary cache is an optimization. Behind a firewall, it is the difference between a deploy that takes ten minutes and one that compiles the world from source.

Nix's superpower — hermetic, reproducible builds — has a price: the first build of nixpkgs on a machine takes hours because every derivation is built in isolation from source. Normally the official cache.nixos.org absorbs that. In an air-gapped cluster there is no normal: nodes behind the firewall either substitute from a cache you run yourself, or each node rebuilds everything it touches, on every deploy. This playbook is the path we took for an air-gapped SCS Kubernetes cluster (K3s on NixOS, HTTP proxy at proxy.internal:3128, Ceph RGW object storage): pick the server, sign it, make upload automatic, and wire every node to it.

Why air-gap changes the math

Nix substitution is a content-addressed download over HTTP. The store protocol is simple: a nix-cache-info endpoint (StoreDir, WantMassQuery, Priority), and a .narinfo file per store path containing the hash, the references, and — critically — a Sig: line. That signature is the entire security model:

• A client only accepts a store path whose public key and hash both match what it asked for.
• Therefore the server signs on-the-fly (Attic's managed signing) — clients never hold the private key; they only carry the public one.
• Anyone can run a server; the trust boundary lives in each client's trusted-public-keys, not in the network.

So the plan is four pieces: a server behind the firewall (signing stays server-side — Attic's managed signing), an automatic upload path (so you never push manually), a client config on every node and build machine, and the public key that the config pins in trusted-public-keys.

Choose the server

Attic wins for a multi-node fleet: server-side signing (one key, never on clients), global dedup across the whole store, per-team tokens for tenants (CI, laptops, nodes), and an S3-compatible backend that maps directly onto the Ceph RGW you already run. nix-serve is listed second for a reason: if you have exactly one machine and one signing key, it is a 10-line module. Everything below assumes Attic.

The server: Attic on Ceph RGW

A minimal NixOS module — the JWT secret and S3 credentials live in an agenix-supplied environment file, the store lives in S3 (the module comes from attic's flake: attic.nixosModules.atticd):

# cache host (NixOS)
{ config, pkgs, ... }:
{
  age.secrets.atticd-env = {
    file = ../secrets/atticd-env.age; # ATTIC_SERVER_TOKEN_HS256_SECRET_BASE64 + S3 creds
    owner = "atticd";
  };

  services.atticd = {
    enable = true;
    environmentFile = config.age.secrets.atticd-env.path; # required by the module
    settings = {
      listen = "[::]:8080";
      storage = {
        type = "s3";               # Ceph RGW — global dedup across teams
        region = "us-east-1";
        bucket = "nix-cache";
        endpoint = "https://ceph-rgw.internal";
        credentials = {
          access_key_id = "attic";
          secret_access_key = "CHANGE_ME"; # from the age secret — never a literal in the store
        };
      };
      chunking.nar-size-threshold = 64 * 1024; # 64 KiB: chunk NARs at or above this
      garbage-collection = {
        interval = "12 hours";
        default-retention-period = "750 days";
      };
    };
  };
}

Two details that bite later:

• garbage-collection must be set. Without it Attic keeps every path forever and the S3 bucket grows unboundedly as CI churns. 12 hours / 750 days says: reap orphans twice a day, discard anything older than two years, and retain GC roots.
• Auth is JWT-based, not password-based. atticd prints a root token on first start; per-team tokens come from atticadm make-token. Anyone who pushes (CI, laptop builders) holds a short-lived token; everyone else just reads — which is what trusted-public-keys is for.

Make upload automatic: post-build-hook

Nothing should upload by hand. Two working mechanisms: the Nix-native post-build-hook (a script Nix fires after every successful build), or Attic's watch-store daemon. The hook first:

# /etc/nix/upload-to-cache.sh — run after every build
#!/bin/sh
set -eu; set -f; export IFS=' '
exec attic push internal:default $OUT_PATHS

# wired in as the post-build-hook — every build auto-pushes:
nix.settings.post-build-hook = "/etc/nix/upload-to-cache.sh";

For always-on machines (CI runners, dedicated builders) the attic watch-store daemon is the cleaner default: it watches the local store and pushes every new path — no hook script to maintain. With that in place, a nixos-rebuild on node A uploads every fresh path; nodes B, C, and D substitute them within minutes. A corresponding post-build-hook or watch-store on every producer guarantees any path produced anywhere is available everywhere.

Client side: subscribe, don't build

Every NixOS node and build machine gets the same two settings — the substituter (where to fetch) and the public key (what to trust):

nix.settings = {
  substituters = [
    "http://cache.internal:8080"
    "https://cache.nixos.org/"     # as fallback for the few machines with egress
  ];
  trusted-public-keys = [
    "cache.internal:...."          # from: attic cache info <cache> → "Public Key:"
    "cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY="
  ];
};

Verify the whole chain with one command on a node:

nixos-rebuild dry-build --flake .#mySystem
# "these N derivations will be built"        ← cache miss, bad sign
# "these N paths will be fetched (S bytes)"  ← hit, what you want

If a node falls back to building, the cache is either missing the path, the key is wrong, or the priority is misconfigured — the dry-run output tells you which before the node burns two hours compiling Chromium.

Bootstrapping the gap itself

The cache host is inside the air gap — so where does its first store come from? The trick is that air gaps are usually network gaps, not total isolation: you have an HTTP proxy (proxy.internal:3128) that maps to a controller node with egress. Bootstrap in three steps:

1. On the controller, run nixos-rebuild with the proxy env vars set — the proxy pulls from cache.nixos.org and fills the local store.
2. attic push the controller's store into the new cache (or let post-build-hook do it — it's already wired).
3. Flip the fleet's substituters to the internal endpoint; egress stays closed on the nodes, NO_PROXY=...cache.internal keeps their traffic off the proxy.

After that, egress on every node can go to zero except the proxy exception — which is a genuinely nice security property for an air-gapped cluster: nodes talk to a handful of endpoints, and almost all of them are yours.

Security checklist

• JWT secret + S3 credentials in agenix (environmentFile) — never a literal in /nix/store; signing stays server-side (managed signing)
• trusted-public-keys pinned in version control on every node (this is the real trust boundary)
• Tenant tokens, short-lived; rotate on team churn or suspected compromise
• Internal-only DNS + NO_PROXY so cache traffic never traverses the proxy
• Cache host has egress for the initial pull-through, then deny it (proxy exception only)
• Garbage collection configured — an unbounded S3 bucket is a cost incident waiting to happen
• Monitor nixos-rebuild dry-build hit rate in CI; a silent drop in substitution is a build-time regression long before it becomes a fire

A signed, deduplicated binary cache is the quiet backbone of an air-gapped fleet. It does not make builds faster — it makes them shared, and sharing is what turns "every node rebuilds the world" into "one machine builds, the fleet subscribes." Set it up once, and the cache pays for itself on the very next deploy.