Disaster Recovery for Self-Hosted Infrastructure

k3s Disaster Recovery Testing and Edge Patterns

Comprehensive DR testing and edge-specific patterns for k3s — failover validation, scheduled snapshots, multi-cluster recovery, and RPO/RTO planning.

If you self-host, you are your own cloud provider. No SLA, no region failover, no support ticket to open at 3 AM. The upside is full control. The downside: when it breaks, it's on you.

Digital sovereignty doesn't mean owning your data — it means being able to get it back.

What "Disaster" Actually Means in a Self-Hosted Context

Before talking tools, define your failure modes:

Your backup strategy should address all of these. Not with the same tool, but with a layered approach.

The Three-Layer Backup Model

Layer 1: Filesystem Snapshots (Restic / Borg)

For general file backup — configs, application data, user home directories — Restic or Borg are the modern standard. They provide:

• Deduplication — incremental forever without storing every version
• Encryption — client-side, before data leaves your server
• Compression — typically 2-5x savings on text/log data

restic init --repo s3:https://s3.eu-central-1.wasabisys.com/my-backup/bucket

# Backup /etc, /var/lib, and application data
restic backup /etc /var/lib/docker/volumes \
  --repo s3:... \
  --password-file /etc/restic/pw \
  --tag monthly-$(date +%Y-%m)

Borg offers stronger deduplication for large file sets; Restic has broader backend support (S3, SFTP, local, REST server). Pick Restic if you want cloud object storage, Borg if you're backing up to a local NAS or remote server.

Layer 2: Database Dumps (PostgreSQL / MySQL)

File-level backup of a database directory is risky — you capture corruption, partial writes, and inconsistent states. Always use the database's native dump tool.

# PostgreSQL: daily full + continuous WAL archiving
pg_dumpall -f /backup/postgres/daily-$(date +%Y-%m-%d).sql

# Or better: pgBackRest for point-in-time recovery
pgbackrest --stanza=main --type=full backup
pgbackrest --stanza=main --type=archived --set=weekly-full backup

For PostgreSQL in production:

1. WAL archiving — stream WAL segments to a separate location (S3, NFS, MinIO). Enables point-in-time recovery down to the second.
2. pgBackRest — parallel backup/restore, checksum validation, delta restore. The de facto standard for serious Postgres.
3. Test your restore — a backup you never restored is a wish.

# Restore to a specific timestamp
pgbackrest --stanza=main --type=time \
  --target="2026-07-10 14:30:00+02" restore

Layer 3: Off-Site / Off-Line

Ransomware that encrypts your server will encrypt your attached backup drive too. Layer 3 breaks that chain.

• Off-site: Push encrypted backups to a different provider (Backblaze B2, Wasabi, rsync.net) using rclone or restic with a different remote.
• Off-line: Rotate a USB drive weekly that stays disconnected. Cheap insurance against crypto.
• 3-2-1 rule: Three copies, two media types, one off-site.

# Restic to a second, off-site repository
restic backup /data \
  --repo sftp://backup-host:/srv/restic-repo \
  --password-file /etc/restic/pw-alt

Automated Backup Schedule

A schedule that works for most self-hosted setups:

Automate with systemd timers or a cron-like tool:

# /etc/systemd/system/restic-backup.service
[Service]
Type=oneshot
ExecStart=/usr/local/bin/restic-backup.sh
Nice=19
IOSchedulingClass=idle

# /etc/systemd/system/restic-backup.timer
[Timer]
OnCalendar=*-*-* 02,10,18,22:00:00
RandomizedDelaySec=600
Persistent=true

Disaster Recovery Playbook

Scenario: Accidental rm -rf / on the Docker data directory

# 1. Stop all containers to prevent writes
docker stop $(docker ps -q)

# 2. Restore /var/lib/docker/volumes from Restic
restic restore latest \
  --path /var/lib/docker/volumes \
  --target / \
  --repo s3:...

# 3. Verify database integrity
docker exec db pgbackrest --stanza=main check

# 4. Start services and validate
docker start $(docker ps -aq)

Scenario: Database corruption from a failed upgrade

# 1. Stop the database container
docker stop postgres

# 2. Restore from pgBackRest to a point before the upgrade
pgbackrest --stanza=main --type=time \
  --target="2026-07-09 03:00:00+02" restore

# 3. Start and verify
docker start postgres
psql -c "SELECT count(*) FROM pg_catalog.pg_tables;"

Scenario: Complete server loss (building fire / theft)

1. Provision new server from your IaC (Terraform/Ansible — yours is version-controlled, right?)
2. Restore system configs from Restic off-site repo
3. Restore databases from pgBackRest off-site repo
4. Verify DNS propagation, TLS certificates, health endpoints

This should take < 4 hours for a medium setup.

Real-World DR Plan: tobias-weiss.org

This site runs on a single-node Docker swarm with the following stack. Here's the actual backup posture for production:

Critical distinction: docker-compose.yml and all .env files live in a private git repo. Losing the server without this repo means rebuilding from memory. Keep your config repo backed up independently.

Backup Script

#!/bin/bash
# /usr/local/bin/daily-backup.sh - runs via systemd timer at 03:00 daily
set -euo pipefail

BACKUP_DIR="/backup/$(date +%Y-%m-%d)"
RESTIC_REPO="s3:https://s3.eu-central-1.wasabisys.com/tobias-backup"
RESTIC_PW="/etc/restic/pw"

mkdir -p "$BACKUP_DIR"

# 1. Dump PostgreSQL
docker exec postgres pg_dumpall -U postgres > "$BACKUP_DIR/postgres-full.sql"

# 2. Export n8n workflows
docker exec n8n n8n export:workflow --all --output=/backup/n8n-workflows

# 3. Backup Docker volumes via Restic
restic backup /var/lib/docker/volumes \
  --repo "$RESTIC_REPO" \
  --password-file "$RESTIC_PW" \
  --tag "$(hostname)-daily-$(date +%Y-%m-%d)" \
  --exclude '*/postgres/data/*'   # excluded — we have SQL dump

# 4. Notify healthchecks
curl -fsS -m 10 --retry 3 "https://hc-ping.com/YOUR-UUID"

Recovery Runbook

## Emergency Recovery: Complete Server Loss

**Target recovery time: 4 hours**

### Phase 1: Infrastructure (30 min)

- [ ] Provision new VPS / bare metal (Intel/AMD, 32GB+ RAM, NVMe)
- [ ] Install Docker + Docker Compose plugin
- [ ] Clone config repo: `git clone git@codeberg:user/server-config`
- [ ] Restore `.env` files from 1Password / Bitwarden

### Phase 2: Restore Data (1 hour)

- [ ] Mount S3 bucket with Restic key
- [ ] `restic restore latest --target / --repo s3:...`
- [ ] Verify volume checksums

### Phase 3: Databases (30 min)

- [ ] Start PostgreSQL container without data dir
- [ ] `pg_restore -U postgres -d postgres /backup/postgres-full.sql`
- [ ] Verify: `docker exec postgres psql -c "SELECT count(*) FROM users;"`

### Phase 4: Services (1 hour)

- [ ] `docker compose up -d` — verify all services healthy
- [ ] Check Traefik dashboard: TLS certs issued?
- [ ] Log into store: transaction history intact?
- [ ] Check listmonk: campaign lists loaded?

### Phase 5: Verification (30 min)

- [ ] curl -I https://tobias-weiss.org — returns 200
- [ ] Run a test Stripe checkout through to redirect
- [ ] Monitor logs: no ERROR entries in first 15 minutes
- [ ] Update DR log: date, actual time, issues encountered

Quarterly DR Drill

Schedule a 2-hour slot every quarter. The drill:

1. Clone the production repo to a staging VPS (DigitalOcean $6 droplet works)
2. Run the recovery runbook start-to-finish
3. Time each phase — if any exceeds the RTO target, update the runbook
4. Test a Stripe payment flow end-to-end
5. Document what broke and fix it before next quarter

Each quarter you don't test, you're accumulating risk. This is not theoretical — a silent Postgres upgrade or an expired S3 credential will only reveal itself during a drill or an actual disaster.

Validating Your Backup

A backup you never test is a disaster waiting to happen. Schedule a quarterly DR drill:

1. Spin up a disposable VM/container
2. Restore the full backup
3. Verify: can the application start? Can you log in? Is data current?
4. Measure: how long did it take? What failed? Update the runbook.

# Quick integrity check between backups
restic check --repo s3:... --password-file /etc/restic/pw
pgbackrest --stanza=main check

Monitoring

Don't rely on "it ran silently." For a deeper look at how production monitoring works across the full stack, see Production Observability Stack and Grafana Dashboards. Monitor your backups:

• Healthchecks.io (or self-hosted Uptime Kuma) — cron sends a ping after each successful backup; if it misses, you get an alert.
• Prometheus + Loki — log backup output to Loki, alert on ERROR in backup logs.
• Simple email — mail -s "Backup failed" in the error path. Better than nothing.

# Prepend to your backup script
trap 'curl -fsS -m 10 --retry 5 \
  https://hc-ping.com/YOUR-UUID/fail' ERR
curl -fsS -m 10 --retry 5 \
  https://hc-ping.com/YOUR-UUID/start

What NOT to Do

• Don't store backups on the same disk array as production data
• Don't rely on RAID as a backup (it's availability, not durability, not integrity)
• Don't assume Docker volumes alone are sufficient — volume backups capture corrupt states; always pair with application-level dumps
• Don't use unencrypted backups (even "private" cloud storage)
• Don't skip restore testing — the first time you try to restore should not be during an actual disaster
• Don't forget to backup your backup config (restic repo keys, pgBackRest configuration, vaultwarden secrets)

### Cross-Links

• Production Observability Stack — infrastructure visibility for self-hosted services
• Grafana Dashboards — building effective monitoring dashboards
• Container Hardening Guide — secure container images reduce disaster surface area
• MinIO S3 Storage — backup target for the off-site layer
• Database Foundation — PostgreSQL point-in-time recovery setup
• Docker Multistage Builds — CI/CD pipeline integration
• SSL/TLS with Traefik — certificate recovery during DR

Summary

Self-hosting means accepting responsibility. A proper DR plan converts a "drop everything and panic" event into a "follow the runbook" procedure. That's the difference between operators and victims.