k3s Disaster Recovery Testing and Edge Patterns

k3s in der Produktion: Upgrades, Backup, Disaster Recovery und bekannte Probleme

Produktionshärtung für k3s-Deployments, einschließlich Upgrade-Strategien (yum/apt, Blue-Green, system-upgrade-controller), Backup und Restore (SQLite vs. etcd), Disaster-Recovery-Planung sowie bekannte Probleme bei einer Skalierung.

Disaster recovery for k3s isn't just about backups — it's about knowing exactly how long recovery will take and testing that process before you need it. This article covers k3s-specific DR testing procedures, edge computing patterns, and tools that simplify automated backups and recovery.

Scheduled Snapshots with etcd-snapshot-schedule-cron

k3s includes the etcd-snapshot-schedule-cron command for automated, periodic backups of embedded etcd.

Basic Cron-Based Scheduling

Schedule backups every 6 hours with retention of 7 days:

sudo nano /etc/rancher/k3s/config.yaml

# Add or modify:
etcd-snapshot-schedule-cron: "0 */6 * * *"
etcd-snapshot-retention: 168  # Keep snapshots for 168 hours (7 days)

Restart k3s to apply:

sudo systemctl restart k3s

Verify Scheduled Snapshots

Check that snapshots are being created:

# List all snapshots
sudo k3s etcd-snapshot ls

# Example output:
# /var/lib/rancher/k3s/server/db/snapshots/etcd-snapshot-2026-07-11_06-00-00.db
# /var/lib/rancher/k3s/server/db/snapshots/etcd-snapshot-2026-07-11_12-00-00.db
# /var/lib/rancher/k3s/server/db/snapshots/etcd-snapshot-2026-07-11_18-00-00.db

Advanced Scheduling: Multiple Cron Expressions

For different backup frequencies (hourly for recent, daily for longer retention):

etcd-snapshot-schedule-cron: |
  - "0 * * * *"    # Every hour
  - "0 0 * * *"    # Every day at midnight
etcd-snapshot-retention:
  hourly: 24 # Keep hourly snapshots for 24 hours
  daily: 30 # Keep daily snapshots for 30 days

Note: This advanced syntax may require k3s v1.30+.

S3 Offload for Remote Storage

Configure automatic upload to S3-compatible storage:

# k3s server config
etcd-snapshot-name: "k3s-server-01"
etcd-snapshot-s3: true
etcd-snapshot-s3-endpoint: "s3.amazonaws.com"
etcd-snapshot-s3-bucket: "k3s-backups"
etcd-snapshot-s3-region: "us-east-1"
etcd-snapshot-s3-skip-ssl-verify: false

Set AWS credentials:

export AWS_ACCESS_KEY_ID="your-access-key"
export AWS_SECRET_ACCESS_KEY="your-secret-key"
export AWS_DEFAULT_REGION="us-east-1"

Restart k3s:

sudo systemctl restart k3s

Verify S3 upload:

aws s3 ls s3://k3s-backups/

# Example output:
# 2026-07-11 12:00:00    2097152 k3s-server-01-2026-07-11_12-00-00.db
# 2026-07-11 18:00:00    2097152 k3s-server-01-2026-07-11_18-00-00.db

Security note: etcd snapshots contain cluster secrets (service accounts, tokens, certificates). Enable S3 server-side encryption on the bucket (SSE-S3 or SSE-KMS) and scope a dedicated IAM user to only s3:PutObject/s3:GetObject on the backup prefix. A public backup bucket is a breach waiting to happen.

Failover Testing: Embedded etcd

Testing embedded etcd failover validates that your k3s cluster can recover from a complete control plane failure.

Test 1: Control Plane Recovery from Backup

Objective: Restore k3s server from an etcd snapshot and verify cluster state.

Prerequisites:

• Working k3s server with embedded etcd
• Valid etcd snapshot (via k3s etcd-snapshot ls)

Procedure:

1. Create test scenario — Simulate data change:

# Create test namespace and deployment
kubectl create namespace dr-test
kubectl create deployment nginx-test --image=nginx -n dr-test
kubectl get pods -n dr-test

# Take snapshot before failure
sudo k3s etcd-snapshot save --name "pre-failure-test"

2. Simulate failure — Stop k3s and corrupt etcd:

# Stop k3s
sudo systemctl stop k3s

# Backup corrupted etcd (for recovery attempt)
sudo cp /var/lib/rancher/k3s/server/db/etcd /var/lib/rancher/k3s/server/db/etcd.corrupted

# Corrupt etcd data
sudo rm /var/lib/rancher/k3s/server/db/etcd

3. Restore from snapshot:

# Restore from snapshot
sudo k3s server \
  --cluster-reset \
  --cluster-reset-restore-path=/var/lib/rancher/k3s/server/db/snapshots/pre-failure-test.db

# Start k3s
sudo systemctl start k3s

# Verify cluster health
sudo k3s kubectl get nodes
sudo k3s kubectl get pods -n dr-test

4. Expected outcome:

• k3s starts successfully
• Nodes report Ready
• nginx-test pod exists and running
• All k3s components (traefik, metrics-server) operational

5. Recovery time measurement:

# Time from failure start to restore completion
# Typical RTO: 2-5 minutes for embedded etcd on small clusters

Test 2: Multi-Server HA Failover

Objective: Test etcd HA failover in a 3-server k3s cluster.

Prerequisites:

• 3-node k3s cluster with embedded etcd HA
• All servers healthy and in sync

Procedure:

1. Verify current cluster state:

# Check all nodes
kubectl get nodes -o wide

# Check etcd members
sudo k3s etcd-snapshot status

# Verify k3s cluster-ID consistency across servers
sudo k3s kubectl get configmap -n kube-system cluster-config -o yaml

2. Simulate failure of server 1:

# On server 1
sudo systemctl stop k3s

# On server 2 or 3 (surviving server)
kubectl get nodes
# Expected: server1 shows NotReady, server2 and server3 Ready

# Verify etcd quorum
sudo k3s etcd-snapshot status
# Expected: 2/3 members healthy (quorum maintained)

3. Test cluster operations during failover:

# Create test resource
kubectl create namespace failover-test
kubectl run test-pod --image=nginx -n failover-test

# Verify pod scheduled
kubectl get pods -n failover-test -o wide
# Expected: Pod scheduled on surviving node

4. Restore server 1:

# On server 1
sudo systemctl start k3s

# Verify cluster re-joins
kubectl get nodes
# Expected: All 3 nodes Ready

# Verify etcd membership
sudo k3s etcd-snapshot status
# Expected: 3/3 members healthy

5. Failover time measurement:

# Time from server1 stop to cluster quorum maintained
# Typical RTO: 30-90 seconds for embedded etcd HA

Failover Testing: External PostgreSQL

For k3s clusters using external PostgreSQL instead of embedded etcd.

Test 1: PostgreSQL Primary Failure

Objective: Test k3s behavior when PostgreSQL primary fails.

Prerequisites:

• k3s cluster configured with external PostgreSQL
• PostgreSQL HA (Patroni or similar) configured
• Valid backup of PostgreSQL

Procedure:

1. Verify current state:

# Check k3s cluster health
kubectl get nodes
sudo k3s kubectl get pods -A

# Check PostgreSQL cluster status
patronictl -c /etc/patroni/patroni.yml list

2. Simulate primary failure:

# Stop PostgreSQL primary
sudo systemctl stop postgresql

# On PostgreSQL replica (now primary)
# Verify automatic failover occurred
patronictl -c /etc/patroni/patroni.yml list

# Expected: New primary elected, replica promoted

3. Test k3s operations during failover:

# Create test deployment
kubectl create deployment test-deploy --image=nginx

# Verify deployment succeeds
kubectl get pods
# Expected: Pods scheduled successfully

4. Recovery time measurement:

# Time from PostgreSQL primary stop to k3s operations resume
# Typical RTO: 1-3 minutes (depends on Patroni failover time)

Test 2: Full PostgreSQL Recovery from Backup

Objective: Restore PostgreSQL from backup and recover k3s cluster.

Prerequisites:

• PostgreSQL backup (pg_dump or physical backup)
• k3s cluster down

Procedure:

# Stop k3s servers
sudo systemctl stop k3s

# Drop and recreate PostgreSQL database
sudo -u postgres psql -c "DROP DATABASE IF EXISTS k3s;"
sudo -u postgres psql -c "CREATE DATABASE k3s;"

# Restore from backup
sudo -u postgres psql k3s < /backup/k3s-postgres-backup.sql

# Start k3s servers
sudo systemctl start k3s

# Verify cluster recovery
kubectl get nodes
kubectl get pods -A

Edge Computing DR Patterns

Edge computing introduces unique DR challenges: intermittent connectivity, limited bandwidth, and physically distributed sites.

Multi-Cluster DR with Rancher Fleet

Rancher Fleet manages multiple k3s clusters centrally. DR patterns differ from single-cluster scenarios.

Pattern 1: Centralized Backup Repository

All edge clusters push backups to a central repository for centralized recovery management.

Architecture:

Edge Site 1 (k3s) ----\
Edge Site 2 (k3s) ----+--> S3 Central Repository ----> Recovery Workstation
Edge Site 3 (k3s) ----/                          |
                                              Backup Validation

Implementation:

# Fleet cluster template for edge sites
apiVersion: v1
kind: ConfigMap
metadata:
  name: k3s-backup-config
  namespace: cattle-system
data:
  config.yaml: |
    etcd-snapshot-schedule-cron: "0 */4 * * *"
    etcd-snapshot-retention: 96  # 4 days
    etcd-snapshot-s3: true
    etcd-snapshot-s3-endpoint: "s3.eu-central-1.amazonaws.com"
    etcd-snapshot-s3-bucket: "edge-k3s-backups"
    etcd-snapshot-s3-region: "eu-central-1"

Deployment via Fleet:

# Apply to all edge clusters
fleetctl apply -f fleet-cluster-template.yaml

# Verify backup across all sites
aws s3 ls s3://edge-k3s-backups/ --recursive

# Example output:
# 2026-07-11 08:00:00    2097152 edge-site-01/k3s-etcd-snapshot.db
# 2026-07-11 08:00:00    2097152 edge-site-02/k3s-etcd-snapshot.db
# 2026-07-11 12:00:00    2097152 edge-site-01/k3s-etcd-snapshot.db
# 2026-07-11 12:00:00    2097152 edge-site-02/k3s-etcd-snapshot.db

Pattern 2: Local-First Backup with Periodic Sync

For edge sites with intermittent connectivity, local backups with periodic sync ensures durability.

Implementation:

#!/bin/bash
# /usr/local/bin/edge-backup-sync.sh

# 1. Take local etcd snapshot
sudo k3s etcd-snapshot save --name "$(date +%Y-%m-%d_%H-%M-%S)"

# 2. Check connectivity to central repository
if ping -c 1 central-repo.example.com &> /dev/null; then
  # 3. Upload recent snapshots
  for snapshot in /var/lib/rancher/k3s/server/db/snapshots/*.db; do
    aws s3 cp "$snapshot" "s3://edge-k3s-backups/$(hostname)/" --only-newer
  done
else
  logger "Central repository unreachable, retaining local backups"
fi

# 4. Prune old local snapshots (keep 24 hours)
find /var/lib/rancher/k3s/server/db/snapshots/ -name "*.db" -mtime +1 -delete

Add to cron:

# crontab -e
# Every 2 hours
0 */2 * * * /usr/local/bin/edge-backup-sync.sh

RPO/RTO Planning for Edge Scenarios

Edge computing requires careful RPO (Recovery Point Objective) and RTO (Recovery Time Objective) planning due to connectivity constraints. Derive the targets from business numbers, not defaults: estimate the cost of an hour of downtime, compare it with the cost of more frequent snapshots, and pick targets you can actually meet and measure. As a rule of thumb, 99.9% availability allows ~8.8 hours of downtime per year; 99.99% allows ~53 minutes. The scenarios below assume targets are measured, not aspirational.

Site-Level Disaster Recovery

Complete site recovery from scratch (hardware failure, natural disaster).

Recovery Kit Preparation:

# /opt/dr-recovery-kit/README.md

# Recovery Kit Contents:
# 1. Latest k3s binary and config
# 2. etcd-snapshot from S3
# 3. Persistent volume backups (if any)
# 4. Network configuration scripts
# 5. DR runbook with step-by-step procedures

# Recovery Procedure:
# 1. Provision new hardware or VM
# 2. Install k3s from recovery kit
# 3. Restore etcd from snapshot
# 4. Restore persistent volumes (if any)
# 5. Verify cluster health
# 6. Test application connectivity

Automated Recovery Script:

#!/bin/bash
# /opt/dr-recovery-kit/recover-site.sh

set -e

SITE_ID="edge-site-01"
K3S_VERSION="v1.30.0+k3s1"
SNAPSHOT_NAME="k3s-latest.db"
S3_BUCKET="edge-k3s-backups"

echo "Starting site recovery for $SITE_ID..."

# 1. Download latest snapshot
aws s3 cp "s3://$S3_BUCKET/$SITE_ID/$SNAPSHOT_NAME" /tmp/

# 2. Install k3s
curl -sfL https://get.k3s.io | INSTALL_K3S_VERSION="$K3S_VERSION" sh -

# 3. Configure k3s for recovery
cat <<EOF | sudo tee /etc/rancher/k3s/config.yaml
cluster-reset: true
cluster-reset-restore-path: /tmp/$SNAPSHOT_NAME
EOF

# 4. Start k3s
sudo systemctl start k3s

# 5. Wait for k3s ready
until sudo k3s kubectl get nodes &> /dev/null; do
  echo "Waiting for k3s to start..."
  sleep 5
done

# 6. Verify cluster health
sudo k3s kubectl get nodes
sudo k3s kubectl get pods -A

echo "Site recovery complete!"

Restic Integration for k3s Persistent Volumes

etcd snapshots only cover cluster state. For application data in persistent volumes, use Restic for backup and recovery.

Restic Setup for k3s PVs

# Install restic
wget https://github.com/restic/restic/releases/download/v0.16.0/restic_0.16.0_linux_amd64.bz2
bunzip2 restic_0.16.0_linux_amd64.bz2
sudo mv restic_0.16.0_linux_amd64 /usr/local/bin/restic
sudo chmod +x /usr/local/bin/restic

# Initialize restic repository (S3)
export RESTIC_REPOSITORY="s3:s3.amazonaws.com/k3s-pv-backups"
export RESTIC_PASSWORD="your-secure-password"

restic init

# Exclude k3s system directories (no need to backup)
cat <<EOF > /tmp/restic-excludes.txt
/var/lib/rancher/k3s/server/db/etcd
/var/lib/rancher/k3s/agent/containerd/io.containerd.content.v1.content
/var/lib/rancher/k3s/agent/containerd/io.containerd.snapshotter.v1.overlayfs
EOF

# Backup k3s persistent volumes
restic backup /var/lib/rancher/k3s/agent/pv --exclude-file=/tmp/restic-excludes.txt --tag "pv-backup"

Automated PV Backup Script

#!/bin/bash
# /usr/local/bin/k3s-pv-backup.sh

set -e

export RESTIC_REPOSITORY="s3:s3.amazonaws.com/k3s-pv-backups"
export RESTIC_PASSWORD="your-secure-password"

# Backup only mounted PVs
for mount in $(findmnt -l | grep /var/lib/rancher/k3s/agent/pv | awk '{print $1}'); do
  pod_name=$(echo "$mount" | cut -d/ -f7)
  pv_name=$(echo "$mount" | cut -d/ -f6)

  echo "Backing up PV: $pv_name (pod: $pod_name)"

  restic backup "$mount" \
    --exclude-file=/tmp/restic-excludes.txt \
    --tag "pv-backup" \
    --tag "pv:$pv_name" \
    --tag "pod:$pod_name"
done

# Prune old backups (keep 30 days)
restic forget --keep-daily 30 --prune

# Check repository integrity
restic check

Schedule daily backups:

# crontab -e
0 2 * * * /usr/local/bin/k3s-pv-backup.sh >> /var/log/k3s-pv-backup.log 2>&1

Restoring PVs from Restic Backup

# Find backup snapshot
export RESTIC_REPOSITORY="s3:s3.amazonaws.com/k3s-pv-backups"
export RESTIC_PASSWORD="your-secure-password"

# List snapshots
restic snapshots --tag "pv:my-app-pv"

# Restore specific PV
restic restore latest --tag "pv:my-app-pv" --target /var/lib/rancher/k3s/agent/pv/my-app-pv

# Restart affected pods to pick up restored data
kubectl delete pod -l app=my-app --force --grace-period=0

From Procedure to Proof: DR Drills as Chaos Engineering

The procedures above are only as good as their last execution. A backup you've never restored is not a backup. DR testing is chaos engineering for infrastructure: you deliberately inject failures (stop the wrong node, corrupt the wrong file, pull the wrong snapshot) and observe whether the system recovers. The same lesson that applies to software engines applies here — example-based testing finds what you thought of; fault injection finds what you didn't. A scheduled snapshot that is never restore-tested is exactly the "config that exists but isn't enforced" trap: it creates false confidence.

Drill Design: Seeded, Deliberate, Varied

• Seed your chaos — a DR drill must be reproducible: record which node failed, which snapshot was restored, what the measured RPO/RTO was. Non-reproducible drills teach nothing.
• Vary the victim — rotate which server stops, which backup you restore from (latest vs. oldest vs. S3-only), and which time of day the drill runs. The bug that hides is the one that only appears with an old snapshot on a cold morning.
• Include the human path — the recovery kit and the runbook are part of the system under test. Half the value of a drill is discovering the runbook is stale or the recovery kit is missing a step.

Measure, Don't Assume

"Typical RTO: 2-5 minutes" is a claim, not a fact. Measure actual recovery time in your environment, on your hardware, with your network:

• time every restore step and record it in the runbook
• Track the RTO/RPO trend across drills — a regression in recovery time is an early warning signal
• Benchmark-verify every performance claim; the "typical" numbers in this article are starting points, not guarantees

Idempotent Restores and Contract Preservation

A restore procedure must be idempotent: running it twice on the same host must not corrupt state. If a restore script assumes a clean slate but a previous restore left artifacts behind, that contract violation will surface at 3 a.m. during a real incident, not during the happy-path drill. Test the contract explicitly: run the restore, run it again, and assert identical cluster health both times.

Validation as a CI-Style Gate

The checklist below should be enforced by automation, not by memory:

• restic check runs nightly and alerts on failure (repository integrity)
• Snapshot freshness alert: fail if the newest etcd snapshot is older than the configured schedule
• S3 offload verification: a scheduled job lists the bucket and asserts the expected snapshot counts per site
• The site-recovery script runs against a sandbox on a schedule — not only when someone remembers to test it

The 3× Rule for DR Readiness

Before declaring DR strategy production-ready, run the full drill three times with three consecutive passes — including at least one restore to a fresh host (not the same VM you corrupted). Two passes can be luck; three is a pattern.

Restore-on-a-Schedule: Continuous Verifiability

Quarterly drills verify your procedures; a weekly scheduled restore verifies your snapshots. Set up a cron job that takes the newest etcd snapshot, provisions a throwaway k3s cluster (same version), restores into it, runs health checks, and destroys it again. If a snapshot is corrupt — discovered hours after the fact by kubectl get nodes failing inside the sandbox — you have a week of snapshots to fall back to, instead of a quarter of false confidence. The sandbox restore is also the perfect rehearsal ground for recover-site.sh: every week it exercises the exact script you will run during a real incident.

DR Validation Checklist

Before declaring your DR strategy production-ready, validate each component.

Embedded etcd Validation

• Scheduled snapshots configured and verified
• S3 offload tested and verified
• Snapshot retention policy configured
• Control plane recovery tested from snapshot
• Recovery time measured and documented
• Snapshot restoration tested on fresh k3s installation

Embedded etcd HA Validation

• 3-server cluster operational
• Quorum failover tested (stop 1 server)
• Split-brain prevention verified
• Cluster re-join tested (restore stopped server)
• Failover time measured (target: < 90 seconds)

External PostgreSQL Validation

• PostgreSQL HA (Patroni) configured
• Primary failover tested
• k3s operations during failover verified
• Backup from PostgreSQL tested
• Recovery from backup tested

Edge Computing Validation

• Multi-cluster backup repository configured
• Local-first backup with periodic sync tested
• RPO/RTO documented for all edge scenarios
• Site-level recovery kit prepared
• Automated site recovery tested in sandbox

Persistent Volume Validation

• Restic repository initialized
• PV backup schedule configured
• Backup retention policy set
• PV restore tested
• Restic integrity checks scheduled

Integration Testing

• Full DR scenario tested (etcd + PV + PostgreSQL)
• End-to-end recovery measured and trend-tracked across drills
• DR runbook documented and reviewed
• DR drills scheduled (quarterly recommended)
• Full drill passes 3× consecutively, including one restore to a fresh host
• Restore procedures proven idempotent (run twice, identical result)
• Automated gates live: restic check nightly, snapshot-freshness alert, S3 offload verification
• Weekly sandbox restore proves snapshots restorable (not just quarterly drills)
• S3 backup bucket encrypted (SSE) with least-privilege IAM

DR Runbook Template

Standardize your DR procedures with a runbook template.

# k3s Disaster Recovery Runbook

## 1. Incident Detection

**Triggers**:

- k3s server down
- etcd corruption detected
- PostgreSQL primary unavailable
- Complete site outage

**Verification**:

```bash
sudo systemctl status k3s
sudo k3s kubectl get nodes
sudo k3s etcd-snapshot status
```

2. Impact Assessment

Business Impact:

• Affected applications: [list]
• Users affected: [count]
• Estimated downtime: [RTO]

Technical Impact:

• Cluster size: [nodes/pods]
• Data at risk: [etcd / PV / PostgreSQL]
• Recovery source: [snapshot / backup / from scratch]

3. Recovery Execution

Option A: etcd Snapshot Recovery

RTO: 2-5 minutes 适用场景: Embedded etcd, recent snapshot available

Steps:

1. [Stop k3s: sudo systemctl stop k3s]
2. [Restore from snapshot: sudo k3s server --cluster-reset --cluster-reset-restore-path=/path/to/snapshot.db]
3. [Start k3s: sudo systemctl start k3s]
4. [Verify recovery: kubectl get nodes]

Option B: PostgreSQL Recovery

RTO: 5-10 minutes 适用场景: External PostgreSQL, Patroni failover failed

Steps:

1. [Stop k3s servers: sudo systemctl stop k3s]
2. [Restore PostgreSQL: psql k3s < /backup/k3s-backup.sql]
3. [Start k3s servers: sudo systemctl start k3s]
4. [Verify cluster: kubectl get pods -A]

Option C: Full Site Recovery

RTO: 30-60 minutes 适用场景: Complete site failure, hardware replacement required

Steps:

1. [Provision new hardware]
2. [Run recovery script: /opt/dr-recovery-kit/recover-site.sh]
3. [Restore PVs from Restic]
4. [Verify application connectivity]

4. Post-Recovery Validation

Validation Steps:

• All nodes Ready
• All pods Running
• Application connectivity verified
• Data integrity checked
• Performance baseline verified

5. Root Cause Analysis

Incident Timeline:

Root Cause:

• [Description]

Preventive Measures:

• [Action items]

6. Communication

Stakeholders Notified:

• Engineering team
• Product team
• Support team
• Customers (if customer-facing)

Communication Channels:

• Slack: #dr-incidents
• Email: dr-notifications@example.com
• Status page: https://status.example.com

7. Lessons Learned

What went well:

• [Observations]

What could be improved:

• [Action items]

DR runbook updates:

• [Documentation changes]

## Further Reading

- [k3s in Production: Upgrades, Backup, and Disaster Recovery](/content/devops/k3s-production-upgrades-backup-dr/) — Comprehensive production patterns for k3s
- [Disaster Recovery Self-Hosted](/content/devops/disaster-recovery-self-hosted/) — Generic DR patterns for self-hosted infrastructure
- [Edge Computing Patterns with k3s](/content/devops/edge-computing-patterns-k3s/) — Multi-cluster management with Rancher Fleet

---

Regular DR testing is the difference between a minor outage and a catastrophic failure. Schedule quarterly DR drills, measure your actual RTO, and update your runbook with lessons learned.