k3s Disaster Recovery Testing and Edge Patterns

Disaster recovery for k3s isn't just about backups — it's about knowing exactly how long recovery will take and testing that process before you need it. This article covers k3s-specific DR testing procedures, edge computing patterns, and tools that simplify automated backups and recovery.

Scheduled Snapshots with etcd-snapshot-schedule-cron

k3s includes the etcd-snapshot-schedule-cron command for automated, periodic backups of embedded etcd.

Basic Cron-Based Scheduling

Schedule backups every 6 hours with retention of 7 days:

# k3s server config
sudo nano /etc/rancher/k3s/config.yaml

# Add or modify:
etcd-snapshot-schedule-cron: "0 */6 * * *"
etcd-snapshot-retention: 168  # Keep snapshots for 168 hours (7 days)

Restart k3s to apply:

sudo systemctl restart k3s

Verify Scheduled Snapshots

Check that snapshots are being created:

# List all snapshots
sudo k3s etcd-snapshot ls

# Example output:
# /var/lib/rancher/k3s/server/db/snapshots/etcd-snapshot-2026-07-11_06-00-00.db
# /var/lib/rancher/k3s/server/db/snapshots/etcd-snapshot-2026-07-11_12-00-00.db
# /var/lib/rancher/k3s/server/db/snapshots/etcd-snapshot-2026-07-11_18-00-00.db

Advanced Scheduling: Multiple Cron Expressions

For different backup frequencies (hourly for recent, daily for longer retention):

etcd-snapshot-schedule-cron: |
  - "0 * * * *"    # Every hour
  - "0 0 * * *"    # Every day at midnight
etcd-snapshot-retention:
  hourly: 24 # Keep hourly snapshots for 24 hours
  daily: 30 # Keep daily snapshots for 30 days

Note: This advanced syntax may require k3s v1.30+.

S3 Offload for Remote Storage

Configure automatic upload to S3-compatible storage:

# k3s server config
etcd-snapshot-name: "k3s-server-01"
etcd-snapshot-s3: true
etcd-snapshot-s3-endpoint: "s3.amazonaws.com"
etcd-snapshot-s3-bucket: "k3s-backups"
etcd-snapshot-s3-region: "us-east-1"
etcd-snapshot-s3-skip-ssl-verify: false

Set AWS credentials:

export AWS_ACCESS_KEY_ID="your-access-key"
export AWS_SECRET_ACCESS_KEY="your-secret-key"
export AWS_DEFAULT_REGION="us-east-1"

Restart k3s:

sudo systemctl restart k3s

Verify S3 upload:

aws s3 ls s3://k3s-backups/

# Example output:
# 2026-07-11 12:00:00    2097152 k3s-server-01-2026-07-11_12-00-00.db
# 2026-07-11 18:00:00    2097152 k3s-server-01-2026-07-11_18-00-00.db

Failover Testing: Embedded etcd

Testing embedded etcd failover validates that your k3s cluster can recover from a complete control plane failure.

Test 1: Control Plane Recovery from Backup

Objective: Restore k3s server from an etcd snapshot and verify cluster state.

Prerequisites:

• Working k3s server with embedded etcd
• Valid etcd snapshot (via k3s etcd-snapshot ls)

Procedure:

1. Create test scenario — Simulate data change:

# Create test namespace and deployment
kubectl create namespace dr-test
kubectl create deployment nginx-test --image=nginx -n dr-test
kubectl get pods -n dr-test

# Take snapshot before failure
sudo k3s etcd-snapshot save --name "pre-failure-test"

2. Simulate failure — Stop k3s and corrupt etcd:

# Stop k3s
sudo systemctl stop k3s

# Backup corrupted etcd (for recovery attempt)
sudo cp /var/lib/rancher/k3s/server/db/etcd /var/lib/rancher/k3s/server/db/etcd.corrupted

# Corrupt etcd data
sudo rm /var/lib/rancher/k3s/server/db/etcd

3. Restore from snapshot:

# Restore from snapshot
sudo k3s server \
  --cluster-reset \
  --cluster-reset-restore-path=/var/lib/rancher/k3s/server/db/snapshots/pre-failure-test.db

# Start k3s
sudo systemctl start k3s

# Verify cluster health
sudo k3s kubectl get nodes
sudo k3s kubectl get pods -n dr-test

4. Expected outcome:

• k3s starts successfully
• Nodes report Ready
• nginx-test pod exists and running
• All k3s components (traefik, metrics-server) operational

5. Recovery time measurement:

# Time from failure start to restore completion
# Typical RTO: 2-5 minutes for embedded etcd on small clusters

Test 2: Multi-Server HA Failover

Objective: Test etcd HA failover in a 3-server k3s cluster.

Prerequisites:

• 3-node k3s cluster with embedded etcd HA
• All servers healthy and in sync

Procedure:

1. Verify current cluster state:

# Check all nodes
kubectl get nodes -o wide

# Check etcd members
sudo k3s etcd-snapshot status

# Verify k3s cluster-ID consistency across servers
sudo k3s kubectl get configmap -n kube-system cluster-config -o yaml

2. Simulate failure of server 1:

# On server 1
sudo systemctl stop k3s

# On server 2 or 3 (surviving server)
kubectl get nodes
# Expected: server1 shows NotReady, server2 and server3 Ready

# Verify etcd quorum
sudo k3s etcd-snapshot status
# Expected: 2/3 members healthy (quorum maintained)

3. Test cluster operations during failover:

# Create test resource
kubectl create namespace failover-test
kubectl run test-pod --image=nginx -n failover-test

# Verify pod scheduled
kubectl get pods -n failover-test -o wide
# Expected: Pod scheduled on surviving node

4. Restore server 1:

# On server 1
sudo systemctl start k3s

# Verify cluster re-joins
kubectl get nodes
# Expected: All 3 nodes Ready

# Verify etcd membership
sudo k3s etcd-snapshot status
# Expected: 3/3 members healthy

5. Failover time measurement:

# Time from server1 stop to cluster quorum maintained
# Typical RTO: 30-90 seconds for embedded etcd HA

Failover Testing: External PostgreSQL

For k3s clusters using external PostgreSQL instead of embedded etcd.

Test 1: PostgreSQL Primary Failure

Objective: Test k3s behavior when PostgreSQL primary fails.

Prerequisites:

• k3s cluster configured with external PostgreSQL
• PostgreSQL HA (Patroni or similar) configured
• Valid backup of PostgreSQL

Procedure:

1. Verify current state:

# Check k3s cluster health
kubectl get nodes
sudo k3s kubectl get pods -A

# Check PostgreSQL cluster status
patronictl -c /etc/patroni/patroni.yml list

2. Simulate primary failure:

# Stop PostgreSQL primary
sudo systemctl stop postgresql

# On PostgreSQL replica (now primary)
# Verify automatic failover occurred
patronictl -c /etc/patroni/patroni.yml list

# Expected: New primary elected, replica promoted

3. Test k3s operations during failover:

# Create test deployment
kubectl create deployment test-deploy --image=nginx

# Verify deployment succeeds
kubectl get pods
# Expected: Pods scheduled successfully

4. Recovery time measurement:

# Time from PostgreSQL primary stop to k3s operations resume
# Typical RTO: 1-3 minutes (depends on Patroni failover time)

Test 2: Full PostgreSQL Recovery from Backup

Objective: Restore PostgreSQL from backup and recover k3s cluster.

Prerequisites:

• PostgreSQL backup (pg_dump or physical backup)
• k3s cluster down

Procedure:

# Stop k3s servers
sudo systemctl stop k3s

# Drop and recreate PostgreSQL database
sudo -u postgres psql -c "DROP DATABASE IF EXISTS k3s;"
sudo -u postgres psql -c "CREATE DATABASE k3s;"

# Restore from backup
sudo -u postgres psql k3s < /backup/k3s-postgres-backup.sql

# Start k3s servers
sudo systemctl start k3s

# Verify cluster recovery
kubectl get nodes
kubectl get pods -A

Edge Computing DR Patterns

Edge computing introduces unique DR challenges: intermittent connectivity, limited bandwidth, and physically distributed sites.

Multi-Cluster DR with Rancher Fleet

Rancher Fleet manages multiple k3s clusters centrally. DR patterns differ from single-cluster scenarios.

Pattern 1: Centralized Backup Repository

All edge clusters push backups to a central repository for centralized recovery management.

Architecture:

Edge Site 1 (k3s) ----\
Edge Site 2 (k3s) ----+--> S3 Central Repository ----> Recovery Workstation
Edge Site 3 (k3s) ----/                          |
                                              Backup Validation

Implementation:

# Fleet cluster template for edge sites
apiVersion: v1
kind: ConfigMap
metadata:
  name: k3s-backup-config
  namespace: cattle-system
data:
  config.yaml: |
    etcd-snapshot-schedule-cron: "0 */4 * * *"
    etcd-snapshot-retention: 96  # 4 days
    etcd-snapshot-s3: true
    etcd-snapshot-s3-endpoint: "s3.eu-central-1.amazonaws.com"
    etcd-snapshot-s3-bucket: "edge-k3s-backups"
    etcd-snapshot-s3-region: "eu-central-1"

Deployment via Fleet:

# Apply to all edge clusters
fleetctl apply -f fleet-cluster-template.yaml

# Verify backup across all sites
aws s3 ls s3://edge-k3s-backups/ --recursive

# Example output:
# 2026-07-11 08:00:00    2097152 edge-site-01/k3s-etcd-snapshot.db
# 2026-07-11 08:00:00    2097152 edge-site-02/k3s-etcd-snapshot.db
# 2026-07-11 12:00:00    2097152 edge-site-01/k3s-etcd-snapshot.db
# 2026-07-11 12:00:00    2097152 edge-site-02/k3s-etcd-snapshot.db

Pattern 2: Local-First Backup with Periodic Sync

For edge sites with intermittent connectivity, local backups with periodic sync ensures durability.

Implementation:

#!/bin/bash
# /usr/local/bin/edge-backup-sync.sh

# 1. Take local etcd snapshot
sudo k3s etcd-snapshot save --name "$(date +%Y-%m-%d_%H-%M-%S)"

# 2. Check connectivity to central repository
if ping -c 1 central-repo.example.com &> /dev/null; then
  # 3. Upload recent snapshots
  for snapshot in /var/lib/rancher/k3s/server/db/snapshots/*.db; do
    aws s3 cp "$snapshot" "s3://edge-k3s-backups/$(hostname)/" --only-newer
  done
else
  logger "Central repository unreachable, retaining local backups"
fi

# 4. Prune old local snapshots (keep 24 hours)
find /var/lib/rancher/k3s/server/db/snapshots/ -name "*.db" -mtime +1 -delete

Add to cron:

# crontab -e
# Every 2 hours
0 */2 * * * /usr/local/bin/edge-backup-sync.sh

RPO/RTO Planning for Edge Scenarios

Edge computing requires careful RPO (Recovery Point Objective) and RTO (Recovery Time Objective) planning due to connectivity constraints.

Scenario	Network	RPO	RTO	Strategy
Urban edge	Fiber 1Gbps	15 min	30 min	Frequent snapshots + S3 sync
Rural edge	LTE 10Mbps	2 hours	4 hours	Local backup + daily sync
Offline edge	No connectivity	24 hours	Manual	Local backup + on-site recovery kit
Satellite edge	Satellite 2Mbps	4 hours	8 hours	Compressed backups + scheduled sync

Site-Level Disaster Recovery

Complete site recovery from scratch (hardware failure, natural disaster).

Recovery Kit Preparation:

# /opt/dr-recovery-kit/README.md

# Recovery Kit Contents:
# 1. Latest k3s binary and config
# 2. etcd-snapshot from S3
# 3. Persistent volume backups (if any)
# 4. Network configuration scripts
# 5. DR runbook with step-by-step procedures

# Recovery Procedure:
# 1. Provision new hardware or VM
# 2. Install k3s from recovery kit
# 3. Restore etcd from snapshot
# 4. Restore persistent volumes (if any)
# 5. Verify cluster health
# 6. Test application connectivity

Automated Recovery Script:

#!/bin/bash
# /opt/dr-recovery-kit/recover-site.sh

set -e

SITE_ID="edge-site-01"
K3S_VERSION="v1.30.0+k3s1"
SNAPSHOT_NAME="k3s-latest.db"
S3_BUCKET="edge-k3s-backups"

echo "Starting site recovery for $SITE_ID..."

# 1. Download latest snapshot
aws s3 cp "s3://$S3_BUCKET/$SITE_ID/$SNAPSHOT_NAME" /tmp/

# 2. Install k3s
curl -sfL https://get.k3s.io | INSTALL_K3S_VERSION="$K3S_VERSION" sh -

# 3. Configure k3s for recovery
cat <<EOF | sudo tee /etc/rancher/k3s/config.yaml
cluster-reset: true
cluster-reset-restore-path: /tmp/$SNAPSHOT_NAME
EOF

# 4. Start k3s
sudo systemctl start k3s

# 5. Wait for k3s ready
until sudo k3s kubectl get nodes &> /dev/null; do
  echo "Waiting for k3s to start..."
  sleep 5
done

# 6. Verify cluster health
sudo k3s kubectl get nodes
sudo k3s kubectl get pods -A

echo "Site recovery complete!"

Restic Integration for k3s Persistent Volumes

etcd snapshots only cover cluster state. For application data in persistent volumes, use Restic for backup and recovery.

Restic Setup for k3s PVs

# Install restic
wget https://github.com/restic/restic/releases/download/v0.16.0/restic_0.16.0_linux_amd64.bz2
bunzip2 restic_0.16.0_linux_amd64.bz2
sudo mv restic_0.16.0_linux_amd64 /usr/local/bin/restic
sudo chmod +x /usr/local/bin/restic

# Initialize restic repository (S3)
export RESTIC_REPOSITORY="s3:s3.amazonaws.com/k3s-pv-backups"
export RESTIC_PASSWORD="your-secure-password"

restic init

# Exclude k3s system directories (no need to backup)
cat <<EOF > /tmp/restic-excludes.txt
/var/lib/rancher/k3s/server/db/etcd
/var/lib/rancher/k3s/agent/containerd/io.containerd.content.v1.content
/var/lib/rancher/k3s/agent/containerd/io.containerd.snapshotter.v1.overlayfs
EOF

# Backup k3s persistent volumes
restic backup /var/lib/rancher/k3s/agent/pv --exclude-file=/tmp/restic-excludes.txt --tag "pv-backup"

Automated PV Backup Script

#!/bin/bash
# /usr/local/bin/k3s-pv-backup.sh

set -e

export RESTIC_REPOSITORY="s3:s3.amazonaws.com/k3s-pv-backups"
export RESTIC_PASSWORD="your-secure-password"

# Backup only mounted PVs
for mount in $(findmnt -l | grep /var/lib/rancher/k3s/agent/pv | awk '{print $1}'); do
  pod_name=$(echo "$mount" | cut -d/ -f7)
  pv_name=$(echo "$mount" | cut -d/ -f6)

  echo "Backing up PV: $pv_name (pod: $pod_name)"

  restic backup "$mount" \
    --exclude-file=/tmp/restic-excludes.txt \
    --tag "pv-backup" \
    --tag "pv:$pv_name" \
    --tag "pod:$pod_name"
done

# Prune old backups (keep 30 days)
restic forget --keep-daily 30 --prune

# Check repository integrity
restic check

Schedule daily backups:

# crontab -e
0 2 * * * /usr/local/bin/k3s-pv-backup.sh >> /var/log/k3s-pv-backup.log 2>&1

Restoring PVs from Restic Backup

# Find backup snapshot
export RESTIC_REPOSITORY="s3:s3.amazonaws.com/k3s-pv-backups"
export RESTIC_PASSWORD="your-secure-password"

# List snapshots
restic snapshots --tag "pv:my-app-pv"

# Restore specific PV
restic restore latest --tag "pv:my-app-pv" --target /var/lib/rancher/k3s/agent/pv/my-app-pv

# Restart affected pods to pick up restored data
kubectl delete pod -l app=my-app --force --grace-period=0

DR Validation Checklist

Before declaring your DR strategy production-ready, validate each component.

Embedded etcd Validation

• Scheduled snapshots configured and verified
• S3 offload tested and verified
• Snapshot retention policy configured
• Control plane recovery tested from snapshot
• Recovery time measured and documented
• Snapshot restoration tested on fresh k3s installation

Embedded etcd HA Validation

• 3-server cluster operational
• Quorum failover tested (stop 1 server)
• Split-brain prevention verified
• Cluster re-join tested (restore stopped server)
• Failover time measured (target: < 90 seconds)

External PostgreSQL Validation

• PostgreSQL HA (Patroni) configured
• Primary failover tested
• k3s operations during failover verified
• Backup from PostgreSQL tested
• Recovery from backup tested

Edge Computing Validation

• Multi-cluster backup repository configured
• Local-first backup with periodic sync tested
• RPO/RTO documented for all edge scenarios
• Site-level recovery kit prepared
• Automated site recovery tested in sandbox

Persistent Volume Validation

• Restic repository initialized
• PV backup schedule configured
• Backup retention policy set
• PV restore tested
• Restic integrity checks scheduled

Integration Testing

• Full DR scenario tested (etcd + PV + PostgreSQL)
• End-to-end recovery measured
• DR runbook documented and reviewed
• DR drills scheduled (quarterly recommended)

DR Runbook Template

Standardize your DR procedures with a runbook template.

# k3s Disaster Recovery Runbook

## 1. Incident Detection

**Triggers**:

- k3s server down
- etcd corruption detected
- PostgreSQL primary unavailable
- Complete site outage

**Verification**:

```bash
sudo systemctl status k3s
sudo k3s kubectl get nodes
sudo k3s etcd-snapshot status
```

2. Impact Assessment

Business Impact:

• Affected applications: [list]
• Users affected: [count]
• Estimated downtime: [RTO]

Technical Impact:

• Cluster size: [nodes/pods]
• Data at risk: [etcd / PV / PostgreSQL]
• Recovery source: [snapshot / backup / from scratch]

3. Recovery Execution

Option A: etcd Snapshot Recovery

RTO: 2-5 minutes 适用场景: Embedded etcd, recent snapshot available

Steps:

1. [Stop k3s: sudo systemctl stop k3s]
2. [Restore from snapshot: sudo k3s server --cluster-reset --cluster-reset-restore-path=/path/to/snapshot.db]
3. [Start k3s: sudo systemctl start k3s]
4. [Verify recovery: kubectl get nodes]

Option B: PostgreSQL Recovery

RTO: 5-10 minutes 适用场景: External PostgreSQL, Patroni failover failed

Steps:

1. [Stop k3s servers: sudo systemctl stop k3s]
2. [Restore PostgreSQL: psql k3s < /backup/k3s-backup.sql]
3. [Start k3s servers: sudo systemctl start k3s]
4. [Verify cluster: kubectl get pods -A]

Option C: Full Site Recovery

RTO: 30-60 minutes 适用场景: Complete site failure, hardware replacement required

Steps:

1. [Provision new hardware]
2. [Run recovery script: /opt/dr-recovery-kit/recover-site.sh]
3. [Restore PVs from Restic]
4. [Verify application connectivity]

4. Post-Recovery Validation

Validation Steps:

• All nodes Ready
• All pods Running
• Application connectivity verified
• Data integrity checked
• Performance baseline verified

5. Root Cause Analysis

Incident Timeline:

Root Cause:

• [Description]

Preventive Measures:

• [Action items]

6. Communication

Stakeholders Notified:

• Engineering team
• Product team
• Support team
• Customers (if customer-facing)

Communication Channels:

• Slack: #dr-incidents
• Email: dr-notifications@example.com
• Status page: https://status.example.com

7. Lessons Learned

What went well:

• [Observations]

What could be improved:

• [Action items]

DR runbook updates:

• [Documentation changes]

## Further Reading

- [k3s in Production: Upgrades, Backup, and Disaster Recovery](/content/devops/k3s-production-upgrades-backup-dr/) — Comprehensive production patterns for k3s
- [Disaster Recovery Self-Hosted](/content/devops/disaster-recovery-self-hosted/) — Generic DR patterns for self-hosted infrastructure
- [Edge Computing Patterns with k3s](/content/devops/edge-computing-patterns-k3s/) — Multi-cluster management with Rancher Fleet

---

Regular DR testing is the difference between a minor outage and a catastrophic failure. Schedule quarterly DR drills, measure your actual RTO, and update your runbook with lessons learned.