k3s in Production: Upgrades, Backup, Disaster Recovery, and Known Issues

The Open-Source Infrastructure Stack: Production-Grade PostgreSQL, Redis, MinIO, n8n, and Observability

Build a complete production infrastructure with open-source tools: PostgreSQL for data, Redis for caching, MinIO for S3-compatible storage, n8n for workflow automation, Prometheus and Grafana for observability - all behind a secure reverse proxy. All components work together in a cohesive architecture.

k3s is lightweight and fast — but running it in production requires the same operational discipline as full Kubernetes. Upgrades, backups, disaster recovery, and capacity planning become critical when edge sites become single points of failure for business operations.

This article covers production-grade operations for k3s: upgrade strategies, backup/restore for both SQLite and etcd backends, disaster recovery patterns, high-availability architecture, and known issues at scale.

Upgrade Strategies

k3s offers multiple upgrade paths depending on your deployment model and tolerance for downtime.

Method 1: yum/apt (Package Manager)

Best for: Stable sites with internet connectivity and external etcd/PostgreSQL datastores.

apt-cache madison k3s

# Pin to specific minor version for controlled rollouts
sudo apt install -y k3s=1.30.2+k3s1

# Or upgrade to latest patch in current minor
sudo apt install -y --only-upgrade k3s

Advantages:

• Automatic rollback via apt downgrade k3s (keeps previous binary)
• systemd handles service restart automatically
• Integrated with existing update management (patch management, security scanners)

Disadvantages:

• Requires internet connectivity (or internal apt mirror)
• No pre-download phase — upgrade happens when you run it
• Cannot test upgrade on a subset of nodes before full rollout

Method 2: Shell Script

Best for: Airgap environments, version pinning, or needing control over the upgrade process.

# Download new version
curl -sfL https://get.k3s.io | INSTALL_K3S_VERSION=v1.30.2+k3s1 sh -

# For airgap, download and install manually
wget https://github.com/k3s-io/k3s/releases/download/v1.30.2+k3s1/k3s
wget https://github.com/k3s-io/k3s/releases/download/v1.30.2+k3s1/k3s-airgap-images-amd64.tar.gz
sudo install k3s /usr/local/bin/
sudo systemctl restart k3s

Advantages:

• Works airgap with pre-downloaded binaries
• Can pre-stage images on edge sites
• Explicit version control

Disadvantages:

• No automatic rollback — must manually restore previous binary
• systemctl restart is immediate (no pre-drain of pods)

Method 3: Blue-Green Upgrades

Best for: Zero-downtime upgrades with full rollback capability.

Concept: Deploy a second k3s cluster on the same hardware, cutover traffic, verify, then retire the old cluster.

Implementation:

# On each node, install k3s-green alongside k3s-blue
sudo mkdir -p /etc/rancher/k3s-green
sudo cat > /etc/rancher/k3s-green/config.yaml <<'EOF'
token: green-cluster-token
datastore-endpoint: mysql://user:pass@mysql-green.example.com:3306/k3s
node-name: ${HOSTNAME}-green
server: https://10.0.1.10:6443
tls-san:
  - k3s-green.example.com
EOF

sudo curl -sfL https://get.k3s.io | INSTALL_K3S_NAME=k3s-green INSTALL_K3S_EXEC="server --config /etc/rancher/k3s-green/config.yaml" sh -

# Cutover traffic (example: update Traefik ingress routing)
kubectl apply -f - <<EOF
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: app-ingress
spec:
  rules:
  - host: app.example.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: app-service
            port:
              number: 80
EOF

# Verify green cluster is healthy
kubectl --kubeconfig /etc/rancher/k3s-green/k3s.yaml get nodes

# Decommission blue cluster after validation period
sudo systemctl stop k3s

Advantages:

• Zero downtime if both clusters run simultaneously
• Full rollback: traffic just switches back to old cluster
• Can validate new version under real load before cutover

Disadvantages:

• Doubles resource requirements during migration window
• Requires external datastore (MySQL/PostgreSQL) to share database
• More complex networking (ingress needs to route to correct cluster)

Method 4: system-upgrade-controller (Zero-Downtime for Single-Node)

Best for: Automated upgrades across many edge nodes with minimal disruption.

The system-upgrade-controller manages k3s upgrades via Plan and Upgrade resources:

apiVersion: upgrade.cattle.io/v1
kind: Plan
metadata:
  name: k3s-upgrade
  namespace: system-upgrade
spec:
  version: v1.30.2+k3s1
  concurrency: 1 # Upgrade one node at a time
  serviceAccountName: system-upgrade-controller

  # Drain pods before upgrade
  drain:
    enable: true
    force: true
    ignoreDaemonSets: true
    deleteLocalData: true

  # Upgrade strategy
  upgrade:
    image: rancher/k3s-upgrade:v1.30.2+k3s1

  # Target nodes
  nodeSelector:
    matchLabels:
      k3s.io/hostname: ${HOSTNAME}

Workflow:

1. Apply Plan resource — controller labels nodes for upgrade
2. Controller drains pods on labeled node
3. Node reboots (via kubelet grace period or explicit reboot command)
4. New k3s version starts on boot
5. Verification checks pass -> controller moves to next node

Advantages:

• Automated, consistent across hundreds of edge nodes
• Self-healing: controller retries on failures
• Can pause/resume mid-rollout

Disadvantages:

• Requires pod disruption during upgrade (short downtime)
• Requires local disk for new binary (needs free space)
• Can get stuck if node doesn't come back online

Backup and Restore

k3s supports two datastores with very different backup semantics: SQLite (default, HA via embedded etcd) and external databases (PostgreSQL/MySQL/etcd).

SQLite + Embedded etcd

k3s uses SQLite for control plane state and embedded etcd for HA cluster state.

Backup via k3s etcd-snapshot CLI (recommended for embedded etcd):

# Manual snapshot
sudo k3s etcd-snapshot save --name backup-$(date +%Y%m%d-%H%M%S)

# Scheduled snapshots (crontab)
0 2 * * * sudo k3s etcd-snapshot save --name daily-backup-\$(date +\%Y\%m\%d-\%H\%M\%S)

# S3 offload (k3s v1.30+)
sudo k3s etcd-snapshot save --s3 \
  --s3-endpoint s3.us-west-2.amazonaws.com \
  --s3-bucket my-k3s-backups \
  --s3-access-key AKIA... \
  --s3-secret-key secret... \
  --s3-region us-west-2

Restore:

# Stop k3s
sudo systemctl stop k3s

# Restore from snapshot
sudo k3s etcd-snapshot restore --name backup-20260711-020000

# Start k3s
sudo systemctl start k3s

# Verify
sudo k3s kubectl get nodes

File-level backup (emergency only):

# Stop k3s first (required for consistent copy)
sudo systemctl stop k3s

# Copy data directory
sudo tar czf k3s-backup-$(date +%Y%m%d).tar.gz /var/lib/rancher/k3s/

# Start k3s
sudo systemctl start k3s

Warning: File-level backups are brittle. They capture etcd data directories in their running state, which may not be portable across k3s versions. Use k3s etcd-snapshot whenever possible.

External Datastore (PostgreSQL/MySQL/etcd)

When k3s uses an external datastore, backup that datastore directly, not k3s.

PostgreSQL:

pg_dump -U k3s_user -h db.example.com k3s_db | gzip > k3s-pg-backup.sql.gz

# Point-in-time recovery (PITR)
pg_dump -U k3s_user -h db.example.com -F c -b k3s_db > k3s-pg-backup.dump

MySQL:

mysqldump -u k3s_user -p k3s_db | gzip > k3s-mysql-backup.sql.gz

External etcd:

# Snapshot (ETCDCTL_API=3 for etcd v3)
ETCDCTL_API=3 etcdctl \
  --endpoints https://etcd1.example.com:2379 \
  --cacert /etc/kubernetes/pki/etcd/ca.crt \
  --cert /etc/kubernetes/pki/etcd/server.crt \
  --key /etc/kubernetes/pki/etcd/server.key \
  snapshot save /backup/etcd-snapshot.db

# Restore
ETCDCTL_API=3 etcdctl snapshot restore /backup/etcd-snapshot.db \
  --data-dir /var/lib/etcd-restore

Restoring to New Clusters

For disaster recovery, you typically restore to new hardware or cloud VMs rather than the original failed node.

Embedded etcd to new cluster:

# On new node, install k3s with token pointing to restore data
sudo curl -sfL https://get.k3s.io | \
  INSTALL_K3S_EXEC="server --token my-cluster-token \
  --server https://10.0.1.10:6443 \
  --cluster-init" \
  sh -

# Then restore snapshot
sudo k3s etcd-snapshot restore --name backup-20260711-020000

External PostgreSQL to new cluster:

# On new k3s cluster, configure to use existing datastore
sudo cat > /etc/rancher/k3s/config.yaml <<'EOF'
token: restored-cluster-token
datastore-endpoint: postgres://k3s_user:password@db.example.com:5432/k3s_db?sslmode=require
server: https://10.0.2.10:6443
EOF

# Start k3s (it will connect to existing datastore)
sudo systemctl start k3s

Caution: Restoring to a new cluster with the same token and server IP will cause the old cluster (if it comes back online) to fight for cluster membership. Ensure failed nodes are fully powered off or have their server IPs changed before bringing up restored nodes.

Disaster Recovery Planning

High Availability Options

Recommendation:

• less than 50 nodes, single site: SQLite + embedded etcd (simpler, k3s-managed)
• 50-500 nodes, multi-site: External PostgreSQL with Patroni
• 500+ nodes: External etcd or PostgreSQL (scales better, independent lifecycle)

RPO/RTO by Business Requirement

Split-Brain Prevention

Split-brain occurs when two subsets of nodes believe they are the authoritative cluster, causing divergence of state (different pod IPs, different configmaps).

k3s built-in prevention (embedded etcd):

# Check etcd membership
sudo k3s kubectl get nodes -o wide
sudo k3s etcd-snapshot ls

# If split-brain suspected, quorum will block writes
# Last-nodes-standing continues (one side wins, other side stops)

External PostgreSQL with Patroni:

Patroni manages leader election and failover for PostgreSQL:

# /etc/patroni/patronictl.yaml
scope: k3s-cluster
name: node1

restapi:
  listen: 0.0.0.0:8008
  connect_address: node1.example.com:8008

postgresql:
  listen: 0.0.0.0:5432
  connect_address: node1.example.com:5432
  data_dir: /var/lib/postgresql/14/main
  bin_dir: /usr/lib/postgresql/14/bin

  config:
    max_connections: 200
    hot_standby: on
    max_wal_senders: 5

bootstrap:
  dcs:
    ttl: 30
    loop_wait: 10
    retry_timeout: 10
    maximum_lag_on_failover: 1048576

  initdb:
    - encoding: UTF8
    - data-checksums

tags:
  nofailover: false
  noloadbalance: false
  clonefrom: false
  nosync: false

Etcd quorum checks:

# Verify etcd cluster health
ETCDCTL_API=3 etcdctl --endpoints https://etcd1.example.com:2379 \
  --cacert /etc/etcd/ca.crt --cert /etc/etcd/server.crt \
  --key /etc/etcd/server.key endpoint health

# Check member list and leader
ETCDCTL_API=3 etcdctl --endpoints https://etcd1.example.com:2379 \
  --cacert /etc/etcd/ca.crt --cert /etc/etcd/server.crt \
  --key /etc/etcd/server.key member list

# If quorum lost (no leader), last-nodes-standing kicks in automatically
# No manual intervention needed if embedded etcd

Failover Testing

Don't wait for a real disaster — test failover regularly.

Test embedded etcd failover:

# On 3-node cluster, kill leader node
sudo k3s kubectl get nodes -o wide
sudo k3s kubectl get cs  # Check which node is etcd leader

sudo systemctl stop k3s  # Stop on leader node

# Verify remaining nodes elect new leader (should take 5-10 seconds)
sudo k3s kubectl get nodes  # On one of the remaining nodes

# Restart stopped node
sudo systemctl start k3s
sudo k3s kubectl get nodes  # Should rejoin automatically

Test PostgreSQL failover (Patroni):

# Check current PostgreSQL leader
sudo patronictl -c /etc/patroni/patronictl.yaml list

# Switchover (manual failover)
sudo patronictl -c /etc/patroni/patronictl.yaml switchover \
  --master node1 --candidate node2

# Verify k3s connects to new leader
sudo k3s kubectl get nodes

Test disaster recovery from backup:

# Stop cluster
sudo systemctl stop k3s  # All nodes

# On one node, restore from recent snapshot
sudo k3s etcd-snapshot restore --name backup-20260711-020000

# Start restored node
sudo systemctl start k3s

# Add other nodes with same token (they'll join restored cluster)
# On new nodes:
sudo curl -sfL https://get.k3s.io | \
  INSTALL_K3S_EXEC="server --token my-cluster-token \
  --server https://10.0.1.10:6443" \
  sh -

Known Issues at Scale

etcd Startup Loop at ~2500 Nodes

Symptom: k3s nodes (running embedded etcd) get stuck in startup loop, logs show request header size too large or snap: failed to recover database.

Root Cause: etcd data directory size grows large (1-2GB) with thousands of pod events. When restoring snapshots or restarting, etcd must load entire data directory into memory before accepting connections. At ~2500 nodes, this exceeds available RAM on small VMs.

Workaround:

# Compact etcd history (retains only recent state)
ETCDCTL_API=3 etcdctl --endpoints 127.0.0.1:2379 \
  compact $(ETCDCTL_API=3 etcdctl --endpoints 127.0.0.1:2379 endpoint status --write-out="json" | jq -r '.[0].Status.header.revision')

# Defragment (reclaims space)
ETCDCTL_API=3 etcdctl --endpoints 127.0.0.1:2379 defrag

# Snapshot (creates clean copy)
sudo k3s etcd-snapshot save --name clean-snapshot

Long-term fix: External etcd with dedicated storage, or limit pod churn and enable event throttling.

kubelet Sandbox Regression in v1.36.0

Symptom: Pods fail to start with FailedCreatePodSandBox error, logs show runtime network not ready.

Root Cause: k3s v1.36.0 included a Kubernetes regression where containerd CRI fails to initialize pod network sandbox on certain kernel versions (5.15+ with cgroup v2).

Workaround: Upgrade to k3s v1.36.1+ where this is fixed:

sudo apt install -y k3s=1.36.1+k3s1
# or
sudo curl -sfL https://get.k3s.io | INSTALL_K3S_VERSION=v1.36.1+k3s1 sh -

Alternative: Downgrade to v1.35.x if immediate upgrade not possible.

Crash Loop on No Default Route

Symptom: k3s crashes repeatedly on nodes with no default route (e.g., multi-homed edge devices with specific routing).

Root Cause: k3s assumes 0.0.0.0/0 default route exists for cluster networking. When absent, API server binds fail or pod CIDR allocation fails.

Workaround:

# Add explicit CIDR routes (even if no default route)
ip route add 10.42.0.0/16 via 192.168.1.1 dev eth0
ip route add 10.43.0.0/16 via 192.168.1.1 dev eth0

# Or add dummy default route
ip route add default via 192.168.1.1 dev eth0 metric 999

Long-term fix: Configure k3s with explicit cluster-cidr and service-cidr that match your routing:

# /etc/rancher/k3s/config.yaml
cluster-cidr: 10.42.0.0/16
service-cidr: 10.43.0.0/16
node-ip: 10.0.1.10

Pod-to-API Server Timeouts at Scale

Symptom: Kubernetes controllers and operators show connection timeout to API server, especially on clusters with hundreds of nodes.

Root Cause: k3s API server default connection limits (max concurrent connections) are tuned for single-node or small HA clusters. At 500+ nodes, controller watch connections exceed limits.

Workaround:

# /etc/rancher/k3s/config.yaml
kube-apiserver-arg:
  - max-mutating-requests-inflight=3000
  - max-requests-inflight=3000
  - event-ttl=72h
kube-scheduler-arg:
  - kube-api-qps=100
  - kube-api-burst=100
kube-controller-manager-arg:
  - kube-api-qps=100
  - kube-api-burst=100

Restart k3s after config change.

Resource Exhaustion on Low-End Hardware

Symptom: k3s consumes 100% CPU or OOM kills on Raspberry Pi 4 with 4GB RAM running 20+ pods.

Root Cause: k3s default resource requests are tuned for server hardware. Edge devices with limited resources struggle with kubelet containerd metrics, etcd compaction, and kube-proxy iptables operations.

Workaround:

# /etc/rancher/k3s/config.yaml
kube-apiserver-arg:
  - default-request-timeout=5m
  - max-request-bytes=3145728
kubelet-arg:
  - container-log-max-files=3
  - container-log-max-size=10M
  - eviction-hard=memory.available_lessthan_200Mi,nodefs.available_lessthan_10%
  - eviction-soft=memory.available_lessthan_500Mi
  - eviction-soft-grace-period=memory.available=1m
kube-scheduler-arg:
  - kube-api-qps=50
  - kube-api-burst=100

Additional tuning:

# Disable k3s features you don't need
sudo cat > /etc/rancher/k3s/config.yaml <<'EOF'
disable:
  - traefik       # If using custom ingress
  - servicelb     # If using MetalLB
  - local-storage # If using Longhorn or Rook-Ceph
  - metrics-server  # If using external Prometheus
EOF

Decision Framework

Use this table to choose the right upgrade, backup, and HA strategy for your production k3s deployment.

Next Steps

• Set up automated backups with k3s etcd-snapshot save --s3 and verify restore
• Schedule quarterly failover drills for HA clusters
• Monitor etcd data directory size and compact monthly
• Implement blue-green upgrades for critical sites if SLAs require zero downtime

k3s is production-ready when you treat it with the same operational discipline as full Kubernetes — upgrades, backups, and disaster recovery are not optional.