Kubernetes on the Edge: K3s, Fleet, and the Art of Running a Cluster Where the Network Lies

Continuum finds, validates, and patches code vulnerabilities: it proves which findings are actually exploitable in your environment, ranks them by business impact, and proposes fixes — with human approval before anything ships.

Kubernetes is a control plane that assumes its nodes are reachable, its etcd is durable, and its operator is a person with a laptop and a VPN. Move it to a factory floor, a cell tower, a retail back room, or a ship, and every one of those assumptions breaks. The network lies. The power blinks. The hardware is small. And the person who deployed it is three hundred kilometers away, probably asleep.

This article is a field guide to running Kubernetes at the edge with the tools that actually work there: K3s as the runtime, Fleet as the fleet manager, GitOps as the delivery mechanism, and a set of practices for surviving the conditions the cloud never has to think about. It is written from the operator's side — what you deploy, what breaks, and what to watch.

What "edge" actually means

The word edge is used for three different things, and conflating them is the first mistake.

• The far edge: hundreds or thousands of sites with a handful of devices each — retail stores, cell towers, ATMs, vending machines. Very constrained compute, intermittent connectivity, no on-site staff.
• The near edge: regional points of presence with modest racks — regional data centers, telco central offices, factory control rooms. Real servers, but limited power, cooling, and bandwidth.
• The device edge: the device itself — a camera, a robot, a sensor — running a single container or a tiny K3s node.

The architecture changes with the tier. A far-edge site runs one K3s node with a handful of pods and a local store-and-forward buffer. A near-edge site runs a small HA cluster. A device-edge deployment is often not Kubernetes at all — it is a container on a device, managed by the same control plane that manages the clusters.

This article focuses on the far and near edge, where Kubernetes earns its keep: many sites, one control plane, declarative configuration, and the ability to roll out a change to a thousand locations without a plane ticket.

Why K3s

K3s is the right answer for most edge cases, and the reasons are not the ones usually cited.

It is not "Kubernetes lite" in the sense of a toy. It is a certified Kubernetes distribution with the API server, scheduler, controller manager, and kubelet in a single binary, with SQLite as the default datastore and an embedded containerd. The footprint is small — roughly 50 MB binary, ~250 MB RAM at idle — and it runs on a Raspberry Pi, an industrial PC, or a VM with two cores.

The reasons that matter at the edge:

1. Single binary, single process. One systemd unit. One thing to restart. One thing to upgrade. At a thousand sites, "how many moving parts" is the only metric that scales.
2. SQLite datastore by default. No external etcd to run, back up, or lose. For single-node edge clusters this is the right trade: durability comes from GitOps and from the fact that the cluster can be rebuilt from the spec in minutes.
3. Embedded containerd. No Docker daemon to manage, no socket to secure. K3s ships its own.
4. Air-gap friendly. A single tarball contains everything. You can install a node with no internet access.
5. HA when you need it. Three K3s nodes with an external datastore (etcd or Postgres) give you a real HA control plane. Use it for near-edge sites with real servers.

The trade-offs: K3s ships with Traefik, CoreDNS, and a local-path provisioner by default. For edge you usually want to disable Traefik and bring your own ingress, and you want to be careful with the local-path provisioner because local storage on edge hardware is unreliable.

# Minimal edge install: no Traefik, no bundled metrics-server, air-gap friendly
curl -sfL https://get.k3s.io | sh -s - \
  --disable traefik \
  --disable metrics-server \
  --write-kubeconfig-mode 644

# Verify
k3s kubectl get nodes -o wide
systemctl status k3s --no-pager

The control plane: one cluster to rule them all

The edge pattern that works is a hub-and-spoke topology. A single management cluster — running in your cloud or on-prem data center — holds the desired state for every edge site. Each edge site runs its own K3s cluster. The management cluster does not run the edge workloads; it runs the controllers that push configuration to them.

The tool for this is Fleet, Rancher's GitOps engine, though Argo CD and Flux work too. Fleet is purpose-built for the "many clusters, one source of truth" problem: it watches a Git repository, bundles the manifests, and rolls them out to a set of target clusters selected by labels.

The reason Fleet fits the edge better than the alternatives:

• It is designed for thousands of clusters, not dozens.
• It handles partial rollout and per-cluster overrides natively.
• It works offline: an edge cluster can pull its bundle when connectivity returns, and the desired state is cached locally.
• It has a simple mental model: Git → bundle → cluster selector → apply.

# fleet.yaml at the root of your GitOps repo
defaultNamespace: edge-apps
helm:
  repo: https://charts.example.com
  chart: edge-agent
  version: "1.4.2"
  values:
    telemetryEndpoint: https://telemetry.example.com
targetCustomizations:
  - name: retail
    clusterSelector:
      matchLabels:
        site-type: retail
    helm:
      values:
        storeId: "{{ .ClusterLabels.storeId }}"
  - name: factory
    clusterSelector:
      matchLabels:
        site-type: factory
    helm:
      values:
        modbusEnabled: true

The cluster selector is the key. You label each edge cluster when it registers — site-type, region, hardware-generation, customer — and Fleet routes the right configuration to the right sites. A new site that registers with the right labels gets the right config automatically. No ticket, no manual step.

GitOps as the only delivery mechanism

At the edge, GitOps is not a preference. It is the only delivery mechanism that survives the conditions.

The reason is simple: the edge cluster cannot be reached. SSH is blocked, the VPN is down, and the person with the keys is on a plane. The only channel that works reliably is outbound HTTPS from the edge to the management plane (or to a Git host). GitOps turns that channel into the entire delivery path: the edge cluster pulls its desired state, applies it, and reports status back. No inbound access required.

The rules that make this work:

1. Everything is in Git. Manifests, Helm charts, Kustomize overlays, Fleet bundles, even the K3s install script. If it is not in Git, it does not exist at the edge.
2. The edge cluster is disposable. A site should be rebuildable from its Git spec plus a K3s install. If you cannot rebuild a site in 30 minutes from a USB stick and a Git URL, your architecture is wrong.
3. Pin versions. Never deploy latest to the edge. Pin chart versions, image digests, and Fleet bundle versions. An uncontrolled update that breaks one site breaks a thousand.
4. Roll out in waves. Use Fleet's rollout strategy to deploy to a canary group first — 1%, then 10%, then 100%. The edge is where a bad release becomes a physical problem.
5. Keep a local cache. The edge cluster should cache its last-known-good bundle. If Git is unreachable, it keeps running the last good config instead of crashing.

# Fleet rollout with canary
rolloutStrategy:
  maxUnavailable: 5%
  maxUnavailablePartitions: 1
  autoPartitionSize: 10%
  partitions:
    - name: canary
      maxUnavailable: 1
      clusterSelector:
        matchLabels:
          canary: "true"
    - name: production
      maxUnavailable: 5%

Surviving the network

The network at the edge is not slow. It is intermittent, asymmetric, and hostile. Cellular backhaul drops packets. Satellite links have 600 ms latency. Factory networks are air-gapped by policy. The design has to assume the link is down most of the time and treat connectivity as a bonus.

Practices that matter:

• Store-and-forward for telemetry. Buffer metrics and logs locally (Prometheus remote-write queue, Vector, or a simple SQLite buffer) and flush when the link returns. Never drop data silently; never block the application waiting for a remote endpoint.
• Local DNS and service discovery. CoreDNS on the edge cluster resolves local services. Do not depend on a remote DNS resolver for anything the application needs to start.
• Offline-capable images. Pre-pull all images to the node. K3s can be configured with an image registry mirror or an air-gapped image tarball. A pod that cannot pull its image is a pod that does not start.
• Idempotent reconciliation. The GitOps controller must tolerate being unable to reach Git for hours or days without degrading. Fleet and Flux handle this; make sure your custom controllers do too.
• Graceful degradation. The application should keep serving its core function when upstream is down. A point-of-sale terminal that stops working because the cloud inventory API is unreachable is a design failure.

# K3s air-gap: private registry mirror config
# /etc/rancher/k3s/registries.yaml
mirrors:
  docker.io:
    endpoint:
      - "https://registry.edge.local:5000"
  "registry.edge.local:5000":
    endpoint:
      - "https://registry.edge.local:5000"
configs:
  "registry.edge.local:5000":
    tls:
      insecure_skip_verify: false
    auth:
      username: edge
      password: "${REGISTRY_PASSWORD}"

Observability on constrained hardware

You cannot run a full Prometheus + Grafana + Loki + Tempo stack on a 2-core, 4 GB edge node. The observability footprint has to be proportional to the hardware.

The pattern that works:

1. Lightweight agent on the edge. VictoriaMetrics agent, or Prometheus with a small scrape interval and short retention (1–2 hours). Scrape only the metrics that matter: node health, pod restarts, disk usage, network up/down.
2. Buffer and forward. Remote-write to a central VictoriaMetrics or Thanos instance. Buffer locally for 24–48 hours of disconnected operation.
3. Central dashboards. Grafana lives in the management cluster, not at the edge. The edge node does not need a UI.
4. Logs: sample and ship. Ship error-level logs and a sample of info-level logs. Full-fidelity logging on a constrained node is a waste of disk and bandwidth.
5. Synthetic probes. A small cronjob on the edge node that curls a health endpoint and reports up/down. This is the single most useful signal: is the site alive?

# VictoriaMetrics agent on the edge
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: vmagent
  namespace: monitoring
spec:
  selector:
    matchLabels:
      app: vmagent
  template:
    metadata:
      labels:
        app: vmagent
    spec:
      containers:
        - name: vmagent
          image: victoriametrics/vmagent:v1.102.0
          args:
            - "-promscrape.config=/config/scrape.yml"
            - "-remoteWrite.url=https://metrics.example.com/api/v1/write"
            - "-remoteWrite.tmpDataPath=/data/buffer"
            - "-remoteWrite.maxDiskUsagePerURL=1GB"
          volumeMounts:
            - name: config
              mountPath: /config
            - name: buffer
              mountPath: /data
      volumes:
        - name: config
          configMap:
            name: vmagent-config
        - name: buffer
          hostPath:
            path: /var/lib/vmagent
            type: DirectoryOrCreate

The maxDiskUsagePerURL setting is the safety net: when the buffer fills, vmagent drops the oldest data instead of crashing or filling the disk. On edge hardware, a full disk is a site outage.

Upgrades: the part that actually hurts

Upgrading a thousand edge clusters is the hardest part of the job, and it is where most edge deployments fail.

• K3s upgrades are simple but risky. A single binary swap, but if it breaks, the site is down and nobody is there to fix it. Test every upgrade in a lab that mirrors the edge hardware. Roll out in waves. Keep the previous binary on disk for rollback.
• Fleet bundle upgrades are the normal path. Application updates go through GitOps. K3s upgrades require a separate mechanism — Rancher's system-upgrade-controller, or a Fleet-managed DaemonSet that runs the install script.
• Hardware upgrades are physical. Plan for 3–5 year hardware lifecycles. Design the software to run on the oldest hardware in the fleet, not the newest.
• OS upgrades are the hidden cost. The OS underneath K3s needs patching. Use an immutable OS (Flatcar, Talos, or a read-only root with A/B updates) so the OS is a versioned artifact, not a snowflake.

# system-upgrade-controller plan for K3s
apiVersion: upgrade.cattle.io/v1
kind: Plan
metadata:
  name: k3s-upgrade
  namespace: system-upgrade
spec:
  concurrency: 10
  cordon: true
  nodeSelector:
    matchExpressions:
      - key: node-role.kubernetes.io/control-plane
        operator: In
        values: ["true"]
  serviceAccountName: system-upgrade
  upgrade:
    image: rancher/k3s-upgrade
    version: v1.30.4-k3s1

The concurrency: 10 is the safety valve. Never upgrade the whole fleet at once.

Security at the edge

Edge sites are physically accessible. A device in a retail back room can be unplugged, opened, or stolen. The security model has to assume the hardware is compromised.

• Encrypt at rest. K3s supports encrypted secrets and etcd (or SQLite) encryption. Use it.
• mTLS everywhere. Fleet communicates over mTLS. The edge cluster's kubeconfig should be scoped to a service account with minimal permissions, not cluster-admin.
• Read-only root filesystem. Where the OS supports it, mount the root filesystem read-only and use a writable overlay for runtime data.
• Remote attestation. For high-security sites, use TPM-based attestation to verify the node's identity before it joins the management plane.
• No secrets in Git. Use SOPS, Sealed Secrets, or an external secrets operator that fetches from a vault at runtime. The edge cluster pulls secrets from the vault over mTLS, never from the Git repo.

What to do instead (the short version)

1. Use K3s. Single binary, SQLite datastore, air-gap friendly. Disable Traefik and bring your own ingress.
2. Use Fleet (or Flux/Argo) for GitOps. Label clusters by site type, region, and hardware. Let the selector route config automatically.
3. Pin everything. Versions, image digests, bundle versions. No latest at the edge.
4. Roll out in waves. Canary first, then production. Never the whole fleet at once.
5. Assume the network is down. Store-and-forward, local DNS, pre-pulled images, graceful degradation.
6. Keep observability small. A lightweight agent, a central dashboard, a synthetic probe. Never run a full stack on a constrained node.
7. Make sites disposable. Rebuildable from Git + a USB stick in 30 minutes.
8. Use an immutable OS. Flatcar, Talos, or A/B updates. No snowflakes.
9. Encrypt and attest. The hardware is physically accessible. Plan for it.

Coda: the cluster is a tool

Kubernetes at the edge is not a smaller version of Kubernetes in the cloud. It is a different thing: a tool that has to survive conditions the cloud never faces, managed by people who are never on site, running on hardware that is old and small. The tools that work — K3s, Fleet, GitOps, immutable OS, lightweight observability — are the ones that accept those conditions instead of fighting them.

The cluster is a tool. Like all tools, it should withdraw into the work. At the edge, the work is keeping a thousand sites running while nobody watches. The art is building something that can do that, and then getting out of its way.

Sources and further reading

• K3s documentation: https://docs.k3s.io
• Fleet documentation: https://fleet.rancher.io
• Rancher system-upgrade-controller: https://github.com/rancher/system-upgrade-controller
• CNCF Edge Computing white paper: https://www.cncf.io/reports/
• Flatcar Container Linux: https://www.flatcar.org
• Talos Linux: https://www.talos.dev
• VictoriaMetrics: https://docs.victoriametrics.com
• SOPS: https://github.com/getsops/sops
• Sealed Secrets: https://sealed-secrets.netlify.app
• KubeCon + CloudNativeCon Kubernetes on Edge Day: https://events.linuxfoundation.org/kubecon-cloudnativecon-north-america/