K3s Distributions Compared: Flannel, Calico, Cilium — CNI Performance and NetworkPolicy Reality | DevOps

K3s is the most popular lightweight Kubernetes distribution for edge, IoT, ARM, and resource-constrained environments — but "k3s" covers a broad ecosystem of installer tools, CNI plugins, and deployment patterns that behave very differently in production.

This article compares the landscape across four dimensions: distribution options, CNI plugin performance and NetworkPolicy support, ingress and storage, and production considerations like HA, scaling limits, and known pitfalls.

If you found out the hard way that NetworkPolicies are silently ignored with Flannel, this article is for you.

K3s Distribution Options

There are four major ways to run k3s. Each targets a different use case:

Feature	k3s Official	k3sup	k3d	AutoK3s
Install method	Shell script on node	SSH from local machine	Docker containers	CLI/UI + provider APIs
Target environment	Any Linux host	Remote VMs via SSH	Local dev / CI	Multi-cloud
HA support	Embedded etcd / external DB	k3sup-pro plan/apply	Multi-server Docker containers	Provider-native HA configs
Airgap support	Manual	Pro feature	N/A	Built-in
GUI	None	None	None	Built-in dashboard
ARM support	Yes	Yes	Yes	Yes
Best for	Production edge	VM provisioning	Development / CI	Multi-cloud mgmt

k3s Official (Rancher / SUSE)

The reference distribution: a single <100 MB binary that bundles everything. Server nodes run k3s server (control-plane + datastore), agent nodes run k3s agent without control-plane overhead.

curl -sfL https://get.k3s.io | sh -

Single-server mode uses embedded SQLite; HA uses embedded etcd (3+ servers) or an external datastore (PostgreSQL, MySQL, external etcd). Bundled defaults include Flannel (VXLAN), Traefik, CoreDNS, ServiceLB (Klipper), Local Path Provisioner, and a kube-router-based NetworkPolicy controller.

k3sup (ketchup)

A client-side SSH tool by Alex Ellis that bootstraps k3s onto any VM from your local machine. Never logs into the remote — uses SSH key-forwarding.

# Install on a remote VM
k3sup install --ip 192.168.1.100 --user root

# Join an agent
k3sup join --ip 192.168.1.101 --server-ip 192.168.1.100 --user root

The Pro version ($25+/mo) adds parallel HA deployment from JSON configs, uninstall, exec, get-config. Best for IaC-style provisioning of bare-metal, EC2, or Raspberry Pi nodes.

k3d

k3s in Docker — creates containerized clusters for local development and CI/CD.

k3d cluster create mycluster --servers 3 --agents 2 --port "8080:80@loadbalancer"

Auto-merges kubeconfig, supports --k3s-arg for any k3s flag, port mappings, registry mounts, and GPU passthrough. Best for testing multi-node behavior on a single machine without provisioning real VMs.

AutoK3s

A management platform (CLI + UI) for deploying k3s across cloud providers (AWS, GCP, Alibaba, Tencent). Supports airgap installation and Helm-based add-on management. Best for multi-cloud deployments with a unified dashboard.

CNI Plugin Deep Dive

The choice of Container Network Interface (CNI) plugin is the most impactful decision for your k3s cluster's performance, security isolation, and operational complexity.

Flannel (Default)

Flannel is the default CNI in k3s. It provides simple overlay networking using VXLAN, host-gw, or WireGuard backends. Minimal overhead, minimal features.

Performance (10GbE testbed, no policies):

Throughput: ~6.8 Gbps
Latency: ~75 μs
Memory: ~50 MB per node

NetworkPolicy support: ✅ With k3s's built-in controller, ❌ Flannel alone

This is the critical distinction that catches most users off guard.

Calico

Calico is the most mature CNI for policy-driven networking. Supports iptables, eBPF, and VPP dataplanes. Full Kubernetes NetworkPolicy + global network policy CRDs.

Performance (10GbE testbed):

Data plane	Throughput	Latency	Policy impact
iptables	~7.5 Gbps	~68 μs	15-20% overhead
eBPF (kernel ≥5.8)	~9.0 Gbps	~45 μs	3-5% overhead

Memory: ~120-180 MB per node.

Cilium

Cilium is eBPF-native. This is not optional — Cilium requires kernel ≥5.8 and runs entirely in the eBPF subsystem. The result is the best performance with the deepest observability.

Performance (10GbE testbed):

Throughput: ~9.2 Gbps
Latency: ~42 μs
Policy impact: 2-3%

CiliumNetworkPolicy CRDs extend L3/L4 policy to L7 (HTTP, gRPC, Kafka, DNS). Hubble provides eBPF-native flow observability — no sidecars, no iptables, no overhead.

Memory: ~200 MB per node (higher than Calico, justified by eBPF dataplane).

Canal (Flannel + Calico Policy)

Canal combines Flannel's VXLAN data plane with Calico's Felix policy engine. You get Flannel's simple overlay with Calico's NetworkPolicy enforcement — but without Calico's eBPF or BGP capabilities.

Performance mirrors Flannel (~6.8 Gbps, ~75 μs) with a ~15% policy overhead from iptables.

Comparison Matrix

Dimension	Flannel	Calico (iptables)	Calico (eBPF)	Cilium	Canal
Throughput	~6.8 Gbps	~7.5 Gbps	~9.0 Gbps	~9.2 Gbps	~6.8 Gbps
Latency (mean)	~75 μs	~68 μs	~45 μs	~42 μs	~75 μs
NetworkPolicy	❌ native / ✅ k3s bunde	✅ Full + Global	✅ Full + Global	✅ L3-L7	✅ Via Calico
eBPF support	❌	❌	✅	✅ (required)	❌
Policy overhead	N/A	15-20%	3-5%	2-3%	~15%
Memory/node	~50 MB	~120-180 MB	~120-180 MB	~200 MB	~170 MB
Observability	Logs only	Basic / Tigera	Basic / Tigera	✅ Hubble	Basic
Kernel req.	Any	Any	≥5.8	≥5.8	Any
K3s integration	Default	Custom	Custom	Custom	Documented

NetworkPolicy Reality Check

Flannel Does NOT Enforce NetworkPolicies

This is the most common k3s gotcha. Flannel is a networking plugin only — it provides overlay connectivity and nothing else. The official Flannel README states:

"Flannel is focused on networking. For network policy, other projects such as Calico can be used."

If you create a NetworkPolicy in a cluster using Flannel without a policy controller, the object is accepted by the API server but never enforced. Your "deny all ingress" policy denies nothing.

What k3s Ships to Fix This

K3s bundles its own NetworkPolicy controller using kube-router's netpol library. This controller is enabled by default and provides basic Kubernetes NetworkPolicy enforcement on top of Flannel.

To verify it's running:

# Check if the kube-router netpol controller is active
kubectl -n kube-system get pods -l k8s-app=kube-router

# Or check k3s startup flags
journalctl -u k3s | grep disable-network-policy

To disable it (needed when using Calico or Cilium, which ship their own policy engines):

curl -sfL https://get.k3s.io | sh -s - --disable-network-policy

Or in the config file:

# /etc/rancher/k3s/config.yaml
disable-network-policy: true

The Three Scenarios

Scenario	NetworkPolicy enforced?	Notes
k3s default (Flannel + built-in netpol)	✅ Yes	kube-router netpol controller enforces basic policies
Flannel alone, no controller	❌ No	Objects accepted but not enforced — silent failure
Calico / Cilium with `--disable-network-policy`	✅ Yes	CNI-owned policy engine, feature-rich

How to Verify NetworkPolicy Is Working

Don't trust configs — verify:

# Deploy a test pod and a deny-all policy
kubectl run test-pod --image=nginx
kubectl run test-client --image=busybox -- sleep 3600

# Apply a policy that denies all ingress
kubectl create networkpolicy deny-all --pod-selector=app=test-pod \
  --policy-type=Ingress

# This should FAIL if NetworkPolicy is enforced
kubectl exec test-client -- wget -qO- http://test-pod:80
# ❌ If connection succeeds: NetworkPolicy is NOT working
# ✅ If connection times out: NetworkPolicy is enforced

Performance Impact of NetworkPolicies

When policies are enforced, performance depends on the CNI data plane:

Cilium (eBPF): 2-3% throughput reduction — negligible
Calico (eBPF): 3-5% — very good
Calico (iptables): 15-20% — noticeable; scales linearly with rule count
Flannel + kube-router netpol: modest overhead for basic policies

Changing CNIs in k3s

K3s makes it straightforward to swap CNIs. From the official docs:

Start K3s with --flannel-backend=none and install your CNI of choice. Most CNI plugins come with their own network policy engine, so it is recommended to set --disable-network-policy as well to avoid conflicts.

Switching to Calico

# 1. Install k3s without flannel and built-in netpol
curl -sfL https://get.k3s.io | sh -s - \
  --flannel-backend=none \
  --disable-network-policy

# 2. Install Calico
kubectl create -f https://raw.githubusercontent.com/projectcalico/calico/master/manifests/calico.yaml

For k3s, Calico needs containerIPForwarding: Enabled:

# calico-installation.yaml
apiVersion: operator.tigera.io/v1
kind: Installation
metadata:
  name: default
spec:
  calicoNetwork:
    containerIPForwarding: Enabled

Switching to Cilium

# 1. Install k3s without flannel and netpol
curl -sfL https://get.k3s.io | sh -s - \
  --flannel-backend=none \
  --disable-network-policy

# 2. Install Cilium via Helm
helm repo add cilium https://helm.cilium.io/
helm install cilium cilium/cilium --namespace kube-system

Important: Before running k3s-killall.sh, manually remove Cilium virtual interfaces or you risk losing network connectivity:

ip link delete cilium_host
ip link delete cilium_net
ip link delete cilium_vxlan

Ingress Options

Traefik (k3s Default)

Traefik is deployed automatically by k3s on ports 80/443 as a LoadBalancer service. Customizable via HelmChartConfig:

# /var/lib/rancher/k3s/server/manifests/k3s-traefik-config.yaml
apiVersion: helm.cattle.io/v1
kind: HelmChartConfig
metadata:
  name: traefik
  namespace: kube-system
spec:
  valuesContent: |-
    service:
      type: LoadBalancer
    ports:
      web:
        port: 80
    logs:
      general:
        level: INFO

Traefik v3 supports Gateway API, has built-in ACME (Let's Encrypt) integration, and a diagnostic dashboard. Memory: ~120 MB for 3 replicas.

To disable Traefik during install: --disable=traefik.

Nginx Ingress

The community ingress-nginx reached EOL in March 2026. F5 maintains the successor: NGINX Ingress Controller (NIC) and NGINX Gateway Fabric (Gateway API-native). RKE2 v1.36 flipped to Traefik as default; v1.37 removes ingress-nginx entirely.

If you need Nginx, use the F5-maintained NIC, not the archived community project.

Ingress Comparison

Feature	Traefik v3	Nginx NIC	HAProxy Ingress
TLS automation	Built-in ACME	cert-manager	cert-manager
Config reload	Zero-downtime	Hot reload	Zero-downtime
Dashboard	✅ Built-in	❌	❌
CRD routing	✅ IngressRoute CRD	❌ (annotations)	❌ (annotations)
Memory (3 replicas)	~120 MB	~180 MB	~90 MB
RPS	~60-70% of Nginx	Highest (C engine)	Close to Nginx

Storage Options with k3s

Local Path Provisioner (Default)

Provisioner: rancher.io/local-path. Access mode: ReadWriteOnce only. Volume binding: WaitForFirstConsumer. No replication, no shared access, node affinity required. Best for dev, single-node, or ephemeral data.

Longhorn

CNCF Incubating (graduated in 2026). Distributed block storage with synchronous replication, snapshots, S3 backups, and a built-in web UI. Memory: ~300 MB per storage node. Minimum 3 nodes for HA. Best for production block storage and databases.

Rook-Ceph

CNCF Graduated. Block (RBD), Filesystem (CephFS), and Object (RGW) in one. On k3s, requires overriding the kubelet path:

helm install rook-ceph rook-release/rook-ceph \
  --set csi.kubeletDirPath=/var/lib/rancher/k3s/agent/kubelet

Memory: 1.5-3 GB per storage node (significantly higher than Longhorn). Minimum 3 nodes with dedicated raw disks. Best for enterprise storage with RWX requirements.

Storage Decision Guide

Use Case	Recommended
Dev / single-node	Local Path Provisioner
Production block (<20 nodes)	Longhorn
Enterprise storage, RWX	Rook-Ceph (CephFS)
Object storage in-cluster	Rook-Ceph (RGW)
Max performance, app-level HA	OpenEBS LocalPV

Production Considerations

HA: Embedded etcd vs External Datastore

Option	Best for	Complexity
Embedded SQLite	Single-node, edge	Minimal
Embedded etcd (3+ servers)	Small HA clusters	Low
PostgreSQL	Cloud-native, managed DB	Medium
MySQL / MariaDB	Existing MySQL infra	Medium
External etcd	Large clusters, strict perf	High

Resource Profiling

Official k3s benchmarks (Intel 8375C):

Role	Min CPU	Min RAM (SQLite)	Min RAM (etcd)
Server + workload	6% core	1596 MB	1606 MB
Server + 1 agent	5% core	1428 MB	1450 MB
Agent only	3% core	275 MB	275 MB

On Raspberry Pi 4: server + workload uses ~30% core, ~1600 MB RAM. Agent only: ~10% core, ~268 MB.

Datastore IOPS requirements are modest — SQLite needs 10 IOPS / <10ms latency; embedded etcd needs 50 IOPS / <5ms latency.

Known Issues & Pitfalls

etcd startup loop at scale (k3s#14130): At ~2500 nodes, clearAlarms() gets rate-limited by etcd under write pressure (>83 writes/sec), causing infinite restart loops.
kubelet sandbox regression (k3s#14078): On v1.36.0, RunPodSandbox times out due to a gRPC module version bump. Workaround: pin to v1.35.x.
Crash loop on no default route (k3s#13895): If k3s starts before a default network route exists, it enters a 5-second restart loop at 100% CPU. Mitigation: systemd drop-in with StartLimitBurst=3 and RestartSec=30s.
Pod-to-apiserver timeout on same node (k3s#13721): Traffic to 10.43.0.1 (kubernetes service) intermittently times out due to k3s's load-balancer logic — SYN reaches cni0 but no ACK returns.
SD card wear on ARM: etcd is write-intensive. Always use an external SSD for Raspberry Pi clusters.

Production Best Practices

Minimum 3 server nodes for HA quorum
Use fast SSDs (NVMe preferred) for /var/lib/rancher/k3s/server/
Isolate datastore IOPS from workload storage
Place a load balancer (HAProxy) in front of server nodes
Use external datastore for clusters >50 nodes
Monitor etcd latency — critical for control plane health
Set --disable-network-policy when using Calico or Cilium
For Cilium: clean up virtual interfaces before k3s uninstall
For Rook-Ceph: always override csi.kubeletDirPath to /var/lib/rancher/k3s/agent/kubelet

Weave Net Is Dead

Weave Net was the fourth major option but is effectively archived. Weaveworks shut down, Kubespray removed Weave support in 2025, and maintenance has been handed to a community fork. Do not use Weave for new clusters.

Decision Framework

Requirement	Best Choice
Quick dev cluster	k3d (local) or k3sup (remote)
Edge / IoT production	k3s official + Flannel + Longhorn
HA production (<50 nodes)	k3s + embedded etcd + Cilium (or Calico eBPF)
Large production (>50 nodes)	k3s + external PostgreSQL + Cilium
Need NetworkPolicy enforcement	Cilium (eBPF, best perf) or Calico (mature, stable)
Maximum throughput	Cilium (9.2 Gbps, eBPF-native)
Simple ingress	Traefik (default, built-in ACME)
Proven ingress at scale	Nginx NIC (F5, migrate from community ingress-nginx)
Block storage (<20 nodes)	Longhorn
Enterprise storage (RWX, large)	Rook-Ceph
Multi-cloud management	AutoK3s

Summary

The k3s ecosystem offers choices at every layer. The default setup (Flannel + Traefik + Local Path) works well for single-node and edge deployments, but as soon as you need NetworkPolicy enforcement, HA, or production performance, you'll need to swap components.

Key takeaways:

Flannel does not enforce NetworkPolicies — k3s ships a kube-router-based controller to fill this gap, but if you disable it, policies are silently ignored
Cilium offers the best performance and deepest policy support (L3-L7) at the cost of kernel requirements and memory
Calico is the most battle-tested CNI with both iptables and eBPF data plane options
Embedded etcd is fine for clusters up to ~50 nodes; beyond that, use an external datastore
Traefik is adequate for most k3s deployments and avoids maintaining a second ingress controller

The right choice depends on your scale, performance requirements, and how much kernel-level control you need — but in every case, verify NetworkPolicy enforcement explicitly. Don't assume it works because you created a NetworkPolicy object.

Cross-Links

Kubernetes GitOps with ArgoCD — managing k3s clusters declaratively
Container Orchestration — comparing orchestration approaches beyond k3s
Container Building and Hardening Guide — secure images for your cluster workloads
Kubernetes Gateway API Migration — next-generation ingress for Kubernetes
CNCF Cloud Native Landscape — where k3s fits in the ecosystem

If you found out the hard way that NetworkPolicies are silently ignored with Flannel, this article is for you.

K3s Distribution Options

There are four major ways to run k3s. Each targets a different use case:

Feature	k3s Official	k3sup	k3d	AutoK3s
Install method	Shell script on node	SSH from local machine	Docker containers	CLI/UI + provider APIs
Target environment	Any Linux host	Remote VMs via SSH	Local dev / CI	Multi-cloud
HA support	Embedded etcd / external DB	k3sup-pro plan/apply	Multi-server Docker containers	Provider-native HA configs
Airgap support	Manual	Pro feature	N/A	Built-in
GUI	None	None	None	Built-in dashboard
ARM support	Yes	Yes	Yes	Yes
Best for	Production edge	VM provisioning	Development / CI	Multi-cloud mgmt

k3s Official (Rancher / SUSE)

The reference distribution: a single <100 MB binary that bundles everything. Server nodes run k3s server (control-plane + datastore), agent nodes run k3s agent without control-plane overhead.

curl -sfL https://get.k3s.io | sh -

k3sup (ketchup)

A client-side SSH tool by Alex Ellis that bootstraps k3s onto any VM from your local machine. Never logs into the remote — uses SSH key-forwarding.

# Install on a remote VM
k3sup install --ip 192.168.1.100 --user root

# Join an agent
k3sup join --ip 192.168.1.101 --server-ip 192.168.1.100 --user root

The Pro version ($25+/mo) adds parallel HA deployment from JSON configs, uninstall, exec, get-config. Best for IaC-style provisioning of bare-metal, EC2, or Raspberry Pi nodes.

k3d

k3s in Docker — creates containerized clusters for local development and CI/CD.

k3d cluster create mycluster --servers 3 --agents 2 --port "8080:80@loadbalancer"

AutoK3s

CNI Plugin Deep Dive

The choice of Container Network Interface (CNI) plugin is the most impactful decision for your k3s cluster's performance, security isolation, and operational complexity.

Flannel (Default)

Flannel is the default CNI in k3s. It provides simple overlay networking using VXLAN, host-gw, or WireGuard backends. Minimal overhead, minimal features.

Performance (10GbE testbed, no policies):

Throughput: ~6.8 Gbps
Latency: ~75 μs
Memory: ~50 MB per node

NetworkPolicy support: ✅ With k3s's built-in controller, ❌ Flannel alone

This is the critical distinction that catches most users off guard.

Calico

Calico is the most mature CNI for policy-driven networking. Supports iptables, eBPF, and VPP dataplanes. Full Kubernetes NetworkPolicy + global network policy CRDs.

Performance (10GbE testbed):

Data plane	Throughput	Latency	Policy impact
iptables	~7.5 Gbps	~68 μs	15-20% overhead
eBPF (kernel ≥5.8)	~9.0 Gbps	~45 μs	3-5% overhead

Memory: ~120-180 MB per node.

Cilium

Cilium is eBPF-native. This is not optional — Cilium requires kernel ≥5.8 and runs entirely in the eBPF subsystem. The result is the best performance with the deepest observability.

Performance (10GbE testbed):

Throughput: ~9.2 Gbps
Latency: ~42 μs
Policy impact: 2-3%

CiliumNetworkPolicy CRDs extend L3/L4 policy to L7 (HTTP, gRPC, Kafka, DNS). Hubble provides eBPF-native flow observability — no sidecars, no iptables, no overhead.

Memory: ~200 MB per node (higher than Calico, justified by eBPF dataplane).

Canal (Flannel + Calico Policy)

Performance mirrors Flannel (~6.8 Gbps, ~75 μs) with a ~15% policy overhead from iptables.

Comparison Matrix

Dimension	Flannel	Calico (iptables)	Calico (eBPF)	Cilium	Canal
Throughput	~6.8 Gbps	~7.5 Gbps	~9.0 Gbps	~9.2 Gbps	~6.8 Gbps
Latency (mean)	~75 μs	~68 μs	~45 μs	~42 μs	~75 μs
NetworkPolicy	❌ native / ✅ k3s bunde	✅ Full + Global	✅ Full + Global	✅ L3-L7	✅ Via Calico
eBPF support	❌	❌	✅	✅ (required)	❌
Policy overhead	N/A	15-20%	3-5%	2-3%	~15%
Memory/node	~50 MB	~120-180 MB	~120-180 MB	~200 MB	~170 MB
Observability	Logs only	Basic / Tigera	Basic / Tigera	✅ Hubble	Basic
Kernel req.	Any	Any	≥5.8	≥5.8	Any
K3s integration	Default	Custom	Custom	Custom	Documented

NetworkPolicy Reality Check

Flannel Does NOT Enforce NetworkPolicies

This is the most common k3s gotcha. Flannel is a networking plugin only — it provides overlay connectivity and nothing else. The official Flannel README states:

"Flannel is focused on networking. For network policy, other projects such as Calico can be used."

If you create a NetworkPolicy in a cluster using Flannel without a policy controller, the object is accepted by the API server but never enforced. Your "deny all ingress" policy denies nothing.

What k3s Ships to Fix This

K3s bundles its own NetworkPolicy controller using kube-router's netpol library. This controller is enabled by default and provides basic Kubernetes NetworkPolicy enforcement on top of Flannel.

To verify it's running:

# Check if the kube-router netpol controller is active
kubectl -n kube-system get pods -l k8s-app=kube-router

# Or check k3s startup flags
journalctl -u k3s | grep disable-network-policy

To disable it (needed when using Calico or Cilium, which ship their own policy engines):

curl -sfL https://get.k3s.io | sh -s - --disable-network-policy

Or in the config file:

# /etc/rancher/k3s/config.yaml
disable-network-policy: true

The Three Scenarios

Scenario	NetworkPolicy enforced?	Notes
k3s default (Flannel + built-in netpol)	✅ Yes	kube-router netpol controller enforces basic policies
Flannel alone, no controller	❌ No	Objects accepted but not enforced — silent failure
Calico / Cilium with `--disable-network-policy`	✅ Yes	CNI-owned policy engine, feature-rich

How to Verify NetworkPolicy Is Working

Don't trust configs — verify:

# Deploy a test pod and a deny-all policy
kubectl run test-pod --image=nginx
kubectl run test-client --image=busybox -- sleep 3600

# Apply a policy that denies all ingress
kubectl create networkpolicy deny-all --pod-selector=app=test-pod \
  --policy-type=Ingress

# This should FAIL if NetworkPolicy is enforced
kubectl exec test-client -- wget -qO- http://test-pod:80
# ❌ If connection succeeds: NetworkPolicy is NOT working
# ✅ If connection times out: NetworkPolicy is enforced

Performance Impact of NetworkPolicies

When policies are enforced, performance depends on the CNI data plane:

Cilium (eBPF): 2-3% throughput reduction — negligible
Calico (eBPF): 3-5% — very good
Calico (iptables): 15-20% — noticeable; scales linearly with rule count
Flannel + kube-router netpol: modest overhead for basic policies

Changing CNIs in k3s

K3s makes it straightforward to swap CNIs. From the official docs:

Start K3s with --flannel-backend=none and install your CNI of choice. Most CNI plugins come with their own network policy engine, so it is recommended to set --disable-network-policy as well to avoid conflicts.

Switching to Calico

# 1. Install k3s without flannel and built-in netpol
curl -sfL https://get.k3s.io | sh -s - \
  --flannel-backend=none \
  --disable-network-policy

# 2. Install Calico
kubectl create -f https://raw.githubusercontent.com/projectcalico/calico/master/manifests/calico.yaml

For k3s, Calico needs containerIPForwarding: Enabled:

# calico-installation.yaml
apiVersion: operator.tigera.io/v1
kind: Installation
metadata:
  name: default
spec:
  calicoNetwork:
    containerIPForwarding: Enabled

Switching to Cilium

# 1. Install k3s without flannel and netpol
curl -sfL https://get.k3s.io | sh -s - \
  --flannel-backend=none \
  --disable-network-policy

# 2. Install Cilium via Helm
helm repo add cilium https://helm.cilium.io/
helm install cilium cilium/cilium --namespace kube-system

Important: Before running k3s-killall.sh, manually remove Cilium virtual interfaces or you risk losing network connectivity:

ip link delete cilium_host
ip link delete cilium_net
ip link delete cilium_vxlan

Ingress Options

Traefik (k3s Default)

Traefik is deployed automatically by k3s on ports 80/443 as a LoadBalancer service. Customizable via HelmChartConfig:

# /var/lib/rancher/k3s/server/manifests/k3s-traefik-config.yaml
apiVersion: helm.cattle.io/v1
kind: HelmChartConfig
metadata:
  name: traefik
  namespace: kube-system
spec:
  valuesContent: |-
    service:
      type: LoadBalancer
    ports:
      web:
        port: 80
    logs:
      general:
        level: INFO

Traefik v3 supports Gateway API, has built-in ACME (Let's Encrypt) integration, and a diagnostic dashboard. Memory: ~120 MB for 3 replicas.

To disable Traefik during install: --disable=traefik.

Nginx Ingress

If you need Nginx, use the F5-maintained NIC, not the archived community project.

Ingress Comparison

Feature	Traefik v3	Nginx NIC	HAProxy Ingress
TLS automation	Built-in ACME	cert-manager	cert-manager
Config reload	Zero-downtime	Hot reload	Zero-downtime
Dashboard	✅ Built-in	❌	❌
CRD routing	✅ IngressRoute CRD	❌ (annotations)	❌ (annotations)
Memory (3 replicas)	~120 MB	~180 MB	~90 MB
RPS	~60-70% of Nginx	Highest (C engine)	Close to Nginx

Storage Options with k3s

Local Path Provisioner (Default)

Longhorn

Rook-Ceph

CNCF Graduated. Block (RBD), Filesystem (CephFS), and Object (RGW) in one. On k3s, requires overriding the kubelet path:

helm install rook-ceph rook-release/rook-ceph \
  --set csi.kubeletDirPath=/var/lib/rancher/k3s/agent/kubelet

Memory: 1.5-3 GB per storage node (significantly higher than Longhorn). Minimum 3 nodes with dedicated raw disks. Best for enterprise storage with RWX requirements.

Storage Decision Guide

Use Case	Recommended
Dev / single-node	Local Path Provisioner
Production block (<20 nodes)	Longhorn
Enterprise storage, RWX	Rook-Ceph (CephFS)
Object storage in-cluster	Rook-Ceph (RGW)
Max performance, app-level HA	OpenEBS LocalPV

Production Considerations

HA: Embedded etcd vs External Datastore

Option	Best for	Complexity
Embedded SQLite	Single-node, edge	Minimal
Embedded etcd (3+ servers)	Small HA clusters	Low
PostgreSQL	Cloud-native, managed DB	Medium
MySQL / MariaDB	Existing MySQL infra	Medium
External etcd	Large clusters, strict perf	High

Resource Profiling

Official k3s benchmarks (Intel 8375C):

Role	Min CPU	Min RAM (SQLite)	Min RAM (etcd)
Server + workload	6% core	1596 MB	1606 MB
Server + 1 agent	5% core	1428 MB	1450 MB
Agent only	3% core	275 MB	275 MB

On Raspberry Pi 4: server + workload uses ~30% core, ~1600 MB RAM. Agent only: ~10% core, ~268 MB.

Datastore IOPS requirements are modest — SQLite needs 10 IOPS / <10ms latency; embedded etcd needs 50 IOPS / <5ms latency.

Known Issues & Pitfalls

etcd startup loop at scale (k3s#14130): At ~2500 nodes, clearAlarms() gets rate-limited by etcd under write pressure (>83 writes/sec), causing infinite restart loops.
kubelet sandbox regression (k3s#14078): On v1.36.0, RunPodSandbox times out due to a gRPC module version bump. Workaround: pin to v1.35.x.
Crash loop on no default route (k3s#13895): If k3s starts before a default network route exists, it enters a 5-second restart loop at 100% CPU. Mitigation: systemd drop-in with StartLimitBurst=3 and RestartSec=30s.
Pod-to-apiserver timeout on same node (k3s#13721): Traffic to 10.43.0.1 (kubernetes service) intermittently times out due to k3s's load-balancer logic — SYN reaches cni0 but no ACK returns.
SD card wear on ARM: etcd is write-intensive. Always use an external SSD for Raspberry Pi clusters.

Production Best Practices

Minimum 3 server nodes for HA quorum
Use fast SSDs (NVMe preferred) for /var/lib/rancher/k3s/server/
Isolate datastore IOPS from workload storage
Place a load balancer (HAProxy) in front of server nodes
Use external datastore for clusters >50 nodes
Monitor etcd latency — critical for control plane health
Set --disable-network-policy when using Calico or Cilium
For Cilium: clean up virtual interfaces before k3s uninstall
For Rook-Ceph: always override csi.kubeletDirPath to /var/lib/rancher/k3s/agent/kubelet

Weave Net Is Dead

Decision Framework

Requirement	Best Choice
Quick dev cluster	k3d (local) or k3sup (remote)
Edge / IoT production	k3s official + Flannel + Longhorn
HA production (<50 nodes)	k3s + embedded etcd + Cilium (or Calico eBPF)
Large production (>50 nodes)	k3s + external PostgreSQL + Cilium
Need NetworkPolicy enforcement	Cilium (eBPF, best perf) or Calico (mature, stable)
Maximum throughput	Cilium (9.2 Gbps, eBPF-native)
Simple ingress	Traefik (default, built-in ACME)
Proven ingress at scale	Nginx NIC (F5, migrate from community ingress-nginx)
Block storage (<20 nodes)	Longhorn
Enterprise storage (RWX, large)	Rook-Ceph
Multi-cloud management	AutoK3s

Summary

Key takeaways:

Flannel does not enforce NetworkPolicies — k3s ships a kube-router-based controller to fill this gap, but if you disable it, policies are silently ignored
Cilium offers the best performance and deepest policy support (L3-L7) at the cost of kernel requirements and memory
Calico is the most battle-tested CNI with both iptables and eBPF data plane options
Embedded etcd is fine for clusters up to ~50 nodes; beyond that, use an external datastore
Traefik is adequate for most k3s deployments and avoids maintaining a second ingress controller

Cross-Links

Kubernetes GitOps with ArgoCD — managing k3s clusters declaratively
Container Orchestration — comparing orchestration approaches beyond k3s
Container Building and Hardening Guide — secure images for your cluster workloads
Kubernetes Gateway API Migration — next-generation ingress for Kubernetes
CNCF Cloud Native Landscape — where k3s fits in the ecosystem

K3s Distribution Options

k3s Official (Rancher / SUSE)

k3sup (ketchup)

k3d

AutoK3s

CNI Plugin Deep Dive

Flannel (Default)

Calico

Cilium

Canal (Flannel + Calico Policy)

Comparison Matrix

NetworkPolicy Reality Check

Flannel Does NOT Enforce NetworkPolicies

What k3s Ships to Fix This

The Three Scenarios

How to Verify NetworkPolicy Is Working

Performance Impact of NetworkPolicies

Changing CNIs in k3s

Switching to Calico

Switching to Cilium

Ingress Options

Traefik (k3s Default)

Nginx Ingress

Ingress Comparison

Storage Options with k3s

Local Path Provisioner (Default)

Longhorn

Rook-Ceph

Storage Decision Guide

Production Considerations

HA: Embedded etcd vs External Datastore

Resource Profiling

Known Issues & Pitfalls

Production Best Practices

Weave Net Is Dead

Decision Framework

Summary

Cross-Links

Never miss a deep-dive

K3s Distribution Options

k3s Official (Rancher / SUSE)

k3sup (ketchup)

k3d

AutoK3s

CNI Plugin Deep Dive

Flannel (Default)

Calico

Cilium

Canal (Flannel + Calico Policy)

Comparison Matrix

NetworkPolicy Reality Check

Flannel Does NOT Enforce NetworkPolicies

What k3s Ships to Fix This

The Three Scenarios

How to Verify NetworkPolicy Is Working

Performance Impact of NetworkPolicies

Changing CNIs in k3s

Switching to Calico

Switching to Cilium

Ingress Options

Traefik (k3s Default)

Nginx Ingress

Ingress Comparison

Storage Options with k3s

Local Path Provisioner (Default)

Longhorn

Rook-Ceph

Storage Decision Guide

Production Considerations

HA: Embedded etcd vs External Datastore

Resource Profiling

Known Issues & Pitfalls

Production Best Practices

Weave Net Is Dead

Decision Framework

Summary

Cross-Links

Never miss a deep-dive