Multi-tenant GPU Kubernetes on simulated hardware with Kamaji and HAMi
~4 min read
devopscloud-nativekubernetesgpukamajihamimulti-tenantkubeflow
Admission control checks whether a spec looks right — not whether a running pod behaves. Nirmata Runtime for Kyverno moves policy into the kernel with eBPF and LSM, and that gap matters most for AI agents whose behavior is only decided at runtime.
GPU teams usually face a bad trade-off. Give every team its own Kubernetes cluster and you pay for a control plane per team. Put everyone in one shared cluster and you get weak isolation. Either way, Kubernetes hands out GPUs as whole cards — so a pod that needs 4 GB of an 80 GB GPU still blocks the entire card.
This post walks a way out by combining two open-source projects:
The whole thing runs on a single machine with no physical GPU, using HAMi's mock device plugin to simulate the cards. Result: two fully isolated tenant clusters, each scheduling fractional-GPU workloads through HAMi, with all eight tests passing. Everything below is reproducible — exact versions, working commands, and the errors hit along the way.
One kind cluster hosts Kamaji and both tenant control planes; each tenant's worker is a separate container that joins over the kind network and runs HAMi. Tenants never share a node or an API server. The only shared pieces are the management cluster and the kamaji-etcd datastore, where each tenant's data stays separate.
Kamaji solves the cluster problem, HAMi solves the GPU problem — they work at different layers, so they don't overlap. The interesting part is where they meet: a tenant's API server lives in the management cluster, but HAMi's webhook lives on the tenant's worker nodes. Every GPU pod creation therefore depends on the API server reaching back into the tenant through Kamaji's Konnectivity tunnel. Proving that path works is the main goal of this lab.
Everything runs as Docker containers on one Linux host — 4 CPU cores, 8 GB free RAM, and 20 GB disk are enough.
How do you test GPU sharing without a GPU? HAMi's real device plugin talks to the NVIDIA driver, so it can't start without hardware. The mock device plugin skips hardware discovery: you describe fake cards in a node annotation, and it registers gpumem and gpucores capacity from that. HAMi's webhook, scheduler, and quota logic then run exactly as they would on real nodes.
Two simulated setups keep tenants easy to tell apart:
The build has five steps; step 3 creates both tenants at once, steps 4–5 repeat per tenant. Plain kubectl talks to the management cluster; --kubeconfig tenant-a.kubeconfig talks to a tenant.
Step 1 — management cluster + cert-manager + MetalLB:
kind create cluster --name kamaji
kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.21.2/cert-manager.yaml
kubectl -n cert-manager wait pod --all --for=condition=Ready --timeout=180s
kubectl apply -f https://raw.githubusercontent.com/metallb/metallb/v0.16.1/config/manifests/metallb-frr-k8s.yaml
kubectl -n metallb-system wait pod --all --for=condition=Ready --timeout=180s
Then give MetalLB a pool from the kind network's IPv4 subnet (docker network inspect kind | grep Subnet), e.g. 172.18.255.200-172.18.255.250 with an L2Advertisement.
Step 2 — install Kamaji:
helm repo add clastix https://clastix.github.io/charts
helm upgrade --install kamaji clastix/kamaji \
--namespace kamaji-system --create-namespace \
--set 'resources=null' --version 0.0.0+latest
kubectl -n kamaji-system get pods
Step 3 — create the tenant control planes (TENANT_VERSION=v1.36.4), then steps 4–5 register each tenant's worker with HAMi's mock device plugin and deploy fractional-GPU workloads as smoke tests.
The split works because isolation and GPU efficiency no longer trade against each other. Tenants keep their own control plane (own API server, own namespaces, own RBAC, own quota) while sharing the physical accelerator fleet through HAMi's device slicing — and the whole thing can be prototyped on a laptop with no GPU at all, because the mock plugin makes the scheduling logic real even when the hardware isn't.
The Konnectivity tunnel between Kamaji's API server and HAMi's per-node webhook is the seam that must hold; this lab proves it works end to end before you spend money on real cards.
Kamaji + HAMi is a compelling answer to the control-plane-per-team versus weak-isolation dilemma — especially for platform teams that need tenant isolation and GPU utilization in the same cluster. And because HAMi's mock plugin runs the full scheduling path without hardware, you can validate the architecture cheaply before procurement.