How AI workloads are reshaping Kubernetes autoscaling
~4 min read
devopscloud-nativekubernetesautoscalinggpukarpenterai
Kubernetes has quietly run much of the world's containerized software on a division of labor between two control loops that rarely had to talk to each other. The scheduler places each pod on a node that already exists; the cluster autoscaler adds nodes when pods have nowhere to fit.
For a decade that separation was sound: compute was largely interchangeable, and either loop could work without knowing much about the other's decisions. AI workloads — and the accelerators they run on — changed the terms.
A pod that should schedule simply will not, even though the cluster is not truly out of capacity. The delay leaves expensive accelerator capacity idle. The instinct is to assume the region has run dry — and sometimes it has. In other cases the capacity is available and the pod still stays pending, because the node the autoscaler is willing to add is not one the scheduler is willing to use.
The autoscaler answers the question it was built for — whether to add a node — when the real question is whether that node is even the right shape for the work.
Several challenges accelerators bring share a thread: something both loops now have to understand the same way. Three show how far the shared answers have come.
Because supply is tight and idle accelerators are expensive, teams reserve capacity ahead of demand. When a scale-up draws from a reservation, the autoscaler must consider not just whether to grow but whether the resulting node uses capacity the team is already paying for. Launching against committed capacity rather than unreserved supply changes the cost calculus the autoscaler optimizes.
Topology is about how devices are wired to one another, not how far apart they sit. An accelerator leans on the parts that feed it — the CPU and memory in its local NUMA domain, the network interface sharing its PCIe path, the high-bandwidth links to its peer accelerators. It runs best when those pieces share locality, so traffic doesn't cross a slower hop.
That locality is a hierarchy: devices on the same bus inside a server, servers in a rack joined by a fast accelerator fabric, racks linked across a data-center network. The largest models span many servers at once, so "keep these close" reaches beyond a single node.
Historically Kubernetes represented accelerator resources as bare counts: a node advertised a number, a pod asked for some, and the scheduler matched two integers. Dynamic Resource Allocation (DRA) supplies what was missing — a workload files a structured claim describing what it needs, including relationships like an accelerator together with the network interface that feeds it. The scheduler reads claims directly, and DRA-aware autoscalers are starting to use the same information when they simulate capacity, though support still varies by implementation.
A DRA-aware autoscaler can account for the constraints the workload declared. Where it can't, the split between scheduling and scaling produces exactly the failure mode above — the autoscaler provisions capacity that the scheduler then refuses, or under-provisions because it can't see what the claim really requires.
The trajectory is clear: the cluster autoscaler is growing from a "how many nodes" answerer into a scheduler-aware partner that understands shape — reservations, interconnect topology, and structured device claims — before spending money on capacity.
AI didn't break the scheduler/autoscaler split; it exposed where the abstraction was leaky all along. Reservations, topology, and DRA are the three fronts where both loops are learning to share the same model of the machine. If you run accelerators on Kubernetes, this is where the interesting capacity-management work will happen over the next releases — and where common pain like "pending pods on a healthy cluster" gets its real fix. Watch the DRA ecosystem and autoscaler implementations that can simulate topology-aware capacity.