Announcing New vLLM and SGLang Inference Engines on TPU

Google Cloud now ships native TPU support for the two most popular open-source inference engines — vLLM and SGLang. The new engines cover the full serving workload spectrum, and several features address agentic workloads specifically.

What's supported

• Agentic & multi-turn — automatic prefix caching (including for hybrid models), KV cache offloading to host memory and Managed Lustre through TPU Sync, tool calling, and structured output.
• Interactive & low latency — speculative decoding with MTP, EAGLE-3, DFlash, DSpark, and n-gram drafters.
• High-throughput batch — data-parallel attention with expert parallelism.
• Long context — context parallelism for prefill and decode, FP8 KV cache, host KV offloading.
• Disaggregated serving — prefill and decode on separate hosts over TPU Sync, including across different parallel layouts.
• Multimodal — vision-language models with image/video input; embedding models.
• Very large models — hybrid and sparse-attention architectures; tensor, data, expert, pipeline, and context parallelism across hosts; FP8, FP4 (MXFP4, NVFP4), INT8, and INT4 checkpoints.

Why agents care about this

Agentic workloads demand several of these features simultaneously. Agents repeatedly resend expanding multi-turn conversations, so serving them efficiently requires finding the cached prefix instead of recomputing it. Both engines keep reusable prefixes in HBM and — with host offload enabled — automatically move older prefixes to host memory as a second tier. That's precisely the pattern that turns the repetitive reshuffling of agent turns from a recompute tax into a cache hit.

Bottom line

For teams running agent workloads or very large models, the combination of automatic prefix caching, KV offload across tiers, and disaggregated prefill/decode is what production serving looks like in 2026. If you've standardized on vLLM or SGLang, TPU is now a first-class target instead of a porting project.