llm-d 1.0 ships: the Kubernetes-native open-source distributed inference stack led by Red Hat, IBM and Google hits production readiness, with vendors reporting 3,000 self-hosted coding agents at 5-10x lower cost than commercial APIs

Background and the problem it targets

Inference traffic is shifting from one-prompt-one-answer chat to multi-step agents and RL rollouts that resend long contexts at every step and arrive in bursts. Behind a single machine or a round-robin load balancer, replicas recompute the same prompts, accelerator memory fills with duplicate state, and one bursty agent loop can stall interactive users. llm-d targets this cluster-level scheduling problem for teams serving their own models.

Architecture and how it works

llm-d does not replace the inference engine; it sits on top of vLLM and SGLang. Its router tracks which replica holds which cached prefix and weighs load and predicted latency; KV cache can be offloaded from accelerator memory to host RAM and then SSD; prefill and decode run as separate pools, with wide expert parallelism for large MoE models. Version 1.0 focuses on Day 2 operations: flow control queues and sheds load by priority and fairness, the router gets hot-standby HA and zero-downtime upgrades, DisaggregatedSet rolls prefill and decode together, the config API graduates to v1, and session-aware routing plus Responses API state arrive for agents.

Benchmarks and measured results

All figures are project or vendor-reported, not independent: a GLM 5.2 deployment peaked at 6.6M output tokens per minute across 3,000 concurrent coding agents at 5-10x lower cost than commercial APIs; on Ironwood TPU median time-to-first-token fell more than 85% with 2.4x throughput; IBM says the router captured 97% of available cache reuse. The GitHub release notes also admit not every tracked goal was completed.

Getting started

1. Teams already running vLLM can start by adding only llm-d's prefix-aware routing in front, then enable flow control, KV offload and prefill/decode disaggregation step by step.
2. GKE users can deploy Google's verified Helm charts or run the router through the managed GKE Inference Gateway.
3. Use the project's Prism dashboard and llm-d-benchmark recipes to reproduce throughput, latency and cost curves on your own cluster before committing.