The open-source llm-d project has reached version 1.0. The release focuses on the day-to-day management of AI inference in production: multi-tenancy, updates without downtime, and serving that takes agents into account. The project is now part of the CNCF and has more than 500 contributors.
llm-d was founded in 2025 by Red Hat, IBM, Google Cloud, CoreWeave, and Nvidia. It is a Kubernetes-native framework for distributed inference of LLMs. At its launch, Red Hat positioned the project as a way to scale vLLM beyond a single server. llm-d itself is not a model server. It distributes requests across vLLM or SGLang instances within a GPU cluster. Since last spring, the Cloud Native Computing Foundation has managed the project.
A number of features had already been in llm-d for some time, such as prefix cache and load-aware routing, KV cache offloading, prefill/decode disaggregation, and wide expert parallelism. With version 1.0, the focus shifts to so-called Day 2 operations, the work that begins once inference is up and running in production.
What’s new?
The first new feature is multi-tenant quality of service. Priority- and fairness-based flow control prevents latency-sensitive requests and spikes in agent traffic from crowding each other out on the same infrastructure. In addition, the router now offers high availability, with a hot standby and rolling updates. Prefill and decode replicas are now managed as a single logical unit and upgraded all at once.
We’re also introducing a stable v1 configuration API. And for agents, there’s session-aware routing: multi-turn conversations remain close to their cached context. Support for the Responses API should reduce how often context needs to be recalculated.
According to CNCF CTO Chris Aniszczyk, AI inference is rapidly becoming a core workload within cloud-native environments. He calls the release an important step toward making open-source inference more operationally mature.
The project members cite two examples. IBM Research used llm-d with open models to research hardware for quantum error correction. This was 11.5 times cheaper than a configuration using only proprietary components. The prefix cache hit rate was approximately 98 percent, at about $0.11 per million tokens.
In a benchmark, the mixture-of-experts model GLM 5.2, with approximately 753 billion parameters, ran on commonly used NVIDIA GPUs. At peak performance, it produced more than 6.6 million output tokens per minute, supporting up to 3,000 coding agents simultaneously and without preemptions. Based on current cloud pricing, this was 5 to 10 times cheaper per token than comparable commercial APIs.