The llm-d project community has announced the release of version 1.0, marking a significant milestone for the open-source initiative designed to help organizations efficiently deploy and scale large language model (LLM) inference on modern infrastructure. Founded in 2025 by Red Hat, IBM, Google Cloud, CoreWeave, and NVIDIA, and now managed by the Cloud Native Computing Foundation (CNCF), llm-d plays a crucial role in building an open platform for enterprises deploying AI at production scale.
As organizations strive to operationalize AI, many find that managing AI at scale can be as complex and costly as developing the models themselves. Businesses need a way to ensure fast user response times, support agentic workloads, and maximize the utilization of increasingly scarce compute resources. llm-d 1.0 is engineered to address these challenges with intelligent, distributed inference capabilities that drive improved performance, increased throughput, and help organizations gain better control over AI serving economics.
Transforming Innovation into Production Reality
The emergence of agentic AI introduces new workload types that place unique demands on infrastructure systems. Agentic workloads constantly access and reuse vast amounts of context while simultaneously supporting parallel agents that can generate unpredictable bursts of activity. Managing this context often consumes more resources than the final generation itself, making latency, throughput, memory efficiency, and infrastructure costs critical factors for achieving AI ROI. llm-d already provides hardware heterogeneity and advanced inference capabilities out-of-the-box for complex infrastructure environments, along with prefix caching, workload-aware routing, KV cache offloading, prefix generation and decoding splitting, wide expert parallelism, and batch/asynchronous inference – across multiple model servers, including vLLM and SGLang.
llm-d 1.0 is built to deliver this performance in the realities of “Day 2” operations and continuous production inference: serving multiple tenants, balancing competing priorities, enabling zero-downtime features, and supporting new agentic workloads – all within a single set of co-developed components that compose a platform. New capabilities in version 1.0 include:
- Quality of Service for Multi-Tenancy: Priority and fairness-aware flow control helps latency-sensitive requests and bursty agentic workloads share a fleet without mutual preemption.
- Zero-Downtime Operations: Highly available hot-standby routers and rolling updates ensure uninterrupted traffic during upgrades and planned maintenance. Managing replicas of prefix generation/decoding as a single logical unit with coordinated deployments and updates in sync makes split inference practical in production.
- Stable Foundation to Build On: A stable v1 configuration API and a battle-tested request processing path across clouds and accelerators provide users with a release they can rely on.
- Agent-Aware Serving: Session-aware routing keeps multi-turn conversations close to their cached context, while turn-priority scheduling and native API response state support help agents reuse context across turns and reduce recomputation.
The result is an infrastructure capable of supporting diverse AI workloads, from latency-sensitive user interactions to large-scale autonomous agent workflows, without forcing organizations to choose between responsiveness and efficiency.
Growing Ecosystem Momentum
llm-d is already seeing adoption by a growing ecosystem of model developers, cloud providers, infrastructure vendors, and enterprise users. To date, the project boasts over 500 contributors from more than 25 organizations collaborating on the future of distributed AI inference and helping to build shared foundations for the entire AI infrastructure stack.
Key users, including organizations engaged in large-scale distributed inference, leading model developers optimizing inference stacks, and cloud providers serving multi-tenant AI endpoints, have already realized measurable benefits from deploying llm-d.
In one recent use case, open frontier models and llm-d enabled IBM Research to explore hardware for quantum error correction with inference costs 11.5x lower than a fully proprietary configuration. Researchers reported a prefix cache hit rate of approximately 98% and a total serving cost of around $0.11 per million tokens.
In a separate recent demonstration using standard workloads, researchers showed that llm-d can efficiently serve large mixture-of-experts models for agentic workloads. The researchers used llm-d to deploy GLM 5.2, an open-weight model of approximately 753 billion parameters, on widely available NVIDIA GPUs, demonstrating how enterprises can leverage existing GPU fleets to scale frontier AI models. On a workload with hundreds of concurrently running agent sessions, the deployment delivered over 6.6 million output tokens per minute at peak on agent tests, serving up to 3,000 simultaneous coding agents without interruption. At current cloud resource rental rates, this deployment resulted in costs that would be 5-10x lower per token than an equivalent commercial API price, with the largest savings on the data-intensive input patterns typical for agentic workloads, underscoring how llm-d can save resources and costs for enterprises deploying AI in production at scale.
With its CNCF governance model and broad industry participation, llm-d aims to accelerate innovation while providing organizations with an open alternative for managing increasingly complex AI infrastructure. Looking ahead, the community will continue to adapt to new hardware and models, evolving the software stack to achieve peak performance on novel architectures. To learn more, check out the announcement blogs from IBM and Red Hat.
Supporting Quotes
Chris Aniszczyk, CTO, CNCF:
“AI inference is rapidly becoming a primary workload in cloud native. llm-d 1.0 helps organizations meet this shift at scale by bringing together open-source inference capabilities with the production scaling requirements that matter, and then addressing the needs of agentic workloads. This is an important step toward making high-performance open-source inference more operationally mature and accessible across the cloud native ecosystem.”
Bryan Stevens, Senior Vice President and Chief Technology Officer, AI, Red Hat:
“As organizations move beyond AI experimentation and into full-scale production, the bottleneck shifts from response generation to managing economics, operational reliability, and the extreme infrastructure demands of complex agentic workloads. Open source collaboration is critical to solving these ‘Day 2’ operational challenges at scale. With llm-d 1.0, the community is delivering a standardized, secure core for distributed inference to empower enterprises to optimize hardware performance, address operational costs, and build high-density AI architectures on an open foundation.”
Priya Nagpurkar, Vice President, AI native systems, IBM Research:
“At IBM, we see firsthand that the challenge of enterprise AI is no longer just about building powerful models, but about scaling them efficiently and reliably. llm-d 1.0 is a critical step in building an open foundation for distributed inference to help organizations get more value from their AI infrastructure and more control over the AI capabilities they depend on. As agentic workloads become increasingly prevalent, open technologies like llm-d will be crucial for improving performance, reducing costs, and accelerating innovation across the AI ecosystem.”
Mark Lohmeyer, Vice President and General Manager, AI & Computing Infrastructure, Google Cloud:
“Agentic AI unlocks tremendous value for enterprises and consumers across a wide array of use cases. However, as these workloads rapidly scale in production, they introduce unique infrastructure requirements and demand new approaches to deliver the performance, reliability, and cost-efficiency our customers need. The release of llm-d 1.0 is a significant industry milestone in addressing these needs. As a founding member of llm-d, Google Cloud is integrating llm-d 1.0 into Google Kubernetes Engine and deploying it across our Google TPU and NVIDIA GPU portfolios, enabling customers to scale multi-tenant environments, optimize inference performance, and significantly reduce their total cost of serving on a secure, open foundation.”
Shadi Saba, VP of Engineering, AI/ML, CoreWeave:
“Every team scaling inference faces the same challenges: context to manage is often more expensive than the output, tenants competing for the same fleet, and upgrades that cannot interrupt traffic. llm-d 1.0 provides the community with a secure, open foundation to solve them once, not separately. CoreWeave helped found the project, and customers can run llm-d on CoreWeave Kubernetes Service alongside NVIDIA Dynamo, choosing a serving stack that fits their workloads.”
Ujjwal Kapasi, VP, Engineering AI Frameworks, NVIDIA:
“The next wave of agentic AI requires inference that delivers performance and efficiency at scale. The addition of llm-d to NVIDIA Vera Rubin will help enterprises build and scale leading AI applications on NVIDIA’s next-generation accelerated infrastructure. Through our collaboration with the llm-d community and contributions from NVIDIA Dynamo open source, we are helping enterprises get more from their AI infrastructure investments.”
Phil Guido, Executive Vice President and Chief Commercial Operations Officer, AMD:
“With llm-d v1.0, we are advancing open, high-performance distributed inference on AMD Instinct GPUs. Agentic workloads require infrastructure that delivers responsiveness and efficiency. By collaborating with the llm-d community, we are delivering the latest inference innovations on an open foundation, providing developers with greater flexibility and choice to deploy AI at scale.”
Assir Locatelli, VP, Modeling, Cohere:
“At Cohere, we face the dual challenge of training frontier language models and efficiently serving them at enterprise scale. llm-d helps us optimize both, whether it’s maximizing throughput for large-scale reinforcement learning workloads or delivering the responsiveness users expect in production. Its flexible, open-source architecture aligns closely with our commitment to sovereign AI, giving organizations greater control over how AI is deployed, operated, and scaled.”
Salman Paracha, SVP, AI Engineering, DigitalOcean:
“DigitalOcean is enthusiastic about the release of llm-d 1.0 and relies on llm-d as the distributed inference backbone for DigitalOcean Serverless Inference and Managed Agents, serving trillions of tokens. Its Kubernetes-native design and integration with Kubernetes Gateway API Inference Extensions provide robust orchestration and endpoint choice, while battle-tested paths for dynamic flow control, inference-aware load balancing, and prefix generation/decoding splitting help us operate reliably at scale. We are working closely with the llm-d community, contributing features and feedback at the core project level, and look forward to llm-d’s continued evolution.”
Vladislav Tankov, Director of AI, JetBrains:
“JetBrains has been working with llm-d since its early releases, and it’s great to see the project reach version 1.0. llm-d is now powering production for our new Junie Lite plan and serving a significant portion of traffic. Our traffic ranges from long coding agent sessions to short queries, and the predicted latency-based scheduling was easy to enable and maintains stable latency for this mix. Junie Lite provides our coding AI agent to developers for free, and llm-d helps make that possible by allowing us to serve more requests on GPUs with lower cost and faster responses. We have enjoyed working with the llm-d team and congratulate them on this release.”
Mathis Felardo, Lucas Fernandez-Martins, and Nikhil Thomas (Mistral AI, Inference Engineers and llm-d Contributors):
“Serving frontier models in high-traffic interactive applications and APIs means handling sharp traffic bursts without bursty workloads degrading latency for premium customers. With llm-d’s prefix-cache-aware routing, flow control, and disaggregated inference support, we can establish strict priority bands for our commercial tiers, automatically holding or dropping lower-priority requests when pools saturate, to keep paid traffic meeting our latency targets. Mistral is a proud supporter of open-source software, and we will continue to contribute to llm-d and DisaggregatedSet projects.”
Nishant Khanna, Sr. Director of Engineering, AI Platform, Roblox:
“At Roblox, efficiently serving large language models across a hybrid of cloud and on-prem GPUs is one of our core infrastructure challenges. llm-d provides us with a Kubernetes-native foundation for disaggregated inference and cache-aware routing on hardware we already use, without locking us into a single inference engine. We are excited to build on llm-d 1.0 and contribute back to the community.”
