llm-d is a distributed inference serving stack for Kubernetes, focusing on intelligent routing, KV-cache management, and batch/offload optimizations to improve throughput and latency for large language models. It targets accelerator-based deployments with production-ready guides and benchmarks.
Collecting history — the radar snapshots this repo daily. The trend line appears after 3 days of data (1 so far).
What it is
llm-d is a high-performance distributed inference serving stack optimized for production deployments on Kubernetes. It aims to improve throughput and latency for OSS large language models across accelerators, using features like intelligent routing, advanced KV-cache management, and batch processing.
How it works
The project integrates with standard technologies such as vLLM and Kubernetes to orchestrate inference across multiple replicas and accelerators. It emphasizes routing optimizations with prefix-cache and load-aware balancing, tiered prefix cache offload, and disaggregated serving techniques to improve hardware utilization and performance. It provides guides and benchmarks to validate production-ready configurations.
Getting started
Read the Quickstart Guide to deploy the first optimized inference service on Kubernetes and validate performance with production-ready benchmarks. Most users begin with the Optimized Baseline for a high-performance foundation.
Recent releases
- v0.8.1 Release v0.8.1 (2026-06-26): LLM-D v0.8.0 Component Summary — Themes include CI coverage, accelerator coverage, multimodal/batch/flow-control production readiness, and initial RL support.
- v0.8.0 Release v0.8.0 (2026-06-24): LLM-D v0.8.0 Component Summary — Themes include CI coverage, accelerator coverage, production readiness for multimodal/batch/flow-control, and initial RL support.
- v0.7.0 Release v0.7.0 (2026-05-12): LLM-D Component Summary with a breaking change related to CUDA runtime updates.
- v0.6.0 Release v0.6.0 (2026-04-03): LLM-D Component Summary with version mappings for components and types.
- v0.5.1 Release v0.5.1 (2026-03-05): LLM-D Component Summary with component version history.
Traction
Stars: 3972; Forks: 649; Open issues: 239
License
Licensed under Apache-2.0.
Getting started (commands)
- Quickstart Guide link provided in docs: follow the Quickstart Guide to deploy your first optimized inference service on Kubernetes.