Nano-vLLM is a lightweight vLLM implementation in Python focused on fast offline inference, readable code, and optimization features. It provides a quick start API compatible with vLLM, a model download workflow, and basic benchmarking.
Collecting history — the radar snapshots this repo daily. The trend line appears after 3 days of data (1 so far).
What it is
Nano-vLLM is a lightweight vLLM implementation built from scratch in Python, aiming for fast offline inference and readable code (approximately 1,200 lines of Python code). It includes an optimization suite with features like prefix caching, tensor parallelism, Torch compilation, and CUDA graph support.
How it works
The repository presents an API that mirrors vLLM's interface with minor differences in the LLM.generate method. It supports an LLM class initialized with a model path and options (eager enforcement and tensor_parallel_size). The included benchmark section demonstrates usage and provides a comparison of inference performance against vLLM.
Getting started
Installation:
pip install git+https://github.com/GeeeekExplorer/nano-vllm.git
Model download (manual):
huggingface-cli download --resume-download Qwen/Qwen3-0.6B \
--local-dir ~/huggingface/Qwen3-0.6B/ \
--local-dir-use-symlinks False
Quick start example:
from nanovllm import LLM, SamplingParams
llm = LLM("/YOUR/MODEL/PATH", enforce_eager=True, tensor_parallel_size=1)
sampling_params = SamplingParams(temperature=0.6, max_tokens=256)
prompts = ["Hello, Nano-vLLM."]
outputs = llm.generate(prompts, sampling_params)
outputs[0]["text"]
Recent releases
The RELEASES section shows the latest release as 0 with no entries listed under it; no additional release notes are provided in the README.
Traction
Stars: 14814 (as reported in the repository metadata).
Behind the repo
The repository is maintained under the GitHub user/org GeeeekExplorer and focuses on deep-learning, inference, LLMs, NLP, PyTorch, and transformer-related topics.
Caveats
License: MIT Created: 2025-06-09 Last push: 2026-04-26 Open issues: 80 Other notes: The README includes example commands and a benchmarking section, but does not list additional release notes beyond the basic usage and benchmark.