agentsop-vllm
Decision SOP for serving LLMs with vLLM. Covers PagedAttention mental model, quantization/parallelism/batching tradeoffs, OOM triage, and when NOT to use vLLM. Activates when a coder-agent is choosing or tuning an inference engine, debugging vLLM throughput/latency/OOM, or comparing vLLM against TGI/SGLang/TensorRT-LLM/llama.cpp.
npx skills add agentsope/SkillAlchemy --skill agentsop-vllm --agent claude-code
Same command for any agent — swap --agent for codex, cursor, copilot.
Weekly change comes from our own snapshots, not the repository page — it measures attention, not adoption.
What it does
Guides decision-making for serving LLMs with vLLM, including when to activate, core mental models (KV cache as virtual memory, continuous batching, prefill vs decode), throughput levers, and a workflow for configuration and benchmarking.
How it works
- Activation triggers: production-grade serving with GPU, comparing engines, vLLM operation issues (OOM, throughput, TTFT), hardware sizing, and prefixes like PagedAttention, prefix caching, continuous batching, chunked prefill, or speculative decoding.
- Core models: describes KV-cache paging with blocks, block tables, and copy-on-write/prefix sharing; explains continuous batching reusing slots per decode step; differentiates prefill vs decode behavior; lists six throughput levers in order of impact (model sizing, hardware scaling, quantization, speculative decoding, parallelism refinement).
- SOP workflow: Step-by-step guidance for selecting model/precision, parallelism, memory envelopes, enabling prefix caching, triaging performance, deciding on speculative decoding, benchmarking, and scaling out.
- Operation model: includes concrete operational steps for diagnose OOM at startup, pick quantization, configure tensor/pipeline parallelism, enable prefix caching, triage performance, and decide on speculative decoding, with explicit outputs and evidence citations.
When to use it
Activates when choosing or tuning an inference engine, debugging vLLM throughput/latency/OOM, or comparing vLLM against other engines.
What it can touch
- Tools referenced: kv cache, flag-like parameters such as --max-model-len, --gpu-memory-utilization, --kv-cache-dtype, --max-num-seqs, --enable-prefix-caching, --enable-prefix-caching, --speculative-config. The workflow and actions presume interaction with vLLM configuration and runtime diagnostics; specific tooling is described via commands and flags in the SOP.
Caveats
- Contains model-specific and vendor-specific guidance; outcomes depend on workload and hardware. The guidance includes cautions about memory usage, latency components, and the need for benchmarking on representative traffic. No guarantees of performance are stated beyond the described steps and outputs; uses cited sources for evidence.
# vLLM Serving SOP ## 1. 何时激活 (When to activate) Activate this skill when any of the following hold: - The user wants to **serve an LLM in production** (multi-user, concurrent requests, throughput-oriented) and has GPU infrastructure. - The user is **comparing inference engines** (vLLM vs TGI vs SGLang vs TensorRT-LLM vs llama.cpp/Ollama). - The user reports a **vLLM operational issue**: CUDA OOM, low throughput, high TTFT, request preemption, multi-GPU setup, quantization choice. - The user is **sizing hardware** for an open-weights model (Llama / Qwen / Mixtral / DeepSeek-V3) and asking about tensor/pipeline parallelism. - The user mentions PagedAttention, prefix caching, continuous batching, chunked prefill, or speculative decoding. **Do NOT activate** for: training/fine-tuning (use accelerate/deepspeed/trl), CPU-only edge inference (use llama.cpp/Ollama), Apple Silicon production (vLLM Metal/MPS is experimental, not production-ready as of 2026) [aimadetools.com 2026], API-only consumption of hosted models (just call the OpenAI/Anthropic SDK). --- ## 2. 核心心智模型 (Core Mental Model) ### 2.1 The OS abstraction: KV cache as virtual memory vLLM's defining insight (Kwon et al., SOSP 20
- 1. 何时激活 (When to activate)
- 2. 核心心智模型 (Core Mental Model)
- 2.1 The OS abstraction: KV cache as virtual memory
- 2.2 The scheduler: iteration-level (continuous) batching
- 2.3 Prefill vs decode are different beasts
- 2.4 Three throughput levers, in order of impact
- 3. SOP 工作流 (SOP Workflow)
- 4. 操作模型 (Operation Model: Trigger / Action / Output / Evidence)
- OP-1: Diagnose vLLM CUDA OOM at startup
- OP-2: Pick quantization for an open-weights model
- OP-3: Configure tensor/pipeline parallelism
- OP-4: Enable prefix caching
- OP-5: Triage live performance (TTFT vs ITL)
- OP-6: Decide on speculative decoding
vllm serve meta-llama/Llama-3.1-70B-Instruct \
What does the agentsop-vllm skill do?
Decision SOP for serving LLMs with vLLM. Covers PagedAttention mental model, quantization/parallelism/batching tradeoffs, OOM triage, and when NOT to use vLLM. Activates when a coder-agent is choosing or tuning an inference engine, debugging vLLM throughput/latency/OOM, or comparing vLLM against TGI/SGLang/TensorRT-LLM/llama.cpp.
How do I install it?
Run `npx skills add agentsope/SkillAlchemy --skill agentsop-vllm --agent claude-code` — it drops the skill into your project so the agent can pick it up. Swap the --agent value for codex, cursor or copilot if you use one of those.
Where does this skill come from?
From agentsope/SkillAlchemy, a repository with 255 stars. We read it straight from the repository tree rather than a submitted listing, so what you see here is what is actually published.
Is a popular skill a good skill?
Not necessarily. Stars measure attention, not adoption — a repository can trend for a week and be abandoned. That is why we show the weekly change from our own snapshots next to the total, instead of a single flattering number.