agentsop-llm-engine-selection
Cross-engine decision rubric for self-hosting or recommending an LLM serving stack. Picks among vLLM, SGLang, TensorRT-LLM, TGI, llama.cpp, Ollama, and MLX as a function of (hardware × workload × constraint), not "which is fastest". Activates whenever a coder-agent must choose, defend, or migrate a serving runtime.
npx skills add agentsope/SkillAlchemy --skill agentsop-llm-engine-selection --agent claude-code
Same command for any agent — swap --agent for codex, cursor, copilot.
Weekly change comes from our own snapshots, not the repository page — it measures attention, not adoption.
What it does
The skill provides a cross-engine decision rubric to select an LLM serving stack from vLLM, SGLang, TensorRT-LLM, TGI, llama.cpp, Ollama, and MLX. It activates when a coder-agent must choose, defend, or migrate a serving runtime, and it explicitly prioritizes a multi-axis fit over a simple speed ranking.
How it works
- Step 0 defines a four-axis constraint vector: Hardware, Workload, Constraint, and Model.
- Step 1 applies hard filters to eliminate incompatible engines based on hardware, OSS requirements, and problem class.
- Step 2 maps workload characteristics to engine strengths (e.g., throughput with mixed traffic → vLLM; structured output → SGLang; NVIDIA-only peak throughput → TensorRT-LLM).
- Step 3 sanity-checks hardware topology (e.g., NVLink vs PCIe) and advises topology-related adjustments.
- Step 4 benchmarks top candidates on representative workload metrics (P50/P99 TTFT, ITL, tokens/s/GPU, cost, memory, load time) when close.
- Step 5 plans escape hatches and re-evaluation cadence (every 6 months).
- The workflow includes many decision cues, such as handling multi-tier deployments, quantization formats, edge/Apple Silicon routing, and structured-output routing with SGLang as primary for constrained decoding.
When to use it
Activate this skill any time a coder-agent must pick a serving stack for a new project, defend an existing stack, migrate between engines, design multi-tier deployments, or audit recommendations that smell like benchmark cherry-picking.
What it can touch
- It references and implies use of the engines (vLLM, SGLang, TensorRT-LLM, TGI, llama.cpp, Ollama, MLX) and related quantization and topology concepts. It also mentions metrics and benchmarks like P50/P99 TTFT, P50/P99 ITL, tokens/s/GPU, and $/M-tokens, but does not itself perform them.
Caveats
- The skill documents an opinionated multi-axes framework rather than a single universal rule. It warns that engines’ relative advantages depend on hardware topology, workload mix, and constraints, and emphasizes re-verification as the landscape evolves. It notes specific historical shifts and maintenance/compatibility caveats for each engine, and it requires up-to-date verification before committing to a choice.
# LLM Engine Selection SOP > **State-of-the-art warning.** This skill is dated **May 2026**. The inference-engine landscape moves in 3–6 month cycles (TGI exited maintenance into deprecation in late 2025; SGLang's RadixAttention regressed vLLM's lead in 2024; TensorRT-LLM dropped its proprietary-engine requirement in 2025; Ollama added concurrent-request support mid-2025). Re-verify before betting a quarter of eng budget on any choice below. --- ## 1. 何时激活 (When to activate) Activate this skill any time a coder-agent must: - **Pick** a serving stack for a new project (production hosting / batch / edge / dev laptop / multi-tenant SaaS / structured-output service). - **Defend** an existing stack against a "let's switch to X" pressure. - **Migrate**: justify or block a swap (e.g. TGI → vLLM, Ollama → vLLM, vLLM → TensorRT-LLM). - **Mix**: design a multi-tier deployment (e.g. premium tier on TensorRT-LLM, free tier on vLLM-AWQ, dev on Ollama). - **Audit** a recommendation that smells like benchmark-cherry-picking ("X is 5× faster"). Do **not** activate for: - Tuning a single chosen engine — defer to the dedicated skill (`vllm`, `sglang`, `tensorrt-llm`, `llama-cpp`). - Training/fine-tu
- 1. 何时激活 (When to activate)
- 2. 核心心智模型 (Core Mental Model)
- 2.1 The four-axis decision space
- 2.2 The default in 2026
- 2.3 "Fastest" is a category error
- 3. SOP 工作流 (SOP Workflow)
- 4. 操作模型 (Operation Model)
- OP-1: Decision matrix lookup by (hardware, workload)
- OP-2: Detect "wrong-tool-for-the-job" symptoms
- OP-3: Quantization-driven engine pick
- OP-4: Structured-output workload routing
- OP-5: Apple Silicon / edge routing
- OP-6: Multi-tier deployment design
- OP-7: Bench-then-pick when the choice is close
What does the agentsop-llm-engine-selection skill do?
Cross-engine decision rubric for self-hosting or recommending an LLM serving stack. Picks among vLLM, SGLang, TensorRT-LLM, TGI, llama.cpp, Ollama, and MLX as a function of (hardware × workload × constraint), not "which is fastest". Activates whenever a coder-agent must choose, defend, or migrate a serving runtime.
How do I install it?
Run `npx skills add agentsope/SkillAlchemy --skill agentsop-llm-engine-selection --agent claude-code` — it drops the skill into your project so the agent can pick it up. Swap the --agent value for codex, cursor or copilot if you use one of those.
Where does this skill come from?
From agentsope/SkillAlchemy, a repository with 255 stars. We read it straight from the repository tree rather than a submitted listing, so what you see here is what is actually published.
Is a popular skill a good skill?
Not necessarily. Stars measure attention, not adoption — a repository can trend for a week and be abandoned. That is why we show the weekly change from our own snapshots next to the total, instead of a single flattering number.