model-serving
LLM and ML model deployment for inference. Use when serving models in production, building AI APIs, or optimizing inference. Covers vLLM (LLM serving), TensorRT-LLM (GPU optimization), Ollama (local), BentoML (ML deployment), Triton (multi-model), LangChain (orchestration), LlamaIndex (RAG), and streaming patterns.
npx skills add majiayu000/claude-skill-registry --skill model-serving-ancoleman-ai-design-components --agent claude-code
Same command for any agent — swap --agent for codex, cursor, copilot.
Weekly change comes from our own snapshots, not the repository page — it measures attention, not adoption.
# Model Serving ## Purpose Deploy LLM and ML models for production inference with optimized serving engines, streaming response patterns, and orchestration frameworks. Focuses on self-hosted model serving, GPU optimization, and integration with frontend applications. ## When to Use - Deploying LLMs for production (self-hosted Llama, Mistral, Qwen) - Building AI APIs with streaming responses - Serving traditional ML models (scikit-learn, XGBoost, PyTorch) - Implementing RAG pipelines with vector databases - Optimizing inference throughput and latency - Integrating LLM serving with frontend chat interfaces ## Model Serving Selection ### LLM Serving Engines **vLLM (Recommended Primary)** - PagedAttention memory management (20-30x throughput improvement) - Continuous batching for dynamic request handling - OpenAI-compatible API endpoints - Use for: Most self-hosted LLM deployments **TensorRT-LLM** - Maximum GPU efficiency (2-8x faster than vLLM) - Requires model conversion and optimization - Use for: Production workloads needing absolute maximum throughput **Ollama** - Local development without GPUs - Simple CLI interface - Use for: Prototyping, laptop development, educational purposes
- Purpose
- When to Use
- Model Serving Selection
- LLM Serving Engines
- ML Model Serving (Non-LLM)
- LLM Orchestration
- Quick Start Examples
- vLLM Server Setup
- Streaming Responses (SSE Pattern)
- BentoML Service
- LangChain RAG Pipeline
- Performance Optimization
- GPU Memory Estimation
- vLLM Optimization
Install pip install vllm Serve a model (OpenAI-compatible API) vllm serve meta-llama/Llama-3.1-8B-Instruct \ Enable quantization (AWQ for 4-bit) vllm serve TheBloke/Llama-3.1-8B-AWQ \ Multi-GPU deployment (tensor parallelism) vllm serve meta-llama/Llama-3.1-70B-Instruct \ python scripts/benchmark_inference.py \
What does the model-serving skill do?
LLM and ML model deployment for inference. Use when serving models in production, building AI APIs, or optimizing inference. Covers vLLM (LLM serving), TensorRT-LLM (GPU optimization), Ollama (local), BentoML (ML deployment), Triton (multi-model), LangChain (orchestration), LlamaIndex (RAG), and streaming patterns.
How do I install it?
Run `npx skills add majiayu000/claude-skill-registry --skill model-serving-ancoleman-ai-design-components --agent claude-code` — it drops the skill into your project so the agent can pick it up. Swap the --agent value for codex, cursor or copilot if you use one of those.
Where does this skill come from?
From majiayu000/claude-skill-registry, a repository with 534 stars. We read it straight from the repository tree rather than a submitted listing, so what you see here is what is actually published.
Is a popular skill a good skill?
Not necessarily. Stars measure attention, not adoption — a repository can trend for a week and be abandoned. That is why we show the weekly change from our own snapshots next to the total, instead of a single flattering number.