tensorrt-llm
Optimizes LLM inference with NVIDIA TensorRT for maximum throughput and lowest latency. Use for production deployment on NVIDIA GPUs (A100/H100), when you need 10-100x faster inference than PyTorch, or for serving models with quantization (FP8/INT4), in-flight batching, and multi-GPU scaling.
npx skills add Orchestra-Research/AI-Research-SKILLs --skill tensorrt-llm --agent claude-code
Same command for any agent — swap --agent for codex, cursor, copilot.
Weekly change comes from our own snapshots, not the repository page — it measures attention, not adoption.
# TensorRT-LLM NVIDIA's open-source library for optimizing LLM inference with state-of-the-art performance on NVIDIA GPUs. ## When to use TensorRT-LLM **Use TensorRT-LLM when:** - Deploying on NVIDIA GPUs (A100, H100, GB200) - Need maximum throughput (24,000+ tokens/sec on Llama 3) - Require low latency for real-time applications - Working with quantized models (FP8, INT4, FP4) - Scaling across multiple GPUs or nodes **Use vLLM instead when:** - Need simpler setup and Python-first API - Want PagedAttention without TensorRT compilation - Working with AMD GPUs or non-NVIDIA hardware **Use llama.cpp instead when:** - Deploying on CPU or Apple Silicon - Need edge deployment without NVIDIA GPUs - Want simpler GGUF quantization format ## Quick start ### Installation ```bash # Docker (recommended) docker pull nvidia/tensorrt_llm:latest # pip install pip install tensorrt_llm==1.2.0rc3 # Requires CUDA 13.0.0, TensorRT 10.13.2, Python 3.10-3.12 ``` ### Basic inference ```python from tensorrt_llm import LLM, SamplingParams # Initialize model llm = LLM(model="meta-llama/Meta-Llama-3-8B") # Configure sampling sampling_params = SamplingParams( max_tokens=100, temperature=0.7, top_p=0.9 ) # Gener
- When to use TensorRT-LLM
- Quick start
- Installation
- Basic inference
- Serving with trtllm-serve
- Key features
- Performance optimizations
- Parallelism
- Advanced features
- Common patterns
- Quantized model (FP8)
- Multi-GPU deployment
- Batch inference
- Performance benchmarks
Docker (recommended) docker pull nvidia/tensorrt_llm:latest pip install pip install tensorrt_llm==1.2.0rc3 Requires CUDA 13.0.0, TensorRT 10.13.2, Python 3.10-3.12 Start server (automatic model download and compilation) trtllm-serve meta-llama/Meta-Llama-3-8B \ Client request curl -X POST http://localhost:8000/v1/chat/completions \
What does the tensorrt-llm skill do?
Optimizes LLM inference with NVIDIA TensorRT for maximum throughput and lowest latency. Use for production deployment on NVIDIA GPUs (A100/H100), when you need 10-100x faster inference than PyTorch, or for serving models with quantization (FP8/INT4), in-flight batching, and multi-GPU scaling.
How do I install it?
Run `npx skills add Orchestra-Research/AI-Research-SKILLs --skill tensorrt-llm --agent claude-code` — it drops the skill into your project so the agent can pick it up. Swap the --agent value for codex, cursor or copilot if you use one of those.
Where does this skill come from?
From Orchestra-Research/AI-Research-SKILLs, a repository with 11,391 stars. We read it straight from the repository tree rather than a submitted listing, so what you see here is what is actually published.
Is a popular skill a good skill?
Not necessarily. Stars measure attention, not adoption — a repository can trend for a week and be abandoned. That is why we show the weekly change from our own snapshots next to the total, instead of a single flattering number.