SageAttention provides optimized kernels and quantization options for attention, with plug-and-play usage and multiple API variants. It targets CUDA GPUs and aims for speedups in end-to-end tasks without accuracy loss.
Collecting history — the radar snapshots this repo daily. The trend line appears after 3 days of data (1 so far).
What it is
SageAttention is an implementation of SageAttention, SageAttention2, and SageAttention2++ that aims to speed up attention on GPUs with maintained accuracy, in a plug-and-play way.
How it works
The repository exposes multiple attention variants and quantization strategies, including INT8 quantization for QK^T and FP16/FP8 for PV, FP16 accumulator for FP8/FP16 PV, two-level accumulation for PV, and support for torch.compile with non-cudagraphs mode and distributed inference. It provides optimized kernels for Ampere, Ada and Hopper GPUs and various backend options (Trtion, CUDA).
Getting started
Installation notes include base environment requirements and CUDA version guidelines:
- python>=3.9, torch>=2.3.0, triton>=3.0.0
- CUDA >=12.0 (Ampere) to enable certain features; higher versions for FP8 on specific GPUs
- flash-attn for benchmarking
Install options:
pip install sageattention==2.2.0 --no-build-isolation
Or from source:
git clone https://github.com/thu-ml/SageAttention.git
cd SageAttention
export EXT_PARALLEL=4 NVCC_APPEND_FLAGS="--threads 8" MAX_JOBS=32 # Optional
python setup.py install
For benchmarking FlashAttention3:
git clone https://github.com/Dao-AILab/flash-attention.git --recursive
git checkout b7d29fb3b79f0b78b1c369a52aaa6628dabfb0d7 # 2.7.2 release
cd hopper
python setup.py install
Usage examples:
from sageattention import sageattn
attn_output = sageattn(q, k, v, tensor_layout="HND", is_causal=False)
Available APIs include sageattn and variants for INT8/FP8 configurations on CUDA or Triton backends.
How to use (example)
import torch.nn.functional as F
+ from sageattention import sageattn
+ F.scaled_dot_product_attention = sageattn
Kernel and benchmarking
There are scripts and benchmarks in the repository to compare SageAttention with FlashAttention variants and to assess kernel performance.
Recent releases
Latest release: v2.0.1 (2025-01-28) SageAttention v2.0.1.
Traction
Stars: 3549
Caveats
License: Apache-2.0 Created: 2024-10-03 Last push: 2026-01-17 Notes:
- The README describes multiple papers and paper-forward references but does not provide detailed per-release feature lists beyond the general features and updates noted.
- Compatibility notes mention CUDA version requirements for different features and GPUs, and optional components like flash-attn for benchmarking.