gptq
Post-training 4-bit quantization for LLMs with minimal accuracy loss. Use for deploying large models (70B, 405B) on consumer GPUs, when you need 4× memory reduction with <2% perplexity degradation, or for faster inference (3-4× speedup) vs FP16. Integrates with transformers and PEFT for QLoRA fine-tuning.
npx skills add Orchestra-Research/AI-Research-SKILLs --skill gptq --agent claude-code
Same command for any agent — swap --agent for codex, cursor, copilot.
Weekly change comes from our own snapshots, not the repository page — it measures attention, not adoption.
# GPTQ (Generative Pre-trained Transformer Quantization) Post-training quantization method that compresses LLMs to 4-bit with minimal accuracy loss using group-wise quantization. ## When to use GPTQ **Use GPTQ when:** - Need to fit large models (70B+) on limited GPU memory - Want 4× memory reduction with <2% accuracy loss - Deploying on consumer GPUs (RTX 4090, 3090) - Need faster inference (3-4× speedup vs FP16) **Use AWQ instead when:** - Need slightly better accuracy (<1% loss) - Have newer GPUs (Ampere, Ada) - Want Marlin kernel support (2× faster on some GPUs) **Use bitsandbytes instead when:** - Need simple integration with transformers - Want 8-bit quantization (less compression, better quality) - Don't need pre-quantized model files ## Quick start ### Installation ```bash # Install AutoGPTQ pip install auto-gptq # With Triton (Linux only, faster) pip install auto-gptq[triton] # With CUDA extensions (faster) pip install auto-gptq --no-build-isolation # Full installation pip install auto-gptq transformers accelerate ``` ### Load pre-quantized model ```python from transformers import AutoTokenizer from auto_gptq import AutoGPTQForCausalLM # Load quantized model from HuggingFac
- When to use GPTQ
- Quick start
- Installation
- Load pre-quantized model
- Quantize your own model
- Group-wise quantization
- Quantization configurations
- Standard 4-bit (recommended)
- High accuracy (3-bit with larger groups)
- Maximum accuracy (4-bit with small groups)
- Kernel backends
- ExLlamaV2 (default, fastest)
- Marlin (Ampere+ GPUs)
- Triton (Linux only)
Install AutoGPTQ pip install auto-gptq With Triton (Linux only, faster) pip install auto-gptq[triton] With CUDA extensions (faster) pip install auto-gptq --no-build-isolation Full installation pip install auto-gptq transformers accelerate Find GPTQ models on HuggingFace
What does the gptq skill do?
Post-training 4-bit quantization for LLMs with minimal accuracy loss. Use for deploying large models (70B, 405B) on consumer GPUs, when you need 4× memory reduction with <2% perplexity degradation, or for faster inference (3-4× speedup) vs FP16. Integrates with transformers and PEFT for QLoRA fine-tuning.
How do I install it?
Run `npx skills add Orchestra-Research/AI-Research-SKILLs --skill gptq --agent claude-code` — it drops the skill into your project so the agent can pick it up. Swap the --agent value for codex, cursor or copilot if you use one of those.
Where does this skill come from?
From Orchestra-Research/AI-Research-SKILLs, a repository with 11,391 stars. We read it straight from the repository tree rather than a submitted listing, so what you see here is what is actually published.
Is a popular skill a good skill?
Not necessarily. Stars measure attention, not adoption — a repository can trend for a week and be abandoned. That is why we show the weekly change from our own snapshots next to the total, instead of a single flattering number.