gguf-quantization
GGUF format and llama.cpp quantization for efficient CPU/GPU inference. Use when deploying models on consumer hardware, Apple Silicon, or when needing flexible quantization from 2-8 bit without GPU requirements.
npx skills add Orchestra-Research/AI-Research-SKILLs --skill gguf --agent claude-code
Same command for any agent — swap --agent for codex, cursor, copilot.
Weekly change comes from our own snapshots, not the repository page — it measures attention, not adoption.
# GGUF - Quantization Format for llama.cpp The GGUF (GPT-Generated Unified Format) is the standard file format for llama.cpp, enabling efficient inference on CPUs, Apple Silicon, and GPUs with flexible quantization options. ## When to use GGUF **Use GGUF when:** - Deploying on consumer hardware (laptops, desktops) - Running on Apple Silicon (M1/M2/M3) with Metal acceleration - Need CPU inference without GPU requirements - Want flexible quantization (Q2_K to Q8_0) - Using local AI tools (LM Studio, Ollama, text-generation-webui) **Key advantages:** - **Universal hardware**: CPU, Apple Silicon, NVIDIA, AMD support - **No Python runtime**: Pure C/C++ inference - **Flexible quantization**: 2-8 bit with various methods (K-quants) - **Ecosystem support**: LM Studio, Ollama, koboldcpp, and more - **imatrix**: Importance matrix for better low-bit quality **Use alternatives instead:** - **AWQ/GPTQ**: Maximum accuracy with calibration on NVIDIA GPUs - **HQQ**: Fast calibration-free quantization for HuggingFace - **bitsandbytes**: Simple integration with transformers library - **TensorRT-LLM**: Production NVIDIA deployment with maximum speed ## Quick start ### Installation ```bash # Clone lla
- When to use GGUF
- Quick start
- Installation
- Convert model to GGUF
- Quantize model
- Run inference
- Quantization types
- K-quant methods (recommended)
- Legacy methods
- Conversion workflows
- Workflow 1: HuggingFace to GGUF
- Workflow 2: With importance matrix (better quality)
- Workflow 3: Multiple quantizations
- Python usage
Clone llama.cpp git clone https://github.com/ggml-org/llama.cpp cd llama.cpp Build (CPU) make Build with CUDA (NVIDIA) make GGML_CUDA=1 Build with Metal (Apple Silicon) make GGML_METAL=1 Install Python bindings (optional)
What does the gguf-quantization skill do?
GGUF format and llama.cpp quantization for efficient CPU/GPU inference. Use when deploying models on consumer hardware, Apple Silicon, or when needing flexible quantization from 2-8 bit without GPU requirements.
How do I install it?
Run `npx skills add Orchestra-Research/AI-Research-SKILLs --skill gguf --agent claude-code` — it drops the skill into your project so the agent can pick it up. Swap the --agent value for codex, cursor or copilot if you use one of those.
Where does this skill come from?
From Orchestra-Research/AI-Research-SKILLs, a repository with 11,391 stars. We read it straight from the repository tree rather than a submitted listing, so what you see here is what is actually published.
Is a popular skill a good skill?
Not necessarily. Stars measure attention, not adoption — a repository can trend for a week and be abandoned. That is why we show the weekly change from our own snapshots next to the total, instead of a single flattering number.