quantizing-models-bitsandbytes
Quantizes LLMs to 8-bit or 4-bit for 50-75% memory reduction with minimal accuracy loss. Use when GPU memory is limited, need to fit larger models, or want faster inference. Supports INT8, NF4, FP4 formats, QLoRA training, and 8-bit optimizers. Works with HuggingFace Transformers.
npx skills add Orchestra-Research/AI-Research-SKILLs --skill bitsandbytes --agent claude-code
Same command for any agent — swap --agent for codex, cursor, copilot.
Weekly change comes from our own snapshots, not the repository page — it measures attention, not adoption.
# bitsandbytes - LLM Quantization ## Quick start bitsandbytes reduces LLM memory by 50% (8-bit) or 75% (4-bit) with <1% accuracy loss. **Installation**: ```bash pip install bitsandbytes transformers accelerate ``` **8-bit quantization** (50% memory reduction): ```python from transformers import AutoModelForCausalLM, BitsAndBytesConfig config = BitsAndBytesConfig(load_in_8bit=True) model = AutoModelForCausalLM.from_pretrained( "meta-llama/Llama-2-7b-hf", quantization_config=config, device_map="auto" ) # Memory: 14GB → 7GB ``` **4-bit quantization** (75% memory reduction): ```python config = BitsAndBytesConfig( load_in_4bit=True, bnb_4bit_compute_dtype=torch.float16 ) model = AutoModelForCausalLM.from_pretrained( "meta-llama/Llama-2-7b-hf", quantization_config=config, device_map="auto" ) # Memory: 14GB → 3.5GB ``` ## Common workflows ### Workflow 1: Load large model in limited GPU memory Copy this checklist: ``` Quantization Loading: - [ ] Step 1: Calculate memory requirements - [ ] Step 2: Choose quantization level (4-bit or 8-bit) - [ ] Step 3: Configure quantization - [ ] Step 4: Load and verify model ``` **Step 1: Calculate memory requirements** Estimate model memory: ``` FP16 me
- Quick start
- Common workflows
- Workflow 1: Load large model in limited GPU memory
- Workflow 2: Fine-tune with QLoRA (4-bit training)
- Workflow 3: 8-bit optimizer for memory-efficient training
- When to use vs alternatives
- Common issues
- Advanced topics
- Hardware requirements
- Resources
pip install bitsandbytes transformers accelerate pip install bitsandbytes transformers peft accelerate datasets Check CUDA version nvcc --version Install matching bitsandbytes pip install bitsandbytes --no-cache-dir
What does the quantizing-models-bitsandbytes skill do?
Quantizes LLMs to 8-bit or 4-bit for 50-75% memory reduction with minimal accuracy loss. Use when GPU memory is limited, need to fit larger models, or want faster inference. Supports INT8, NF4, FP4 formats, QLoRA training, and 8-bit optimizers. Works with HuggingFace Transformers.
How do I install it?
Run `npx skills add Orchestra-Research/AI-Research-SKILLs --skill bitsandbytes --agent claude-code` — it drops the skill into your project so the agent can pick it up. Swap the --agent value for codex, cursor or copilot if you use one of those.
Where does this skill come from?
From Orchestra-Research/AI-Research-SKILLs, a repository with 11,391 stars. We read it straight from the repository tree rather than a submitted listing, so what you see here is what is actually published.
Is a popular skill a good skill?
Not necessarily. Stars measure attention, not adoption — a repository can trend for a week and be abandoned. That is why we show the weekly change from our own snapshots next to the total, instead of a single flattering number.