Open Source Radar
Mixtral offloading implements efficient inference for Mixtral-8x7B models using mixed quantization and per-layer MoE offloading with an LRU cache to keep active experts on GPU. The project provides a demo notebook for Colab and desktops; no local CLI is available yet.
2.3kstars
225forks
26issues
MITlicense
2023since
Star historydaily snapshots by VibeCrowd
Collecting history — the radar snapshots this repo daily. The trend line appears after 3 days of data (1 so far).
Alternatives & relatedmatched by topic overlap
Reviewgenerated from repository data · Aug 5, 2026
What it is
This project implements efficient inference of Mixtral-8x7B models. It uses mixed quantization with HQQ and a MoE offloading strategy to fit the model into combined GPU and CPU memory, keeping active experts in an LRU cache.
How it works
- Mixed quantization with HQQ applies separate quantization schemes for attention layers and experts.
- MoE offloading stores each expert per layer separately and brings them back to GPU only when needed, using an LRU cache to reduce GPU-RAM communication during activations for adjacent tokens.
Getting started
- Running: The README points to a demo notebook for trying the model: "./notebooks/demo.ipynb" and provides a Colab link. It also notes that there is no command-line script available for running the model locally yet, but you can create one using the demo notebook as a reference. That being said, contributions are welcome!
The README does not provide a install or setup script, and there is no explicit environment setup section beyond referencing the demo notebook.
Recent releases
- RELEASES (latest 0): - none
Traction
- Stars: 2334
- Forks: 225
- Open issues: 26
Behind the repo
- License: MIT
- Language: Python
- Created: 2023-12-15
- Last push: 2024-04-08
- Topics include colab-notebook, deep-learning, google-colab, language-model, llm, mixture-of-experts, offloading, pytorch, quantization
Caveats
- There is no dedicated command-line script currently available for running locally.
- Some techniques described in the technical report are not yet available in this repo, with ongoing work and upcoming features listed (e.g., support for other quantization methods, speculative expert prefetching).