mistral.rs is a Rust-based framework for fast LLM inference that supports multiple models and quantization methods. It offers a zero-config CLI, a built-in web UI, and SDKs for Python and Rust, with CUDA, Metal, and multi-model support.
Collecting history — the radar snapshots this repo daily. The trend line appears after 3 days of data (1 so far).
What it is
mistral.rs is a fast, flexible LLM inference engine written in Rust. It supports Hugging Face models, GGUF, UQFF, and various quantization methods, and provides tools for multi-model serving, agentic capabilities, and a built-in web UI.
How it works
The project exposes a CLI (run, serve, bench) that detects model architecture, quantization, and chat templates. It supports continuous batching, CUDA with FlashAttention, Metal, and distributed inference. It includes features for per-layer quantization topology and auto-selects quant methods for hardware. It also provides server endpoints compatible with OpenAI and Anthropic interfaces and an agentic runtime with tool-calling support.
Getting started
Install:
curl --proto '=https' --tlsv1.2 -sSf https://raw.githubusercontent.com/EricLBuehler/mistral.rs/master/install.sh | sh
Run a model:
mistralrs run -m Qwen/Qwen3-4B
Start a server with UI:
mistralrs serve -m google/gemma-4-E4B-it
CLI usage examples show auto-detection and a single binary for chat, server, benchmarks, and UI:
# Diagnose issues
mistralrs doctor
Getting started (continued)
The Quick Start mentions a Windows PowerShell command for installation as well:
irm https://raw.githubusercontent.com/EricLBuehler/mistral.rs/master/install.ps1 | iex
The project emphasizes zero-config usage and supports multiple deployment paths including Docker (prebuilt CPU and CUDA images).
Recent releases
Latest release: v0.9.0 (2026-07-07) with changes:
- Fix CUDA version extraction in install.sh by @zaqxsw-dev in https://github.com/EricLBuehler/mistral.rs/pull/2287
- feat(metal): improve runtime and aot compilation system by @EricL Previous: v0.8.23 (2026-06-25) with CUDA 13.3 support; v0.8.22 (2026-06-22) with CUDA improvements; v0.8.21 (2026-06-22) core fixes; v0.8.20 (2026-06-21) release workflow and quant optimization
Traction
Stars: 7565
Behind the repo
The project has links to a documentation site, quickstart guides, and GitHub issues for model requests, but no company affiliation is provided in the facts.
Caveats
License: MIT Created: 2024-02-26 Last push: 2026-07-29 Open issues: 381 Languages: Rust