oMLX is a Python-based LLM inference server optimized for Apple Silicon, featuring continuous batching and a two-tier KV cache managed from a macOS menu bar. It provides a CLI and Homebrew workflow, with a web admin UI and OpenAI-compatible API endpoints.
Collecting history — the radar snapshots this repo daily. The trend line appears after 3 days of data (1 so far).
What it is
oMLX is an LLM inference server optimized for Apple Silicon, with continuous batching and a two-tier KV cache (hot RAM and cold SSD) managed via a macOS menu bar interface. It supports text LLMs, vision-language models, OCR, embeddings, and rerankers, and offers an OpenAI-compatible API surface with a built-in admin dashboard and model management features.
How it works
The server runs a FastAPI-based backend that exposes OpenAI/Anthropic-compatible endpoints and uses a multi-engine setup (EnginePool) with BatchedEngine, VLMEngine, EmbeddingEngine, and RerankerEngine. It implements a tiered cache where hot blocks stay in RAM and cold blocks are stored on SSD in safetensors format, with Copy-on-Write and prefix sharing. It supports continuous batching via mlx-lm BatchGenerator and per-model settings including TTL, pinning, and manual load/unload. It also includes an admin dashboard for monitoring, model management, chat, benchmarks, and per-model configuration.
Getting started
Install options include:
- macOS App: Download the .dmg from Releases and install; includes in-app auto-update and a CLI shim at ~/.omlx/bin/omlx.
- Homebrew: provided commands to tap, install, upgrade, and run as a background service. Example:
brew tap jundot/omlx https://github.com/jundot/omlx
brew install omlx
# Upgrade to the latest version
brew update && brew upgrade omlx
# Run as a background service (auto-restarts on crash)
omlx start
- From Source: clone, install core with
pip install -e ., and optionallypip install -e ".[mcp]"for MCP support. You can enable custom kernels viaOMLX_WITH_CUSTOM_KERNEL=1during install. - CLI usage examples:
omlx start
omlx stop
omlx restart
omlx serve --model-dir ~/models
Requires macOS 15.0+, Python 3.11–3.13, and Apple Silicon.
All settings can be configured via the web admin panel at /admin, and are persisted to ~/.omlx/settings.json.
Recent releases
Latest release: v0.5.7 (2026-08-04) OMLX 0.5.7. Hotfix history includes 0.5.6 and 0.5.5 addressing DeepSeek V4 cache/signature and prompt consistency issues.
Highlights from 0.5.4 include native support and acceleration for DeepSeek V4 Flash 0731, Inkling Small, Step-3.7-Flash, MiMo V2.5, Laguna S-2.1, and expanded MTP, DFlash, SpecPrefill, and context batching.
Traction
18432 stars; 1582 forks; 788 open issues.
Behind the repo
Not applicable here as no linked startup/company is described beyond the project page.
Caveats
License: Apache-2.0. Language: Python. Created 2026-02-13; last push 2026-08-04. Requires macOS 15.0+ and Apple Silicon. Commands and install notes are as shown in the README including the exact CLI examples and Homebrew usage.