mesh-llm

What it is

Mesh LLM pools GPUs and memory across machines and exposes the result as one OpenAI-compatible API at http://localhost:9337/v1. Start one node, add more nodes later, and let the mesh decide whether a model runs locally, routes to a peer, or uses Skippy stage splits for models that are too large for one box.

How it works

• Single-machine fit first: if one node can host the full model, it serves locally.
• Mesh routing: all nodes expose the same /v1 API; requests are routed by the model field to a peer that can serve that model.
• Owner-control plane: operator config/inventory actions use an additive mesh-llm-control lane; public mesh join, gossip, routing, and inference stay on the public mesh plane.
• Skippy stage splits: large models can load as package-backed layer stages with planned contiguous layer ranges.
• Layer packages: model-package.json plus GGUF fragments; peers fetch needed pieces for their stage.
• Public discovery: published meshes advertise through Nostr discovery; private meshes use invite-token based discovery.

For a deeper operator guide, see docs/USAGE.md and for commands see docs/CLI.md.

Local model-only serving

Use the direct topology when a process should expose one complete local model through the OpenAI API without becoming a mesh node:

mesh-llm serve \
  --local-model-only \
  --model /models/model.gguf \
  --port 9337

This mode starts the OpenAI frontend and one local Skippy model runtime. It does not start QUIC, discovery, peer maintenance, split planning, plugins, release lookup, the web console, or the management API. Add --listen-all only when the OpenAI endpoint must bind beyond loopback.

Getting started

Quick start commands include:

• Install the latest release executable:

curl -fsSL https://raw.githubusercontent.com/Mesh-LLM/mesh-llm/main/install.sh | bash

• On Windows, use PowerShell:

irm https://raw.githubusercontent.com/Mesh-LLM/mesh-llm/main/install.ps1 | iex

• Finish setup:

mesh-llm setup

• Join the public mesh and start serving:

mesh-llm serve --auto

• Check available models:

curl -s http://localhost:9337/v1/models | jq '.data[].id'

• Send an OpenAI-compatible request:

curl http://localhost:9337/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"GLM-4.7-Flash-Q4_K_M","messages":[{"role":"user","content":"hello"}]}'

• To run in headless server deployments, use:

mesh-llm serve --auto --headless

Getting started (workflow options)

Various short commands are provided to pick workflows, such as joining a private mesh, running a local model, or running a split-model setup. See the Quick Start and Pick the workflow you need sections in the README for details.

Releases

Latest releases include v0.75.0-rc1 (2026-07-30), v0.74.0 (2026-07-27), v0.74.0-rc8 (2026-07-27), v0.74.0-rc6 (2026-07-26), and v0.73.1 (2026-07-14).

Install and build notes

Tagged releases publish macOS bundles and multiple Linux/Windows runtimes. Build from source with just:

git clone https://github.com/Mesh-LLM/mesh-llm
cd mesh-llm
just build

Source builds require just, cmake, Rust, and Node.js 24 + npm. Example release-bound build:

just release-host-build
just release-runtime-build metal # or cpu, cuda, rocm, vulkan
MESH_LLM_NATIVE_RUNTIME_BUNDLE_DIR="$PWD/dist/native-runtimes" \
MESH_LLM_NATIVE_RUNTIME_CACHE_DIR="$(mktemp -d)" \
  ./target/release/mesh-llm runtime list

How to install artifacts

The README describes artifact naming for Linux ARM64 and CUDA variants, and notes that portable archives work offline. It also describes a verification process using cargo run -p xtask -- release-attestation inspect with keys. See the install/build notes in the README for details.