xorbitsai/

inference

GitHub Website

Xorbits Inference provides a single-command setup to serve language, speech, and multimodal models via Xinference. It supports self-hosting with Docker, Kubernetes, and pip-based installation, and exposes OpenAI-compatible RESTful API, RPC, CLI, and WebUI interfaces.

9.5kstars

851forks

51issues

Apache-2.0license

2023since

Star historydaily snapshots by VibeCrowd

Collecting history — the radar snapshots this repo daily. The trend line appears after 3 days of data (1 so far).

Alternatives & relatedmatched by topic overlap

yangjianxin1/

Firefly

Firefly: 大模型训练工具，支持训练Qwen2.5、Qwen2、Yi1.5、Phi-3、Llama3、Gemma、MiniCPM、Yi、Deepseek、Orion、Xverse、Mixtral-8x7B、Zephyr、Mistral、Baichuan2、Llma2、Llama、Qwen、Baichuan、ChatGLM2、InternLM、Ziya2、Vicuna、Bloom等大模型

6.7k584Pythonquiet

hiyouga/

LlamaFactory

▲ 162/wk

Unified Efficient Fine-Tuning of 100+ LLMs & VLMs (ACL 2024)

74k9.0kPython

unslothai/

unsloth

▲ 524/wk

Unsloth is a local UI for training and running Kimi K3, Gemma 4, Qwen3.6, DeepSeek-V4, GLM and other models.

70k6.3kPython

ollama/

ollama

▲ 640/wk

Get up and running with Kimi-K2.6, GLM-5.2, MiniMax, DeepSeek, gpt-oss, Qwen, Gemma and other models.

178k17kGo

khoj-ai/

khoj

▲ 106/wk

Your AI second brain. Self-hostable. Get answers from the web or your docs. Build custom agents, schedule automations, do deep research. Turn any online or local LLM into your personal, autonomous AI (gpt, claude, gemini, llama, qwen, mistral). Get started - free.

36k2.3kPython

datawhalechina/

self-llm

▲ 87/wk

《开源大模型食用指南》针对中国宝宝量身打造的基于Linux环境快速微调（全参数/Lora）、部署国内外开源大模型（LLM）/多模态大模型（MLLM）教程

32k3.1kJupyter Notebook

Compareinference vs Firefly inference vs LlamaFactory inference vs unsloth

Reviewgenerated from repository data · Aug 5, 2026

What it is

Xorbits Inference (Xinference) is a library to serve language, speech recognition, and multimodal models. It aims to let users deploy and serve models with a single command.

How it works

The project enables a unified inference API for various models and supports heterogeneous hardware (GPUs/CPUs) and multiple interfaces (OpenAI-compatible RESTful API, RPC, CLI, WebUI). It includes built-in model integrations and supports distributed deployment across devices.

Getting started

Docker

docker run --name xinference -d -p 9997:9997 -e XINFERENCE_HOME=/data -v </on/your/host>:/data --gpus all xprobe/xinference:latest xinference-local -H 0.0.0.0

K8s via helm

# add repo
helm repo add xinference https://xorbitsai.github.io/xinference-helm-charts

# update indexes and query xinference versions
helm repo update xinference
helm search repo xinference/xinference --devel --versions

# install xinference
helm install xinference xinference/xinference -n xinference --version 0.0.1-v<xinference_release_version>

Quick Start

pip install "xinference[all]"

$ xinference-local

Check docs for running locally via web UI, curl, CLI, or Python client.

Recent releases

v3.1.0 (2026-07-31): New features include a SSO/OIDC login button for the login page.
v3.1.0.rc1 (2026-07-29): New features include a SSO/OIDC login button.
v3.0.0 (2026-07-19): Frontend switched to Next.js app.
v2.12.0 (2026-07-04): Gate model remote-code execution behind a config switch.
v2.11.0 (2026-06-19): UI updates and feature additions.

Traction

Stars: 9477 Forks: 851 Open issues: 51

Behind the repo

Linked to enterprise features and integrations with Xagent, Dify, FastGPT, RAGFlow, and MaxKB.

Caveats

License: Apache-2.0 Created: 2023-06-14 Last push: 2026-08-04

SharePost on X LinkedIn