Inference & Serving

vllm-project/

vllm

A high-throughput and memory-efficient inference and serving engine for LLMs

nomic-ai/

gpt4all

GPT4All: Run Local LLMs on Any Device. Open-source and available for commercial use.

hiyouga/

LlamaFactory

Unified Efficient Fine-Tuning of 100+ LLMs & VLMs (ACL 2024)

ray-project/

ray

Ray is an AI compute engine. Ray consists of a core distributed runtime and a set of AI Libraries for accelerating ML workloads.

sgl-project/

sglang

SGLang is a high-performance serving framework for large language models and multimodal models.

gitleaks/

gitleaks

Find secrets with Gitleaks 🔑

liguodongiot/

llm-action

本项目旨在分享大模型相关技术原理以及实战经验（大模型工程化、大模型应用落地）

Lightning-AI/

litgpt

20+ high-performance LLMs with recipes to pretrain, finetune and deploy at scale.

RyanCodrai/

turbovec

A vector index built on TurboQuant, written in Rust with Python bindings

bentoml/

OpenLLM

Run any open-source LLMs, such as DeepSeek and Llama, as OpenAI compatible API endpoint in the cloud.

openvinotoolkit/

openvino

OpenVINO™ is an open source toolkit for optimizing and deploying AI inference

Tiiny-AI/

PowerInfer

High-speed Large Language Model Serving for Local Deployment

bentoml/

BentoML

The easiest way to serve AI apps and models - Build Model Inference APIs, Job queues, LLM apps, Multi-model pipelines, and more!

InternLM/

lmdeploy

LMDeploy is a toolkit for compressing, deploying, and serving LLMs.

ai-dynamo/

dynamo

A Datacenter Scale Distributed Inference Serving Framework

katanemo/

plano

Plano is an AI-native proxy and data plane for agentic apps — with built-in orchestration, safety, observability, and smart LLM routing so you stay focused on your agents core logic.

algorithmicsuperintelligence/

openevolve

Open-source implementation of AlphaEvolve

flashinfer-ai/

flashinfer

FlashInfer: Kernel Library for LLM Serving

kserve/

kserve

Standardized Distributed Generative and Predictive AI Inference Platform for Scalable, Multi-Framework Deployment on Kubernetes

Michael-A-Kuykendall/

shimmy

⚡ Pure-Rust WebGPU inference engine — OpenAI-API compatible, GGUF native, runs on any GPU. No Python. No llama.cpp. Single binary.

cactus-compute/

cactus

Low-latency AI engine for mobile devices & wearables

xlite-dev/

Awesome-LLM-Inference

📚A curated list of Awesome LLM/VLM Inference Papers with Codes: Flash-Attention, Paged-Attention, WINT8/4, Parallelism, etc.🎉

gpustack/

gpustack

A GPU cluster manager for high-performance AI model serving (vLLM, SGLang) and on-demand SSH-accessible GPU instances.

superduper-io/

superduper

Superduper: End-to-end framework for building custom AI applications and agents.

FellouAI/

eko

Eko (Eko Keeps Operating) - Build Production-ready Agentic Workflow with Natural Language - eko.fellou.ai

lemonade-sdk/

lemonade

Lemonade helps users discover and run local AI apps by serving optimized LLMs right from their own GPUs and NPUs. Join our discord: https://discord.gg/5xXzkMu8Zk

ruvnet/

RuVector

RuVector is a High Performance, Real-Time, Self-Learning Ai, Vector GNN, Memory DB built in Rust.

algorithmicsuperintelligence/

optillm

Optimizing inference proxy for LLMs

NVIDIA/

GenerativeAIExamples

Generative AI reference workflows optimized for accelerated infrastructure and microservice architecture.

predibase/

lorax

Multi-LoRA inference server that scales to 1000s of fine-tuned LLMs