Trending AI repositories

ai-dynamo/

dynamo

A Datacenter Scale Distributed Inference Serving Framework

kserve/

kserve

Standardized Distributed Generative and Predictive AI Inference Platform for Scalable, Multi-Framework Deployment on Kubernetes

OpenBMB/

UltraRAG

A Low-Code MCP Framework for Building Complex and Innovative RAG Pipelines

xlite-dev/

Awesome-LLM-Inference

📚A curated list of Awesome LLM/VLM Inference Papers with Codes: Flash-Attention, Paged-Attention, WINT8/4, Parallelism, etc.🎉

gpustack/

gpustack

A GPU cluster manager for high-performance AI model serving (vLLM, SGLang) and on-demand SSH-accessible GPU instances.

katanaml/

sparrow

Structured data extraction, instruction calling and agentic workflows with ML, LLM and Vision LLM

ovg-project/

kvcached

Virtualized Elastic KV Cache for Dynamic GPU Sharing and Beyond

jmaczan/

tiny-vllm

Build your own high performance LLM inference engine in C++ and CUDA - a smaller version of vLLM

harleyszhang/

llm_note

LLM notes, including model inference, transformer model structure, and llm framework code analysis notes.

pgalko/

BambooAI

A Python library powered by Language Models (LLMs) for conversational data discovery and analysis.

openinfer-project/

openinfer

Pure Rust + CUDA LLM inference engine — no PyTorch, OpenAI-compatible, serves Qwen3 to Kimi-K2

Chen-zexi/

vllm-cli

A command-line interface tool for serving LLM using vLLM.

huawei-csl/

KVarN

KVarN is a native vLLM KV-cache quantization backend for your agents: 3-5x more context, throughput above FP16, and FP16-level accuracy. Calibration-free, one flag.

AEON-7/

Qwen3.6-27B-AEON-Ultimate-Uncensored-DFlash

Fully uncensored, capability-enhanced abliteration of Qwen3.6-27B. NVFP4 + z-lab DFlash speculative decoding (n=12) on the unified ghcr.io/aeon-7/aeon-vllm-ultimate:latest container, tuned for long-context draft acceptance on DGX Spark. 6 HF variants (BF16/NVFP4/MTP/MTP-XS), docker-compose, and QuickStart.

devnen/

qwen3.6-windows-server

One-click Qwen3.6-27B inference on Windows. 158 tok/s on RTX 5090, 72 tok/s on RTX 3090. Native, no WSL, no Docker, no telemetry.

0xSero/

glm-5.2-sm120

GLM-5.2-NVFP4-REAP-469B serving on SM120 (4× RTX PRO 6000 Blackwell) — one-command vLLM launch recipe, 250K context, DeepSeek Sparse Attention + MTP speculative decode

Sandermage/

sndr_core_engine

SNDR Core Engine (Genesis) — vLLM runtime patch-overlay for Qwen3.6 + Gemma4 on consumer NVIDIA (Ampere sm_86, 2× A5000/3090). Qwen3.6-35B-A3B FP8 ~240 tok/s, 27B-int4 hybrid GDN+Mamba, Gemma4 26B/31B AWQ, 256K ctx. 321 patches: TurboQuant k8v4 KV, MTP/DFlash spec-decode, FULL cudagraph, hybrid GDN. vLLM pin dev424 + Control Center GUI.

generative-computing/

granite-switch

Granite Switch — Build AI models like you build software

Bike4Mind/

bike4mind

The open-core AI workbench — notebooks, agents, RAG, voice, and images across any model: OpenAI, Anthropic, Google, xAI, or local via Ollama/vLLM. BSL 1.1, auto-converting to Apache-2.0 on a two-year clock. Your AI keeps running when theirs doesn't.

wuxuedaifu/

vllm-chatterbox-stream

OpenAI-compatible multilingual TTS server — Chatterbox on vLLM with real-time PCM audio streaming, low time-to-first-byte (~0.7 s), voice cloning, and 23 languages.