Trending AI repositories

vllm-project/

vllm

A high-throughput and memory-efficient inference and serving engine for LLMs

sgl-project/

sglang

SGLang is a high-performance serving framework for large language models and multimodal models.

flashinfer-ai/

flashinfer

FlashInfer: Kernel Library for LLM Serving

gpustack/

gpustack

A GPU cluster manager for high-performance AI model serving (vLLM, SGLang) and on-demand SSH-accessible GPU instances.

beam-cloud/

beta9

Ultrafast serverless GPU inference, sandboxes, and background jobs

jmaczan/

tiny-vllm

Build your own high performance LLM inference engine in C++ and CUDA - a smaller version of vLLM

zhihu/

ZhiLight

A highly optimized LLM inference acceleration engine for Llama and its variants.

andrewkchan/

yalm

Yet Another Language Model: LLM inference in C++/CUDA, no libraries except for I/O

Avarok-Cybersecurity/

atlas

Pure Rust Inference Engine

yassa9/

qwen600

Static suckless single batch CUDA-only qwen3-0.6B mini inference engine

zjhellofss/

KuiperLLama

校招、秋招、春招、实习好项目，带你从零动手实现支持LLama2/3和Qwen2.5的大模型推理框架。

zengxiao-he/

tessera

From teacher to tiles — a from-scratch LLM distillation & serving engine: custom Triton/CUDA kernels, FSDP distillation, paged-KV continuous batching, speculative decoding, a Rust gateway, a JAX oracle, and interpretability tooling.

openinfer-project/

openinfer

Pure Rust + CUDA LLM inference engine — no PyTorch, OpenAI-compatible, serves Qwen3 to Kimi-K2

avifenesh/

bw24

From-scratch Rust+CUDA inference engine, bit-exact by construction — NVFP4, MoE, MTP speculative decoding, tuned against measured limits of one RTX 5090 Laptop (sm_120a).

Sandermage/

sndr_core_engine

SNDR Core Engine (Genesis) — vLLM runtime patch-overlay for Qwen3.6 + Gemma4 on consumer NVIDIA (Ampere sm_86, 2× A5000/3090). Qwen3.6-35B-A3B FP8 ~240 tok/s, 27B-int4 hybrid GDN+Mamba, Gemma4 26B/31B AWQ, 256K ctx. 321 patches: TurboQuant k8v4 KV, MTP/DFlash spec-decode, FULL cudagraph, hybrid GDN. vLLM pin dev424 + Control Center GUI.

hwdsl2/

self-hosted-ai-stack

Deploy a complete self-hosted AI stack with Docker Compose: Ollama, LiteLLM, AnythingLLM, Whisper, WhisperLive, Kokoro, Embeddings, Docling and MCP Gateway. Local-first, private by default, with lightweight stacks, optional HTTPS and NVIDIA CUDA acceleration. Multi-arch: amd64, arm64.

RightNow-AI/

AutoMegaKernel

An agent harness that compiles a model into one provably-correct, self-retargeting CUDA megakernel and self-tunes it past cuBLAS at batch-1 LLM decode, paper: https://arxiv.org/abs/2606.09682

wuxuedaifu/

vllm-chatterbox-stream

OpenAI-compatible multilingual TTS server — Chatterbox on vLLM with real-time PCM audio streaming, low time-to-first-byte (~0.7 s), voice cloning, and 23 languages.