Trending AI repositories

vllm-project/

vllm

A high-throughput and memory-efficient inference and serving engine for LLMs

ray-project/

ray

Ray is an AI compute engine. Ray consists of a core distributed runtime and a set of AI Libraries for accelerating ML workloads.

liguodongiot/

llm-action

本项目旨在分享大模型相关技术原理以及实战经验（大模型工程化、大模型应用落地）

bentoml/

OpenLLM

Run any open-source LLMs, such as DeepSeek and Llama, as OpenAI compatible API endpoint in the cloud.

bentoml/

BentoML

The easiest way to serve AI apps and models - Build Model Inference APIs, Job queues, LLM apps, Multi-model pipelines, and more!

superduper-io/

superduper

Superduper: End-to-end framework for building custom AI applications and agents.

gpustack/

gpustack

A GPU cluster manager for high-performance AI model serving (vLLM, SGLang) and on-demand SSH-accessible GPU instances.

predibase/

lorax

Multi-LoRA inference server that scales to 1000s of fine-tuned LLMs

microsoft/

aici

AICI: Prompts as (Wasm) Programs

ovg-project/

kvcached

Virtualized Elastic KV Cache for Dynamic GPU Sharing and Beyond

zhihu/

ZhiLight

A highly optimized LLM inference acceleration engine for Llama and its variants.

rohan-paul/

LLM-FineTuning-Large-Language-Models

LLM (Large Language Model) FineTuning

openinfer-project/

openinfer

Pure Rust + CUDA LLM inference engine — no PyTorch, OpenAI-compatible, serves Qwen3 to Kimi-K2

Sandermage/

sndr_core_engine

SNDR Core Engine (Genesis) — vLLM runtime patch-overlay for Qwen3.6 + Gemma4 on consumer NVIDIA (Ampere sm_86, 2× A5000/3090). Qwen3.6-35B-A3B FP8 ~240 tok/s, 27B-int4 hybrid GDN+Mamba, Gemma4 26B/31B AWQ, 256K ctx. 321 patches: TurboQuant k8v4 KV, MTP/DFlash spec-decode, FULL cudagraph, hybrid GDN. vLLM pin dev424 + Control Center GUI.