Trending AI repositories

andrewkchan/

yalm

Yet Another Language Model: LLM inference in C++/CUDA, no libraries except for I/O

Avarok-Cybersecurity/

atlas

Pure Rust Inference Engine

devflowinc/

uzi

CLI for running large numbers of coding agents in parallel with git worktrees

rohan-paul/

LLM-FineTuning-Large-Language-Models

LLM (Large Language Model) FineTuning

felladrin/

MiniSearch

Minimalist web-searching platform with an AI assistant that runs directly from your browser. Demo: https://felladrin-minisearch.hf.space

NotPunchnox/

rkllama

Ollama alternative for Rockchip NPU: An efficient solution for running AI and Deep learning models on Rockchip devices with optimized NPU support ( rkllm )

yassa9/

qwen600

Static suckless single batch CUDA-only qwen3-0.6B mini inference engine

zjhellofss/

KuiperLLama

校招、秋招、春招、实习好项目，带你从零动手实现支持LLama2/3和Qwen2.5的大模型推理框架。

openinfer-project/

openinfer

Pure Rust + CUDA LLM inference engine — no PyTorch, OpenAI-compatible, serves Qwen3 to Kimi-K2

microsoft/

sarathi-serve

A low-latency & high-throughput serving engine for LLMs

Chen-zexi/

vllm-cli

A command-line interface tool for serving LLM using vLLM.

huawei-csl/

KVarN

KVarN is a native vLLM KV-cache quantization backend for your agents: 3-5x more context, throughput above FP16, and FP16-level accuracy. Calibration-free, one flag.

jia-gao/

leanctx

Drop-in prompt compression for production LLM apps. Cut your token bill 40-60% without changing your code. Python SDK, LLMLingua-2, MIT.

avifenesh/

bw24

From-scratch Rust+CUDA inference engine, bit-exact by construction — NVFP4, MoE, MTP speculative decoding, tuned against measured limits of one RTX 5090 Laptop (sm_120a).

devnen/

qwen3.6-windows-server

One-click Qwen3.6-27B inference on Windows. 158 tok/s on RTX 5090, 72 tok/s on RTX 3090. Native, no WSL, no Docker, no telemetry.

0xSero/

glm-5.2-sm120

GLM-5.2-NVFP4-REAP-469B serving on SM120 (4× RTX PRO 6000 Blackwell) — one-command vLLM launch recipe, 250K context, DeepSeek Sparse Attention + MTP speculative decode

ahammadmejbah/

Awesome-Datasets-Hub

A curated collection of datasets for Large Language Models (LLMs), covering medical AI, NLP, multimodal learning, instruction tuning, reasoning, code generation, and evaluation benchmarks.

Sandermage/

sndr_core_engine

SNDR Core Engine (Genesis) — vLLM runtime patch-overlay for Qwen3.6 + Gemma4 on consumer NVIDIA (Ampere sm_86, 2× A5000/3090). Qwen3.6-35B-A3B FP8 ~240 tok/s, 27B-int4 hybrid GDN+Mamba, Gemma4 26B/31B AWQ, 256K ctx. 321 patches: TurboQuant k8v4 KV, MTP/DFlash spec-decode, FULL cudagraph, hybrid GDN. vLLM pin dev424 + Control Center GUI.

RightNow-AI/

AutoMegaKernel

An agent harness that compiles a model into one provably-correct, self-retargeting CUDA megakernel and self-tunes it past cuBLAS at batch-1 LLM decode, paper: https://arxiv.org/abs/2606.09682

generative-computing/

granite-switch

Granite Switch — Build AI models like you build software

PMZFX/

intel-arc-pro-b70-benchmarks

Benchmark results and performance data for the Intel Arc Pro B70 GPU (Xe2/Battlemage) - LLM inference, video generation, dual-GPU scaling.

OpenCPIL/

prima.cpp

[Official] prima.cpp: Fast 30-70B LLM inference on heterogeneous and everyday home devices

MADEVAL/

FerryAI

Native AI inference for PHP 8.3+ - run ONNX, GGUF (llama.cpp) and RubixML models directly in your PHP process via FFI. Chat, streaming, embeddings, RAG and vector search. No Python, no HTTP microservices, no Docker sidecars.

wuxuedaifu/

vllm-chatterbox-stream

OpenAI-compatible multilingual TTS server — Chatterbox on vLLM with real-time PCM audio streaming, low time-to-first-byte (~0.7 s), voice cloning, and 23 languages.

AdamBien/

lightmetal

Apple Silicon mlx with Zero Dependency Java