Inference & Serving
Running models fast and cheap: inference engines, serving stacks, quantization and edge deployment.
Add a real-time analytics node to your operational database. Spice is a portable, accelerated SQL query, search, and LLM-inference engine in Rust for data-grounded AI apps and agents.
Distributed LLM inference. Connect home devices into a powerful cluster to accelerate LLM inference. More devices means faster inference.
Medusa: Simple Framework for Accelerating LLM Generation with Multiple Decoding Heads
Official Implementation of EAGLE-1 (ICML'24), EAGLE-2 (EMNLP'24), and EAGLE-3 (NeurIPS'25).
An open coding agent for your terminal, built by a community collective rather than a company. Bring your own model, keep your code on your machine, and owe nothing to anyone.
AICI: Prompts as (Wasm) Programs
The Agentic Framework of the PHP ecosystem to build production-ready AI driven applications. Connect components (LLMs, Tools, vector DBs, memory) to agents that can interact with your data.
Run any Llama 2 locally with gradio UI on GPU or CPU from anywhere (Linux/Windows/Mac). Use `llama2-wrapper` as your local llama2 backend for Generative Agents/Apps.
OpenLake is a high performance storage engine for efficient LLM inference and GPU Training
Ultrafast serverless GPU inference, sandboxes, and background jobs
Declarative way to run AI models in React Native on device, powered by ExecuTorch.
A high-performance inference engine for LLM, VLM, DiT and REC models, optimized for diverse AI accelerators. It is hosted in OpenAtom Foundation.
Run local LLMs like llama, deepseek-distill, kokoro and more inside your browser
List of software that allows searching the web with the assistance of AI: https://hf.co/spaces/felladrin/awesome-ai-web-search
A curated collection of top-tier penetration testing tools and productivity utilities across multiple domains. Join us to explore, contribute, and enhance your hacking toolkit!
The non profit fostering AI in India through credits, grants, resources
LLMs as Copilots for Theorem Proving in Lean
Highly Performant, Modular, Memory Safe and Production-ready Inference, Ingestion and Indexing built in Rust 🦀
Home for "How To Scale Your Model", a short blog-style textbook about scaling LLMs on TPUs
Streamlines and simplifies prompt design for both developers and non-technical users with a low code approach.
Virtualized Elastic KV Cache for Dynamic GPU Sharing and Beyond
Local AI app and inference engine for agents. Run open-weight LLMs locally — private, 100% offline on your computer.
OpenAlpha_Evolve is an open-source Python framework inspired by the groundbreaking research on autonomous coding agents like DeepMind's AlphaEvolve.
Build your own high performance LLM inference engine in C++ and CUDA - a smaller version of vLLM
Pure C++ implementation of several models for real-time chatting on your computer (CPU & GPU)
List of awesome hosting sorted by minimal plan price
A highly optimized LLM inference acceleration engine for Llama and its variants.
LLM notes, including model inference, transformer model structure, and llm framework code analysis notes.
LLM.swift is a simple and readable library that allows you to interact with large language models locally with ease for macOS, iOS, watchOS, tvOS, and visionOS.
“AI-Compass”将为社区指引在 AI 技术海洋中航行的方向，无论你是初学者还是进阶开发者，都能在这里找到通往 AI 各大方向的路径。旨在帮助开发者系统性地了解 AI 的核心概念、主流技术、前沿趋势，并通过实践掌握从理论到落地的全过程。