The Production AI Stack is an opinionated, self-hosted reference for deploying end-to-end AI systems with a Postgres-first approach, focusing on retrievable pipelines, structured outputs, and replayability. It provides architecture guidance and concrete layer choices across model serving, embeddings, gatewaying, retrieval, and parsing.
Collecting history — the radar snapshots this repo daily. The trend line appears after 3 days of data (1 so far).
What it is
The Production AI Stack is an opinionated map of what runs in production, covering model serving, embeddings, gateway and routing, retrieval and storage, and document parsing. It emphasizes Postgres-first design, self-hostable components, replayability, and traceable evals.
How it works
The stack proposes default components for each layer and switches when constraints apply:
- Model serving: Default vLLM; Switch to SGLang for prefix-heavy workloads and structured JSON output; other tiers include TensorRT-LLM, NVIDIA Dynamo/llm-d, and Ollama for local/dev.
- Embeddings and rerankers: TEI for serving with Infinity as an alternative; rerankers on the top candidates improve quality.
- Gateway and routing: Default LiteLLM; switch to Bifrost for low-latency high-RPS use; can also use OpenRouter or cloud gateways.
- Retrieval and storage: Default Postgres + pgvector; switch to Qdrant when vectors are central; additional options include OpenSearch/Elasticsearch, Milvus, LanceDB, Chroma.
- Document ingestion and parsing: Default Docling; switch to MinerU for hostile documents; other options include marker, VLM-based parsers, Azure/Mistral OCR, and visual retrieval approaches.
It stresses that the gateway is a single point of failure and highlights latency, data governance, and replayability considerations.
Getting started
The README excerpt outlines layer-by-layer choices and the rationale for defaults and switches. It emphasizes evaluating with your traffic shape, pinning versions, and avoiding unverified upgrades in gateways. Specific commands for setup are not included in the truncated portion provided.
Recent releases
RELEASES (latest 0): - none
Traction
Stars: 68 (as of repository snapshot) with 0 open issues reported in the provided data.
Behind the repo
No startup or company linkage is provided in the supplied facts.
Caveats
License: none listed. Created 2026-07-17; last push 2026-07-17. Traps include gateway single point of failure, need to pin versions, and careful evaluation of quantization and recall across tasks. Specific warnings about LiteLLM supply-chain incident in March 2026 are noted.