Infinity is a Python-based REST API server for high-throughput, low-latency serving of text embeddings, reranking, and multi-modal models, with CLI and Docker deployment options.
Collecting history — the radar snapshots this repo daily. The trend line appears after 3 days of data (1 so far).
What it is
Infinity is a high-throughput, low-latency REST API for serving text-embeddings, reranking models, clip, clap and colpali. It is developed under the MIT License and provides a CLI and Docker deployment paths.
How it works
The project orchestrates multiple models (embeddings, reranking, multi-modal) and supports dynamic batching. It is built on FastAPI and integrates with HuggingFace models, ONNX/TF backends via optimum, and CTranslate2. It enables multi-model, multi-task serving and can launch models via environment variables or CLI arguments.
Getting started
Getting started materials include:
pip install infinity-emb[all]
To run the CLI directly after installation:
infinity_emb v2 --model-id BAAI/bge-small-en-v1.5
infinity_emb v2 --help
To use a pre-built Docker container:
port=7997
model1=michaelfeil/bge-small-en-v1.5
model2=mixedbread-ai/mxbai-rerank-xsmall-v1
volume=$PWD/data
docker run -it --gpus all \
-v $volume:/app/.cache \
-p $port:$port \
michaelf34/infinity:latest \
v2 \
--model-id $model1 \
--model-id $model2 \
--port $port
The cache path inside the container is set by the environment variable HF_HOME.
Specialized docker images
Details for CPU, ROCm, and Onnx-GPU/TensorRT images are provided in separate docker details.
Advanced CLI usage
The v2 CLI supports launching multiple models simultaneously and environment-variable based configuration.
# Example usage patterns described in docs
Getting started (continued)
Environment variable usage is supported, with variables prefixed by INFINITY_. Example: INFINITY_MODEL_ID.
Chosing the fastest engine
Options include --engine torch, --engine optimum, and --engine ctranslate2, each with model-type prerequisites.
Telemetry opt-out
# Disable
export INFINITY_ANONYMOUS_USAGE_STATS="0"
Getting started (models and specs)
Infinity provides tested embedding and reranking models, with documentation indicating supported model families and deployment patterns.
Release history
Latest releases include:
- 0.0.77 (2025-08-22): Update ColQwen2_5 to IMAGE_COL_MODELS and Colpali engine; minor model additions.
- 0.0.76 (2025-03-16): torch=2.6.0 update; 5-10% faster attention on hopper; note about torch.compile compatibility.
- 0.0.75 (2025-01-22): Better transformers support; updated verbose parameter for ModelWorker initialization.
- 0.0.74 (2025-01-03): CLI moved to extra file; macOS tests; Modernbert support with latest HuggingFace.
- 0.0.73 (2024-12-10): Support for matryoshka embeddings; optimum option for PredictEngine.
Traction
- Stars: 2904
Behind the repo
(This section omitted because no linked startup/company information is provided in the FACTS.)
Caveats
- License: MIT
- Age: Created 2023-10-11; last_push 2026-03-24
- Open issues: 130