Text Embeddings Inference is a Rust-based toolkit for deploying and serving open source text embeddings models with feature-rich REST/gRPC interfaces and configurable batching. It emphasizes high-performance inference with support for CUDA, SafeTensors, and multiple pooling options.
Collecting history — the radar snapshots this repo daily. The trend line appears after 3 days of data (1 so far).
What it is
Text Embeddings Inference (TEI) is a toolkit for deploying and serving open source text embeddings and sequence classification models. TEI enables high-performance extraction for various models and includes features such as fast inference, small docker images, and support for token-based dynamic batching.
How it works
TEI provides a serving server (text-embeddings-router) with options to specify model-id or local model paths, and configuration for tokenization, data types, pooling, batching, and concurrency. It supports multiple backends and optimizations like Flash Attention, Candle, cuBLASLt, and Safetensors for weight loading. The Docker/docker-compose usage shown in the README demonstrates running a container with a model ID and exposing ports for REST requests to an /embed endpoint.
Key operational aspects include:
- Token-based dynamic batching and options to control max-concurrent-requests, max-batch-tokens, and max-client-batch-size
- Configurable pooling method (cls, mean, splade, last-token) and the ability to override model pooling configuration
- Optional density of inputs via auto-truncate and max-batch-tokens controls
- Options for dense module paths, HF token handling, and API security keys
Getting started
- Docker usage example:
model=Qwen/Qwen3-Embedding-0.6B
volume=$PWD/data # share a volume with the Docker container to avoid downloading weights every run
docker run --gpus all -p 8080:80 -v $volume:/data --pull always ghcr.io/huggingface/text-embeddings-inference:cuda-1.9 --model-id $model
- Request example:
curl 127.0.0.1:8080/embed \
-X POST \
-d '{"inputs":"What is Deep Learning?"}' \
-H 'Content-Type: application/json'
- The README notes that GPUs require the NVIDIA Container Toolkit and CUDA compatibility (12.2+), and demonstrates a full options list for text-embeddings-router with various flags like --model-id, --dtype, --pooling, --max-batch-tokens, --max-concurrent-requests, --default-prompt, --dense-path, --hf-token, --hostname, --port, and others.
Recent releases
- v1.9.3 (2026-03-23): Use rust-toolchain.toml before rustup on Dockerfile-{cuda,cuda-all} and related fixups
- v1.9.2 (2026-02-25): Fix auto-truncate false setting; add support for pad_token_id nullable and rope parameters
- v1.9.1 (2026-02-17): Fix support for containers with CUDA 13.0+ (related to release process for ghcr.io)
- v1.9.0 (2026-02-17): Release notes include a breaking change section
- v1.8.3 (2025-10-30): Bug fixes including handling empty requests and max_input_length edge cases
Traction
- Stars: 4973; Forks: 421; Open issues: 203
Behind the repo
- Language: Rust
- License: Apache-2.0
- Created: 2023-10-13; Last push: 2026-07-24
- Topics include ai, embeddings, huggingface, llm, ml
Caveats
- License: Apache-2.0
- Open issues: 203
- Development activity as of last push: 2026-07-24