Ultravox is a Python-based multimodal LLM designed for real-time voice interactions, with weights hosted on Hugging Face and multiple releases since 2024. It emphasizes streaming text from audio input and supports training adapters for open-weight LLMs.
Collecting history — the radar snapshots this repo daily. The trend line appears after 3 days of data (1 so far).
What it is
Ultravox is a fast multimodal LLM designed for real-time voice interactions. It accepts audio input and emits streaming text, without requiring a separate ASR stage. The default model is built on top of Llama 3.3 70B, with an 8B variant also available. Weights are available on the Hugging Face page at https://huggingface.co/fixie-ai/.
How it works
Ultravox extends an open-weight LLM with a multimodal projector that converts audio directly into the high-dimensional space used by LLMs, enabling faster responses by avoiding separate ASR steps. It can be trained against open-weight models by training only the adapter/projector while keeping the LLM and audio encoder frozen. The documentation references usage for both training and evaluation workflows, including config-driven training and multi-node options.
Getting started
Key setup and development steps are described under Environment Setup and Training. Notable commands and sections include:
/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"
brew update
brew install just
just install
poetry run python -m ultravox.training.train --config_path ultravox/training/configs/example_config.yaml
TRAIN_ARGS="--config_path ultravox/training/configs/example_config.yaml"
poetry run python -m ultravox.training.helpers.prefetch_weights $TRAIN_ARGS
poetry run torchrun --nproc_per_node=8 -m ultravox.training.train $TRAIN_ARGS
just eval --config_path ultravox/evaluation/configs/eval_config.yaml
Recent releases
Latest releases listed in the repo:
- v0.7 (2025-12-) release referenced in Latest News and Hugging Face collections
- v0.6 (2025-08-18) weights pushed to Hugging Face and Ultravox Realtime APIs documentation
- v0.5 (2025-02-11) weights pushed to Hugging Face
- v0.4.1 (2024-11-12) weights pushed to Hugging Face and updated datasets for training
- v0.4 (2024-08-27) weights pushed to Hugging Face
- v0.3 (2024-08-23) weights pushed to Hugging Face
Traction
Stars: 4507 Forks: 385 Open issues: 61
Behind the repo
Not applicable; no startup/company link provided beyond project scope in the given data.
Caveats
License: MIT Created: 2024-05-29 Last push: 2025-12-12 Language: Python Topics: ai, llm, slm, speech Releases indicate ongoing development and weight distributions on Hugging Face. Training notes indicate MosaicML platform usage with end-of-July 2025 shutdown mention and migration guidance. Documentation references specific environment and config tooling, including Poetry and Just-based workflows.