LLM Foundry is a Python-based toolkit for training, finetuning, evaluating, and deploying LLMs using MosaicML Composer and related tooling. It includes scripts for data prep, training, inference, and evaluation, with Docker-based and AMD/Intel considerations.
Collecting history — the radar snapshots this repo daily. The trend line appears after 3 days of data (1 so far).
What it is
LLM Foundry is code for training, finetuning, evaluating, and deploying LLMs with Composer and the MosaicML platform. It includes directories for models, datasets, callbacks, utilities, and scripts to run workloads, plus a TUTORIAL and MCLI integration.
How it works
The repository provides:
- llmfoundry/ for models, datasets, callbacks, utilities
- scripts/train for training or finetuning HuggingFace and MPT models
- scripts/inference for model conversion and response generation
- scripts/eval for evaluating LLMs on ICL tasks
- mcli/ to launch workloads via MosaicML platform
Getting started
Installation (summary commands exactly as written in README):
git clone https://github.com/mosaicml/llm-foundry.git
cd llm-foundry
pip install -e ".[gpu]" # or `pip install -e .` if no NVIDIA GPU.
Docker path (recommended):
git clone https://github.com/mosaicml/llm-foundry.git
cd llm-foundry
pip install -e ".[gpu]" # or `pip install -e .` if no NVIDIA GPU.
Without Docker:
git clone https://github.com/mosaicml/llm-foundry.git
cd llm-foundry
# Creating and activate a virtual environment
python3 -m venv llmfoundry-venv
source llmfoundry-venv/bin/activate
pip install cmake packaging torch # setup.py requires these be installed
pip install -e ".[gpu]" # or `pip install -e .` if no NVIDIA GPU.
TransformerEngine and amp_fp8 support
pip install flash-attn --no-build-isolation
pip install git+https://github.com/NVIDIA/TransformerEngine.git@stable
AMD (BETA) and Intel Gaudi notes are provided in README with branch references and commands.
Quickstart (example workflow)
cd scripts
# Convert C4 dataset to StreamingDataset format
python data_prep/convert_dataset_hf.py \
--dataset allenai/c4 --data_subset en \
--out_root my-copy-c4 --splits train_small val_small \
--concat_tokens 2048 --tokenizer EleutherAI/gpt-neox-20b --eos_text '<|endoftext|>'
# Train an MPT-125m model for 10 batches
composer train/train.py \
train/yamls/pretrain/mpt-125m.yaml \
variables.data_local=my-copy-c4 \
train_loader.dataset.split=train_small \
eval_loader.dataset.split=val_small \
max_duration=10ba \
eval_interval=0 \
save_folder=mpt-125m
# Convert the model to HuggingFace format
python inference/convert_composer_to_hf.py \
--composer_path mpt-125m/ep0-ba10-rank0.pt \
--hf_output_path mpt-125m-hf \
--output_precision bf16 \
# --hf_repo_for_upload user-org/repo-name
# Evaluate the model on a subset of tasks
composer eval/eval.py \
eval/yamls/hf_eval.yaml \
icl_tasks=eval/yamls/copa.yaml \
model_name_or_path=mpt-125m-hf
# Generate responses to prompts
python inference/hf_generate.py \
--name_or_path mpt-125m-hf \
--max_new_tokens 256 \
--prompts \
"The answer to life, the universe, and happiness is" \
"Here's a quick recipe for baking chocolate chip cookies: Start by"
Recent releases
- v0.22.0 (2025-07-29): test changes using foundry images; bias in QKV input projections for Qwen2 model series
- v0.21.0 (2025-05-31): Torch version bumped to 2.7.0; Support FSDP2 via ENV VAR: FSDP_VERSION=2
- v0.20.0 (2025-04-29): Dev 0.20.0.dev0; update example yamls to 0.19.0
- v0.19.0 (2025-04-07): Python 3.12 bump; deprecate Python 3.9; use llmfoundry ima
- v0.18.0 (2025-03-18): Torch bumped to 2.6.0; sparse support adjustments
Traction
4435 stars, 589 forks, 66 open issues
Behind the repo
Not provided in the README facts block
Caveats
License: Apache-2.0 Created: 2023-04-28 Last push: 2026-03-25 Languages: Python Dependencies/compatibility notes include PyTorch versions and Docker image recommendations; AMD and Intel Gaudi sections exist with branch references for experimental support.