dLLM is a Python library that unifies training, evaluation, and inference for diffusion-based language models, with multi-engine training and evaluation support and various example pipelines.
Collecting history — the radar snapshots this repo daily. The trend line appears after 3 days of data (1 so far).
What it is
dLLM is a library that unifies the training and evaluation of diffusion language models, bringing transparency and reproducibility to the entire development pipeline. It provides training pipelines (based on transformers Trainer) with support for LoRA, DeepSpeed, and Fully Sharded Data Parallel (FSDP), unified evaluation pipelines (lm-evaluation-harness), and minimal training/inference recipes for open-weight models.
How it works
It relies on components such as transformers-based training, PEFT for LoRA, DeepSpeed, and FSDP for scalable training, and lm-evaluation-harness for evaluation. It includes training modules (e.g., MDLMTrainer) and inference samplers (MDLMSampler) along with multiple example pipelines for different model families (llada, llada2, llada21, dream, a2d, editflow, fastdllm, rl). The repository includes entry points and training scripts that demonstrate typical usage.
Getting started
Installation
# create and activate conda environment
conda create -n dllm python=3.10 -y
conda activate dllm
# install pytorch with CUDA 12.4 (other pytorch/cuda versions should also work)
conda install cuda=12.4 -c nvidia
pip install torch==2.6.0 torchvision==0.21.0 torchaudio==2.6.0 \
--index-url https://download.pytorch.org/whl/cu124
# install dllm package
pip install -e .
(optional) Evaluation setup
# initialize `lm-evaluation-harness` submodule
git submodule update --init --recursive
# install submodule in editable mode with IFEval & Math dependencies
pip install -e "lm-evaluation-harness[ifeval,math]"
(optional) Slurm setup
For Slurm users, update scripts/train.slurm.sh for your cluster:
- #SBATCH --partition=mllm_safety # Note: adjust this for your cluster
- #SBATCH --quotatype=spot # Note: adjust this for your cluster
+ #SBATCH --partition=YOUR_PARTITION
+ #SBATCH --quotatype=YOUR_QUOTATYPE
Files
The repository contains modules for training/sampling, with pipelines for llada, llada2, llada21, dream, a2d, bert, fastdllm, rl, and editflow, plus entry points under examples for training, inference, and evaluation.
Training
A typical training entry script is shown in examples/llada/sft.py, including model/tokenizer setup, dataset handling, and creation of an MDLMTrainer. Training can be launched locally with accelerate or submitted to Slurm. Example commands illustrate 4-bit quantization, LoRA, and various accelerate configurations.
Inference
A typical inference script (examples/llada/sample.py) demonstrates loading a model and tokenizer, creating an MDLMSampler, and performing sample generation with prepared inputs. The repository also shows interactive chat scripts and a Fast-dLLM-based inference path for caching and decoding thresholds.
Evaluation
Evaluation commands use accelerate to run eval.py with specified tasks and model arguments, and there are scripts to evaluate LLaDA, Dream, and BERT-Chat on benchmarks, including examples/llada/eval.sh and related fastdllm evaluation scripts.
Traction
2660 stars, 282 forks, 20 open issues. Language: Python. License: Apache-2.0.