A Python-based, from-scratch transformer training pipeline for LLMs using PyTorch, supporting pretraining, SFT, reward models, and RL methods with configurable JSON-based stages and a Streamlit UI. Repository provides code structure, data prep, and training scripts. Latest push: 2026-06-24.
Collecting history — the radar snapshots this repo daily. The trend line appears after 3 days of data (1 so far).
What it is
Train LLM From Scratch provides a transformer model implemented in PyTorch, built from small pieces, to train a language model from raw text to an aligned assistant. It includes data processing, model components (MLP, attention, transformer blocks, full transformer), and post-training steps (SFT, reward model, PPO/DPO/GRPO). The README describes an end-to-end pipeline from raw text to a base model, SFT, reward modeling, and evaluation, with a focus on training a billion-parameter or million-parameter LLM on a single GPU.
How it works
- The project builds the Transformer from small modules (MLP, attention, transformer_block, transformer) and stacks them. Code structure shows modules under src/models and a post_training directory for SFT, reward model, and RL methods.
- Data preparation includes tokenization via r50k_base, converting text to token IDs, and generating loss masks for training. Data flows from raw text to token IDs, then to training-ready arrays for pretraining and post-training stages.
- The pipeline includes steps for pretraining data processing and later post-training stages (SFT, reward model, PPO, DPO, GRPO) driven by JSON config files in configs/ and CLI overrides.
Getting started
- Install in editable mode:
git clone https://github.com/FareedKhan-dev/train-llm-from-scratch.git
cd train-llm-from-scratch
pip install -e .
- Optional extras for specific parts:
pip install -e ".[train]" # datasets + wandb, for downloading data and logging
pip install -e ".[ui]" # streamlit + pandas + altair, for the control panel
pip install -e ".[docs]" # mkdocs, for the documentation site
pip install -e ".[all]" # everything
- There are two config systems:
- config/config.py (legacy pretraining)
- config/post_training_config.py plus JSON files in configs/ (all post-training stages)
- For fast checks there is a tiny configs/smoke/ variant per stage.
Getting started (example commands from README)
- Pretraining data download and preprocessing (legacy path):
python scripts/data_download.py # downloads the validation file + 1 training shard
python scripts/data_preprocess.py # tokenizes to data/train/pile_train.h5 and data/val/pile_dev.h5
- Pretraining data via the faster path:
python scripts/prepare_pretrain_data.py --split val --out data/pile_dev.h5
python scripts/prepare_pretrain_data.py --split train --num_shards 1 --out data/pile_train.h5
- SFT/DPO/RL data prep pipelines:
python scripts/prepare_sft_data.py # Alpaca + Dolly + GSM8K -> sft_packed.h5
python scripts/prepare_preference_data.py # HH-RLHF + UltraFeedback -> preferences.jsonl
python scripts/prepare_rl_prompts.py # GSM8K + arithmetic -> rl_prompts.jsonl
Recent releases
- RELEASES (latest 0): - none
Traction
- Stars: 8904
Behind the repo
- License: MIT
- Language: Python
- Created: 2025-01-12
- Last push: 2026-06-24
- Topics include: gemini, large-language-models, llm, openai, training, transformers
Caveats
- Issues: 4
- No listed releases beyond latest marker; README describes multiple training stages and scripts, with configuration driven via JSON files and CLI overrides.