Datawhalechina/llms-from-scratch-cn provides notebooks and tutorials to build GPT-like models from scratch, covering GLM4, Llama3, and RWKV6 architectures. It includes code implementations, step-by-step learning, and architecture discussions across multiple chapters and model types.
Collecting history — the radar snapshots this repo daily. The trend line appears after 3 days of data (1 so far).
What it is
LLMs From Scratch: Hands-on Building Your Own Large Language Models is a tutorial project that guides from zero coding to constructing large language model components. It includes code to create GPT-like models, pretraining, and fine-tuning, with notebooks and example implementations for multiple architectures (GLM4, Llama3, RWKV6).
How it works
The project presents chapters covering data handling, attention mechanisms, GPT from scratch, pretraining with unlabeled data, and fine-tuning for text classification and human feedback. It also provides architecture discussions for ChatGLM, Llama, and RWKV variants, with a directory of notebooks and scripts illustrating model construction and training workflows.
Getting started
- It contains notebooks and code under paths like Codes/ch02, Codes/ch03, Codes/ch04, and Model_Architecture_Discussions for model explorations. Specific files mentioned include ch02.ipynb, dataloader.ipynb, exercise-solutions.ipynb, ch03.ipynb, multihead-attention.ipynb, gpt.py, train.py, generate.py, and various rwkv-vX guides.
- The README references datasets and code implementations intended to help users build and train small, functioning LLMs to understand underlying principles.
Recent releases
- Releases: latest 0 with entry "- none". No separate release artifacts are listed.
Traction
- Stars: 4301
- Forks: 590
- Open issues: 14
Behind the repo
- Language: Jupyter Notebook
- License: none listed
- Created: 2024-02-28
- Last push: 2026-03-26
- Topics include glm, llama, llm, llms-from-scratch, rwkv
Caveats
- License: none listed in the repository metadata; a license file is not indicated in the provided facts.
- The README shows multiple chapters and forthcoming sections labeled as 即将发布 (to be released).