training-llms-megatron

npx skills add Orchestra-Research/AI-Research-SKILLs --skill megatron-core --agent claude-code

Same command for any agent — swap --agent for codex, cursor, copilot.

# Megatron-Core - Large-Scale LLM Training ## Quick start Megatron-Core trains LLMs from 2B to 462B parameters with up to 47% Model FLOP Utilization on H100 GPUs through advanced parallelism strategies. **Installation**: ```bash # Docker (recommended) docker run --gpus all -it --rm nvcr.io/nvidia/pytorch:25.04-py3 # Or pip pip install megatron-core ``` **Simple distributed training**: ```bash # Train with 2 GPUs using data parallelism torchrun --nproc_per_node=2 examples/run_simple_mcore_train_loop.py # Or LLaMA-3 8B training ./examples/llama/train_llama3_8b_fp8.sh ``` ## Common workflows ### Workflow 1: Train LLaMA-style model with 3D parallelism Copy this checklist: ``` LLaMA Training Setup: - [ ] Step 1: Choose parallelism configuration - [ ] Step 2: Configure training hyperparameters - [ ] Step 3: Launch distributed training - [ ] Step 4: Monitor performance metrics ``` **Step 1: Choose parallelism configuration** Model size determines parallelism strategy: | Model Size | GPUs | Tensor Parallel | Pipeline Parallel | Data Parallel | Context Parallel | |------------|------|-----------------|-------------------|---------------|------------------| | 7B | 8 | 1 | 1 | 8 | 1 | | 13B |

Docker (recommended)
docker run --gpus all -it --rm nvcr.io/nvidia/pytorch:25.04-py3
Or pip
pip install megatron-core
Train with 2 GPUs using data parallelism
torchrun --nproc_per_node=2 examples/run_simple_mcore_train_loop.py
Or LLaMA-3 8B training
train_llama_70b.sh
LLaMA 70B configuration
Training hyperparameters