llm-pretraining
Plan and execute large language model pretraining from data preparation to checkpoint management
npx skills add majiayu000/claude-skill-registry --skill llm-pretraining --agent claude-code
Same command for any agent — swap --agent for codex, cursor, copilot.
Weekly change comes from our own snapshots, not the repository page — it measures attention, not adoption.
# LLM Pretraining End-to-end pretraining workflow: data mixing, tokenizer co-design, scaling law estimation, compute budgeting, checkpoint management, and training stability monitoring. ## Decision Table | Scale (params) | Data Budget | Batch Size (tokens) | LR | Key Considerations | |---|---|---|---|---| | 100M-500M | 5-10B tokens | 256K-512K | 3e-4 | Quick experiments, single-node | | 1-3B | 20-60B tokens | 512K-1M | 2e-4 | Chinchilla ~20 tok/param | | 7-13B | 140-500B tokens | 1M-2M | 1.5e-4 | Multi-node, activation ckpt | | 30-70B | 1-2T tokens | 2M-4M | 1e-4 | TP+PP, quality annealing phase | | 200B+ | 4T+ tokens | 4M-8M | 6e-5 | MoE likely, multi-phase curriculum | ## Data Mixing and Curriculum ### Multi-Phase Training Configuration ```python from dataclasses import dataclass @dataclass class DomainSource: name: str path: str weight: float # fraction of tokens from this source quality_threshold: float # min quality score to include dedup_strategy: str # "exact", "fuzzy", "none" @dataclass class TrainingPhase: name: str token_budget: int sources: list[DomainSource] batch_size_tokens: int learning_rate: float def validate(self): total = sum(s.weight for s in self.sources) asser
- Decision Table
- Data Mixing and Curriculum
- Multi-Phase Training Configuration
- Tokenizer-Data Co-Design
- Scaling Law Estimation
- Checkpoint Management
- Loss Spike Detection
- Gotchas
What does the llm-pretraining skill do?
Plan and execute large language model pretraining from data preparation to checkpoint management
How do I install it?
Run `npx skills add majiayu000/claude-skill-registry --skill llm-pretraining --agent claude-code` — it drops the skill into your project so the agent can pick it up. Swap the --agent value for codex, cursor or copilot if you use one of those.
Where does this skill come from?
From majiayu000/claude-skill-registry, a repository with 534 stars. We read it straight from the repository tree rather than a submitted listing, so what you see here is what is actually published.
Is a popular skill a good skill?
Not necessarily. Stars measure attention, not adoption — a repository can trend for a week and be abandoned. That is why we show the weekly change from our own snapshots next to the total, instead of a single flattering number.