spark-training-gotchas
Preflight and diagnose the ten known failure modes for ML training on NVIDIA DGX Spark. Use when a training run on DGX Spark fails to start, OOMs below the 128GB limit, slows down mid-run, or before any multi-hour training job on GB10.
Profile →npx skills add wshobson/agents --skill spark-training-gotchas --agent claude-code
Same command for any agent — swap --agent for codex, cursor, copilot.
Weekly change comes from our own snapshots, not the repository page — it measures attention, not adoption.
# Spark Training Gotchas DGX Spark's GB10 chip (Grace Blackwell, SM121, 128GB unified memory, aarch64) has ten recurring failure modes across launch, memory, thermals, bandwidth, and precision. Each is named G1–G10 so it can be checked by number — the numbering is load-bearing for tooling that runs these checks. Read this before a long run, not after hour six. ## When to Use This Skill - A training run fails to start, with an import error or a segfault that doesn't point at the real cause. - A run OOMs while `nvidia-smi` still shows headroom. - Throughput degrades partway through a run that started fine. - Before any multi-hour or multi-epoch job on GB10. - Wiring two Sparks together, before picking a parallelism strategy. - Choosing between FP8 and NVFP4 for a Spark-hosted run. ## Common Issues Quick Reference | # | Symptom | Fix | |---|---|---| | G1 | undefined symbol / segfault | cu130 wheel or container | | G2 | flash-attn wrong backend used | skip pip build; monkeypatch on NGC | | G3 | OOM despite headroom | drop page cache | | G4 | throughput drop / reboot | expect ~100W sustained cap | | G5 | memory-bound step slow | budget 180–192 GB/s | | G6 | cache evicted mid-run | one G
- When to Use This Skill
- Common Issues Quick Reference
- The Ten Gotchas
- G1: CUDA 12/13 ABI Mismatch
- G2: flash-attn — Skip the pip Build, Watch Unsloth's Auto-Detect
- G3: UMA OOM Below 128GB
- G4: Thermal Throttling
- G5: Bandwidth Ceiling
- G6: Global UMA Resource Contention
- G7: NVFP4 Slower Than FP8 on SM121
- G8: Stale Official Playbooks
- G9: Container-First, Not Bare Pip
- G10: Dual-Spark Is DDP/FSDP Only
- Fast Triage
python3 -c "import torch; print(torch.version.cuda)" # expect 13.x (G1); NGC builds have no +cu130 tag — that's not a failure
What does the spark-training-gotchas skill do?
Preflight and diagnose the ten known failure modes for ML training on NVIDIA DGX Spark. Use when a training run on DGX Spark fails to start, OOMs below the 128GB limit, slows down mid-run, or before any multi-hour training job on GB10.
How do I install it?
Run `npx skills add wshobson/agents --skill spark-training-gotchas --agent claude-code` — it drops the skill into your project so the agent can pick it up. Swap the --agent value for codex, cursor or copilot if you use one of those.
Where does this skill come from?
From wshobson/agents, a repository with 38,479 stars. We read it straight from the repository tree rather than a submitted listing, so what you see here is what is actually published.
Is a popular skill a good skill?
Not necessarily. Stars measure attention, not adoption — a repository can trend for a week and be abandoned. That is why we show the weekly change from our own snapshots next to the total, instead of a single flattering number.