debug-distributed

npx skills add majiayu000/claude-skill-registry --skill debug-distributed --agent claude-code

Same command for any agent — swap --agent for codex, cursor, copilot.

# Debug Distributed Training Debugging guide for distributed training issues in AReaL (FSDP2, TP, CP, EP). ## When to Use This skill is triggered when: - Training hangs or deadlocks - Results differ across ranks or are numerically wrong - OOM errors in distributed settings - NCCL/communication errors or device mesh issues ## Debugging Principles ### Minimal Reproduction **Always follow the minimal demo principle**: Reproduce with the least amount of code to narrow down the issue faster. ```python # Bad: Debug in full training loop # Good: Create minimal script import torch import torch.distributed as dist dist.init_process_group("nccl") rank = dist.get_rank() # Reproduce the exact operation that fails tensor = torch.ones(10).cuda() dist.all_reduce(tensor) # <-- Isolate the failing op print(f"Rank {rank}: {tensor}") ``` **Reduction strategy:** 1. Remove unrelated model components 1. Use small tensor sizes 1. Reduce world_size to minimum (e.g., 2 GPUs) 1. Remove torch.compile if possible 1. Disable activation checkpointing ## Step-by-Step Debugging Guide ### 1. Hang Debugging (Deadlocks, Synchronization) **Environment Variables for Debugging**: ```bash # Full debug logging export TOR

Full debug logging
export TORCH_DISTRIBUTED_DEBUG=DETAIL
export NCCL_DEBUG=INFO
export NCCL_DEBUG_SUBSYS=ALL
torch.compile debugging
export TORCH_LOGS="+dynamo,recompiles"
export TORCHDYNAMO_VERBOSE=1
Find process IDs
ps aux | grep python
Dump call stack of specific rank