pytorch-distributed
Distributed training strategies including DistributedDataParallel (DDP) and Fully Sharded Data Parallel (FSDP). Covers multi-node setup, checkpointing, and process management using torchrun. (ddp, fsdp, distributeddataparallel, torchrun, nccl, rank, process-group)
npx skills add majiayu000/claude-skill-registry --skill pytorch-distributed-cuba6112-skillfactory-fbadb72e --agent claude-code
Same command for any agent — swap --agent for codex, cursor, copilot.
Weekly change comes from our own snapshots, not the repository page — it measures attention, not adoption.
## Overview PyTorch Distributed enables training models across multiple GPUs and nodes. `DistributedDataParallel` (DDP) is the standard for multi-process data parallelism, while `Fully Sharded Data Parallel` (FSDP) shards model state to allow training models too large for a single GPU. ## When to Use Use DDP for general multi-GPU training on a single or multiple nodes. Use FSDP when model parameters, gradients, and optimizer states exceed the memory of a single GPU. ## Decision Tree 1. Does your model fit on one GPU? - YES: Use `DistributedDataParallel` (DDP). - NO: Use `Fully Sharded Data Parallel` (FSDP). 2. Are you launching the job? - USE: `torchrun` to handle environmental setup and fault recovery. 3. Are you saving a checkpoint? - DO: Only save on `rank == 0` to avoid file corruption and redundant I/O. ## Workflows 1. **Setting Up a DDP Training Job** 1. Initialize the process group using `dist.init_process_group()` with appropriate backend (e.g., 'nccl'). 2. Set the current device for the process using `torch.cuda.set_device(rank)`. 3. Wrap the model with `DistributedDataParallel`. 4. Wrap the dataset with a `DistributedSampler` to ensure unique data shards per process. 5. C
- Overview
- When to Use
- Decision Tree
- Workflows
- Non-Obvious Insights
- Evidence
- Scripts
- Dependencies
- References
What does the pytorch-distributed skill do?
Distributed training strategies including DistributedDataParallel (DDP) and Fully Sharded Data Parallel (FSDP). Covers multi-node setup, checkpointing, and process management using torchrun. (ddp, fsdp, distributeddataparallel, torchrun, nccl, rank, process-group)
How do I install it?
Run `npx skills add majiayu000/claude-skill-registry --skill pytorch-distributed-cuba6112-skillfactory-fbadb72e --agent claude-code` — it drops the skill into your project so the agent can pick it up. Swap the --agent value for codex, cursor or copilot if you use one of those.
Where does this skill come from?
From majiayu000/claude-skill-registry, a repository with 534 stars. We read it straight from the repository tree rather than a submitted listing, so what you see here is what is actually published.
Is a popular skill a good skill?
Not necessarily. Stars measure attention, not adoption — a repository can trend for a week and be abandoned. That is why we show the weekly change from our own snapshots next to the total, instead of a single flattering number.