nemo-automodel-distributed-training
Guide for selecting and configuring distributed training strategies in NeMo AutoModel, including FSDP2, Megatron FSDP, DDP, and parallelism settings.
npx skills add NVIDIA/skills --skill nemo-automodel-distributed-training --agent claude-code
Same command for any agent — swap --agent for codex, cursor, copilot.
Weekly change comes from our own snapshots, not the repository page — it measures attention, not adoption.
What it does
Recommends and documents distributed training strategies for NeMo AutoModel, focusing on FSDP2, Megatron FSDP, and DDP, and describes how to configure device meshes, parallelism (TP, PP, CP, EP), and related sub-configs.
How it works
Describes the overall distributed training model where a single MeshContext holds device meshes, strategy configs, and axis names. Provides strategy selection guidance and the exact YAML fields to set for TP, PP, CP, EP, and optional sub-configs like pipeline and moe. Includes examples showing the required fields and how world_size relates to dp_size, tp_size, pp_size, and cp_size. Outlines how MoE introduces a moe_mesh when ep_size > 1, and how MegatronFSDP differs in feature support. Covers various configuration blocks (distributed, pipeline, moe) and how they map to internal config objects and managers (FSDP2Manager, MegatronFSDPManager, DDPManager) during initialization.
When to use it
Allows adding or modifying distributed training strategies (FSDP2, HSDP, DDP), debugging multi-GPU or multi-node failures, configuring context or tensor parallelism, or tuning sharding settings.
What it can touch
Touches YAML keys under distributed (strategy, tp_size, pp_size, cp_size, ep_size, dp_size, dp_replicate_size, sequence_parallel, activation_checkpointing, defer_fsdp_grad_sync), as well as optional sub-configs:
- pipeline (pp_schedule, pp_microbatch_size, scale_grads_in_schedule)
- moe (reshard_after_forward, ignore_router_for_ac, wrap_outer_model)
Caveats
License: Apache-2.0. Strategy choices depend on model size, topology, and whether EP/PP/CP are supported. MegatronFSDP does not support EP, PP, or sequence_parallel. MoE requires ep_size > 1 and introduces moe_mesh; ep_size must divide dp_size * cp_size. The YAML fields and constraints are as described in the skill; no guarantees of training outcomes are stated.
# Distributed Training in NeMo AutoModel ## Purpose NeMo AutoModel uses PyTorch-native distributed training. All parallelism is orchestrated through a single `MeshContext` object that holds device meshes, strategy configs, and axis names. <!-- NVSkills catalog signing requested after PR #2937 (2026-07-31). --> ## Instructions For conceptual distributed-training questions, answer directly from the
What does the nemo-automodel-distributed-training skill do?
Guide for selecting and configuring distributed training strategies in NeMo AutoModel, including FSDP2, Megatron FSDP, DDP, and parallelism settings.
How do I install it?
Run `npx skills add NVIDIA/skills --skill nemo-automodel-distributed-training --agent claude-code` — it drops the skill into your project so the agent can pick it up. Swap the --agent value for codex, cursor or copilot if you use one of those.
Where does this skill come from?
From NVIDIA/skills, a repository with 2,789 stars. We read it straight from the repository tree rather than a submitted listing, so what you see here is what is actually published.
Is a popular skill a good skill?
Not necessarily. Stars measure attention, not adoption — a repository can trend for a week and be abandoned. That is why we show the weekly change from our own snapshots next to the total, instead of a single flattering number.