Agent skill · AI & Agents

hyperpod-nccl

Diagnose NCCL failures and adjacent training-pod failures on HyperPod GPU clusters (EKS or Slurm) — training hangs, AllReduce / collective-op timeouts, EFA or libfabric errors, rendezvous failures, EFA TCP fallback, /dev/shm or memlock issues, NCCL version mismatch across pods, container OOM / exit-137 / OOMKilled, GPU OOM (CUDA out of memory), CrashLoopBackOff / Pending pods, MASTER_ADDR DNS, NetworkPolicy blocking. Not for single-node hardware faults (→ hyperpod-node-debugger § G) or cluster-creation EFA / SSM failures (→ hyperpod-cluster-debugger § A / § F).

Amazon Web Services - Labs12,649★ · +18/wk · 2 repos on radarProfile →
claude-codeships scriptsApache-2.0
Install
npx skills add awslabs/agent-plugins --skill hyperpod-nccl --agent claude-code

Same command for any agent — swap --agent for codex, cursor, copilot.

Facts
Files in the skill folder: 6
SKILL.md size: 13 KB
Bundled scripts: yes
Version: 0.0.1
Path: plugins/sagemaker-ai/skills/hyperpod-nccl/SKILL.md
Open the folder on GitHub →
Where it comes from
Stars: 850
Language: Python

Weekly change comes from our own snapshots, not the repository page — it measures attention, not adoption.

From the SKILL.md

# HyperPod NCCL Debugger **Operating policy.** Run read-only diagnostics yourself. Never run a command that changes cluster, node, or workload state — present each one as a **Suggested command (run this yourself)** block and wait for the customer. Destructive order: **investigate → reboot → replace** (replace destroys root + secondary volumes; not supported on Slurm controller nodes). Never discard training state on speculation. Diagnose NCCL failures on SageMaker HyperPod (EKS and Slurm). `scripts/nccl-diagnose.sh` reads state via AWS APIs, kubectl, and SSM, then prints each issue as `[FAIL] ... → references/<file>.md § <section>`. Read-only. **Signal sourcing:** `list-cluster-events` carries infrastructure-level state only (lifecycle, bootstrap, EFA health check, capacity, replacement, reboot, AMI rollback). It does **not** carry NCCL timeouts, GPU XID/ECC, or per-pod training signals — those come from pod logs, CloudWatch training streams, on-node SSM probes, and NCCL env audit. "No events" on a training-time NCCL issue is expected, not a clean bill of health. --- ## Workflow 1. Collect cluster name, region, namespace/job (EKS), exact NCCL error string. 2. Run the diagnostic (al

What's inside
Steps it walks through
  1. Workflow
  2. Step 1: Authenticate kubectl (EKS)
  3. Step 2: Run the diagnostic
  4. Remediation index
  5. Prerequisites
  6. Defaults
  7. Error handling
  8. IAM permissions
  9. Scale strategy
  10. Skill delegation
  11. Escalate to AWS Support
  12. Before opening the case
  13. Include in the case
  14. References
Ships with 5 files
  • references/debugging-guide.md
  • references/error-patterns-quick-ref.md
  • references/operations.md
  • references/performance-testing.md
  • scripts/nccl-diagnose.sh
Commands it runs
aws eks update-kubeconfig --name "$EKS_NAME" --region <REGION>
kubectl get nodes
bash scripts/nccl-diagnose.sh --cluster <HYPERPOD-NAME> --region <REGION>
Scope to an EKS job/namespace:
bash scripts/nccl-diagnose.sh --cluster <NAME> --region <REGION> --namespace <NS> --job <JOB>
Force orchestrator:
bash scripts/nccl-diagnose.sh --cluster <NAME> --region <REGION> --orchestrator slurm
Larger hardware sample (default 3):
bash scripts/nccl-diagnose.sh --cluster <NAME> --region <REGION> --sample-nodes 10
Specific node only:
More from agent-plugins
All skills →
dsqlBuild with Aurora DSQL — manage schemas, execute queries, handle migrations, diagnose query plans…Securitydataset-transformationGenerates code that transforms datasets between ML schemas for model training or evaluation. Use…Data & Analyticsaws-transformMigrate, modernize, and upgrade codebases to AWS. Run analysis on repos for tech debt, security…Securityaws-lambda-managed-instancesEvaluate, configure, and migrate workloads to AWS Lambda Managed Instances (LMI). Triggers on…DevOps & Clouddataset-evaluationValidates dataset formatting and quality for SageMaker model fine-tuning (SFT, DPO, or RLVR). Use…Data & Analyticsfinetuning-techniqueSelects a fine-tuning technique (SFT, DPO, RLVR, or RLAIF) for the user's use case and validates it…AI & Agents
About this skill
What does the hyperpod-nccl skill do?

Diagnose NCCL failures and adjacent training-pod failures on HyperPod GPU clusters (EKS or Slurm) — training hangs, AllReduce / collective-op timeouts, EFA or libfabric errors, rendezvous failures, EFA TCP fallback, /dev/shm or memlock issues, NCCL version mismatch across pods, container OOM / exit-137 / OOMKilled, GPU OOM (CUDA out of memory), CrashLoopBackOff / Pending pods, MASTER_ADDR DNS, NetworkPolicy blocking. Not for single-node hardware faults (→ hyperpod-node-debugger § G) or cluster-creation EFA / SSM failures (→ hyperpod-cluster-debugger § A / § F).

How do I install it?

Run `npx skills add awslabs/agent-plugins --skill hyperpod-nccl --agent claude-code` — it drops the skill into your project so the agent can pick it up. Swap the --agent value for codex, cursor or copilot if you use one of those.

Where does this skill come from?

From awslabs/agent-plugins, a repository with 850 stars. We read it straight from the repository tree rather than a submitted listing, so what you see here is what is actually published.

Is a popular skill a good skill?

Not necessarily. Stars measure attention, not adoption — a repository can trend for a week and be abandoned. That is why we show the weekly change from our own snapshots next to the total, instead of a single flattering number.

Keep going
All skills →
The full catalog, by category and by tool.
Open Source Radar →
The repositories these skills come from, watched daily.
The people behind them →
Makers and builders across the ecosystem.