hyperpod-performance-debugger
Diagnose performance issues on Amazon SageMaker HyperPod clusters — uneven NCCL bandwidth across nodes and poor filesystem throughput. Read-only. Surfaces host-side signals (Xid, ECC, NVLink, EFA reachability, FSx saturation) and routes to the appropriate sibling skill (hyperpod-node-debugger, hyperpod-nccl, hyperpod-version-checker, hyperpod-issue-report) for any remediation. Triggers on uneven NCCL across nodes, straggler node, FSx slow, checkpoint slow, dataloader slow, filesystem bottleneck, FSx throughput, cross-AZ latency, topology mismatch.
Profile →npx skills add awslabs/agent-plugins --skill hyperpod-performance-debugger --agent claude-code
Same command for any agent — swap --agent for codex, cursor, copilot.
Weekly change comes from our own snapshots, not the repository page — it measures attention, not adoption.
# HyperPod Performance Debugger 1. **Uneven NCCL performance across nodes** — workload faster on some node sets than others, pairwise bandwidth variance, suspected straggler. 2. **Poor filesystem performance** — training stalled on data loading, checkpoint save/load dominating step time, FSx throughput saturated. ## Scope and delegation Route findings outside the two in-scope scenarios to the owner skill below. | Concern observed | Route to | | ---------------------------------------------------------------------- | ------------------------------------------------------------ | | GPU hardware fault, ECC, NVLink, Xid, DCGM diagnostics, drain/replace | `hyperpod-node-debugger` (§ F Hardware/Auto-Repair, § G GPU) | | `Cannot allocate memory` at `os.fork()`, root volume exhausted | `hyperpod-node-debugger` (§ I Resource Exhaustion) | | NCCL timeouts, hangs, AllReduce stalls, EFA TCP fallback, RDMA memlock | `hyperpod-nccl` | | EFA / NCCL / CUDA / NVIDIA driver version drift across nodes | `hyperpod-version-checker` | | EFA self-referencing security-group rule missing — single node | `hyperpod-node-debugger` § A (EFA / Security Group) | | EFA self-referencing security-group rule missing
- Scope and delegation
- Operating policy
- Workflow
- Step 1: Run the snapshot
- Step 2: Match signal → section
- A: Uneven NCCL Performance
- Diagnostic pass (read-only)
- Pairwise NCCL bandwidth test
- Topology verification
- B: Poor Filesystem Performance
- Branches
- Continuous health coverage
- References
bash scripts/perf-snapshot.sh --cluster <CLUSTER_NAME_OR_ARN> --region <REGION> Scope to one suspect node: bash scripts/perf-snapshot.sh --cluster <C> --region <R> --node <INSTANCE_ID> aws cloudwatch get-metric-statistics --region <REGION> \
What does the hyperpod-performance-debugger skill do?
Diagnose performance issues on Amazon SageMaker HyperPod clusters — uneven NCCL bandwidth across nodes and poor filesystem throughput. Read-only. Surfaces host-side signals (Xid, ECC, NVLink, EFA reachability, FSx saturation) and routes to the appropriate sibling skill (hyperpod-node-debugger, hyperpod-nccl, hyperpod-version-checker, hyperpod-issue-report) for any remediation. Triggers on uneven NCCL across nodes, straggler node, FSx slow, checkpoint slow, dataloader slow, filesystem bottleneck, FSx throughput, cross-AZ latency, topology mismatch.
How do I install it?
Run `npx skills add awslabs/agent-plugins --skill hyperpod-performance-debugger --agent claude-code` — it drops the skill into your project so the agent can pick it up. Swap the --agent value for codex, cursor or copilot if you use one of those.
Where does this skill come from?
From awslabs/agent-plugins, a repository with 850 stars. We read it straight from the repository tree rather than a submitted listing, so what you see here is what is actually published.
Is a popular skill a good skill?
Not necessarily. Stars measure attention, not adoption — a repository can trend for a week and be abandoned. That is why we show the weekly change from our own snapshots next to the total, instead of a single flattering number.