hyperpod-slurm-debugger
Diagnostic-only skill for Slurm scheduler and node-daemon issues on Amazon SageMaker HyperPod Slurm clusters. Scope mirrors the HyperPod troubleshooting guide. Invoke when the user reports a Slurm node stuck in down/drain, "Node unexpectedly rebooted" after auto-repair, slurmd not running, jobs stuck PENDING with REASON=Resources while sinfo shows idle nodes, jobs stuck COMPLETING after node replacement, GRES/GPU counts wrong, scontrol ping failing, slurmctld unresponsive, an Action:Reboot/Replace request that did not trigger HyperPod auto-recovery, or auto-resume not restarting a job. Also tr
Profile →npx skills add awslabs/agent-plugins --skill hyperpod-slurm-debugger --agent claude-code
Same command for any agent — swap --agent for codex, cursor, copilot.
Weekly change comes from our own snapshots, not the repository page — it measures attention, not adoption.
# HyperPod Slurm Debugger Diagnostic-only. Identify and classify Slurm scheduler and node-daemon issues on HyperPod Slurm clusters. Do not run, recommend, or print any state-mutating command. For remediation, link to the official AWS or Slurm documentation. ## When to invoke Invoke when the user reports any of the symptoms in the [decision table](#decision-table). ## When NOT to invoke - Cluster has `Orchestrator.Eks` — invoke `hyperpod-node-debugger` or `hyperpod-nccl`. - Single-node hardware fault with healthy Slurm scheduler — invoke `hyperpod-node-debugger`. - NCCL training-hang investigation — invoke `hyperpod-nccl`. - Node unreachable via SSM — invoke `hyperpod-ssm`. ## Constraints - Read-only. Do not run, recommend, or print state-mutating commands. - For any remediation, link to AWS or Slurm docs. The user authorizes and executes. - IaC-managed cluster (Terraform / CloudFormation / CDK): warn that direct mutation drifts the live state from the IaC plan. Canonical recovery URLs: [references/slurm-details.md → Authoritative recovery documentation](references/slurm-details.md). ## Prerequisites - AWS CLI v2, authenticated for the target account and region with permissions: - `
- When to invoke
- When NOT to invoke
- Constraints
- Prerequisites
- Procedure
- Step 1 — Collect inputs
- Step 2 — Confirm orchestrator
- Step 3 — Run the diagnostic script
- Step 4 — Map findings → docs
- Decision table
- Defaults
- Error handling
- A: Node Down
- B: Unexpected Reboot
aws sagemaker describe-cluster --cluster-name <NAME/ARN> --region <REGION> \ bash scripts/slurm-diagnose.sh --cluster <NAME> --region <REGION> Scope to a node: bash scripts/slurm-diagnose.sh --cluster <NAME> --region <REGION> --node <SLURM_NODE>
What does the hyperpod-slurm-debugger skill do?
Diagnostic-only skill for Slurm scheduler and node-daemon issues on Amazon SageMaker HyperPod Slurm clusters. Scope mirrors the HyperPod troubleshooting guide. Invoke when the user reports a Slurm node stuck in down/drain, "Node unexpectedly rebooted" after auto-repair, slurmd not running, jobs stuck PENDING with REASON=Resources while sinfo shows idle nodes, jobs stuck COMPLETING after node replacement, GRES/GPU counts wrong, scontrol ping failing, slurmctld unresponsive, an Action:Reboot/Replace request that did not trigger HyperPod auto-recovery, or auto-resume not restarting a job. Also tr
How do I install it?
Run `npx skills add awslabs/agent-plugins --skill hyperpod-slurm-debugger --agent claude-code` — it drops the skill into your project so the agent can pick it up. Swap the --agent value for codex, cursor or copilot if you use one of those.
Where does this skill come from?
From awslabs/agent-plugins, a repository with 850 stars. We read it straight from the repository tree rather than a submitted listing, so what you see here is what is actually published.
Is a popular skill a good skill?
Not necessarily. Stars measure attention, not adoption — a repository can trend for a week and be abandoned. That is why we show the weekly change from our own snapshots next to the total, instead of a single flattering number.