remote-gpu-trainer
Deploy, monitor, and debug long GPU jobs on RENTED/remote instances (AutoDL, RunPod, vast.ai, Lambda, Slurm, K8s): teardown/billing safety, spot resilience, resumable checkpointing, OOM/NaN triage.
Profile →npx skills add sickn33/agentic-awesome-skills --skill remote-gpu-trainer --agent claude-code
Same command for any agent — swap --agent for codex, cursor, copilot.
Weekly change comes from our own snapshots, not the repository page — it measures attention, not adoption.
What it does
Deploy and babysit long-running GPU jobs on rented boxes you don't own, across any platform, and get the result off the box before the meter or a preemption kills it. The skill enforces a six-phase lifecycle, platform profiles, and per-platform billing/teardown rules to detach work, preserve results, and stop billing safely. It emphasizes checkpoint-to-durable + idempotent resume, platform-aware teardown, and monitoring patterns for long jobs. It supports long-running tasks such as training, evaluation, ablations, batch inference, or large data processing on AutoDL, RunPod, vast.ai, Lambda, Slurm, or Kubernetes, including Chinese platforms and bare SSH boxes.
How it works
- Phase-based workflow: Phase 0 environment audit; Phase 1 SSH credentials; Phase 2 CPU smoke gate; Phase 3 detached launch; Phase 4 durable monitoring with a four-layer architecture; Phase 5 aggregate/verify and teardown with a strict teardown gate.
- Uses a per-platform profile (profiles/<platform>.md) that maps the abstract verbs up/push/run/watch/pull/down to concrete commands and billing actions.
- Encourages reusing existing tooling and prebuilt image stacks; does not perform new package installations on rental boxes unless needed.
- Implements checkpointing to durable storage and idempotent resume to survive SSH drops, Slurm/K8s rescheduling, or spot preemption.
- Enforces user-facing safeguards before destructive actions: no release/terminate/destroy until checkpoints are pulled and verified locally with explicit user approval.
When to use it
Use when deploying, training, monitoring, or troubleshooting long-running GPU jobs on a RENTED or remote instance you do not own, including AutoDL, RunPod, vast.ai, Lambda, Paperspace, Chinese platforms (China), bare SSH boxes, Slurm, or Kubernetes; supports single or multi-instance scenarios.
What it can touch
- Requires a shell + SSH (or platform CLI/API) to drive the remote box; scripts are bash/python.
- Tools declared: claude-code, codex, cursor.
- Phase steps reference scripts/ssh_transport.md, references/monitoring_patterns.md, scripts/health_patrol.sh.template, and various profile-specific files. The agent interacts with remote boxes using SSH or cloud APIs and runs wrapper/training code in scripts under the skill. The exact touchpoints include running commands, uploading code/data, launching detached sessions, and pulling results.
Caveats
- Safe risk classification; license MIT.
- Per-platform behavior varies by profile (e.g., stop/terminate semantics and disk retention), and users should verify platform-specific teardown rules before action.
- The teardown and meter-stop actions are irreversible on many platforms; confirmation and local verification via load/verify are required before releasing resources.
- Requires compatible agent (e.g., Claude Code, Codex, Cursor) and an environment that supports SSH or platform API access.
# remote-gpu-trainer — Remote GPU Job Orchestration ## Overview Deploy and babysit long-running GPU jobs on **rented boxes you don't own**, across any platform, and get the result off the box before the meter or a preemption kills it. The core insight: **you are a short-term tenant on someone else's machine** — so the job is to *detach the work, make the result outlive the instance, and stop the meter safely*, not to provision a cluster. This skill is **platform-agnostic at the core, platform-specific at the edges**: a fixed set of operating principles + a 6-phase lifecycle that hold everywhere, plus one **profile per platform** (`profiles/<platform>.md`) that owns every concrete path, proxy, billing verb, and spot semantic. Its defensible value is the union the big orchestrators skip: **Chinese cgroup-isolated rentals + bare-SSH cheap boxes + the disk-budget / monitoring / teardown reality** that *is* the job on metered hardware. ## When to Use This Skill Use whenever the user deploys, trains, monitors, or troubleshoots a long-running GPU job on a **RENTED or remote instance they do not own** — training, eval, ablation sweeps, batch inference, or large data processing — on AutoDL,
- Overview
- When to Use This Skill
- When NOT to use — and what to use instead
- Operating principles (the WHY — 10 invariants)
- Code discipline (the wrapper & training scripts you write)
- Pick your platform profile FIRST
- Default workflow (6 phases)
- Parallel ablation fan-out
- Quick reference — the four facts that bite per platform
- Common gotchas (top 8 inline — full catalog in references/)
- When training itself breaks (the model, not the platform)
- Companion skills (separate installs; REQUIRED reading where present)
- Getting better over time (capture new gotchas + personalize)
- Limitations
What does the remote-gpu-trainer skill do?
Deploy, monitor, and debug long GPU jobs on RENTED/remote instances (AutoDL, RunPod, vast.ai, Lambda, Slurm, K8s): teardown/billing safety, spot resilience, resumable checkpointing, OOM/NaN triage.
How do I install it?
Run `npx skills add sickn33/agentic-awesome-skills --skill remote-gpu-trainer --agent claude-code` — it drops the skill into your project so the agent can pick it up. Swap the --agent value for codex, cursor or copilot if you use one of those.
Where does this skill come from?
From sickn33/agentic-awesome-skills, a repository with 44,414 stars. We read it straight from the repository tree rather than a submitted listing, so what you see here is what is actually published.
Is a popular skill a good skill?
Not necessarily. Stars measure attention, not adoption — a repository can trend for a week and be abandoned. That is why we show the weekly change from our own snapshots next to the total, instead of a single flattering number.