mlsys-reproducibility
Use when hardening the reproducibility of MLSys performance claims, pinning the full system layer from driver to interconnect, separating ML randomness from systems noise, choosing repetition counts and variance reporting for throughput and latency numbers, and disclosing hardware, workloads, and cost so strangers can re-measure results.
Profile →npx skills add brycewang-stanford/Awesome-Journal-Skills --skill mlsys-reproducibility --agent claude-code
Same command for any agent — swap --agent for codex, cursor, copilot.
Weekly change comes from our own snapshots, not the repository page — it measures attention, not adoption.
# MLSys Reproducibility Use this while experiments are still running — reproducibility at this venue is a measurement-design property, not a packaging afterthought. An MLSys claim is typically "system A beats system B by X% on workload W on hardware H," and every one of those four variables can silently drift. The venue's culture (badge-based artifact evaluation, the MLPerf benchmark lineage published in its own proceedings) means reviewers assume performance numbers will eventually be re-measured by someone else. ## Two kinds of nondeterminism — control them separately | Source | Examples | Control | |---|---|---| | ML randomness | Init seeds, data order, dropout, sampling temperature | Fix and log seeds; report across-seed variation where accuracy matters | | Systems noise | Clock boosting/thermal state, co-tenant interference, NUMA/PCIe placement, network jitter, filesystem caches | Warmup phases, repeated trials, exclusive nodes, pinned placement, reporting distributions | Papers routinely fix seeds meticulously while leaving thermal state and placement uncontrolled — backwards for a performance paper, where systems noise usually dwarfs seed effects on latency numbers. ## The e
- Two kinds of nondeterminism — control them separately
- The environment pin — deeper than requirements.txt
- Measurement harness discipline
- Disclosure floor for the paper itself
- Common measurement bugs this venue catches
- Pre-submission reproducibility drill
- Cycle-volatility warning
- Output format
What does the mlsys-reproducibility skill do?
Use when hardening the reproducibility of MLSys performance claims, pinning the full system layer from driver to interconnect, separating ML randomness from systems noise, choosing repetition counts and variance reporting for throughput and latency numbers, and disclosing hardware, workloads, and cost so strangers can re-measure results.
How do I install it?
Run `npx skills add brycewang-stanford/Awesome-Journal-Skills --skill mlsys-reproducibility --agent claude-code` — it drops the skill into your project so the agent can pick it up. Swap the --agent value for codex, cursor or copilot if you use one of those.
Where does this skill come from?
From brycewang-stanford/Awesome-Journal-Skills, a repository with 984 stars. We read it straight from the repository tree rather than a submitted listing, so what you see here is what is actually published.
Is a popular skill a good skill?
Not necessarily. Stars measure attention, not adoption — a repository can trend for a week and be abandoned. That is why we show the weekly change from our own snapshots next to the total, instead of a single flattering number.