mlsys-experiments
Use when designing or auditing the evaluation of an MLSys paper, selecting representative workloads and hardware, tuning baselines symmetrically, reporting throughput, latency tails, memory, cost, and quality together, structuring ablations that attribute gains to mechanisms, and building scaling and sensitivity evidence reviewers trust.
Profile →npx skills add brycewang-stanford/Awesome-Journal-Skills --skill mlsys-experiments --agent claude-code
Same command for any agent — swap --agent for codex, cursor, copilot.
Weekly change comes from our own snapshots, not the repository page — it measures attention, not adoption.
# MLSys Experiments Use this before the evaluation is frozen. An MLSys evaluation exists to answer one compound question: *does the named mechanism deliver the claimed system-level payoff on workloads that matter, at acceptable cost, for understood reasons?* Each clause needs its own evidence, and this venue — which published the MLPerf Training methodology in its own proceedings — holds measurement to benchmark-committee standards. ## Workload selection: the make-or-break choice "Unrepresentative workload" is the modal fatal objection at this venue. Defend the choice explicitly: - Anchor on recognizable workloads: MLPerf-family tasks, widely used open models, or published request traces. Bespoke workloads need a characterization section showing their statistics (request rates, sequence lengths, arrival burstiness, model mix). - Cover the axis your mechanism exploits: a scheduler for bursty load must be tested on bursty *and* uniform load; the uniform case bounds your claim, it does not weaken it. - Include at least one workload where you expect little or no gain, and show it. A non-win row is the cheapest credibility purchase available. - State workload licensing/provenance — trac
- Workload selection: the make-or-break choice
- Baseline discipline
- The reporting quartet
- Ablation = attribution
- Scaling and sensitivity evidence
- Statistical floor for systems numbers
- Evaluation-design review before running
- Common reject patterns to design against
- Output format
What does the mlsys-experiments skill do?
Use when designing or auditing the evaluation of an MLSys paper, selecting representative workloads and hardware, tuning baselines symmetrically, reporting throughput, latency tails, memory, cost, and quality together, structuring ablations that attribute gains to mechanisms, and building scaling and sensitivity evidence reviewers trust.
How do I install it?
Run `npx skills add brycewang-stanford/Awesome-Journal-Skills --skill mlsys-experiments --agent claude-code` — it drops the skill into your project so the agent can pick it up. Swap the --agent value for codex, cursor or copilot if you use one of those.
Where does this skill come from?
From brycewang-stanford/Awesome-Journal-Skills, a repository with 909 stars. We read it straight from the repository tree rather than a submitted listing, so what you see here is what is actually published.
Is a popular skill a good skill?
Not necessarily. Stars measure attention, not adoption — a repository can trend for a week and be abandoned. That is why we show the weekly change from our own snapshots next to the total, instead of a single flattering number.