pldi-reproducibility
Use when hardening a PLDI paper's measurements against the SIGPLAN Empirical Evaluation Guidelines — warmup and steady-state discipline, variance and confidence reporting, principled benchmark choice, pinned toolchains, cross-platform validity, and a measurement log that survives artifact evaluation.
Profile →npx skills add brycewang-stanford/Awesome-Journal-Skills --skill pldi-reproducibility --agent claude-code
Same command for any agent — swap --agent for codex, cursor, copilot.
Weekly change comes from our own snapshots, not the repository page — it measures attention, not adoption.
# PLDI Reproducibility PLDI's methodological yardstick is written down: the SIGPLAN Empirical Evaluation Guidelines and their one-page checklist (Blackburn, Hauswirth, Berger, Hicks, Krishnamurthi, 2018; sigplan.org/Resources/EmpiricalEvaluation/, read 2026-07-08). Reviewers and artifact evaluators both reach for it. This skill turns the checklist into compiler-bench practice; `pldi-experiments` covers what to measure, this covers whether anyone can trust and repeat the measurement. ## Checklist, translated to PL systems | Guideline item | What it means for a compiler/runtime paper | |---|---| | Clearly stated claims | "1.17x geomean on suite S vs baseline B at -O2" — never "significant speedups" | | Suitable comparison | The strongest sensible baseline configuration, tuned as its authors intend | | Principled benchmark choice | The suite is justified; exclusions are listed with reasons, not silently dropped | | Adequate data analysis | Repetitions, variance, and an aggregation rule (geomean for ratios) stated in the paper | ## The measurement sins PLDI reviewers hunt - **No warmup discipline.** JIT-compiled and cache-sensitive workloads need documented warmup iterations before tim
- Checklist, translated to PL systems
- The measurement sins PLDI reviewers hunt
- A protocol worth writing down
- Compile-time and memory are claims too
- Tie-in to badges
- Output format
protocol.sh — executed, not described set -euo pipefail lscpu > results/env/cpu.txt; uname -a > results/env/os.txt cc --version > results/env/toolchain.txt for b in $(cat benchmarks/suite.list); do for i in $(seq 1 5); do ./run.sh "$b" >/dev/null; done # warmup for i in $(seq 1 30); do ./run.sh "$b" >> "results/raw/$b.csv"; done done python3 scripts/aggregate.py --stat geomean --ci 95 results/raw/
What does the pldi-reproducibility skill do?
Use when hardening a PLDI paper's measurements against the SIGPLAN Empirical Evaluation Guidelines — warmup and steady-state discipline, variance and confidence reporting, principled benchmark choice, pinned toolchains, cross-platform validity, and a measurement log that survives artifact evaluation.
How do I install it?
Run `npx skills add brycewang-stanford/Awesome-Journal-Skills --skill pldi-reproducibility --agent claude-code` — it drops the skill into your project so the agent can pick it up. Swap the --agent value for codex, cursor or copilot if you use one of those.
Where does this skill come from?
From brycewang-stanford/Awesome-Journal-Skills, a repository with 984 stars. We read it straight from the repository tree rather than a submitted listing, so what you see here is what is actually published.
Is a popular skill a good skill?
Not necessarily. Stars measure attention, not adoption — a repository can trend for a week and be abandoned. That is why we show the weekly change from our own snapshots next to the total, instead of a single flattering number.