sigir-reproducibility
Use when strengthening the reproducibility of a SIGIR paper or preparing a SIGIR Reproducibility track submission — pinning the retrieval pipeline, seeds and variance for neural rankers, documenting collection versions and index settings, and structuring a reproduction study of published IR results with honest divergence analysis.
Profile →npx skills add brycewang-stanford/Awesome-Journal-Skills --skill sigir-reproducibility --agent claude-code
Same command for any agent — swap --agent for codex, cursor, copilot.
Weekly change comes from our own snapshots, not the repository page — it measures attention, not adoption.
# SIGIR Reproducibility Reproducibility is unusually load-bearing at SIGIR for two reasons. First, the venue runs a dedicated **Reproducibility track** (its own track in 2026, split out of the former combined Resource & Reproducibility track — budget and dates 待核实 on the current page), so reproduction studies are publishable first-class work. Second, the field's own literature documents how often reported IR gains fail to replicate under matched tuning — reviewers of *regular* papers therefore read reproducibility signals as a proxy for whether the gains are real. ## Why IR results drift: the usual suspects | Drift source | Typical symptom | Pin it by | |---|---|---| | Collection version | "MS MARCO" numbers off by points | Exact version/split ids, `ir_datasets` identifiers, checksums | | Index-time analysis | BM25 baseline differs across papers | Scripted index build; record stemmer, stopwords, k1/b | | Doc processing for neural models | Recall@k shifts | Max length, stride, title concatenation recorded as config | | Truncated vs judged pools | Inflated dense-retrieval scores | State pooling; report judged@k alongside nDCG | | Seeds and nondeterminism | ±0.005 nDCG run-to-run | Mu
- Why IR results drift: the usual suspects
- Minimum reporting block for any empirical SIGIR paper
- Writing a Reproducibility track paper
- Decay planning: the three-year audience
- Reproducibility as review defense for regular papers
- Quick self-audit before submission
- Output format
What does the sigir-reproducibility skill do?
Use when strengthening the reproducibility of a SIGIR paper or preparing a SIGIR Reproducibility track submission — pinning the retrieval pipeline, seeds and variance for neural rankers, documenting collection versions and index settings, and structuring a reproduction study of published IR results with honest divergence analysis.
How do I install it?
Run `npx skills add brycewang-stanford/Awesome-Journal-Skills --skill sigir-reproducibility --agent claude-code` — it drops the skill into your project so the agent can pick it up. Swap the --agent value for codex, cursor or copilot if you use one of those.
Where does this skill come from?
From brycewang-stanford/Awesome-Journal-Skills, a repository with 984 stars. We read it straight from the repository tree rather than a submitted listing, so what you see here is what is actually published.
Is a popular skill a good skill?
Not necessarily. Stars measure attention, not adoption — a repository can trend for a week and be abandoned. That is why we show the weekly change from our own snapshots next to the total, instead of a single flattering number.