icse-experiments
Use when designing or auditing the evaluation of an ICSE research-track paper, covering subject and benchmark selection, baseline fairness, statistical tests and effect sizes as SE reviewers expect them, qualitative-methods rigor, ablations for AI-based techniques, and threats-driven study design.
Profile →npx skills add brycewang-stanford/Awesome-Journal-Skills --skill icse-experiments --agent claude-code
Same command for any agent — swap --agent for codex, cursor, copilot.
Weekly change comes from our own snapshots, not the repository page — it measures attention, not adoption.
# ICSE Experiments Design the study to survive an empiricist's audit, because that is who reviews it. ICSE's rigor criterion scores "thoroughness and completeness of an evaluation" (2027 call wording, read 2026-07-08), and the community has strong default expectations per study type that function as unwritten checklists. ## Evidence proportional to claim | Claim shape | Minimum evidence ICSE reviewers expect | |---|---| | "Technique X finds more bugs than Y" | Real subject programs, Y actually run (not quoted from its paper), same budget/timeout, statistics + effect size | | "Developers struggle with Z" | Systematic observation: survey with stated sampling, interviews to saturation, or instrumented behavior — not anecdote | | "LLM-based tool solves task T" | Ablations over prompts/models, contamination discussion, cost reporting, non-LLM baseline where one exists | | "Metric M predicts defects" | Multiple projects, time-aware splits, comparison against trivial baselines (size, churn) | | "Our benchmark/dataset enables research" | Construction protocol, quality validation sample, license clarity, comparison with existing sets | The recurring failure is claim-evidence mismatch: a gen
- Evidence proportional to claim
- Subjects and benchmarks
- Baselines and fairness
- Statistics as the SE community practices them
- Qualitative rigor
- Threats-driven design
- Sanity experiments that catch your own bugs
- Ablations for AI-era SE papers
- Output format
What does the icse-experiments skill do?
Use when designing or auditing the evaluation of an ICSE research-track paper, covering subject and benchmark selection, baseline fairness, statistical tests and effect sizes as SE reviewers expect them, qualitative-methods rigor, ablations for AI-based techniques, and threats-driven study design.
How do I install it?
Run `npx skills add brycewang-stanford/Awesome-Journal-Skills --skill icse-experiments --agent claude-code` — it drops the skill into your project so the agent can pick it up. Swap the --agent value for codex, cursor or copilot if you use one of those.
Where does this skill come from?
From brycewang-stanford/Awesome-Journal-Skills, a repository with 909 stars. We read it straight from the repository tree rather than a submitted listing, so what you see here is what is actually published.
Is a popular skill a good skill?
Not necessarily. Stars measure attention, not adoption — a repository can trend for a week and be abandoned. That is why we show the weekly change from our own snapshots next to the total, instead of a single flattering number.