statistical-analysis
Hypothesis testing, power analysis, ablation design, and multiple comparison corrections for ML experiment evaluation.
npx skills add majiayu000/claude-skill-registry --skill statistical-analysis-jlaws-dotfiles --agent claude-code
Same command for any agent — swap --agent for codex, cursor, copilot.
Weekly change comes from our own snapshots, not the repository page — it measures attention, not adoption.
# Statistical Analysis for ML Experiments ## Test Selection ### Decision Table: Which Test to Use | Scenario | Test | Assumptions | When to Use | |----------|------|-------------|-------------| | Compare 2 models, paired runs | Paired permutation test | None (non-parametric) | Default choice for ML | | Compare 2 models, unpaired | Mann-Whitney U | Independent samples | Different datasets/splits | | Compare 2 means, normal data | Paired t-test | Normality, paired | Large N (>30) runs | | Compare >2 models | Friedman test | Paired, ordinal | Ranking across datasets | | Post-hoc after Friedman | Nemenyi test | Same as Friedman | Pairwise model comparison | | Compare proportions | McNemar's test | Paired binary | Classification correct/incorrect | | Correlation of metrics | Spearman's rho | Monotonic relationship | Rank-based, robust to outliers | | Effect of hyperparameters | ANOVA / Kruskal-Wallis | See specific test | Factorial ablation design | ### Decision Table: Confidence Interval Method | Situation | Method | N Seeds | |-----------|--------|---------| | Metric approximately normal | t-interval | >= 5 | | Unknown distribution | Bootstrap percentile | >= 10 | | Small N, heavy tai
- Test Selection
- Decision Table: Which Test to Use
- Decision Table: Confidence Interval Method
- Bootstrap Confidence Intervals
- Paired Permutation Test
- Multiple Comparison Corrections
- When to Correct
- Implementation
- Power Analysis
- Ablation Study Design
- Reporting Ablation Results
- Gotchas and Anti-Patterns
- P-Hacking via Metric Selection
- Underpowered Experiments
What does the statistical-analysis skill do?
Hypothesis testing, power analysis, ablation design, and multiple comparison corrections for ML experiment evaluation.
How do I install it?
Run `npx skills add majiayu000/claude-skill-registry --skill statistical-analysis-jlaws-dotfiles --agent claude-code` — it drops the skill into your project so the agent can pick it up. Swap the --agent value for codex, cursor or copilot if you use one of those.
Where does this skill come from?
From majiayu000/claude-skill-registry, a repository with 534 stars. We read it straight from the repository tree rather than a submitted listing, so what you see here is what is actually published.
Is a popular skill a good skill?
Not necessarily. Stars measure attention, not adoption — a repository can trend for a week and be abandoned. That is why we show the weekly change from our own snapshots next to the total, instead of a single flattering number.