experimentation-analytics
How to read experiment results without fooling yourself. Confidence intervals, p-values, multiple testing, sequential testing, CUPED, heterogeneous treatment effects, ratio metrics, network effects, dashboard reconciliation, and the interpretation failures that produce confidently wrong shipping decisions.
Profile →npx skills add rampstackco/claude-skills --skill experimentation-analytics --agent claude-code
Same command for any agent — swap --agent for codex, cursor, copilot.
Weekly change comes from our own snapshots, not the repository page — it measures attention, not adoption.
What it does
Explains how to read experiment result panels to avoid misinterpretation. It emphasizes understanding confidence intervals, p-values, multiple testing corrections, sequential testing, CUPED, heterogeneous treatment effects, ratio metrics, and dashboard reconciliation. It targets product managers and data analysts who interpret results together and need a shared vocabulary that highlights common statistical pitfalls.
How it works
Describes what a result panel should expose: per-variant metrics, lift with both absolute and relative values, p-values or Bayesian probabilities, variance reduction techniques like CUPED, guardrail statuses, pre-registered segments, test status, and a time series of lift. Provides interpretation rules for confidence intervals, explains the meaning and limits of p-values, covers multiple testing corrections (Bonferroni, BH), and explains sequential testing concepts (mSPRT, group sequential designs, anytime-valid CIs). It discusses CUPED adjustments and how they affect point estimates vs. precision, and outlines heterogeneous treatment effects, including how to pre-register segments and avoid overinterpreting post-hoc findings. It also notes platform-specific implications and the need for dashboard reconciliation when panel numbers diverge from BI sources.
When to use it
Uses when you are reading an experiment result panel and about to make a ship/kill/iterate decision.
What it can touch
Not specified in this excerpt beyond mentioning that platform results panels, raw data, and pre-experiment data are involved. The skill references various platforms (e.g., Statsig, Eppo, GrowthBook, PostHog, Amplitude, Optimizely) and general analytics surfaces, but does not enumerate executable touch rules here.
Caveats
Assumes experiment design is solid and platform results are technically correct. Warns that misreads can arise from underpowered tests, peeking, or using ratio metrics without proper delta-method corrections. It emphasizes pre-registration of primary metrics/segments and explains limitations of p-values and CIs without prescribing guarantees.
# Experimentation Analytics A data-team-mentor's playbook for interpreting experiment results without fooling yourself. The result panel is the moment-of-truth for an experiment. The numbers on it determine whether you ship, kill, or iterate. They also expose every shortcut taken in the design phase: an underpowered test produces wide confidence intervals; a peeked test produces a too-narrow p-value; a ratio metric without delta-method correction produces overconfident lift estimates. Most ship-the-wrong-thing decisions trace back to misreading the result panel. This skill is the discipline that prevents misreading. It assumes the experiment was designed well (see the `experiment-design` skill). It assumes the platform's results panel is technically correct (most modern platforms are; some older ones are not). It assumes you can read a number off a screen. The hard part is knowing what each number actually means and what it does not, and that is what is here. When to use this skill: any time you are reading an experiment result panel and about to make a ship, kill, or iterate decision. --- ## What this skill is for This skill covers result interpretation, the statistical concepts t
- What this skill is for
- The result panel: what every modern platform should expose
- Confidence intervals: the most important number
- P-values: what they mean and what they do not
- Multiple testing corrections
- Sequential testing math
- CUPED variance reduction
- Heterogeneous treatment effects (HTE) and segments
- Ratio metrics and the delta method
- Bayesian vs frequentist results panels
- Network effects and SUTVA violation
- Dashboard metric vs experiment metric reconciliation
- Long-term effect estimation
- Common interpretation failures
What does the experimentation-analytics skill do?
How to read experiment results without fooling yourself. Confidence intervals, p-values, multiple testing, sequential testing, CUPED, heterogeneous treatment effects, ratio metrics, network effects, dashboard reconciliation, and the interpretation failures that produce confidently wrong shipping decisions.
How do I install it?
Run `npx skills add rampstackco/claude-skills --skill experimentation-analytics --agent claude-code` — it drops the skill into your project so the agent can pick it up. Swap the --agent value for codex, cursor or copilot if you use one of those.
Where does this skill come from?
From rampstackco/claude-skills, a repository with 515 stars. We read it straight from the repository tree rather than a submitted listing, so what you see here is what is actually published.
Is a popular skill a good skill?
Not necessarily. Stars measure attention, not adoption — a repository can trend for a week and be abandoned. That is why we show the weekly change from our own snapshots next to the total, instead of a single flattering number.