experiment-design
A discipline for designing experiments (A/B tests, multivariate, holdouts) so the results actually answer the question you asked. Hypothesis writing, sample size, duration, segment analysis, interpretation, decision-making, and the common failure modes that produce confidently wrong shipping decisions.
Profile →npx skills add rampstackco/claude-skills --skill experiment-design --agent claude-code
Same command for any agent — swap --agent for codex, cursor, copilot.
Weekly change comes from our own snapshots, not the repository page — it measures attention, not adoption.
What it does
The skill provides a discipline and blueprint for designing experiments (A/B tests, multivariate, holdouts) to ensure results answer the original question. It covers hypothesis writing, sample size, duration, segment analysis, interpretation, decision-making, and common failure modes that lead to shipping decisions based on confounded or inconclusive results.
How it works
- Emphasizes designing experiments with a working experimentation platform in mind (examples of platforms are listed).
- Outlines a twelve-section framework for trustworthy results, including: Hypothesis discipline; Sample size and minimum detectable effect (MDE); Test duration; What NOT to A/B test; Segment analysis; Interaction effects; Ratio metrics and variance estimation; Network effects and two-sided markets; Sequential testing and the peeking problem; Pre-commitment vs p-hacking; Reading results and making the call; Common failures and fixes.
- For each consideration, it describes what to design or pre-commit before launching and how to interpret outcomes.
- Specifies that the skill designs experiments and that the platform implements execution.
When to use it
Use it any time you are about to design or interpret an experiment. Read the relevant section before you start, not after the test is running.
What it can touch
The skill assumes you have a working experimentation platform (Statsig, PostHog, GrowthBook, Optimizely, Amplitude, Eppo, Kameleoon; the platform does not matter for the principles). It does not cover feature flag operational mechanics or platform-specific tooling; references to other skills cover those areas. The skill produces the experiment design; the platform implements it.
Caveats
- Does not cover feature flag operational mechanics; see the feature-flagging skill for flag taxonomy, environment management, and stale-flag cleanup.
- Does not cover statistical analysis depth; see the experimentation-analytics skill for delta methods and Bayesian alternatives.
- For orchestration of which experiments to run and in what order, see the forthcoming experimentation-platform-orchestrator skill.
- License: MIT
# Experiment Design A senior product manager's playbook for running experiments that produce trustworthy decisions. The default state of experimentation in most companies is sloppy. PMs run tests against vague hypotheses, look at results too early, ignore guardrails, stratify into noise, and ship features whose lift is mostly measurement error. The cost is real: ship the wrong thing, kill the righ
What does the experiment-design skill do?
A discipline for designing experiments (A/B tests, multivariate, holdouts) so the results actually answer the question you asked. Hypothesis writing, sample size, duration, segment analysis, interpretation, decision-making, and the common failure modes that produce confidently wrong shipping decisions.
How do I install it?
Run `npx skills add rampstackco/claude-skills --skill experiment-design --agent claude-code` — it drops the skill into your project so the agent can pick it up. Swap the --agent value for codex, cursor or copilot if you use one of those.
Where does this skill come from?
From rampstackco/claude-skills, a repository with 515 stars. We read it straight from the repository tree rather than a submitted listing, so what you see here is what is actually published.
Is a popular skill a good skill?
Not necessarily. Stars measure attention, not adoption — a repository can trend for a week and be abandoned. That is why we show the weekly change from our own snapshots next to the total, instead of a single flattering number.