data-warehouse-experimentation
Running experiments out of the data warehouse instead of via dedicated experiment platforms. SQL-based assignment, exposure logging discipline, metric definitions in dbt models, statistical analysis in SQL or Python, variance reduction with CUPED, sequential testing, and the operational tradeoffs vs platforms like Statsig and Optimizely. Triggers on warehouse-native experimentation, run experiments in BigQuery, run experiments in Snowflake, dbt experiments, SQL t-test, CUPED variance reduction, exposure log, sample ratio mismatch, sequential testing, mSPRT, doubly robust estimation, build vs b
Profile →npx skills add rampstackco/claude-skills --skill data-warehouse-experimentation --agent claude-code
Same command for any agent — swap --agent for codex, cursor, copilot.
Weekly change comes from our own snapshots, not the repository page — it measures attention, not adoption.
What it does
Implements warehouse-native experimentation by guiding assignment directly in SQL, logging exposure events to the warehouse, and defining metrics via dbt models. It supports statistical analysis in SQL or Python and includes variance reduction (CUPED), sequential testing, and contrasts with external platforms. It also outlines triggers for choosing between warehouse-native vs platform-based experimentation and describes the four architecture components: Assignment, Exposure logging, Metric definitions, and Analysis.
How it works
- Assignment patterns are described: Deterministic hash assignment via a SQL expression, Feature flag assignment by reading from a flag service, and Randomized assignment table creation with a precomputed user-to-variant mapping.
- The exposure log section specifies that exposure events must fire exactly when the user experiences variant-specific behavior, including required fields like experiment_id, variant_id, user_id, exposed_at, and optional context_* fields. It emphasizes firing exposure only once per user per experiment at first exposure.
- Metric definitions in dbt models are used to compute primary and secondary metrics, enabling version control, testability, composability, and a single source of truth. Examples show usage of exp_metrics_* and fct_* naming patterns.
- Statistical analysis in SQL includes a basic Welch's t-test over per-variant metrics, with guidance to convert to p-values or CIs via a function or Python. It notes limitations for complex methods (CUPED, sequential testing) and suggests performing advanced stats in Python when needed.
- Statistical analysis in Python provides a typical workflow: pull aggregated data, split by variant, perform Welch's t-test, compute confidence intervals for the mean difference, and report lift and p-values, leveraging libraries like scipy, numpy, and potentially statsmodels.
When to use it
Use this skill when warehouse-native experimentation is the right call, such as when cost, custom metrics, complex segmentation, regulatory transparency, or strong in-house data capabilities favor in-warehouse control. It is recommended when deciding between a platform and warehouse-native, building warehouse-native infrastructure, auditing an existing setup, or running a metric that a platform cannot handle.
What it can touch
The skill references SQL queries, dbt models, and Python notebooks for analysis. It specifies using a warehouse (e.g., BigQuery, Snowflake) and a dbt-based metric definition workflow, with optional Python-based statistical analysis.
Caveats
The material cautions that some methods (e.g., complex CUPED, bootstrap, doubly robust estimation, sequential testing) are easier in Python. It emphasizes correct exposure discipline (fire exactly one exposure per user per experiment at first exposure) and warns about traps like delayed exposure and always-fire exposure if not properly handled.
# Data Warehouse Experimentation A senior data scientist's playbook for running experiments natively out of BigQuery, Snowflake, or any modern data warehouse, with metric definitions in dbt and statistical analysis in SQL or Python. Most companies that run experiments at scale use a dedicated platform. Statsig, Optimizely, LaunchDarkly with experimentation, PostHog, Amplitude Experiment. The platforms are good. They handle assignment, instrumentation, and analysis in one product, and the SQL-savvy data team does not have to reinvent the variance reduction wheel. There is a different operational model that mature data teams increasingly choose: warehouse-native experimentation. Assignment happens in code or via feature flags. Exposure events fire to the warehouse like any other event. Metrics are defined as dbt models. Statistical analysis runs as SQL or in a Python notebook against warehouse data. The "experiment platform" is just your existing data stack. This skill covers when warehouse-native is the right call, the architecture, and the specific techniques that make it work: assignment patterns, exposure logging discipline, metric definitions in dbt, t-tests and CUPED in SQL, se
- What this skill is for
- When warehouse-native is the right call
- The architecture
- Assignment patterns
- The exposure log
- Metric definitions in dbt models
- Statistical analysis in SQL
- Statistical analysis in Python
- Variance reduction: CUPED and beyond
- Pre-experiment power analysis
- Sequential testing patterns
- Common pitfalls
- The framework: 12 considerations for warehouse-native experimentation
- Reference files
What does the data-warehouse-experimentation skill do?
Running experiments out of the data warehouse instead of via dedicated experiment platforms. SQL-based assignment, exposure logging discipline, metric definitions in dbt models, statistical analysis in SQL or Python, variance reduction with CUPED, sequential testing, and the operational tradeoffs vs platforms like Statsig and Optimizely. Triggers on warehouse-native experimentation, run experiments in BigQuery, run experiments in Snowflake, dbt experiments, SQL t-test, CUPED variance reduction, exposure log, sample ratio mismatch, sequential testing, mSPRT, doubly robust estimation, build vs b
How do I install it?
Run `npx skills add rampstackco/claude-skills --skill data-warehouse-experimentation --agent claude-code` — it drops the skill into your project so the agent can pick it up. Swap the --agent value for codex, cursor or copilot if you use one of those.
Where does this skill come from?
From rampstackco/claude-skills, a repository with 515 stars. We read it straight from the repository tree rather than a submitted listing, so what you see here is what is actually published.
Is a popular skill a good skill?
Not necessarily. Stars measure attention, not adoption — a repository can trend for a week and be abandoned. That is why we show the weekly change from our own snapshots next to the total, instead of a single flattering number.