The implementation gap most experiment-analysis material skips over: computing standard errors, CUPED covariates, and SRM checks directly in SQL against a real data warehouse, not a small Python dataframe.
The three tables every experiment analysis actually rests on — who was assigned, who was actually exposed, and the metric events joined against both — and why conflating them is where most SQL experiment bugs start.
Rolling event-level data up to one row per user, and computing a two-sample t-test directly in SQL — means, variances, a pooled standard error, and a z-statistic — with no data ever leaving the warehouse.
Why a per-user ratio metric's variance can't be computed by averaging per-user ratios, what the delta method actually does about it, and the exact SQL to compute a ratio metric's standard error at scale.
Detecting Sample Ratio Mismatch with a chi-square test in SQL, plus the pre-treatment balance checks that catch a broken assignment pipeline before it ever reaches a results readout.
Capping extreme values at a percentile threshold computed per-metric, in pure SQL window functions — and the ordering mistakes that quietly damage the exact high-value tail the metric is supposed to capture.
Building the pre-period covariate table, joining it against assignment without leaking post-treatment data, and computing the CUPED-adjusted metric and its variance reduction, end to end in SQL.
Why recomputing an experiment's cumulative metrics from day one on every refresh gets more expensive every day it stays open, and the state-table-plus-merge pattern (built on sufficient statistics, not raw means) that fixes it.
Why checking each guardrail metric independently at the 5% level inflates the odds of a false alarm, the Bonferroni correction that fixes it, and one SQL query that reduces every metric's z-statistic to a single ship/hold verdict.
Why a correlated per-user timestamp filter can't be pruned by the query planner, the static partition filter that fixes it, and why that gap is what actually lets Chapter 7's incremental table keep cost flat as an experiment ages.
Turning every query in this book into a dbt project — staging models, an incremental model built on Chapter 7's pattern, and a results mart guarded by Chapter 4's SRM check as an automated test that can actually fail a run.