Most of what's on this blog is a question that eventually got resolved — the muddled starting point, the thing I mixed up, the habit that turned out to be wrong, and then the fix. This one's different. It's just the current state of the board: everything I'm actually chewing on right now, most of it unresolved, written down mostly so I can come back to it later and see which threads actually went somewhere.
Where Frequentist Metrics Stop Being Simple
Everyone seems to be running some version of goal metric / secondary metric / guardrail metric. Doing it properly under a frequentist framework turns out to be harder than it looks from the outside. And whether a p-value threshold of 0.05 is even the right bar, business-wise, seems to depend entirely on the situation the company is actually in — not something you can answer in the abstract.
If the honest answer is "nothing else matters, just raise revenue per user," then that's genuinely the only thing that needs raising — a single-metric problem. But if that same revenue number is really a guardrail, and what actually needs improving is usability, then you're stuck holding two metrics — a goal and a guardrail — at the same time, which is a meaningfully harder statistical problem than either one alone. And on the same thread, extending this further, if what's actually needed is a real understanding of the user —
— that's as far as I've gotten with that one. I don't have the next sentence yet. It's on the list because it feels like the right extension of the same question, not because I know what it resolves to.
Expected Return Wants Bayesian, and Bayesian Wants More Reading
Once the actual goal is computing expected return rather than clearing a significance bar, the p-value stops being the right tool for the job — a point I'd already landed on a while back. Trying to use Bayesian methods in a way that's actually statistically defensible, rather than just gesturing at "chance to win," turns out to require learning a lot more than I'd assumed going in.
Concretely: say a launch decision actually has 2 goal metrics, 4 secondary metrics, and 2 guardrail metrics on the table at once. Does that call for a Bayesian hierarchy across all of them? This question didn't start from the statistics — it started from the relationships between the metrics themselves. Which metric do you actually need to move to get retention, or revenue per user, to move with it? That's a modeling question before it's a testing question, and I don't think it has a clean answer yet.
The Real Question Is Which Metric to Move
There's a related, more operational question: how do you actually run experiments fast? Working through it, "how do we experiment faster" turns out to reduce almost entirely to "which metric do we actually want to raise" — speed is downstream of clarity about the target, not a separate problem. What we actually want to raise is customer lifetime value: the total amount a user pays into the service before they churn for good. Once that's the real target, a lot of the "which metric" confusion is really "which metric is actually a good short-run proxy for that."
Guardrails, Again: What Happens at Two
And then there's the question I keep circling back to in a different form each time: what do you actually do once there are two guardrail metrics instead of one? I've worked through the two-kinds-of-test version of this question and the pure sample-size version of it, but the version that shows up here — inside an actual Bayesian, multi-metric decision — is a different shape of the same problem, and I don't think I've actually closed it out yet.
Too Many Domains, So I Picked One
There are genuinely a lot of different teams running experiments, across a lot of different domains, and I don't yet have a good way to understand all of them well enough to build experimentation infrastructure that actually helps rather than just adding process. Since getting a real handle on every individual team isn't realistic right now, I've scoped down to just guardrail metrics for the moment — and applied sequential testing to them specifically. Worth noting: Spotify's framing of a "deterioration metric" fits this use case better than the generic guardrail-metric language I'd been using — it's more precise about what the test is actually there to catch.
Recommendations: the Candidate Pool Problem
On the recommendations side, there's a concept I need to actually understand properly: the candidate pool — the set of items a model is even allowed to choose from, which is different at training time than it is at serving time. Understanding the backend API that constructs that pool turns out to be a harder problem than understanding the ML model sitting on top of it. The model is the part that gets the attention; the pool is the part that quietly determines what the model was ever able to recommend in the first place.
Reading GrowthBook's Source
I've been going through GrowthBook's codebase, which is written in TypeScript. The open question there is practical: how do you modify it in a way that still merges cleanly the next time GrowthBook ships an update, instead of maintaining a permanently diverged fork? CUPED is the concrete case that's pushing this. As shipped, if the metric is revenue, the covariate is just prior revenue for that same metric — nothing more. I want to build a monthly revenue mart and combine it with other metrics as additional covariates, which should make for a meaningfully better CUPED adjustment. But doing that means actually changing the source, not just configuring around it — which is exactly the customization-vs-mergeability tension above, just showing up concretely instead of abstractly.
Why Write This Down Unfinished
None of the sections above end in a resolution, and that's the point of this one. The rest of this blog is what these threads look like after they've been pulled apart and actually answered. This is what the desk looks like before that happens — which threads are live, which ones are already connected to each other, and which one sentence I genuinely don't have a follow-up for yet.