Holdout group design: the incrementality tool most lifecycle programs skip

Imagine you send a winback email — a message designed to bring back lapsed users — to 100,000 dormant customers, and 3,000 of them come back and buy. Lovely number, ships well in a deck. But the question finance will eventually ask is the awkward one: how many of those 3,000 were going to come back anyway, with or without the email? If the answer is 2,800, your program produced 200 incremental customers, not 3,000. The deck stops shipping quite so well.

This is the gap a holdout closes. Attribution models — first-touch, last-touch, multi-touch (different rules for which marketing touchpoint gets credit when a sale happens) — divide credit across touchpoints that occurred before a conversion. They do not answer the causal question: would the conversion have happened without any of those touchpoints? For lifecycle, the honest answer is "yes, for a meaningful share of revenue." Users who would have returned, renewed, or bought anyway show up in attribution reports as lifecycle wins. Nobody's fault, exactly. Nobody's telling you either.

A holdout strips the confusion at the root. Random assignment — splitting users by something like a coin flip rather than by who they are or how they behave — means the holdout group is, on average, identical to the messaged group on every dimension except message exposure. The revenue gap is incremental by construction. No attribution model required. No debate with finance about which model is "right."

5–10% of the eligible audience is the operator default, and for good reasons. 10% reaches statistical power— the test's ability to detect a real effect rather than missing it as noise — faster. 5% loses less revenue while the test runs, in the case where the program turns out to be a winner. For a program with tight monthly revenue targets, start at 5%. For a program where you're genuinely testing whether to keep running it, 10% gets you to the answer sooner.

The holdout has to be large enough to detect the effect size — how big the difference between the two groups needs to be before you trust it isn't random — that you actually care about. Ballpark: to detect a 10% incremental lift at 95% confidence (the standard threshold for "this almost certainly isn't a fluke") over a quarter, you typically need around 5,000 users in the holdout. The exact number moves with your baseline conversion rate. Below 5,000 the test is underpowered, and you'll land on a number that isn't reliably different from zero — which is worse than running no test at all, because people will still quote it.

The Orbit Experiment Design skillhandles the power calculation against your specific baseline and expected lift. Skip that step and you're running a test that can't answer the question it was built to ask.

How you put users into the holdout matters more than most teams realise. All three of the rules below can break silently — the test will keep running, the dashboard will keep producing numbers — and you won't find out until someone asks a hard question in a quarterly business review.

Stable assignment. A user assigned to the holdout today stays in the holdout for the full measurement period. Users flickering in and out of the holdout contaminate the read — you're no longer comparing two clean groups, you're comparing two blurry ones. Use a persistent random integer — a number permanently attached to each user that doesn't change over time. Braze's Random Bucket Number(a number from 0–9999 assigned once per user and never reshuffled) is built for exactly this. Don't use a random value that recalculates on every audience refresh.

Random assignment. "Users who haven't engaged in the last 30 days" is not a random cut. It's selection bias— picking users by a trait that already correlates with the outcome you're measuring — with a severity rating. The holdout has to be statistically equivalent to the treatment group on every dimension except message exposure. Random Bucket Number or equivalent. No shortcuts, no "but we only want to hold out the engaged users."

Global assignment. Exempting specific programs from the holdout — "we won't hold out onboarding because it feels cruel" — compromises the measurement. Either you hold out, or you don't. A global holdoutmeans every marketing send respects the same exclusion list. Transactional sends — order receipts, password resets, the messages you'd send regardless of marketing — are exempt, obviously. Marketing isn't.

Mistake 1 — holdout leakage. Users in the holdout occasionally get mail because a broadcast ignored the flag — a one-off newsletter that bypassed the audience filter, a re-engagement campaign that pulled from a different list. Two percent leakage is enough to invalidate the measurement.You're no longer comparing messaged versus unmessaged — you're comparing heavily-messaged versus lightly-messaged, and the gap shrinks accordingly. Audit broadcasts monthly for holdout compliance. Every month. No exceptions.

Mistake 2 — seasonal confounds. A confoundis something other than the message that changes the outcome — and Black Friday is the obvious one. A holdout that runs only in November will show enormous incremental lift, because the volume spike does the heavy lifting, not your emails. The number won't generalise to the rest of the year, but it'll get quoted as if it does. Run holdouts across full quarters or full years so you average seasonal effects out.

Mistake 3 — reading before statistical power.A two-week holdout result is almost never significant — there isn't enough volume yet for the gap between groups to be distinguishable from noise. Leadership asks for an update, the team produces a number anyway, and that number gets cited as the incrementality forever. The fix is simple and politically hard: don't publish interim reads. Publish once at the end of the measurement period, with the full analysis attached.

A holdout produces a single most-important number: incremental revenue per user, the average extra dollars each messaged user generated above and beyond the holdout baseline. Multiply that by audience size for total program contribution. Divide by program cost for true ROI. This is the number that goes in front of finance and replaces the attribution-model figures that always, eventually, get questioned.

The retention economics guide covers how to frame this number in a CFO conversation. A defensible quarterly holdout study is usually more persuasive than six quarters of attribution spreadsheets, because it answers the causal question rather than the correlational one. Run one annually at minimum. Programs that run holdouts annually have budget conversations that go differently from programs that don't.

On the mechanics question that comes up constantly — in Braze, use Random Bucket Number filters. A fixed slice (RBN < 500 for a 5% holdout) is stable, random, and respected across every program in the instance. The same attribute that underpins IP warm-up is what you want here. It was designed for this job.

What if the holdout shows zero incremental lift? Don't panic. Don't bury it either. Zero is honest information, and it's worth investigating before it turns into a narrative. Is the program targeting users who were going to convert anyway? Is the offer too weak to change behaviour? Is the timing wrong? Zero is rarely the final answer on a well-designed program — but when it is, the program needs rethinking, not another quarter of the same cadence with a redesigned header image.