Incrementality testing: the measurement that tells you if a program actually works

Picture how most lifecycle programs report success. A dormant user gets a win-back email on Tuesday. They click. They buy on Thursday. A reporting model called last-click attribution — assigning revenue to whatever the user touched most recently before purchasing — credits the email. Tidy story. Easy to put in a slide.

Trouble is the implicit claim underneath the story: without that email, the user wouldn't have bought. Wrong often enough to matter. A meaningful share of those dormant users were already drifting back — checking the app, opening a competitor's tab, half-thinking about a purchase. Your email caught a wave that was already cresting. Your program did something. Just nothing like what your dashboard says.

Incrementality testing fixes this. Cleanest way to picture it: a clinical drug trial. Half the patients get the drug, half get a sugar pill — the placebo arm — and whatever difference shows up between the two groups is what the drug actually did. Lifecycle works the same way. A random fraction of eligible users — the holdout, your placebo arm — gets no program. Everyone else gets it as normal. Whatever difference shows up in their downstream behaviour is the lift — the slice of revenue your program genuinely caused rather than merely sat near.

Attribution still earns its keep for daily operations. Incrementality calibrates it. Run attribution as the daily pulse, run incrementality as the annual or quarterly recalibration. Different instruments, different questions.

One distinction worth nailing first. An A/B test compares one variant of a message against another — subject line A versus subject line B, both groups receive the program. It tells you which version wins. An incrementality test compares with program against no program. It tells you whether the program is worth running at all. Both are useful. Each answers a different question, and people confuse them constantly.

Core design follows the clinical-trial scaffolding directly. Five steps, in order:

1. Define the program. The specific sequence, trigger, or treatment being tested. "The 3-message win-back flow triggered at 60 days of dormancy." Not "winback" as a vague concept — name the exact artefact going live.

2. Define the eligible population. Users who would normally qualify. "Users dormant exactly 60 days and not unsubscribed." Write it as if describing a SQL filter — the database query that pulls these users out — because at some point you will be.

3. Random assignment to treatment versus control. Usually 80/20 or 90/10 treatment-to-control. Treatment gets the program, control gets nothing. Random is load-bearing — both groups need to be statistically interchangeable on everything except the program itself, the way a clinical trial randomises patients to drug versus placebo. Everything else stays identical for both.

4. Pre-register the outcome metric and measurement window. "Purchase rate within 30 days of trigger." Decide before the test starts. Write it down. Reasons matter and we'll get to them.

5. Measure. Treatment minus control. That difference is the incremental lift. A proportion test or t-test — standard statistical checks for whether two groups genuinely differ — tells you if it's significant or noise.

One technical principle that matters once you're running this in the real world: intent-to-treat. Anyone assigned to treatment counts as treatment, even if they never opened the email. Anyone in the control counts as control. You don't exclude people who didn't engage — that's how teams accidentally over-credit their program by quietly removing the users it failed to reach. Randomisation is what makes the comparison fair. Don't break it after the fact.

Your holdout — the placebo arm, users who get nothing — needs to be large enough to detect the lift you expect to see. Usually 5–20% of eligible users, sized to the effect.

Expected lift 5% or more (strong programs): 5% holdout is usually enough.

Expected lift 2–5% (typical): 10–15% holdout.

Expected lift under 2% (marginal): 20%+ holdout. While you're at it, consider whether the program is earning its place at all.

Maths is in the sample size guide; same formulas apply here. One thing to hold in mind: your control group is the bottleneck. Treatment runs at full volume; control size determines statistical power — the ability to detect a real effect rather than miss it as noise.

The harder problem is human, not statistical. Someone in finance or growth will look at the holdout and see lost revenue: 10% of eligible users not getting the program means 10% of the program's sends not happening. Framing that works is risk management, not measurement cost. A program with no incrementality number has unknown value — nobody can tell whether trimming 10% of the budget hurts anything or saves money. A program with a measured lift produces a defensible figure that survives skeptical questioning. Those 10–15% "lost" sends are the price of measurement, and the figure they buy you is usually worth far more in budget defensibility than the sends themselves would have generated.

Run the test long enough that your measured effect represents real lift, not a short-term shift in timing. Right duration is set by the metric, not the calendar:

Short effects (purchase within 7 days): 2–4 weeks of assignment, then analyse.

Medium effects (purchase within 30 days): 4–8 weeks.

Long effects (retention, subscription renewal): 3–6 months.

Now the numbers themselves. Headline figure is revenue per user: total revenue in the window divided by users in each group, treatment versus control. Your CFO wants a per-head impact they can multiply by your eligible population to size the program's genuine contribution. Secondary is conversion rate, the percentage making any purchase, which tells you whether lift is driven by more buyers or bigger baskets. Collect program-level engagement (opens, clicks, attributed conversions) for context, but don't let attribution numbers confuse the incrementality read — they measure different things on purpose.

Formula: (revenue/user in treatment − revenue/user in control) / revenue/user in control. A 5% incremental lift means the program genuinely added 5% to per-user revenue over what would have happened without it.

Control contamination. Holdout users still receiving the program through another channel — a push notification, an in-app banner, a Meta ad. Audit every trigger and suppression rule. Audit whether other programs overlap with the one under test. If your placebo arm is secretly getting the drug through a side door, the test reads zero lift even when the program works.

External events. A product launch, sale, or news cycle during the test affects both groups — but unevenly if the groups are already behaviourally different. Balance on pre-test characteristics where possible (purchase history, tenure, segment) so both arms are genuinely matched at the start.

Measurement window too short. Measuring 7-day purchase rate on a program that drives long-term retention misses the effect entirely. Match your window to the mechanism — what is the program actually trying to do, and how long does that effect take to show up?

Post-hoc metric shopping. A win-back flow might not lift 30-day purchase rate but might cut unsubscribe rate. Pre-register what you'll measure. Picking the winning metric after you've seen results is how teams convince themselves of things that aren't true. Same discipline pre-registration enforces in clinical trials, and for the same reason.

Rough benchmarks for a healthy incrementality number, by program type, so you have something to compare against. Welcome flow: 5–15%. Win-back: 3–10%. Cart abandonment: 8–20%. Browse abandonment: 5–15%. Newsletters: 1–3% directly plus indirect effects. Your program against those bands tells you where it stands. Absolute numbers drift with audience and category, so treat the bands as orientation, not target.

One more thing worth saying plainly. Skip incrementality testing on transactional mail — password resets, order confirmations, shipping notifications. Nobody is holding back a password reset for measurement, and an order confirmation isn't earning its keep on incremental revenue anyway. Transactional value is functional, not promotional. Incrementality testing belongs in lifecycle programs where there's a genuine choice about whether to send.

If your measured lift comes back smaller than expected, treat that as information, not failure. Either the program is less impactful than attribution claimed (common; trim budget or rebuild), or the test was underpowered or contaminated (audit methodology). Small or null incrementality on programs "everyone knows work" is the single most common finding in this entire discipline — and, unsurprisingly, the most valuable one. Whatever number survives the holdout is the number you can defend.

covers incrementality as a quarterly practice. Every major lifecycle program deserves at least an annual incrementality read to calibrate the attribution number it's been quietly generating all year.