Ship or kill: a 30K-user experiment

CONTEXT

A product team ran an onboarding experiment across roughly 30,000 users, split evenly between control and variant. The variant looked better in the topline numbers. The team needed a ship-or-kill call, with the confidence to defend it.

QUESTION

Is the variant's lift real, or is it noise that will vanish after launch?

THE WORK

The core of the analysis is a two-proportion z-test. The explorer below is live: drag the sliders and watch the sampling distributions, the p-value, and the verdict respond. The defaults are the experiment's actual numbers.

FIG 3.1 — SIGNIFICANCE EXPLORER

Control conversion12.5%

Variant conversion14.1%

Sample size per arm15,000

Z-Score

4.08

P-Value

< 0.001

Detectable

1.07pp

SHIP

Shaded band: 95% confidence interval on the variant. When the control mean sits outside it, the lift is unlikely to be noise.

p < 0.05 — the variant's lift from 12.5% to 14.1% is real.

FIG 3.2 — GUARDRAIL PANEL

Guardrail metricControlVariantStatus

Median page load1.42s1.44s✓ Healthy

Crash-free sessions99.6%99.6%✓ Healthy

Support contact rate0.8%0.7%✓ Healthy

Day-1 uninstall rate2.1%2.0%✓ Healthy

No guardrail degraded. The lift is not being bought with a worse experience elsewhere.

FIG 3.3 — MDE CALCULATOR

Sample size per arm15,000

α = 0.05 · power = 80% · baseline 12.5%

Minimum detectable effect

1.07pp

8.6% relative lift detectable

At 15,000 users per arm, the test could reliably detect the 1.6pp lift it was designed to find.

DECISION

Ship the variant.p < 0.05 — the variant's lift from 12.5% to 14.1% is real. Guardrails held. The recommendation went out in writing with the confidence level attached.

RESULT

The variant shipped to 100% of users. Post-launch monitoring confirmed the lift held outside the experiment window.

WHAT I'D DO DIFFERENTLY

Pre-register the MDE and stopping rule before the first user enters the test. This experiment was healthy, but the discipline should live in the design, not the analysis.

METHODOLOGY ▸

Two-proportion z-test with pooled standard error. Two-tailed p-value against α = 0.05. A non-significant result is judged against the test's design sensitivity, not post-hoc power: the explorer computes the minimum detectable effect at the current sample size (α = 0.05, 80% power) and reports "no effect detected" only when that is sharp enough to have caught the pre-registered 1.6pp lift. Otherwise it reports "keep testing".

Post-hoc power is deliberately not used. It is a monotone function of the observed p-value, so it adds no information beyond p, and thresholding on it would make "no effect detected" unreachable.

The explorer recomputes z, p, the detectable effect, and the verdict on every slider input, using the same formulas as the offline analysis.