Lift Lab

Sample size (proportions). Two-proportion normal approximation: n = (z₁₋α/2·√(2·p̄(1-p̄)) + z₁₋β·√(p₁(1-p₁)+p₂(1-p₂)))² / (p₂-p₁)², where p₂ = p₁·(1+MDE). Unequal splits inflate the total by 1/(4·s·(1-s)).

Sample size (revenue). n/arm = 2·(z₁₋α/2+z₁₋β)²·CV²/MDE². Revenue is heavily skewed, so the coefficient of variation (CV) does the heavy lifting; CV of 3-6 is typical for mobile ARPU.

Reading a test (proportions). Lift = rate_test − rate_control. Standard error √(p_c(1-p_c)/n_c + p_t(1-p_t)/n_t), and a two-sided z gives the p-value. The selected confidence level sets the significance bar.

Bayesian readout. Each arm's true rate gets a Beta(conv+1, non-conv+1) posterior. We draw 8,000 Monte-Carlo samples to get the probability the test arm is genuinely better and the credible interval of the lift. This is what stays useful when traffic is too thin for a clean p-value. Revenue tests use a normal approximation instead: P(better) = Φ(diff/SE), and the chart samples 4,000 normal draws.

Incremental economics. Incremental conversions = (rate_test − rate_control)·n_test, i.e. only the conversions the ads actually caused. iCPI = spend / incremental conversions; iROAS = incremental revenue / spend. The naive version counts every exposed conversion as caused, which is the over-counting the platforms bank on.

Reconciliation. Each source is bias-corrected (value ÷ bias) then combined as a trust-weighted average: Σ(corrected·weight)/Σ(weight). The band runs from the lowest to the highest corrected value. It's an opinion with structure, not a measurement.

The honest limit. A calculator designs and reads tests; it can't run a clean holdout for you, and a contaminated split produces confident nonsense. At very low spend, no test can detect realistic lifts. The tool will tell you when you're in that zone instead of pretending otherwise.