Free A/B Test Significance Calculator – Z-Score, P-Value & Verdict

How to Read A/B Test Significance the Honest Way

An A/B test compares two versions of a page, email, or ad to see which converts better, but a raw difference in conversion rate means nothing on its own. If version A converts 10 percent and version B converts 13 percent, the gap could be a genuine improvement or it could be plain luck from a small sample. Statistical significance is the discipline that tells the two apart. This calculator runs the standard two-proportion z-test on your numbers and reports the z-score, the p-value, a confidence level, the relative and absolute lift, and a plain verdict, so you can decide whether B truly beat A or whether you are about to ship a coin flip. The math starts with the two conversion rates, p̂₁ = conversions₁ ÷ visitors₁ for the control and p̂₂ = conversions₂ ÷ visitors₂ for the variant. It then pools them, p_pool = (conversions₁ + conversions₂) ÷ (visitors₁ + visitors₂), and builds a standard error, SE = √(p_pool × (1 − p_pool) × (1 ÷ visitors₁ + 1 ÷ visitors₂)). The z-score is simply how many standard errors apart the two rates sit: z = (p̂₂ − p̂₁) ÷ SE. Feeding that z into the standard normal distribution gives the p-value, the probability of seeing a gap this large if the two versions were really identical. A two-tailed p-value tests for any difference in either direction, while a one-tailed p-value only asks whether B is better; the tool shows both, because quietly reporting a one-tailed result as two-tailed is the most common way people fool themselves into a false win. Take a real example. The control gets 100 conversions from 1,000 visitors, a 10 percent rate, and the variant gets 130 from 1,000, a 13 percent rate. Pooling gives 0.115, the standard error works out to 0.014267, and z = (0.13 − 0.10) ÷ 0.014267 = 2.10. The two-tailed p-value is 0.0355, meaning there is only a 3.55 percent chance of a gap this big by luck, so the result is significant at the 95 percent level and the variant's 30 percent relative lift is real. Change the variant to 55 conversions from 1,000 against a 50-conversion control, though, and z falls to 0.50 with a p-value of 0.62, nowhere near significant, so that apparent win is noise you should not act on. Marketers and CRO teams use this test to decide when a test has actually finished. Comparing the p-value against a 0.05 threshold stops the habit of peeking at an early lead and declaring victory before the data has settled. The required-sample-size output does the opposite job, telling you before you launch how many visitors per variant you need to reliably detect the lift you care about, so you can estimate how long a test must run. For the 10-versus-13 percent example, detecting that lift at 80 percent power needs roughly 1,772 visitors in each group, which is a far cry from calling the test at a few hundred. A product team testing a new checkout button, an email marketer testing subject lines, and a landing-page designer testing headlines all lean on the same numbers to separate signal from noise. The honest caveats matter as much as the formula. With small samples the normal approximation is shaky, so the tool shows a prominent warning when either group has too few conversions, and a low p-value on tiny numbers should never be trusted. Statistical significance is not the same as business significance either: a change can be statistically real yet too small to be worth shipping, so weigh the lift against the effort. Running many tests or peeking repeatedly inflates your chance of a false positive, so fix your sample size in advance where you can. This is a frequentist two-proportion z-test, not a Bayesian model, and it assumes each visitor is counted once and independently. Treat the verdict as decision support, not a guarantee, and never as a substitute for sound experiment design. Every calculation runs in your browser with guards against zero or invalid inputs, and none of your test data is uploaded or stored.

A/B Test Significance Calculator Formula & Method

🔥 Trending Tools