How many visitors do I need for an A/B test?

The number of visitors required depends on the base conversion rate, minimum expected improvement, confidence level and statistical power. The smaller the effect you expect to see, the more visitors are needed. By switching to "Plan sample size" and entering these values, the calculator will give you the number of visitors required for each version as well as an estimate of how long the test is likely to take.

What does statistical significance mean in this context?

This means that the difference between two versions is unlikely to be due to random variation. The test provides a p-value, which indicates the probability of observing a difference at least as large as this if the actual conversion rates for both versions were equal. If the p-value falls below a threshold, then the results are statistically significant. A threshold of 0.05 corresponds to a confidence level of 95%.

Should a one-sided or two-sided test be used?

A two-sided test is the more robust standard choice as it can detect both worsening and improving performance. A one-sided test only looks for improvement so may need slightly fewer visitors but risks overlooking changes that are secretly hurting conversion rate.

Can the test be stopped once significant results have been achieved?

No. If the results are checked frequently and the test is stopped as soon as a significant value occurs, then the rate of false positives will increase. Set the sample size and test duration before starting the test and wait until the test is completed. The test should be run for at least one week or two weeks to cover a weekly behavior cycle.

What is the minimum detectable effect size (MDES)?

The MDE is the smallest change in conversion rate that you want to be able to detect reliably with your test. The smaller the MDE, the larger the sample size required and the longer the test will take. You can enter this value as either a percentage change or an absolute change.

A/B Test Calculator

A/B test mode

What do you want to do?

Pick what you are doing: reading results you already have, or planning how big a test needs to be.

Analyze test results

Plan sample size

Your test results

Enter the totals for each version. Conversions are whatever action you count as a success.

Visitors in A (control)

Conversions in A (control)

Visitors in B (variant)

Conversions in B (variant)

Statistical settings

Confidence level

How sure you want to be before calling a result real. 95% is the common standard.

Hypothesis

Two-sided also catches a variant that performs worse; one-sided only looks for an improvement.

Result

Relative uplift (B vs A)

Z-score

P-value

Variant B wins. The lift is statistically significant at your confidence level, so the difference is unlikely to be chance.

Variant B converts at 12% against 9% for the control, a relative change of 33.33%.

Relative uplift

Control rate (A): %
Variant rate (B): %
Absolute difference: %

Significance: %
Difference: low end: %
Difference: high end: %

The A/B test calculator answers two questions you face with any experiment. Before starting, it tells you how many visitors and how long to run the test for, and after collecting data it judges whether the difference between the two versions is a real effect or just random noise.

This tool has both of these features. By switching between Analyze Test Results and Plan Sample Size at the top of the page, the input fields will change accordingly.

Outcome analysis - Is the difference real?

Once the data is available, the calculator compares the conversion rates of both versions using a two-sample z-test. The conversion rate is the number of conversions divided by the number of visitors.

p_A = \frac{x_A}{n_A} \qquad p_B = \frac{x_B}{n_B}

Relative improvement tells you how much better B is than A in percentage terms, with the conversion rate of A as a base. If it goes from 9% to 12%, then the relative improvement is 33%. The 3 points are the absolute improvement range.

\text{uplift} = \frac{p_B - p_A}{p_A}

To assess whether this difference exceeds random fluctuation, the test first integrates the data assuming that the conversion rates of both groups are equal, calculates the standard error, and then converts the difference into a z-score.

\hat{p} = \frac{x_A + x_B}{n_A + n_B} \qquad SE = \sqrt{\hat{p}(1-\hat{p})\left(\tfrac{1}{n_A}+\tfrac{1}{n_B}\right)}

z = \frac{p_B - p_A}{SE}

The Z-score can then be converted into a p-value. The p-value is the probability of observing a difference at least as large as the observed one, assuming that the true conversion rates for both versions are equal. If the p-value falls below a predetermined threshold, the results are statistically significant. This threshold is typically set to 5%, corresponding to a confidence level of 95%.

The calculator also shows the confidence interval for the difference in true conversion rates. If the entire interval is above zero then version B is actually superior. If the interval crosses zero no clear winner can be determined yet.

Test planning: How many visitors will be needed and how long should the test run?

Four parameters must be set before the test begins: the base conversion rate, the minimum detectable effect to investigate, the confidence level and statistical power. The calculator will return the number of visitors required for each version.

n = \frac{\left(z_{\alpha}\,\sigma_1 + z_{\beta}\,\sigma_2\right)^2}{\delta^2}

Here δ represents the absolute change in conversion rate to be detected, zα is determined by the confidence level and test hypothesis, while zβ is set by statistical power. The two σ terms each represent the variability of the conversion rates before and after the effect has occurred.

The minimum detectable effect size (MDES) is the smallest improvement that you are interested in detecting. It can either be entered as a relative value (e.g., "If performance increases by 20%, then conversion rate will increase from 10% to 12%") or an absolute value (e.g., "An increase of 20 points results in a 30% increase in the conversion rate"). Which definition is used can be selected via a switch. The smaller the desired effect size, the more visitors are required. For example, if the MDES is halved, then the sample size requirement roughly quadruples.

Dividing the total number of visitors by your weekly traffic will give you an estimate for how long to run a test. Because testing multiple variations at once can lead to problems with multiple comparisons, the calculator uses a Bonferroni correction to make the significance level more stringent and adds one group of visitors as a sample size for each additional variation.

Symbol	Meaning	Example
p	Baseline conversion rate	10 percent
MDE	Minimum detectable effect	20 percent relative
alpha	False-positive rate (1 minus confidence)	0.05
power	Chance of catching a real effect	80 percent
n	Visitors needed per variant	result

How to correctly interpret results:

Statistical significance is not the same as business relevance. Even if a test shows a very small improvement, that doesn't necessarily mean it's worth doing. First establish what size of effect would be really important to your business, then use that effect size to determine sample size and run the full test.

Don't stop the test just because a p-value falls below some threshold. If you keep checking results and only stop when there's a significant result, then the rate of false positives will be much higher than the 5% that was originally accepted. Determine your sample size in advance and collect enough data before making any decisions. The test should run for at least one week, ideally two weeks, with both weekday and weekend activity included in the data.

This calculator is for planning and learning purposes only. It uses a normal approximation to the binomial distribution, which is fairly accurate for typical visitor counts in real tests, but its accuracy is lower with extremely small samples. It is not a substitute for a well-designed experimentation platform or professional statistical advice.

Frequently asked questions

How many visitors do I need for an A/B test?: The number of visitors required depends on the base conversion rate, minimum expected improvement, confidence level and statistical power. The smaller the effect you expect to see, the more visitors are needed. By switching to "Plan sample size" and entering these values, the calculator will give you the number of visitors required for each version as well as an estimate of how long the test is likely to take.
What does statistical significance mean in this context?: This means that the difference between two versions is unlikely to be due to random variation. The test provides a p-value, which indicates the probability of observing a difference at least as large as this if the actual conversion rates for both versions were equal. If the p-value falls below a threshold, then the results are statistically significant. A threshold of 0.05 corresponds to a confidence level of 95%.
Should a one-sided or two-sided test be used?: A two-sided test is the more robust standard choice as it can detect both worsening and improving performance. A one-sided test only looks for improvement so may need slightly fewer visitors but risks overlooking changes that are secretly hurting conversion rate.
Can the test be stopped once significant results have been achieved?: No. If the results are checked frequently and the test is stopped as soon as a significant value occurs, then the rate of false positives will increase. Set the sample size and test duration before starting the test and wait until the test is completed. The test should be run for at least one week or two weeks to cover a weekly behavior cycle.
What is the minimum detectable effect size (MDES)?: The MDE is the smallest change in conversion rate that you want to be able to detect reliably with your test. The smaller the MDE, the larger the sample size required and the longer the test will take. You can enter this value as either a percentage change or an absolute change.

Related calculators

Critical Value CalculatorFind the critical value for Z, t, chi-square, and F tests at any significance level, for left-tailed, right-tailed, and two-tailed hypothesis tests.

Benford's Law CalculatorLook up the Benford probability of any leading digit, or test a dataset's first-digit counts against Benford's law with a chi-square goodness-of-fit test and the mean absolute deviation.

Sample Variance CalculatorFind the sample and population variance of your data set, along with the standard deviation, mean, and sum of squares. Enter your values and read the full breakdown.

Weighted Average CalculatorFind the weighted average (weighted mean) of up to eight values and weights, compare it with a plain average, and plan the value you need next to reach a target.

Percentile CalculatorFind the value at any percentile of your data set, or the percentile rank of a score, plus the quartiles, median, and interquartile range. Enter your values and read the full summary.

Z-Score CalculatorFind the z-score of a value, or solve for the value, mean, or standard deviation from a z-score, and convert any z-score to a percentile and tail probabilities.

Disclaimer: This calculator is provided for general informational and educational purposes only. Our calculators are under active development, and results may be inaccurate, incomplete, or unsuitable for your situation. Always verify the figures independently and seek advice from a qualified professional before relying on them. We make no warranties and accept no liability for any loss or decision arising from use of this tool.

References

Kohavi, Tang & Xu: Trustworthy Online Controlled Experiments
Standard reference on running and analyzing A/B experiments.
Evan Miller: How Not To Run An A/B Test
Why fixed sample sizes and no peeking keep false positives in check.
NIST/SEMATECH e-Handbook: Comparing two proportions
The two-proportion test and sample-size formulas.