Google Ads A/B tests: a +25% CVR lift needs 66 days at 420 clicks/day (power calc)

Before running any A/B test in Google Ads (landing page, ad copy, bid strategy), there is a calculation almost nobody does. It takes thirty seconds and it determines whether the test can succeed.

Here it is for one account. Baseline conversion rate 2.88%, 420 clicks a day, split 50/50, two-sided test at 95% confidence and 80% power:

A 25% improvement in conversion rate (which anyone would call a meaningful win), takes 66 days to prove at this volume. Not to see; to prove, to the standard where you'd act on it.

Sixty-six days is longer than most tests are allowed to run. It is longer than most account managers stay on an account. It is long enough that the campaign budget, the competitive set, the season and the bidding strategy will all have changed underneath the test.

At single-campaign volume it's worse

The table above is for a test across six campaigns. A test on one campaign, 154 clicks a day, looks like this:

Four months to prove a 25% lift. Most single-campaign tests in most accounts are in this regime, and most of them are called after two weeks.

What "called after two weeks" looks like

The same account ran a landing page test that started on a Saturday. Here is the variant's apparent effect on conversion rate, read at three points on the second day:

Nine hours apart, the same test went from "the new page is worse" to "the new page is dramatically better." The variant picked up three conversions in an afternoon while the control picked up none. Both readings were built on fewer than ten conversions per arm. Both were noise. Anyone checking the dashboard at lunch and again at dinner would have reached opposite conclusions, and either would have been wrong.

At 246 total variant clicks against the ~7,800 needed, the test was about 3% of the way to an answer. It looked, on the interface, like it had one.

The three honest responses

Once you've done the calculation, there are three positions you can take. All three are defensible. What is not defensible is running the test anyway and reading it early.

Make a bigger change. A +100% lift reads in five days. If the variant is a full rebuild (different structure, a fifth the page weight, one form instead of six), it might genuinely be that different. Test big changes; ship small ones on judgement.

Ship on judgement and say so. If the change is a headline tweak, it's not going to move conversion rate by half. Make the change, don't call it a test, and don't claim a result. "We changed this because we think it's better" is an honest sentence. "We tested this and it won" is not, if the test couldn't have detected the win.

Don't make the change. If it's not worth shipping blind and not big enough to test, it's not worth doing.

Pre-register the bar

The other thing the calculation gives you is a number to write down before the data arrives. In the account above, the success bar was set at +50%, 2.88% to 4.32%, and recorded on the day the test launched.

That matters because a test read without a pre-set bar will be talked into a win. The variant is up 12%? That's a win. Up 8%? Still directionally positive. The control is up 5%? Inconclusive, let's extend. Without the number written down first, the result is whatever the person reading it wanted.

The calculation

For anyone who wants to run it, the required clicks per arm for a two-proportion test:

n = (z_α + z_β)² × [p₁(1−p₁) + p₂(1−p₂)] / (p₂ − p₁)²

where p₁ is the baseline rate, p₂ the target rate, z_α = 1.96 for 95% confidence, z_β = 0.84 for 80% power. Any online calculator does the same thing. It takes thirty seconds, and it will tell you, before you've spent anything, whether the test you're about to run can ever tell you what you want to know.

Most can't. That is the finding. Run the ones that can.