A/B testing landing pages in Google Ads: setup, traps and a checklist

We put two rebuilt landing pages into a Google Ads A/B test for an Australian solar installer. Within a day both arms were serving evenly, and the variant was getting 55% more clicks per impression than the control. The ad copy in both arms was identical.

That number should not be possible, and it turned out to be the most useful thing the test produced in its first day. This post covers the setup, the three traps we hit in the first 24 hours, and the checklist we now run on every landing page test. If you would rather have someone run this for you, that is what our Google Ads management service does.

The setup

The client sells and installs residential solar and battery systems across four Australian states. Their account had a clear diagnosis: ad relevance scored above average on almost all spend, while landing page experience scored below average on 96% of it. The ads were fine. The pages were not.

Two rebuilt pages went up. Both made the same structural change rather than a cosmetic one.

Six forms on one page deserves a mention. Nobody designed that. It accumulated over time, and it meant we could not be sure every form fired the conversion trigger. That is a reason to consolidate on its own, separate from speed.

Two tests went live: a battery page across six campaigns and a panel page on one campaign, both at a 50/50 split.

Use ad variations, not a campaign experiment

Google Ads gives you two ways to test a URL. The obvious one is the wrong one.

Campaign experiments

A custom experiment clones the campaign, lets you change anything in the clone, and splits traffic between the two. You get full control, but it splits your conversion signal in half.

One battery campaign had just moved to a target CPA bid strategy and had 33 conversions in the last 30 days. Smart bidding wants around 30. Halving that puts both arms under the threshold. You would break the bidding to test the page.

Ad variations

An ad variation modifies the ads in place. One campaign, one budget, one bid strategy, one conversion pool. For a change that only touches the final URL it is the better tool, and it can run across many campaigns in a single experiment, so clicks accumulate faster.

Check this first. Ad variations rewrite the ad's final URL. If any keyword has its own final URL, that keyword overrides the ad and quietly stays on the old page. Your test then runs on a subset of traffic you cannot identify. We checked all 2,120 enabled keywords before starting. None had keyword-level URLs. If yours do, fix that before testing anything.

The API cannot create ad variations

We build and verify through the Google Ads API where we can. For this step we could not. In API v25:

• There is no ad variation service or resource. The only experiment services are campaign_draft, experiment and experiment_arm.

• ExperimentArm exposes campaigns, control and traffic_split, and no URL field of any kind.

• AD_VARIATION does appear in ExperimentTypeEnum, which is what makes this confusing. It is there so you can read variations that already exist. There is no create path.

So creation is a manual job in the interface. Everything after that, including enrolment, status and performance, reads back through the API. That is how we verified the setup rather than trusting the confirmation screen. Worth knowing before you plan an automated testing pipeline around it.

Work out the sample size before you start

This is the step most people skip, and it decides whether the test is worth running at all. At the account's baseline conversion rate and 420 clicks a day split evenly, with a two-sided test at 95% confidence and 80% power:

A 25% improvement, which most people would call a clear win, takes 66 days to prove at this volume. By then the account has changed underneath you. Budgets shift, seasonality moves, competitors enter.

The practical consequence is simple. If your rebuild is a tweak, do not test it. Either make a change large enough to detect in a sensible window, or accept that you are shipping on judgement rather than evidence. Pretending a 66-day test will happen is the worst of the three.

We set the bar at +50% before any data arrived and wrote it down. Setting it afterwards is how tests get talked into being wins.

Trap one: the staggered start

A variant ad is a new ad as far as Google is concerned, even when only the URL changed. It goes into review. Ours were created at 20:15 on a Friday and were still in review at 22:06. They were approved some time before 16:30 on Saturday, roughly a 20-hour delay.

Meanwhile the control kept serving all day Friday and banked conversions the variant never had the chance to compete for. Compare lifetime totals and you get this:

Read that on a Monday morning and you kill a good page. It is entirely an artefact of the review delay. 14.4 of those control conversions landed on a day the variant was not serving at all.

The fix is simple, and it belongs in your reporting rather than your memory: window both arms from the variant's first serving day and discard everything before it. On the one day both arms actually ran, the picture was ordinary. 91 control clicks against 83 variant clicks, one conversion each. Nothing to see, which is the correct answer at that sample size.

Trap two: identical ads, different click-through rates

Once both arms were serving, the split looked healthy. Battery: 91 control clicks against 83 variant. Panel: 342 control impressions against 362 variant. That is as even as a 50/50 split gets at small volume.

Then we looked at click-through rate.

The two arms run the same responsive search ads. Same headlines, same descriptions, same display path. The only difference is the final URL.

Click-through rate is decided before the click. The searcher sees an ad and decides whether to click it. They have not reached the landing page yet, so the landing page cannot influence that decision. A 50% CTR gap between arms should be impossible.

Here is why it matters more than the conversion numbers right now. CTR is a property of the auctions an ad wins: which queries matched, what position it took, on what device, at what hour. If the variant's CTR is genuinely higher, the variant is winning a different mix of auctions than the control. Different auctions mean different queries, intent, devices and times of day. That is a different population of visitors. If the two arms are drawing different visitors, any difference in conversion rate is confounded. You cannot tell how much came from the page and how much came from the traffic.

The randomisation unit in an ad variation is the auction, not the user. Nothing in the mechanism guarantees the two arms draw comparable query mixes if Google's rotation treats a brand-new ad differently from an established one.

Three possible explanations

• New-ad exploration. Google gives fresh creative extra impressions in favourable slots while it learns. Ad rank includes expected CTR, and a new ad without history gets an estimate rather than a measurement. This is the most likely explanation, and it should fade.

• Small samples. 1,521 and 362 variant impressions. Any rate computed from those has wide confidence intervals.

• A real serving asymmetry. The one that would compromise the test. Less likely, but nothing we have seen rules it out.

This is testable, and the test itself provides the evidence. If the gap shrinks toward parity as impressions accumulate, it was the exploration boost and the test is sound. If it persists at thousands of impressions per arm, the arms are drawing different traffic and the comparison is invalid. At that point the honest move is to stop, not to publish a conversion-rate delta we cannot attribute.

It costs nothing to check, and almost nobody does. CTR parity is the assumption every URL-only A/B test rests on, and it is rarely stated.

Trap three: the same page converts six times better in one state than another

Before designing the test we looked at how the existing page performed across the campaigns already pointing at it. Same URL, same three ads, eleven days:

Roughly 45% of clicks came from segments converting under 1%. We checked whether the page was regionally specific. It was not. The title names three capital cities and the body treats the states about evenly. So the cause is something else: brand recognition outside the home state, competitive intensity, keyword mix.

For test design the cause matters less than the consequence. A blended readout averages a strong-state win against a weak-state null and reports something in between that describes neither. If the rebuild fixes the weak states, that is a much bigger result than a uniform lift, and a blended number hides it completely.

So the segments get committed before the data arrives, alongside the success bar. Deciding how to cut the data after seeing it is how you find a win in noise.

Where SEO fits in

Two things in this test point outside paid search.

First, the rebuild itself. Cutting page weight from 245 KB to 45 KB, halving load time and collapsing six forms into one is the same work an organic page needs. Google Ads scores it as landing page experience. Google Search scores it as page experience and Core Web Vitals. One rebuild serves both, and the organic side keeps paying after the campaign budget stops.

Second, the state-level gap. VIC converted at 0.5% on the same page that converted at 5.4% in regional NSW. The most likely reason is that people in Melbourne have not heard of the brand, so a paid click lands cold. Paid search cannot fix that on its own. Ranking organically for the same terms in that market can, because a brand that shows up in both the ads and the organic results is a brand the searcher has now seen twice. That is the case for pairing a paid test like this with SEO in Melbourne, or whichever market your weakest state happens to be.

It also changes the maths on the 66-day problem. A landing page test can only prove a lift the paid budget can afford to buy. Organic traffic to the same page compounds over the same window, and it is the cheaper way to reach the click volume that makes small lifts detectable at all.

The checklist we now run

1. Confirm no keyword-level final URLs exist, or the variation silently misses that traffic.

2. Use ad variations for URL-only tests. Reserve campaign experiments for changes that need one, and never on a campaign near its smart-bidding conversion threshold.

3. Compute the sample size first. Write down the success bar before any data lands. If the required window exceeds a month, change the page more, or ship on judgement and say so.

4. Expect a review delay of up to a day. Window both arms from the variant's first serving day and discard everything before it.

5. Check CTR parity every time you check conversion rate. A persistent gap on identical creative invalidates the comparison.

6. Commit to segments (geography, device, campaign) before the readout.

7. Verify the variant records at least one conversion early. Clicks with zero conversions is a tracking break far more often than it is a bad page, and it looks identical in the report.

Where the test stands

At the time of writing the battery test has around 102 variant clicks against the roughly 3,900 per arm it needs. Both p-values sit comfortably in noise. The battery test reads on 25 September, the panel test on 8 October, and neither will be looked at seriously before then.

What the first 24 hours did produce was an instrument check: ads approved, both arms serving, the split even, and a conversion registered against each new page, which confirms tracking survives on both. That was the real question for day one.

The CTR gap is the open thread. It may be nothing more than Google's exploration boost on new creative, in which case it will fade and the test stands. But it is the kind of thing that quietly ruins an experiment while every dashboard reports it as healthy. If we had only been watching conversion rate, as almost every guide to landing page testing recommends, we would never have seen it.

If you want this kind of testing run on your account, with the sample size worked out before the money is spent, see our Google Ads management service.

Client details anonymised. Figures are from live Google Ads accounts and are accurate as reported at the time of writing. Conversion counts include fractional attribution and will restate as conversion lag resolves. Sample-size calculations use a two-proportion test, two-sided, alpha 0.05, power 0.80.