Geo holdout testing

What a geo holdout test is

Your ad platform credits a channel with a number of orders. Some of those orders would have happened anyway. A geo holdout test finds out how many, by switching the channel off in a set of regions (the holdout) and leaving it on everywhere else. Total orders, not platform-reported ones, are the outcome, because the question is what the business sold, not what the platform counted.

The holdout is deliberately the minority: regions covering about 15 to 25% of your orders. Only the tested channel pauses there. Everything else keeps running, and the channel's budget can shift into the regions still running so national spend stays flat.

The sample account's split: five regions held out, seven keep advertising. The stronger the colour, the more orders a region carries.

How the answer is worked out

Before the test, every region has months of weekly sales history. From that history the analysis learns how the regions that keep advertising normally track the regions that will pause: a weighted blend of the paused regions that follows the advertising regions week by week, plus a correction for any slow drift between them. During the test, that blend is what the advertising regions would have sold if nothing had changed. The gap between what they actually sold and the blend is the lift the channel drove.

The interval around that lift is not assumed from a formula. The same analysis is run on windows of the pre-test history where nothing changed, and the spread of the answers it gives there is the spread you should expect from noise alone. The test's lift is judged against that spread. Ninety per cent intervals, two-sided: the channel can turn out to be worth more or less than the platform claims, and the interval is the honest width for a budget decision rather than a drug trial.

For the analyst: the estimator is synthetic difference-in-differences (Arkhangelsky et al., 2021), unit weights from the pre-period fit and time weights for the intercept, with a placebo-in-time standard error. The weekly total is the inference unit. Every calibration figure on this site regenerates from a Monte Carlo harness in the codebase, and any change to the estimator has to pass it.

The two questions a result answers

A result is compared against two reference points, not one. Zero is the traditional question: did the ads do anything at all. One hundred per cent is the platform's implicit claim: every credited order was incremental. A channel measured at 40% incremental with a tight interval answers both. The ads work, and the platform is crediting them with two and a half times what they earn.

The two questions are not equally hard. Convicting an over-credited channel is easy: a wide interval around 10% is hopeless against zero but decisive against 100. Acquitting a genuinely incremental channel is the hard case, because an interval around 85% has to be tight enough not to reach 100. Rates above 100% are real and come from under-credited channels, typically upper-funnel prospecting and video. Brand search is the classic over-credited channel and lands well below.

A worked example

The sample account on this site sells nationally with orders in every UK region. Five regions were held out for eight weeks. In the regions that kept advertising, total orders came in 11% above the blend of the paused regions, an interval of 10.2% to 11.9%, which is 7,504 more orders than they would have had.

Over the same weeks the ad platform credited the tested campaigns with 18,000 orders on £64,000 of spend. So the channel is 42% incremental: 7,504 real orders out of 18,000 claimed. The reported cost per conversion was £3.56. The true cost per incremental conversion is £8.53.

That last number is the one to run the account on. The platform's dashboard keeps reporting £3.56, and the 42% carries across every campaign in the channel until the next test says otherwise. This is the result page itself, on that sample:

What it costs to run

The holdout regions stop seeing the tested channel for the length of the test. That is the cost, and it is bounded by the thing being measured: if the platform is over-crediting the channel, the test costs less than the dashboard implies, because the orders the dashboard says you will lose were mostly going to happen anyway. A genuinely incremental channel costs real orders to test. Shifting the holdout regions' budget into the running regions keeps national spend flat and leaves the efficiency gap on the shifted pounds as the remaining cost.

How long it runs

Four clean weeks is the default; three to eight is the verified range. A clean week is seven complete days with the channel paused, so the days the exclusions are applied and removed never enter the measurement. Longer is not better past the point where the interval stops narrowing, and every extra week is suppressed revenue and another chance for someone to touch the campaigns. How long a geo test should run shows the working.

What you need

• Orders by postcode and date. Twelve weeks at the least, eighteen months if you have it. A Shopify or BigCommerce export works as downloaded (how to get the file).
• Enough volume. The channel's share of your orders has to clear the change your total volume lets a test detect. The readiness checker answers that in two numbers.
• A channel you can pause by region. Google Ads and Meta both support location exclusions. The exclusions have to be applied to every campaign in the channel and removed on the day the test ends.
• Nothing else changing. One test at a time. A new campaign launched mid-test, a rebrand or a sale that lands in only some regions will show up in the result.

Mistakes that quietly break a test

• Measuring platform-reported orders instead of total orders. The channel is off in the holdout, so its reported orders there are zero by construction. That produces a mechanical 100% lift and proves nothing.
• Holding out one big region. London against the rest of the country is a comparison between two different economies. Spread the holdout across regions whose history tracks the rest.
• Cumulative totals in the data. A running total looks like normal data and produces a confident wrong answer. Each row has to be that day's count.
• Trusting the pause. Exclusions can fail to apply, or apply to some campaigns and not others. Check delivery in the holdout regions during the test, not after it.