Stop Google Ads for four weeks and most Shopify stores lose less revenue than the ROAS column led them to expect. Sometimes far less. A holdout test is how you find out by how much, before you commit another ₹20 lakh to a channel that may be taking credit for orders you were going to get anyway. The test is simple: turn the ads off for one set of customers, leave them on for a comparable set, and compare what the two groups actually bought.
That's it. The hard part is the setup, and the fact that almost nobody does it because the platform already reports a number that looks fine.
Why the number in the dashboard isn't the number you spend against
Google Ads reports conversions it can associate with a click or an impression. It has no way to report conversions that would have happened without it. Those two things overlap heavily on branded search, on remarketing, and on Performance Max campaigns that quietly harvest your own brand terms.
So you end up with a campaign showing 4x ROAS that is, in truth, buying back traffic your organic listing, your email flows and your WhatsApp broadcasts already earned. The spend is real. The revenue is real. The causal link between them is the thing in question.
Incremental ROAS is the answer to a different question: for every extra rupee I put into this channel, how much extra revenue arrives that would not have arrived otherwise? On brand campaigns we routinely see incremental ROAS come in at a third to a half of platform-reported ROAS. On cold prospecting it's usually closer, because there's less to cannibalise.
Three ways to run a holdout, from weakest to strongest
Before-and-after pause. Turn everything off on the 1st, compare the next 28 days to the previous 28. Cheap, fast, and contaminated by everything else that changed: weather, a competitor's sale, a reel that took off, Diwali. Use it only when your spend is too small for anything better, and treat the output as a hint rather than a finding.
Geo holdout. Split your delivery geography into two matched groups, run ads in one, suppress them in the other, compare. This is the workhorse. Google Ads has a built-in geo experiment setup, and you can also do it crudely with location targeting and negative locations. Either way, Shopify order data by shipping state is your source of truth, not the Ads interface.
User-level conversion lift. Google randomises exposure at the user level and reports lift back to you. Cleaner statistically, but availability and volume thresholds mean most Indian D2C accounts under a few lakh a month won't qualify, and you're taking the platform's word on the maths of its own performance.
We start with a geo holdout on brand search nine times out of ten. It's the test with the biggest gap between what founders expect and what happens.
A worked geo holdout, with the arithmetic on the page
Illustrative numbers, round for readability, but the method is exactly what we run.
A store ships across India, spends about ₹4.2 lakh a month on Google, AOV around ₹2,400. We take the 20 cities that make up most of the revenue and split them into two groups of 10, matched on the last 90 days of Shopify revenue so the arms are roughly equal. Group A keeps ads. Group B goes dark for 28 days.
Pre-period, normalised to 28 days:
- Group A: ₹17,80,000
- Group B: ₹17,50,000
A was already running 1.7% ahead of B (17.8 ÷ 17.5 = 1.017). Hold that ratio.
Test period:
- Group A, ads on: ₹18,40,000
- Group B, ads off: ₹16,10,000
If Google contributed nothing, Group A should have landed at 16,10,000 × 1.017 = ₹16,37,000. It landed at ₹18,40,000. Incremental revenue is ₹2,03,000. Spend in Group A over those 28 days was roughly half the monthly total, so ₹2,10,000.
Incremental ROAS: 2,03,000 ÷ 2,10,000 = 0.97.
Now put margin on it. At 55% gross margin, ₹2,03,000 of revenue is about ₹1,12,000 of gross profit, against ₹2,10,000 of media. The channel, as configured, is burning roughly ₹98,000 a month of contribution. Meanwhile the Ads dashboard is reporting something in the region of 4x and everyone in the Monday meeting is pleased.
The finding is not "Google Ads doesn't work". The finding is that this account's mix, at this budget, is buying mostly its own demand. Which is a fixable problem, and a very different one from a traffic problem.
Start with brand search, because that's where the money hides
If you rank first organically for your own name and no reseller or marketplace is bidding on it, pausing brand search usually recovers most of those clicks for free. Customers typing your name are coming to you either way. The paid click is a toll you're paying to yourself.
The exception matters though. If Amazon, Nykaa, a rogue distributor or a competitor is bidding on your brand, going dark hands them the top of the page. We've seen brand holdouts where the dark cities lost 15-20% of branded conversions, and every rupee of that was defensive spend doing its job. You cannot know which case you're in without running the test, and the answer changes when someone new starts bidding, so it's worth re-checking twice a year.
While the test runs, this is also when the state of your organic brand result stops being an SEO abstraction and starts being a P&L line. A thin title tag, no sitelinks, no reviews in the SERP, a slow first paint on the homepage: all of that decides how much of the paid click you recover for free. Worth fixing before the test, not after. Our SEO and content work usually gets pulled into these projects for exactly that reason.
How much spend you need, and how long to sit still
Our working rule: each arm needs to be doing at least ₹8-10 lakh of revenue across the test window for a 10% swing to be legible above normal week-to-week noise. Below that, the confidence interval is wider than the effect you're trying to measure and you'll read randomness as insight.
Duration: 21 to 28 days minimum. Two full weekly cycles plus a buffer. Anything shorter and you're measuring a payday cycle.
And do not touch anything else. No new creative, no price change, no free-shipping threshold experiment, no influencer drop, no Meta budget shift. This is the discipline that kills most holdouts. Someone always wants to change one small thing in week two.
What breaks the test
COD and RTO. If you measure placed orders instead of delivered revenue, a geo holdout on an India-wide store will lie to you. Tier-2 and Tier-3 cities skew COD-heavy with higher return-to-origin, so a group weighted towards those markets books more and keeps less. Match your arms on delivered revenue, and reconcile after the RTO tail closes, which for most stores means waiting an extra 10-14 days past the test window before you read the result.
Festive season. Don't run a holdout between late September and mid-November. The auction, your discounting and consumer intent all move at once. June-July or February are usually quiet enough.
Spillover. Meta and Google feed each other. Suppressing Google in Group B while Meta runs nationally means Meta picks up some of the slack, and your holdout under-reads Google's true contribution. It's an acceptable compromise, but say so out loud in the readout rather than pretending the arms are clean.
Reallocated budget. If you pause brand in half the country and Google's smart bidding pushes that budget into Performance Max nationally, you haven't run a holdout. You've run a reshuffle. Cap the campaigns you're not testing.
Small geography. Single-city brands and most GCC stores can't split by geo meaningfully. Fall back to a time-based on/off pattern, alternating weeks over eight to ten weeks, and accept the wider error bars.
Reading the result without kidding yourself
Three numbers come out of a holdout: incremental revenue, incremental ROAS, and incremental contribution after margin. The third one decides the budget. A 1.4 incremental ROAS on a 35% margin product is a loss. A 1.4 on a 70% margin skincare line is fine and you should spend more.
Then act on it properly. A weak result on brand search means restructure brand, not kill Google. A weak result on Performance Max usually means the campaign is eating brand traffic, so exclude brand terms and re-test. A strong result on non-brand shopping means your budget cap is the thing holding you back.
One more note, because it comes up in every readout we present: if the holdout says the channel is barely incremental, the bottleneck is often the store, not the ad account. Cold traffic converts at a fraction of branded traffic, so a 3.2s LCP on mobile or a collection page with no usable filters shows up as poor incremental ROAS. Fixing store speed raises the ceiling on every channel at once, which is why we tend to sequence it before a budget increase.
A reasonable first move
Pull your last 90 days of Shopify orders by shipping state. Pull Google Ads spend and reported conversions by campaign for the same window. Work out what share of your reported conversions come from campaigns that touch your brand name, including Performance Max. If that share is above 40%, you have a holdout worth running in the next quiet month, and probably a budget that's overstated.
If you'd rather have someone else do the split, run the test and sit in the room when the numbers land, our free store audit covers the ad-account side alongside the storefront. We're a Google Partner, which mostly means we've seen enough accounts to know what the dashboard tends to overstate.