Incrementality Testing for Shopify Ad Spend
Incrementality testing measures the sales your ads actually caused. It answers one question: which outcomes did this ad or campaign cause that would not have happened otherwise? Google calls its version Conversion Lift, a tool that measures conversions directly driven by people seeing your ads.
Your ROAS column cannot show that. It reports the sales the platform attributes to your ads, not the sales the ads caused, and only a controlled experiment separates the two.
What Is Incrementality Testing?
In short, incrementality testing means running a controlled experiment on your own ad spend.
The experiment has two groups. People who see your ads are the treatment arm, and people who do not are the control arm. Google calls the difference between them the lift, meaning the increase in conversions caused by the ads.
Meta's Conversion Lift uses the same design, with a test group that sees your ads and a control group that does not. The groups are randomised when the Facebook campaign is created, not afterwards.
That is why incrementality cannot be pulled from a report after the fact. The test has to be set up before the campaign spends, with your budget as the treatment.
Why Reported ROAS Is Higher Than Incremental ROAS
Platform-reported ROAS is not a causal number. Research shows how far it can drift from what the ads actually caused.
The strongest evidence is peer reviewed, not vendor content. Gordon, Zettelmeyer, Bhargava and Chapsky, two of whom worked at Facebook, published their study in Marketing Science in 2019. It covered 15 US advertising experiments at Facebook, with 500 million user-experiment observations and 1.6 billion ad impressions.
The authors compared those experiments with ordinary attribution-style models. The models often failed to match the experiment results, even after accounting for detailed demographic and behavioral data.
In half the studies, the modelled increase in purchases was off by a factor of three. The models generally overestimated what the ads did.
Checkout conversions fared worst, and those are the conversions a Shopify store counts. In 7 of the 14 studies that measured checkout conversions, estimates were off by more than a factor of three.
Paid search shows the same pattern. Blake, Nosko and Tadelis ran experiments at eBay, published as NBER working paper 20171. They found that brand-keyword ads had no measurable short-term benefit.
Non-brand ads did worse. Most of the spend went to frequent buyers whose purchases the ads did not change, and average returns came out negative. Our Shopify ad benchmarks include a modelled version of the same gap.
Incrementality Testing vs A/B Testing: What Is the Difference?
The two tests are easy to confuse, but they answer different questions.
An A/B test compares two things that both run. It picks a winning creative, audience or landing page. An incrementality test compares advertising against no advertising.
That is why Google groups Brand Lift, Search Lift and Conversion Lift together as lift studies. They are built to go beyond metrics like clicks and impressions.
| Question | Right Test |
|---|---|
| Which headline performs better? | A/B test |
| Which audience is cheaper? | A/B test |
| Did this campaign create sales at all? | Incrementality test |
| Would we lose revenue if we paused this? | Incrementality test, holdout design |
Meta Incrementality Testing: How Conversion Lift Works
Meta's tool for Facebook incrementality testing is Conversion Lift. Advertisers who meet Meta's criteria can run a Conversion Lift test themselves, using the self-serve option in Meta Ads Manager. Meta does not publish those criteria, so check Ads Manager to see if the option is open to you.
Meta's Conversion Lift page also makes two performance claims, with no study shown behind them. Businesses running 15 experiments a year saw 30% higher ad performance than those running none. Meta also cites a 15% return-on-ad-spend correction from experimentation-based calibration.
Meta's A/B testing guidance sets its own rules. Run a test for 2 weeks to 30 days, and expect results after at least 100 events. One rule from that guidance limits contamination: do not use the test audience for any other campaign running at the same time.
Google Incrementality Testing: Which Lift Tests You Can Run Yourself
Google's incrementality tools split into two groups: some need a rep and others you can set up yourself. That split decides what you can run this quarter.
Conversion Lift is not open to every Google Ads account. To use it, you have to contact your Google account representative.
Custom experiments in Google Ads are self-serve. Google recommends a 50% traffic split for the best comparison. Choose the cookie-based split, which makes sure each user sees only the original campaign or the experiment.
Custom experiments have two hard limits. You can schedule up to 5 experiments for a campaign, but only one runs at a time. They also only work for Search, Display, Video and Hotel Ads campaigns.
Performance Max and Shopping are not on that list. Those are the campaign types most Shopify stores run.
Performance Max experiments only half fill the gap. They come in several subtypes, and the Uplift experiment is the one that measures incrementality.
The Uplift experiment does not support Performance Max with a Merchant Center (GMC) feed. It is only available for the Online Sales (Non-feed), Store goals (Offline) and Lead Gen marketing objectives.
So a Shopify store running Performance Max from a Merchant Center product feed cannot use it. The subtype that does cover Merchant Center is the Upgrade experiment, which tests Standard Shopping against Performance Max. Both arms run ads, so it is an A/B test between campaign types, not an incrementality test.
| Test | Self-Serve? | Covers PMax or Shopping? |
|---|---|---|
| Conversion Lift, user based | No, rep required | Not by default. Google lists Video, Discovery and Demand Gen as available, and says to ask your rep about Shopping and PMax |
| Conversion Lift, geo based | No, rep required | Yes, both listed |
| Custom experiments | Yes | No |
| PMax Uplift experiment (incrementality) | Yes | No, excludes PMax with GMC feed |
| PMax Upgrade experiment, Shopping vs PMax | Yes | Yes, but both arms advertise, so not incrementality |
Geo-based lift supports a broader list of campaign types than the user-based version, including Performance Max and Shopping. That makes the geo version the better fit for most Shopify stores.
Our Google Ads for Shopify guide argues Performance Max should be judged on incremental return. For a feed-based Performance Max campaign, geo-based Conversion Lift or a manual geo holdout is how you get that number.
What Is a Geo Lift Test?
A geo lift test splits regions instead of people. Some cities or postal areas keep the ads while others go dark. You then compare total sales across the two sets.
Google offers this as geography-based Conversion Lift, and supported campaign types include Performance Max and Shopping. It still needs a rep, at least one compatible conversion action, and single-country targeting.
Google's November 2025 measurement update puts the minimum spend for an incrementality experiment at $5,000. The same experiment could once cost more than $100,000.
The geo Conversion Lift page names no budget figure. It only says geo budgets tend to be higher than budgets for user-based studies.
You can run a manual holdout without a rep. Google Ads location targeting reaches regions, cities and postal codes. Small towns can be a problem, because Google only allows targeting for locations that meet minimum privacy thresholds.
Google's public geo lift deck gives two rules for drawing regions. Both aim to minimise contamination, where a shopper crosses from a control area into a test one.
- Build regions from grouped postal codes, with boundaries drawn through the least populous areas possible.
- Keep those boundaries from crossing well-known commutes.
Geo results use pre-attributed conversions, which are not linked to a specific cookie or ad interaction. That is why a geo test survives tracking loss and a pixel report does not.
How Long Should an Incrementality Test Run?
Google recommends running an experiment for at least 4 to 6 weeks, or longer if you have a long conversion delay. It gives the same advice when a Performance Max experiment comes back inconclusive.
New experiments also need a 7 to 14 day ramp-up, and data can be volatile during that period. Do not judge the results until the ramp-up is over.
Google publishes no minimum duration for Conversion Lift, and the 4 to 6 week figure is for experiments, not lift studies. Meta publishes no duration for Conversion Lift either. Its 2-week to 30-day window and 100-event floor are A/B testing rules that do not carry over to a lift test.
Common Incrementality Testing Mistakes
- Intervening on a small spend gap. Google treats a spend discrepancy under 20% as normal auction behavior and recommends not intervening.
- Forgetting the traffic cost. An existing Performance Max campaign placed in an experiment will only serve to 50% of its eligible traffic.
- Changing things mid-flight. During a geo test, avoid regional marketing, budget shifts to other channels, and changes to tracking, creative or audiences.
- Testing on a list that is too small. Cookie-based splits that use audience lists need at least 10,000 users in the list.
How to Calculate iCPA and iROAS
Incremental cost per acquisition is total spend divided by incremental conversions: iCPA = total spend / incremental conversions.
Incremental ROAS is the mirror image: iROAS = incremental conversion value / total spend.
Compare iROAS against your break-even number, not against reported ROAS. Our break-even ROAS calculator gives you that threshold. Pair it with blended MER and your CAC calculation.
Check significance before you act, because an inconclusive result means you have not learned anything yet. Google splits each arm's data into twenty buckets and applies jackknife resampling. Significance is two-tailed, against a 95% confidence interval.
Does Agency AI Run Incrementality Tests?
No, we do not run lift studies. Our dashboard shows platform-reported and blended numbers together, so you can see where they disagree.
The AI Strategist has a starter prompt for this exact problem: "Why does my ROAS not match Meta Ads Manager?" Our guide to what MER measures covers the dashboard side.
Our app costs $59 per month or $492 per year on the Shopify App Store, with a 30-day free trial.
Final Thoughts
Google's Conversion Lift needs an account representative, and Meta's self-serve route depends on criteria it does not publish. Google's self-serve Uplift experiment excludes Performance Max with a Merchant Center feed, so most Shopify stores cannot use it.
That leaves three routes to try:
- A manual geo holdout using Google Ads location targeting, which needs no rep.
- Meta's self-serve Conversion Lift, if Ads Manager offers it to your account.
- Geo-based Conversion Lift, if your Google rep can set it up.
Read whatever result you get against your break-even number, not against the ROAS column in your dashboard. If the result is inconclusive, do not treat it as a green light.
Frequently Asked Questions
Sources
- Google Ads Help, Conversion Lift pages: the definitions of Conversion Lift, lift, iCPA and iROAS, the rep requirement, geo-based campaign types, the single-country rule and higher geo budgets.
- Google's November 2025 incrementality testing update: the $5,000 minimum spend.
- Google Ads custom experiments page: the 50% split, the five-experiment cap, the supported campaign types and the 10,000-user audience floor.
- Google's Performance Max experiments pages and experiments FAQs: the experiment subtypes and the Uplift limits.
- The same Google pages: the 4 to 6 week duration, the 7 to 14 day ramp-up, the 50% traffic warning and the 20% spend-gap guidance.
- Google's experiment statistics page: jackknife resampling, twenty buckets per arm and the 95% confidence interval.
- Google's public geo lift deck on services.google.com: region boundaries, contamination and pre-attributed conversions.
- Meta's Conversion Lift page and documentation: the incrementality question in our intro, the test and control design, the self-serve criteria, and the 30% and 15% figures. Those two figures are Meta's own claims, with no study shown.
- Meta's A/B testing guidance: the 2 weeks to 30 days window, the 100-event floor and the audience rule.
- Gordon, Zettelmeyer, Bhargava and Chapsky, Marketing Science, 2019: the factor-of-three findings. Two co-authors were Facebook employees.
- Blake, Nosko and Tadelis, NBER working paper 20171: the branded search and negative-returns findings.
- Our own app: the dashboard and the AI Strategist starter prompt. Pricing and the free trial come from our Shopify App Store listing, checked 25 August 2026.
Free to install. Launch campaigns and pay nothing for your first 30 days.
