Incrementality Testing for Ecommerce: How to Plan Holdout Tests Around Black Friday ift Guides, Codes, and Links
Incrementality testing for ecommerce answers one question attribution can't: how many of these sales would have happened anyway? The timing problem is that most brands start asking it in mid-November, when budgets are peaking and the data is at its least readable. Black Friday lands on November 27 this year and Cyber Monday on November 30. If you want lift numbers you can actually use for your 2026 peak, the useful testing windows are the weeks on either side of it, not the week itself.
This post covers the practical design: how to split, how long to run, what budget you need for a readable result, and how to compare lift results against click-based attribution. For the broader comparison of MMM, attribution, and incrementality as measurement methods, see our guide to MMM vs attribution vs incrementality. This one is about scheduling and running the test.
Why does a holdout test during BFCM week give noisy results?
A holdout test run during Cyber Week measures a period that looks nothing like the rest of your year, so the lift it reports rarely transfers. Three things break at once.
- Baseline demand jumps. Adobe forecasts US online spend of $47.5 billion across Cyber Week, about 17% of the entire November and December season packed into five days. Shoppers who were already planning to buy from you will buy whether or not they see an ad, so your control group converts at an unusually high rate.
- Your offer is doing the work. A 30% sitewide discount moves people on its own. The test can't cleanly separate the ad's effect from the promo's effect when both are running at full strength.
- Auction costs swing daily. CPMs and CPCs climb sharply into the peak, so the same budget buys different reach on Tuesday than on Friday. One agency guide on geo holdouts puts it bluntly: Q4 is exactly the wrong time to run a test you plan to base annual budgets on.
There's a reasonable counterpoint. Some practitioners argue that promo-period buying may be largely organic, which is exactly why lower-funnel campaigns deserve testing in Q4. That's fair. The resolution is to test the same campaign before and after the peak, then treat any in-peak read as a separate, holiday-only number rather than your baseline.
When should you run incrementality tests around Black Friday?
Run one test before the peak to set expectations and one after to confirm them. A workable 2026 calendar looks like this:
- October 14 to November 15: pre-peak test. Four weeks, ending before early Black Friday promos distort demand. This tells you how incremental prospecting and retargeting are at normal prices.
- November 16 to December 7: blackout. No new tests launch. If one is mid-flight, extend it rather than reading it inside the peak.
- December 8 to 21: optional holiday read. Only for gifting brands where December is the real season, and only if you report it as a seasonal figure.
- January 5 to early February: post-peak test. Repeat the October design. Starting after the first week of January avoids lingering return and gift card behavior.
The before-and-after pair is the useful output. If Meta prospecting shows similar lift in October and January, you can plan next year's BFCM budget on it with some confidence. If the two numbers diverge sharply, you've learned that the channel's incrementality is seasonal, which is worth knowing before you scale into the next peak. Our recap of what BFCM 2025 teaches us about 2026 covers the budget pacing side of that decision.
How to design a holdout test for Meta ads
A holdout test for Meta ads withholds ads from a random slice of your audience or geography and compares conversions between the exposed and unexposed groups. You have two practical ways to do it.
Option 1: User-level Conversion Lift
Meta's Conversion Lift study randomly assigns accounts to a test group that sees your ads and a control group that doesn't, then reports the difference in conversions. Meta runs the randomization, which makes setup easy. The trade-off is that Meta is also the one reporting the result, and the conversions it counts come from its own pixel and CAPI signal. That's useful, but you'll want to check it against your store's order data.
Option 2: Geo holdout
You turn ads off in a set of regions and compare sales against matched regions where ads stay on. Because the outcome is measured in your own Shopify, BigCommerce, or WooCommerce orders, the result doesn't depend on any ad platform's reporting. It takes more setup, and our companion post on matched-market geo lift tests walks through it.
Split, length, and budget minimums
- Split: Hold out 10% to 20% of the audience or addressable market. Smaller holdouts protect revenue but widen the confidence interval. For a first test on a major channel, lean toward 20%.
- Length: At least four weeks, and ideally three to four times your typical purchase cycle. A two-week test on a product with a 21-day consideration window misses late conversions.
- Volume: As a rough planning rule, one practitioner guide suggests a channel needs around 500 conversions a week across the market for a four-week geo design, and that channels under about $5,000 a month in spend may not be testable yet. Treat those as starting points, not hard limits.
- Freeze everything else: No new creative launches, offer changes, or bid strategy switches mid-test. Each one adds variance you can't explain later.
AdBeacon shows click-based ROAS beside platform-reported ROAS for the same campaigns, so your lift factor comes from one screen.
How to read lift results against click-based attribution
Lift tells you how many conversions a channel caused. Click-based attribution tells you which orders a channel touched. Comparing the two over the same window gives you a conversion factor you can apply to daily reporting.
The math is simple. Divide the incremental conversions from the test by the conversions AdBeacon attributed to that channel during the same period. If a Meta prospecting test shows 400 incremental orders and AdBeacon's click-based attribution credited Meta prospecting with 500, your factor is 0.8. Going forward, you multiply attributed results by 0.8 to estimate incremental results without running a test every week.
A few patterns show up often:
- Factor near 1.0: Click attribution is tracking close to causal impact. Common for prospecting to cold audiences.
- Factor well below 1.0: The channel is touching orders that would have happened anyway. Retargeting and branded search often land here.
- Factor above 1.0: The channel drives sales it doesn't get click credit for, often through view-driven brand search or other channels. Worth a closer look at view vs click attribution before you act on it.
Run the comparison against platform-reported conversions too. Platform numbers that count view-through conversions usually produce a much lower factor than click-based numbers do, which is a direct measure of how much each source over-credits. AdBeacon shows its click-based ROAS beside platform-reported ROAS for the same campaigns, so you can calculate both factors from one screen, and the attribution model you choose (Lighthouse, first click, last click, linear) changes the denominator, so note which one you used.
What to do with the results before Cyber Week
If your October test finishes by mid-November, you still have time to act on it before Black Friday:
- Shift budget away from campaigns with low incrementality factors, especially retargeting that mainly reaches people already in your cart flow. AdBeacon's Recent Buyers Exclusion audience can keep spend off customers who just purchased.
- Hold or raise budget on prospecting that showed real lift, using MER and new customer CAC (nCAC) as your peak-week guardrails rather than in-platform ROAS.
- Set your daily reporting to apply the factor, so a Cyber Monday ROAS spike isn't read as pure ad performance.
Then rerun the identical design in January. One test is a data point. Two tests on either side of the peak start to look like a planning input.
Plan the test now, read it before the peak
Incrementality testing works best around Black Friday, not during it: one clean test before the peak, one after, and a click-attribution factor that carries the results into daily decisions. If you want to see how AdBeacon's click-based numbers line up against your platform reporting before you set up a holdout, book a demo with the AdBeacon team and we'll walk through it on your own account.
FAQ
What is incrementality testing in ecommerce?
Incrementality testing measures how many sales an ad channel actually caused by comparing a group exposed to ads against a group that wasn't. The difference between the two is the incremental lift.
Should I run a holdout test during Black Friday week?
Usually not for baseline planning. Promo-driven demand, auction volatility, and organic buying intent make Cyber Week results hard to apply to the rest of the year, so run tests before and after the peak instead.
How long should a holdout test for Meta ads run?
Plan for at least four weeks, or three to four times your average purchase cycle, whichever is longer. Shorter tests tend to miss delayed conversions.
What size holdout group should I use?
Most brands hold out 10% to 20% of their audience or market. Larger holdouts give a clearer read at the cost of some short-term revenue.
How do I compare lift results to attribution data?
Divide the test's incremental conversions by the conversions your attribution tool credited to that channel over the same window. The result is a factor you can apply to ongoing click-based reporting.