Short answer: yes, you can now split-test Black Friday offers on your own storefront before you commit to one, because discounts joined Shopify Rollouts on 2 October 2026. You assign a share of traffic to each offer, schedule the start and end, and roll back without touching code. The harder part hasn't changed. Most stores don't have enough November traffic to tell a 10% lift from noise, and the test that looks like it worked is usually the one that was contaminated.
So here's how we run Black Friday discount testing on Shopify for US merchants: what to test, how much traffic you need before the result means anything, and the four or five mistakes that turn a clean experiment into an expensive guess.
What Rollouts changed, and what it didn't
Before this, testing an offer meant one of three bad options. Flip the sitewide discount at 9pm and compare to last year. Build a duplicate theme and route traffic with an app. Or pay for an experimentation tool that sat in front of the storefront and slowed it down.
Now the discount itself is the thing being rolled out. You can put 50% of sessions on 20% off sitewide and 50% on free shipping over $75, set it to start Wednesday at 00:01 store time and end Friday at 23:59, and have a rollback path that doesn't involve someone awake at midnight deleting an automatic discount by hand.
What it didn't change: statistics, attribution, and the fact that your email list doesn't know it's in an experiment. The platform now gives you the mechanism. It does not give you a valid result.
Sample size for offer testing: do the arithmetic before you build anything
This is the step almost everyone skips, and it's the one that decides whether the test is worth running at all.
A workable approximation for a two-arm conversion test at 95% confidence and 80% power:
n per arm ≈ 16 × p(1−p) ÷ Δ²
where p is your baseline conversion rate and Δ is the absolute lift you want to detect.
Say your storefront converts at 2.5% and you'd like to catch a 10% relative improvement, so 2.5% to 2.75%. Δ = 0.0025. p(1−p) = 0.024375.
16 × 0.024375 ÷ 0.00000625 = 62,400 sessions per arm. 124,800 total for two arms.
If your whole BFCM week does 40,000 sessions, you cannot detect a 10% lift. Full stop. Run the same formula backwards with 13,333 sessions per arm (40,000 split three ways) and you get Δ ≈ 0.0054, which is 2.5% to 3.04%. A 22% relative lift is the smallest thing you'd reliably see. Offers that good exist, but if you're comparing 20% off to 25% off, you're not going to find one.
Two consequences. First, test big, structurally different offers, not neighbouring numbers. Second, three arms is expensive: splitting traffic three ways cuts each arm's power, and three pairwise comparisons push your false-positive rate from 5% to about 14% (1 − 0.95³). We usually talk clients down to two arms plus a small holdout.
A/B testing discount offers when the metric is money, not clicks
Conversion rate is the wrong primary metric for an offer test, and it's the default everywhere, which is why so many stores pick the offer that sold more units at less profit.
Use contribution margin per session. Revenue minus COGS minus the discount minus shipping cost minus payment fees, divided by sessions in that arm. Revenue per session is an acceptable stand-in if your margins are flat across the catalogue. They usually aren't.
The catch: revenue per session has far more variance than conversion rate, because one $400 order moves it. So the sample size you calculated above is a floor, not a target. In practice we look at both — conversion rate for the directional signal, margin per session for the decision — and we refuse to call a winner when they disagree and the margin gap is inside a few percent.
Free shipping threshold vs percentage off: run the numbers on your own basket
General advice here is worthless because the answer depends entirely on your AOV and your parcel weight. Do this with your own figures. Here's the shape of it with a $70 AOV and 60% gross margin, so $42 of gross profit per order before shipping.
- 20% off sitewide. Revenue drops to $56, COGS stays at $28. Gross profit $28, and you still pay shipping. The customer gets the best headline number and you take the full hit on every single order, including the ones that would have converted anyway.
- Free shipping, no threshold. Revenue $70, COGS $28, shipping cost around $8 for a one-to-two-pound ground parcel in zones 2–5. Gross profit $34. Better than the percentage off, and it reads well in an ad. But it subsidises your smallest orders hardest.
- Free gift over $85. If the threshold actually moves average order value to $88, revenue is $88, COGS at 40% is $35.20, the gift costs you $6 landed, shipping $8. Gross profit $38.80. You're also moving a unit of inventory you chose rather than one the customer chose.
On paper the threshold wins. In the test it often doesn't, because the AOV lift is smaller than the model assumes and some share of customers bounce at the gap between their cart and the threshold. That gap is the whole game. Set the threshold about 15–25% above current AOV and the cart nudge does real work; set it at double AOV and you've just made your offer invisible.
One US-specific wrinkle worth knowing before you compare margins: shipping charges are taxable in a number of states, so zeroing out shipping changes the tax base slightly and your arms won't be perfectly comparable on net revenue. It's small. It is not zero, and if you're running Shopify Tax across twenty-odd nexus states, pull the figures net of tax before you compare.
How long to run a BFCM offer test
Not during BFCM. That's the honest answer and nobody likes it.
The Wednesday-to-Monday window is five days of wildly different traffic mixes — paid, email, organic, affiliates, Honey extensions firing on the checkout — with shifting ad spend on top. Day-of-week effects don't average out over four days. And the cost of being wrong is your entire year's peak.
What works: run the test in early-to-mid November on a smaller promotion with the same structure. If you want to know whether a threshold beats a flat percentage, find out on a 7–10 day November campaign, with a full number of complete weeks so weekday and weekend traffic are balanced. Then take the winner into BFCM as a scheduled change, not an experiment.
If you insist on testing during the event, keep it to a single clean comparison, run it across complete 24-hour cycles, and decide the stopping date in advance. Checking twice a day and stopping the moment something crosses significance is how you manufacture a winner that doesn't exist. Peeking inflates the false positive rate far past the 5% you think you're accepting.
Scheduling discount changes so nobody has to be awake
Set start and end times in the store's timezone and then write the actual times out in Eastern and Pacific next to the schedule, because someone on the team will misread it otherwise. We've seen a Cyber Monday offer go live three hours late because the store was configured to Pacific and the marketing calendar was in Eastern.
The scheduling checklist we use:
- Discount scheduled with an explicit end time, not an open-ended one someone has to remember to kill.
- Banner, cart message and homepage hero on the same schedule. A test where one arm sees a sitewide banner advertising the other offer is dead on arrival.
- Email and SMS sends matched to the schedule. If you blast the full list a discount code at 6am, that code applies to both arms and your experiment is over.
- A named rollback: who does what if checkout errors climb, and what the trigger number is.
- Inventory floor per arm. If the gift SKU in the threshold arm runs out on day two, that arm becomes a different offer halfway through.
Staged discount launch on Shopify Plus
If you're on Plus, the useful pattern isn't really a test, it's a staged launch. Put the offer in front of 5% of sessions, watch checkout completion rate and support ticket volume for an hour, go to 25%, then 50%, then everyone. You're not looking for a lift. You're looking for breakage: discount stacking with an existing automatic discount, a Function that throws on gift cards, a subscription line that gets discounted when it shouldn't.
That's a genuinely good reason to use progressive rollout on peak day, and it's separate from the measurement question. It costs you nothing in statistical power because you're not comparing arms, you're checking for errors.
Worth saying plainly: none of this is a reason to upgrade to Plus. Checkout extensibility, B2B and scripting are reasons. Staged discount rollout on its own is not, and the Plus bill at peak season rarely pays for itself on one feature.
The failure modes that make the result meaningless
In rough order of how often we see them:
The email list overrides the test. You split the storefront 50/50, then send 80,000 subscribers a single code. Those sessions carry one offer regardless of assignment. Either split the send too, or exclude email traffic from the analysis entirely.
Code leakage. Public codes end up in browser extensions and deal forums within hours. Automatic discounts tied to the rollout are much harder to leak than codes, which is one reason we lean on them.
Cross-device and returning visitors. Someone browses on a phone on Thursday and buys on a laptop on Friday. They can land in the other arm. This is noise you can't eliminate, only account for by running longer.
Site speed collapsing under load. Peak traffic plus a heavy theme and a dozen apps and your LCP goes from 2.4s to well over 4s. Conversion falls in both arms, but not necessarily equally, because one arm's cart drawer does more work. If your storefront isn't already fast in October, fix that before you worry about offer testing — we cover the mechanics in Shopify speed optimization, and SwiftStore will at least tell you where you stand and keep watching the score through the season.
No holdout. Two discounts beating each other tells you which discount is better. It doesn't tell you whether either beat doing nothing. Hold back 5% of traffic with no offer. On BFCM weekend that costs you very little and it's the only way to see what the promotion actually bought you.
Testing offers without wrecking margin
Three things keep the cost of learning small.
Cap the downside per arm. Exclude your thinnest-margin SKUs and anything already on markdown from the discount. A sitewide percentage applied to clearance stock is a transfer from your bank account to nobody's benefit.
Put the aggressive offer on the smaller slice. A 90/10 split with your proven offer at 90% costs far less than 50/50 and still produces a readable result if the effect is large. You give up power, so only do this when you're testing something structurally different, not a 5% change in discount depth.
Kill it early on margin, not on conversion. Set a floor before you start — say, contribution margin per session no more than 8% below control — and stop the arm if it breaches for two consecutive days. This is not statistically pure. It's commercially sane, and in November commercial sanity wins.
Questions we get asked in October
Can I test three offers at once? You can. You usually shouldn't. Three arms on a store doing under 100k sessions a month means the smallest detectable difference is so large that only a disaster or a runaway winner shows up. Pick the two you genuinely can't choose between.
How much traffic do I need? Use the formula above with your own conversion rate. As a rule of thumb at a 2.5% baseline: roughly 15,000 sessions per arm to see a 20% relative lift, roughly 62,000 per arm to see a 10% one.
Does a percentage discount or a free shipping threshold convert better? Percentage off typically lifts conversion rate; a threshold typically lifts AOV and protects margin. Which nets more money depends on your basket and your parcel cost, which is exactly why it's worth testing rather than arguing about.
Can I just compare to last year? No. Different ad spend, different list size, different competitive environment, different Meta CPMs. Year-over-year is a business report, not an experiment.
What if the result is flat? That's a real result and a useful one. It means pick the offer that costs you less and spend the energy on something that moves — creative, email segmentation, the first-purchase experience.
Where to start this week
Pull your last 30 days of sessions and conversion rate from Shopify analytics, run the sample size formula, and find out what size of effect you can actually detect. That single number tells you whether to run a two-arm test in early November, a 90/10 split, or no test at all.
If you'd rather have someone else do the arithmetic and set up the schedule, our team works with US brands on exactly this kind of peak-season prep — see how we work with Shopify merchants in the US, or book a free store audit and we'll tell you what your traffic will and won't support before you build anything.