Shopify A/B testing: how to run a test that tells you something

A test is a prediction you commit to before the traffic arrives

An A/B test on a Shopify store splits visitors into two groups, shows each a different version of one thing, and compares a single number. The comparison is only worth reading if four decisions were made before the first visitor was split: what changes, which number decides it, how many visitors it needs, and how long it runs. Most tests that "won" and then failed to show up in revenue skipped one of those four.

Google describes the mechanism in plain terms. In its guidance on website testing, an A/B test is one where you try two or more variations of a change and collect data about how users react to each version. You can run it by sending some visitors to a separate URL, or without changing the URL by inserting the variation dynamically with JavaScript. Both routes work on Shopify. They fail in different ways, which is most of what follows.

Change one thing, and write down which number decides it

Pick one change and one primary metric. For a product page that is usually add-to-cart rate; for the cart it is the share of sessions that reach checkout; for the checkout it is completed orders. Secondary numbers are for reading afterwards, not for rescuing a test that missed.

Two changes in one variant is a different experiment. If a new hero image and a new button label go out together and the variant wins, you have learned that the bundle works and nothing about which half did the work. That is fine when you only want the bundle, and useless when you want to apply the lesson to the next page.

Write the test down in one sentence before you build it: "Moving the add-to-cart button above the fold on 390 by 844 mobile viewports will raise the add-to-cart rate on product pages." A sentence that cannot be written this plainly is not ready to run.

Work out the sample size first, because it decides whether you can test at all

The smallest effect you can detect depends on your baseline conversion rate and how many visitors each variation gets. Evan Miller's sample size calculator shows the shape of it. Its pre-filled example is a 10.2% baseline with a result of 13.2% as the smallest difference worth detecting, and it reports 2,545 visitors per variation. That is a three-point lift on a high baseline. Detecting a much smaller lift, or starting from a baseline closer to what most stores see on a product page, needs a larger number, because the gap you are trying to see is smaller relative to the noise.

Do the arithmetic for your own store before you build anything:

If the answer is many months, the honest conclusion is that this page cannot be tested for a small lift. Test a bigger change, test on a page with more traffic, or skip the test and apply the fix on its merits. A change you can justify from a measurement, such as a hero image that is the largest contentful paint element or an add-to-cart button that is below the fold on a phone, does not need a test to be worth doing.

Fix the run length in advance and do not stop when the chart looks good

A test that is checked daily and stopped the first time one variant is ahead will often declare a winner that is just noise passing through. The calculator's number is the stopping rule. Decide the visitors per variation, divide by the traffic, round up to whole weeks so that every weekday appears the same number of times in each group, and do not call it earlier.

Round up for a second reason. Behaviour on a store differs by day of week and by what is going on in the calendar: a sale, a newsletter send, a holiday. A test that runs from Monday to Thursday has seen a different week from one that ran Friday to Monday. Whole weeks smooth out the first problem; keeping promotions off the tested page during the run handles the second.

Google makes the same point from the search side: a good testing tool tells you when you have gathered enough data to draw a reliable conclusion, and once the test ends you should update the site with the winning version and remove the test's alternate URLs and scripts as soon as possible. Google adds that a test left running for an unnecessarily long time may be read as an attempt to deceive search engines. The run length you fixed in advance is also your clean-up date.

Split the traffic so the groups are comparable

Two things break the comparison without any visible error.

Returning visitors who see both versions. If assignment is not sticky, a shopper who sees one version on their phone and the other on their laptop, or after a cleared cookie, is counted in both. Check that your tool stores the assignment in a cookie or a customer-level identifier, and ask what it does for visitors who block cookies.

Unequal groups. If you asked for a 50/50 split and the visitor counts differ by more than chance would explain, something is routing visitors unevenly: a redirect that fails on slow connections, a script that does not load in one browser, a variant that crashes on one device type. Compare the two counts on day one. A mismatch in how many people entered each group means the groups no longer started equal, and no amount of analysis fixes that afterwards.

Flicker is the failure the metric will not show you

Client-side tests work by loading the original page and then changing it in the browser. If the change arrives late, the visitor sees the original for a moment before it swaps. That flash is a problem twice over: it can change behaviour, so the test measures the flash as well as the variant, and it moves content, which counts against cumulative layout shift. The web.dev guide to optimizing Cumulative Layout Shift defines the metric as the combination of how much visible content shifted and how far the affected elements moved, so a swapped block that changes height after paint registers.

The usual remedy is an anti-flicker snippet that hides the page until the variant is ready. It trades one cost for another: the page is invisible until the testing script has loaded, so a slow script delays everything. That is the same cost described in third-party scripts on Shopify, and it lands on the largest contentful paint of every page the snippet runs on, not only the tested one.

Three ways to keep the cost down:

• Test on templates where the variant can be rendered by the theme itself, with a section setting or a Liquid condition, rather than rewritten in the browser. No swap, no flicker, no extra script.
• Load the testing script only on the templates under test. A tag that fires on every page taxes the ones you are not testing.
• Measure the page with and without the test live: run the product page through PageSpeed Insights in both states and compare the largest contentful paint and the layout shift. If the test costs more than the lift you expect, the maths of the test has already failed.

Test the page, but check the search side first

Most conversion tests do not touch search results. Google notes that small changes such as the size, colour or placement of a button, or the call to action text, often have little or no impact on a page's snippet or ranking. A test changes what Google sees only when it changes content that Google reads or the URL it reads it at. For those, Google's guidance gives four rules:

1. Do not cloak. Never show one set of content to Googlebot and a different set to people. Google counts cloaking as a spam-policy breach whether or not it is part of a test, and it notes that Googlebot generally does not support cookies, so a cookie-based assignment shows it the version for browsers that refuse cookies.
2. Use rel="canonical" on the alternate URLs when a test runs on multiple URLs, pointing at the original, rather than a noindex.
3. Use a 302 redirect, not a 301, when sending visitors from the original URL to a variant, so the original stays in the index.
4. Remove the test when it ends. Alternate URLs, scripts and markup go as soon as the result is in.

In practice that means that testing a button label or a block order on a product template is low risk for search, and splitting a collection page onto a second URL is the case that needs the canonical and the 302 set up correctly.

Read the result with the checks you set at the start

When the run ends, read in this order:

1. Did each group reach the sample size you fixed? If not, the result is not a result.
2. Are the two group sizes close to the split you asked for? If not, discard and find the routing fault.
3. Did the primary metric move, and by more than the smallest lift you said was worth acting on? A difference smaller than that can be real and still not worth shipping.
4. Did the secondary numbers move the wrong way? A variant that raises add-to-cart and lowers completed orders has moved the problem down the funnel, not removed it.
5. Does the page still load as fast? Re-run the measurement you took before the test.

Then apply the winner in the theme, remove the testing code, and write down the sentence, the sample, the run dates and the result. A log of tests with their outcomes is more useful in a year than any one of them, and it is the evidence to ask for from anyone who tests for you, as set out in what a Shopify CRO specialist does.

What to do first, and how long it takes

Start with the arithmetic: baseline rate, weekly sessions, the sample the calculator asks for. That is twenty minutes and it tells you whether the page can be tested at all. If it can, write the one-sentence prediction, check the split and the assignment in the tool, and measure the page before the test goes live; an hour for the setup, then however many whole weeks the sample needs.

If the page cannot be tested, the right use of the time is the measured fixes on the services page and in the audit checklist, which need no sample size. And if you would rather have the store's test candidates found and ranked for you, that is what the Store Teardown is for.