How to Run Theme Tests in Shopify

Ever wondered if your store might be better off using a different Shopify theme?

Now, there’s no need to wonder. 

In addition to testing specific pages and templates, you can use Shogun A/B Testing to see what happens when you switch over to an entirely new theme for your store. By sending some visitors to this new theme and others to your original theme over the same period of time, you’ll be able to objectively determine which theme performs better.

Theme tests can be used to compare two different variations of the same theme as well. In this guide, we’ll go over everything else you should know about Shopify theme testing, including:

• Use Cases for Theme Testing in Shopify
• Setting up a Theme Test Using Shogun A/B Testing

Use Cases for Theme Testing in Shopify

There are many different scenarios where you might want or need to test your whole Shopify theme rather than just a page or template, such as:

• Implementing entire site redesigns: If you want to use an entirely different theme for your site, you would be wise to test it out before you publish it for everyone. What if your new theme ends up performing worse than your original design? In that case, you’ll know that you need to look for issues that you might have missed initially, such as sluggish load times or design elements that don’t render right — or you might discover that you should just scrap the redesign and stick with your original theme.
• Launching new sales funnels: Even if you plan to keep using your current theme, you might want to test performance across your entire site — this would require a theme test, as page tests and template tests are limited to just a single page or set of pages, respectively. For example, if you start advertising on a platform for the first time, you might consider making some tweaks to your overall site design to accommodate these new visitors. A theme test would help you figure out if that’s a good idea.
• Testing new site navigation: Want to make some changes to your site’s main menu or other navigation options? With a theme test, you can see whether these changes are actually steering visitors in the right direction. 
• Making other changes to your header or footer: In addition to your menu and navigation options, there may be other elements on your site’s header or footer that you want to experiment with, such as social media links or information about payment options. A theme test will give you the ability to do just that.
• Comparing translations: If a significant share of your sales come from buyers who speak a different language, then it’s especially important for the content on your store to be translated accurately. Such merchants can use a theme test to compare the results of different machine translation services or see how the work of a human translator stacks up.
• Evaluating mobile-friendliness: If you want to test changes intended to make it easier to use your site on smartphones and tablets, then you’ll probably want to run an experiment on your whole theme instead of just one page or template. After all, these changes will usually affect your entire site (making text bigger, increasing the contrast between the color of text and the color of the page background, etc.), so it only makes sense to test your entire site as well. 

Setting up a Theme Test Using Shogun A/B Testing

With Shogun A/B Testing, you can create a new theme test by simply following these steps:

• After installing the Shogun A/B Testing app, select the Apps option in the left sidebar of the main Shopify control panel.
• Open Shogun A/B Testing.
• Click on the “Create test” button near the top-right corner of the page. This will open up a dropdown menu — select the “Theme test” option here.

By default, the name of your test will be the name of the theme you’re currently using, but you can change this to whatever you want.

In the “Choose which theme to test” section of the theme test settings, you’ll see your current theme displayed in the “Original” box. In the “Variant” box next to it, click on the “Select another theme” button and choose which theme you want to test. Before this step, you should have already made any edits that you want to apply to your new theme (or a copy of your current theme, if that’s what you want to test here) through Shopify’s built-in theme editor.

Shogun A/B Testing allows you to customize how much traffic goes to your original theme and how much goes to the new one you’re testing out.

By default, this traffic split will be set to an even 50/50. But you should strongly consider sending more traffic to the original version of the page. This is because there’s always a chance that switching over to a new theme will result in worse performance for your store — by sending fewer people to the new theme, you’ll be able to minimize the damage to your bottom line as well as your reputation if your test ends up not working out well. 

Shogun A/B Testing also allows you to select how you want to measure success for your test. Your options include:

• Conversion: The percentage of visitors who complete a purchase.
• Clickthrough: The percentage of visitors who click a specific element on the page.
• Add-to-cart: How often visitors add items to their cart. 
• Form submission: How often a specific form on the page is completed (to track this metric, you’ll need to plug in the form ID, which can be found by using your browser’s developer tools to inspect the HTML code of the page). 
• Revenue per visitor: Total sales divided by the number of visitor sessions.
• Average order value: Total sales divided by the number of orders (discounts are accounted for in this metric). 

The metrics that you didn’t select as your test goal will still be tracked while you’re running your test, so don’t worry if there are two or more of these metrics that you’re interested in. 

Finally, Shogun A/B Testing gives you the option of only performing your experiment for a particular type of visitor if you prefer. There are several pre-built audiences to choose from, and you can create your own custom audience as well. 

Shogun allows you to choose from the following types of conditions for your audience:

• URL: Only perform your test for visitors if their URL contains (or doesn’t contain) a certain set of characters. This is useful for influencer marketing, Google Shopping ads, and all other forms of digital marketing where you can add a suffix to the end of your inbound link for tracking purposes.
• Logged-in customer: You can target visitors based on whether this condition is true or false.
• Device: Target visitors based on whether they’re using a desktop or mobile device.
• Referrer: Only perform your test if the URL of the page that sent the visitor to your site contains (or doesn’t contain) a certain set of characters.
• New visitor: As with logged-in customers, you can target visitors based on whether this condition is true or false.
• Returning visitor: For this condition, you can determine the exact number of hours or days it takes for someone to be considered a returning visitor.
• Location: Target your test to visitors who are located in a specific country, region, or city.

It’s worth mentioning that you can set multiple conditions for the same audience segment. And in that case, you’ll be able to determine whether all of the conditions or only one condition must be met in order for the visitor to be included in your audience.

Once you’re done configuring your test, click on the “Review” button. You’ll then get a chance to rename your test again — once you’ve confirmed the name, select “Start test”. 

While your test is running, you’ll be able to keep track of real-time visitor data for both the original theme and your new theme. There’s a section at the top of the test results page that lists the total sessions for each theme as well as the performance in terms of the main goal that you’re tracking.

Further down the test results page, you’ll be able to see the performance of each theme for all the metrics that Shogun A/B Testing keeps track of. This data is updated every five minutes.

You’ll want to keep running your test until each theme has received enough visitors to generate statistically significant results. The less data you have, the more likely it is that your results are just due to random chance. Conversely, as you collect more data, the results will eventually reach statistical significance — that means you can be reasonably confident that the results actually prove (or disprove) the hypothesis you’re trying to test.

How can you tell if your results are statistically significant yet? Don’t worry, you don’t need to be a professional data scientist to figure this out. There are free online tools available that can do the math for you. All you need to do is plug in details such as the number of visitors and the conversion rate for each theme, and these tools will tell you whether the results are statistically significant. If they aren’t yet, just keep collecting data and check again later.

Whenever you’re ready to conclude your A/B test, just click on the “End” button. If you’ve determined that you want to stick with your original theme, then no further action is needed. But if you want to switch to the new theme, you’ll need to do so through the Shopify admin:

• Select the sales channel you want to customize (e.g., “Online Store”) in the left sidebar of the main Shopify control panel.
• Scroll down to the “Theme library” section of this settings page.
• Click on the “Publish” button next to the theme that you want to switch to.

As mentioned above, Shogun A/B Testing also allows you to run page tests and template tests. If you’d like to run one of these types of tests while your theme test is running, here are a few things you should know:

• Only one theme test can be run at a time, but they can indeed be run simultaneously with page tests and template tests.
• Once your theme test is running, visitors who meet your audience criteria will be assigned to either the original theme or the new theme. This assignment will stick whenever the visitor returns, so that they will continue to see the same theme during the entire test. 
• If a visitor was assigned to the original theme, then they may also be served a variant of a page test or template test. If they were assigned to the new theme, then they will not be involved in any other test.

At this point, you are fully prepared to start running your first theme test with Shogun A/B Testing. The insights provided by this tool can help you increase your conversion rate, generate more sales, and achieve whatever else you want for your Shopify store.