What is a Split Test?
While this post isn’t meant as an introduction, I’ll cover it very briefly. A split test is where you show some people one version of a page and others another and measure which one produces the best results. It’s arguably the single most effective way to improve your conversion rate. It doesn’t matter what your customers say about your shopping cart (your friends will almost always so “oh it’s so easy to use”), and what your opinion is. What’s important is to measure what people actually do. Even the experts pick the best performing version incorrectly much of the time, which they will readily admit, so testing real performance is critical.
The Challenge of Split Testing Templates
Usually when people talk about testing, they are talking about testing a single landing page, or a page in your checkout. Technically they are easy to test. Where it gets tricky is when you want to test your product template in your shopping cart, or a page template in your CMS. In most systems, it’s very hard to create a second version of the template and test. So what’s the solution? That’s what this blog post is about.
How to Split Test a Template Based Site
We’ll talk about testing with Google Website Optimizer. It’s an excellent tool, widely used, and free. Win!
1. Log into Google Website Optimizer with a Google Account at https://www.google.com/analytics/siteopt.
2. Choose “Create a New Experiment”
3. You are given the choice between an A/B test and Multivariate test. Google Website Optimizer informs you that A/B Experiments are simple and Multivariate tests are robust. However, ignore the suggestions Google Website Optimizer makes, included the recommendation that you need at least 1,000 page views per week. We are abusing this tool to get it work the way we want so that isn’t relevant. That’s the first trick.
4. Next, we are at the “New Multivariate Experiment” page. Fill in the fields as follows:
“Experiment Name” – give the experiment a meaningful name. In my case, I’ll call it “Product Page Add to Cart Button”.
“Identify Your Test Page” – in this box, put in the URL of any page that’s using the template you want to test. In my case, I’m testing the product template, so I add any product page, it doesn’t matter which one. For example, you’d put in www.mystore.com/brand/red-widget.html. We are testing the template, not just this page. We just need to trick Google.
“Identify Your Conversion Page” – put in the conversion page, that is, the page that users will reach when they’ve achieved the goal you want to boost. For an ecommerce site, this is often the “thank you” page after completing the purchase. In this case I am testing the “add to cart” button, and the shop doesn’t have a huge amount of sales per day, so it would take a long time to know whether the button was improved or not. So instead, I’ll put the conversion page as the page you are shown after clicking the add to cart button. There’s all sorts of clever customizations you can do around tracking the conversion page, but they are beyond the scope of this post.
5. Next you are asked who will be installing it. For the purposes of this post I’ll assume it’s you. The process is pretty similar if it’s not you.
You’ll be shown a few snippets of JavaScript with instructions on how to put them in.
What you want to do is to put this in your template file. In this case, where I am talking about making a change to a product page, I’ll find the products template and put it in there. So to be clear, we are making the change to every product page, not just one. This is just what we want! Remember how on the last screen it asked us for one specific page? Don’t worry about that. It’ll work.
The instructions can be difficult to implement in some template systems. For example, “Paste the follow at the beginning of your test page’s source code”. What you ideally need to do is put the code directly after the <head> tag in your template (detailed instructions are in the Google Help site if you want to read more). With some shopping cart templates it’s hard to control the exact place you put the code – it could be the <head> section is in a generic template that controls the whole site. In that case, just put it as close to the top of the product template as possible. Even just after the <body> tag will work. While this isn’t officially supported, I’ve put the scripts in all sorts of odd places depending on the shopping cart or CMS, and never had a problem.
Page Sections
The next interesting section is the page sections.
This is where you define the section(s) you want to test. This is the crucial piece of code. Multivariate tests are designed to test different parts of the page simultaneously. However, unless you have a lot of traffic and a lot of sales, then multivariate is impractical. So you need to decide what the test is. This could be something very simple like my test where I’m just trying different “add to cart” buttons, all the way through to testing complete new layouts. I’ll cover how to do some of this trickier stuff at the end, and stick with the simple example for now.
Here’s what my code looks like:
I’ve named the snippet “CartButton”, and put the code for the current button inside the snippet. The name “CartButton” will show up in the Google Website Optimizer later, so make sure the name is obvious. To stress again, unless you have a lot of sales (at least 50+per day) I recommend testing only one thing at a time.
When you go into Website Optimizer, you’ll see it appear there.
What you do now is click the “Add new variation” and put in the code you want to test. In this case, I’m putting in another add to cart button. You can add as many variations as you want, but be warned again, the more variations you have the longer it will take to test. Unless you know what you are doing, first time around stick to one variation.
That’s it! You have created your test! Just go through the preview and launch and your test is live.
Check back next day to make sure it’s all working. You should be seeing conversions coming in and with time you’ll be able to tell which is the winning version.
How to Test Layout Changes
So you say “ok, that’s fine, I can swap out an image or whatever, but I want to try more advanced stuff”. No problem! It gets slightly trickier, but not too bad. Some tests you might want to try:
- Changing the layout of the page
- Changing a style e.g. making the headline more dominant
- Testing the presence or absence of an item, for example, a testimonial, security seal, or similar
How do you do this? Let’s start with testing whether an item makes a difference by being there. This is pretty simple. Let’s say you have a testimonial section on your page:
<div id=”testimonials”>
Lorem ipsum dolor sit amet, consectetur adipiscing elit. Curabitur euismod adipiscing eleifend. In non justo velit, aliquam tempus tortor. Aenean mauris sem, faucibus nec venenatis ac, faucibus eu mauris.
</div>
Simply add a test section to the page, and add some CSS to turn it off.
<script>utmx_section(“testimonials”)</script>
<style type=”text/css”>
#testimonials
{display:none;}
</style>
</noscript>
Then when you create a page section variation in Website Optimizer, simply leave it blank. By not showing the CSS that turns it off, then it’s on.
How about making a headline more dominant? It’s basically the same as above. Instead of testing CSS to turn off a section, just use CSS to change the style.
Changing the page layout is a bit trickier. Ideally this would be done with an A/B test. However, that’s tricky to do with many shopping carts and content management systems without some major code hacking. The solution is to insert a great big chunk of CSS similar to the two examples above, but use that CSS to significantly override the main style sheet and replace it with the new test style. You may need to use the !important declaration at times if needed. If your layout is still done using tables, this might be so difficult as to be in practice impossible. So, sorry, you’re out of luck! Time to get your code up to scratch!
If you are doing advanced CSS tests, don’t forget to preview your tests across different browsers to make sure all users have a good experience.
Categories: Conversion, Shopping Carts, Usability | Date: Nov 25th, 2010 | Author: Mark | 1 Comment
The Magento Development Cookbook has the tagline
Magento 1.4 Themes Design addresses the extremely powerful Magento theming engine. This is well deserving of a book in itself as this is both a very powerful part of Magento, and a place where developers are likely to spend a lot of their time.
The “EG-1001″ shopping cart is made by NewSky Tech. They appear to be based in China. The EG-1001 seems to be a relatively unconventional design, unusually deep and not like the flatter bed shopping carts I have used in the past at my weekly grocery run.
