I was doing some work with a client recently. We were looking at their homepage and trying to work out how to optimize the amount of traffic going to one important section of the site. I looked at the homepage, and decided the best thing to do was change the link text as it was a little ambiguous. At first I thought “Oh this is sure to increase click throughs, I’ll just do it”. I consulted with a people who all agreed that the change was a definite improvement.

However, I decided I should do an A/B test, showing one text to half the users, and the other to the other half of users, and tracking how many of them clicked on the link. Full of pride, I wanted to be able to prove my infinite cleverness and demonstrate how much I improved click throughs  with one simple change. Would it be 10%? 25%? 50%? Who knows! I was getting excited.

After running the test for a few days, I had enough data for an answer.  And the result was… a staggering -26% improvement. A great number, except that pesky minus sign at the front. Yes, my content genius had decreased click throughs by 26%.

The point here is that despite my experience in this field, I am still consistently wrong. I wish I could say this was an unusual event but it’s not. The more I test, the more I realize I just don’t know the answer. It’s not just me. I spoke at a conference a while ago to a bunch of site owners. I showed them three versions of a site and asked them to guess which one would have the best sales. About 80% of them picked the same one, which was the same one I had picked when I first saw it. It also happens that one was, by quite a margin, the worst performing of the three. So, don’t guess, test. It pays off – quite literally.