Most Unexpected A/B Test Results and What Could Cause Them

How A/B Testing Could Go Wrong

Running A/B split test experiments is not only a useful exercise to help you increase conversion rates but it can also be quite fun. Especially when your experiment shows some unexpected results. What could be the most unusual surprises?   

  • Control performs better than any treatment variation

You spent all that time planning your experiment and driving visitors to the site and at the end of the experiment the best performing option is your control. Try convincing your boss you need A/B tests after that! So what could have gone wrong? Most likely changes that are made in variations are not significant enough to increase conversions. You need to think of better things to run experiments on. Also it would make sense to look at the data in more details to identify if there were any other factors that might have affected the experiment. AB Split Test

  • Gender of visitors affects test results

If traffic is not homogeneous it is likely that your results might be affected by how men and women react to different images. A research called “What’s Psychology Worth?” found that using a professional picture of a woman had an effect on men but no effect on women, the response rate of a mail marketing campaign increased by 4.5% amongst men when a picture of a woman was shown. Take into account your target audience when testing a variation containing an image. 

  •  Change in navigation of a website performs worse than expected

When you introduce a new navigation your conversion could fall. Even though everybody in your company might agree that the navigation is much better than the old one it still might not perform as well as the control variation. The most common reason is that visitors are used to the old navigation and the novelty adds a confusion. Analogy here is a new layout of a store where you make weekly shopping. The minute the layout is changed it makes you uncomfortable, takes you longer to find what you are looking for until you get used to the new layout. Therefore, it’s better to test changes in navigation on new visitors first excluding any returning visitors from the test.   

  • Results are skewed by unpredicted factors

If your results are not what you expected don’t take them for granted. Try drilling down to what might have caused the experiment to end this way. There are certain things that could have an impact on results which you might not anticipate. For example, website down time, JavaScript errors and visits by bots or employees. In one of its tests Microsoft noticed that the new page design had a much lower conversion rate (click on the “buy” button was used as a conversion) but the number of page views per user was a lot higher for the treatment variation. When investigating further they discovered that the experimenting site had a monitoring system which was simulating clicks to determine if clicks fail, it tried several times before raising an alarm. In the new treatment variation clicks from a monitoring system didn’t work and it made many attempts which reduced the click-through rate.

To conclude, if results of a test look suspicious to you there might be a reason for that. Don’t take things for granted unless there were no possible factors influencing the test. Don’t stop experimenting.


Do you really know why visitors leave your site?

Eliminate the guesswork and boost your conversion rate with this simple tool

This is not just another tool in your online marketing armoury, it could literally be the only thing you need to have if you are serious about maximising the ROI of your existing online marketing activities.

The best part is that even a tiny improvement in conversion rates makes a big difference in the bottom line. There are numerous analytics providers that can help you improve the website usability, to name just a few: Crazyegg, Clickdensity and Google Analytics. However, my personal favourite is ClickTale.

ClickTale has become an industry leader very rapidly due to its great features, ease of use and integration. This is great you would say but how this is going to help me increase the conversion rate?

You can view recordings of what each visitor does on your website as if you were sitting behind him and looking over his shoulder. This can be quite useful for example when:

  • Members of your team can’t agree about the design of a landing page (happens with everyone)
  • Your had a recent redesign of a website
  • There is a high dropout rate on your check out page

If you have a highly trafficked website watching every visitor might become overwhelming over time but there are simple ways to analyse the data and quickly identify issues with your landing pages. The way to get round this is to use reports, identify where the issues are and then view recordings only of visitors that experienced these issues. Some of the reports I would recommend to pay attention to:

Conversion funnels – one of the most important reports, lets you see at which stage users leave the conversion process. You can identify issues in the conversion funnel by viewing recordings of users who left the site at a certain stage. Also great for comparing conversion rates for different marketing channels. Conversion funnel reports are highly customisable and could be scheduled to be sent by email.

Conversion funnel report. Analyse when visitors drop out and where they go after leaving each stage.

Form analytics – you can find out how users interact with your forms: which form fields make your users abandon the funnel, how long users spend on each field, which fields are left unfilled. Extremely useful for designing and optimising forms. When using ClickTale for optimising an application process for a job board I realised that users ignored/didn’t notice a tick box to agree with the terms and conditions which was a mandatory field. Making the tick box more prominent increased the conversion rate by 100% overnight. A simple change made such a big difference that it was hard to believe.

Form analytics. Find out how visitors interact with your forms.

Custom Events – lets you set up events which could be used for filtering visitor recordings. For example, events could be a visit to a registration page and a submitted form. This allows you to segment your visitors further. You might want to look at recordings of all visitors who came to the registration page but did not submit the form or you might want to identify recordings of users who submitted the form to see how they interacted with it.

Heatmaps – I find this particularly useful for new designs. You can quickly identify what remains unnoticed by visitors, how far down the page they scroll and which links and content are ignored. That certainly gives you a lot of room for optimisation.

Heatmaps. Discover what attracts visitors’ attention and what remains unnoticed.

Happy optimising! Watch your conversion rate increase and be prepared to meet a growth in demand.

Note: I have no affiliation with ClickTale, I simply find it to be a helpful tool for the CRO