Flawed A/B testing could be costing online retailers $13bn per year

Poorly implemented A/B testing could be causing online retailers to fail to capitalise on billions in potential revenue every year, according to CXM (Customer Experience Management) outfit Qubit.

A/B testing simply means testing A against B in terms of making a change to a website, and attempting to measure whether that theoretical change (B) makes a positive impact on sales compared to the original scheme of things (A) on the currently live website. If it looks good, the retailer can then adopt the change.

However, Qubit points out that while this is a pretty simple concept, there are a number of ways that A/B testing can be badly executed, leading to a potential positive impact being missed, or indeed a false positive impact being indicated.

Qubit has done some research into the figures, and claims that properly carried out testing has demonstrated a 12 per cent uplift in sales, which would total a $13 billion increase across US businesses.

Graham Cooke, CEO of Qubit, commeted: “A/B testing has been sold as a way to scientifically test the impact of changes to your website. However, as with all science, unless your experimental methodology is robust, the results of your testing will be meaningless. Many A/B testing approaches take a statistically naïve approach to methodology, leading to test results that are inaccurate at best and actively damaging at worst.”

Qubit lays out three major reasons for A/B testing failures. The first is insufficient statistical power, which is caused by failing to correctly judge the sample size for the test. There’s also the practice of running multiple simultaneous tests, which is an approach more likely to lead to false positives, Qubit argues.

Finally, there’s regression to the mean. Qubit explains: “Over time, many apparently positive changes will see a declining return. This is due to the well-known but rarely applied, statistical phenomenon of regression to the mean – which essentially states that false positives are inevitable in small testing samples but that anomalous results will regress towards the average, accurate result over time.”

For more on this topic, see our article on A/B testing and ensuring a long-term relationship between your website and customers.