What is A/B Testing?

You have probably heard the term “A/B testing” thrown around in conversations about Google, Amazon, and other leading tech companies known for their impressive ability to optimize just about everything they do. But what really is A/B testing and how does it work?

Put simply, an A/B test is a way of running an experiment that is designed to compare the performance of two different versions of something, typically on a website or app. For example, are people more likely to click “subscribe” if I make the button red or green? Will I see a higher conversion rate if this sneaker is priced at $109 or $99?

To run an A/B test, it’s important to first define what you want to test, and how you hope it will move the needle for your business. You should enter a test with a hypothesis of what you expect will happen, and an understanding of what metrics you want to track.

If you are testing price, common KPIs to track are total revenue, total margin, conversion rate, average order volume, and blending those together, average revenue per site visit and average margin per site visit. Depending on the test, things like product mix or % of customers buying a subscription may also be useful.

The next step is to actually run the test. When a potential customer arrives at your site, they are randomly assigned to one of two groups that will experience a different version of the element you are testing. Sticking with our shoes example, this means that Group A will see your sneakers priced at $99, while Group B will see the sneakers priced at $109. By tracking conversion rate, it’s possible you may find that counterintuitively, shoppers are more attracted to the higher $109 price tag as it implies quality. If this result ‘holds’ as statistically significant across a large number of customers, that allows for $10 of additional profit for each pair of sneakers. Once you have this ‘winner,’ you can roll it out to all of your potential buyers and site visitors.

While A/B testing is actually quite a simple concept, execution can be more challenging. Tracking all of the necessary data and performing statistical analysis is hard. True randomization is critical both in terms of generating rigorous data, but also ensuring that you don’t inadvertently run into price discrimination issues (e.g., charging more based on zip code, which correlates with race). Ultimately, A/B testing is best done through specialized A/B testing software to automate the process and deliver clear insights. Just ask https://www.liveliina.com/ !

Let us know if you ever want to discuss A/B testing on your e-commerce store. Get in touch at info@intelligems.io, sign up on our website or find us on the Shopify App Store!