A/B testing | A Practical Approach

Testing is trying out something different to see if it works. In software development testing takes centre stage and plays an important role in building user driven products. If you think about it there are three ways to test your software.

Both 1) and 2) are types of acceptance testing and similar in some sense. 3) is A/B testing and has a completely different objective.

What is A/B test

An A/B test consist of three parts

A: This is what the current state of the product is

B: This will be the state of the product after the release of the update

Test: Measuring the impact of the change. This also forms your hypothesis

Why do you need A/B testing

Companies spend a lot of time and resources building new capabilities and making big changes to their product. Change is great but change whose impact cannot be measured is meaningless. There are reasons why sudden massive changes are not good:

That’s where A/B tests come in. They empower product managers to accurately measure impact through small incremental changes.

Examples of A/B tests 

A/B testing is useful for making data driven decisions. Some areas in which A/B testing can help you make informed choices are

Key parts of an A/B test

Goal & Motivation

What is your driver behind conducting an A/B test? Your primary focus could be improving user experience or driving revenue. Making it clear in the A/B test helps align the team toward the goal.

Hypothesis

You cannot run an A/B test without forming and stating your hypothesis. Your hypothesis should be short, crisp and to the point. It should not run into a couple of pages. You can use the following format for writing an A/B test hypothesis:

“By making “X” change, we expect “Y” result”

Test groups

An A/B test has two groups:

Randomisation

Users should be put into control and variant groups at random without any specific rule. This enables the experiment to run without bias being induced unknowingly that can skew the results.

Traffic split

As part of the experiment you need to specify how the traffic needs to be split between control and variant.

Duration

A/B experiments need to have a start and end date. Keep seasonality in mind when you decide to run your experiment.

Variants

In most cases you will have a control group and variant. But at times you might want to test more than one variant. In such cases you can do an A/B/C test where B and C are two variants to find out the winner!

Platforms and Devices

Users behave differently across platforms and devices. You will have to run different tests for each type to prove your hypothesis. What works for Desktop might not work for Mobile .For example what might work for Android users might not work for iOS user. You cannot extrapolate the results of one platform to say that it will work for the other.

Statistical Significance

Survey Monkey explains this really well. According to them

In the context of AB testing experiments, statistical significance is how likely it is that the difference between your experiment’s control version and test version isn’t due to error or random chance.

You can use this online tool from survey monkey to calculate statistical significance.

Metrics Tracked

It is important to not confuse this with the impact. These are metrics you track for your product on a regular basis. Your A/B experiment should not hurt other important metrics at the cost of increasing the impact of the experiment.

How to setup A/B tests

Write the test proposal

The first step is to write up your test proposal which succinctly describes each part mentioned in the above section.

Ask yourself the following:

Download our A/B test template to get started

Get Stakeholder Buy-In

You have to get everyone on the same page that running the A/B test is important. Setup time with all the key stakeholders to go over your test proposal. Listen to their feedback and update the test where it makes sense.

Track Progress

You should check how your test is performing atleast once a week with the key stakeholders. At the end of the test make sure you communicate the final results and next steps. Sometimes you have so many tests running at once that it’s easy to lose track of changes.

Find the right Application

Now that you know everything about A/B testing the big question is  how do you go about setting one up for your product. Don’t worry you don’t have to build from scratch. There are some great tools out there which you can use for the A/B test. If this is the first time your company is running A/B test I have listed a few of them

Depending of your specific use case one tool might work better for you than the other.

Challenges with A/B testing

While A/B testing is a great framework in our arsenal(go gooners!! ⚽️ ) it come with their own set of challenges. Lets look at some of them:

Conclusion

So after reading all this one question comes to mind. Does it make sense for you to do A/B testing?

It depends!!

🤦‍

I know that’s not the answer you were looking for. Does your organization faces any of the challenges above? At the end of the day you have to weigh the pros and cons. For some people doing A/B testing might not be practically possible. Often teams run into the bad habit of testing without purpose. Try to keep it in check. Create a centralised test approval and launch team. Yes I said centralized😇 ! In this scenario it actually does help. It provides a unified view of all the a/b tests running in the company and new tests that different teams want to run.