A/B testing | A Practical Approach

A/B testing | A Practical Approach

Testing is trying out something different to see if it works. In software development testing takes centre stage and plays an important role in building user driven products. If you think about it there are three ways to test your software.

1.Testing the product by the Quality Assurance team before it is released to users and customers. The goal of this testing is to find out if the product works as per requirements. All companies do this type of testing. The amount of effort they put in it varies a lot.
2.Testing of the product by external users. The goal of this testing is to make sure that it is consistent with the expected user experience.
3.Testing to find out if the update or change in the product provides the expected results

Both 1) and 2) are types of acceptance testing and similar in some sense. 3) is A/B testing and has a completely different objective.

What is A/B test

An A/B test consist of three parts

A: This is what the current state of the product is

B: This will be the state of the product after the release of the update

Test: Measuring the impact of the change. This also forms your hypothesis

Why do you need A/B testing

Companies spend a lot of time and resources building new capabilities and making big changes to their product. Change is great but change whose impact cannot be measured is meaningless. There are reasons why sudden massive changes are not good:

1.Big changes require a lot of resources: You have a finite number of resources. Spending them on one big update can lead to other areas of the product getting neglected.
2.Large updates are riskier: When you make a lot of changes to your code the risk of something not working increases.
3.Big change takes time: Customers today both B2B and B2C expect companies to evolve everyday. If you do not improve your user experience on a regular cadence you run the risk of becoming obsolete. Larger updates require more testing and is an additional drain on time and resources
4.Hard to measure: When you release a big update it’s hard to figure out the contribution of each part of the change on the total impact. Large updates have more than one moving part and therefore become hard to measure.

That’s where A/B tests come in. They empower product managers to accurately measure impact through small incremental changes.

Examples of A/B tests 

A/B testing is useful for making data driven decisions. Some areas in which A/B testing can help you make informed choices are

1.Call to Action Buttons: See which CTA design, text and location converts more users for your webpages.
2.Site Banners: Test different types of banners for promotional offers.
3.Site Layout/Cards: Compare how changing the layout of your site elements impacts conversion
4.User Journey: Make changes to flow of different actions that users take on your site to reduce the drop off rates
5.Email Campaigns:  Test the language, design and timing of your promotional emails to increase open and click through rates
6.Landing Pages: Find out which landing page performs better to convert site visitors to leads.

Key parts of an A/B test

Goal & Motivation

What is your driver behind conducting an A/B test? Your primary focus could be improving user experience or driving revenue. Making it clear in the A/B test helps align the team toward the goal.


You cannot run an A/B test without forming and stating your hypothesis. Your hypothesis should be short, crisp and to the point. It should not run into a couple of pages. You can use the following format for writing an A/B test hypothesis:

“By making “X” change, we expect “Y” result”

Test groups

An A/B test has two groups:

1.Control Group: When an A/B test begins users in this group do not see the new change. They continue to use the product in its original avatar.
2.Variant Group: Users in this group see the product in its new updated version.


Users should be put into control and variant groups at random without any specific rule. This enables the experiment to run without bias being induced unknowingly that can skew the results.

Traffic split

As part of the experiment you need to specify how the traffic needs to be split between control and variant.


A/B experiments need to have a start and end date. Keep seasonality in mind when you decide to run your experiment.


In most cases you will have a control group and variant. But at times you might want to test more than one variant. In such cases you can do an A/B/C test where B and C are two variants to find out the winner!

Platforms and Devices

Users behave differently across platforms and devices. You will have to run different tests for each type to prove your hypothesis. What works for Desktop might not work for Mobile .For example what might work for Android users might not work for iOS user. You cannot extrapolate the results of one platform to say that it will work for the other.

Statistical Significance

Survey Monkey explains this really well. According to them

In the context of AB testing experiments, statistical significance is how likely it is that the difference between your experiment’s control version and test version isn’t due to error or random chance.

You can use this online tool from survey monkey to calculate statistical significance.

Metrics Tracked

It is important to not confuse this with the impact. These are metrics you track for your product on a regular basis. Your A/B experiment should not hurt other important metrics at the cost of increasing the impact of the experiment.

How to setup A/B tests

Write the test proposal

The first step is to write up your test proposal which succinctly describes each part mentioned in the above section.

Ask yourself the following:

1.What do you want to improve on your site?
2.How would you quantify the improvement?
3.Is the change worth the effort?

Download our A/B test template to get started

Get Stakeholder Buy-In

You have to get everyone on the same page that running the A/B test is important. Setup time with all the key stakeholders to go over your test proposal. Listen to their feedback and update the test where it makes sense.

Track Progress

You should check how your test is performing atleast once a week with the key stakeholders. At the end of the test make sure you communicate the final results and next steps. Sometimes you have so many tests running at once that it’s easy to lose track of changes.

Find the right Application

Now that you know everything about A/B testing the big question is  how do you go about setting one up for your product. Don’t worry you don’t have to build from scratch. There are some great tools out there which you can use for the A/B test. If this is the first time your company is running A/B test I have listed a few of them

Depending of your specific use case one tool might work better for you than the other.

Challenges with A/B testing

While A/B testing is a great framework in our arsenal(go gooners!! ⚽️ ) it come with their own set of challenges. Lets look at some of them:

1.Traffic Volume: Statistically significant results require a good amount of traffic on your site and users. If you don’t have the volumes then you will not be able to use A/B testing for measuring impact.
2.Test Duration: If you have lower volumes of users in your test group then you will have to run tests for a longer time. This means you will have to wait longer to find out if all your users can benefit from the change. Also, if the test runs too long it runs the risk of new variables like seasonality of business polluting the results.
3.Number of tests: If you run more than a few tests at one point of time you will run out of traffic pretty quickly. In such cases your update might be ready but you be unable to release it. As a result your users will have to wait longer to benefit from the change.


So after reading all this one question comes to mind. Does it make sense for you to do A/B testing?

It depends!!


I know that’s not the answer you were looking for. Does your organization faces any of the challenges above? At the end of the day you have to weigh the pros and cons. For some people doing A/B testing might not be practically possible. Often teams run into the bad habit of testing without purpose. Try to keep it in check. Create a centralised test approval and launch team. Yes I said centralized😇 ! In this scenario it actually does help. It provides a unified view of all the a/b tests running in the company and new tests that different teams want to run.

You may also like