Embedding NLP AI in A/B Testing

Overview:

The key to successful learning lies in understanding the core principles first, as they can be applied to any subject. As an example, I have a strong grasp on the product management life cycle and am now employing that same methodology in writing my debut book. Similarly, in this piece, I will break down the complex concept of A/B testing using a similar approach, allowing for easier comprehension and retention of information.

What is A/B testing?

A/B simply is made up of two words which to a lay man: A/B means two things, or two versions and testing means an experiment.

A/B testing, also known as split testing, is a powerful marketing tool that allows you to conduct a controlled experiment with your audience. By splitting your audience into two groups and showing them different versions of your campaign, you can gather valuable data and insights on which version performs better. It's like conducting a science experiment, but instead of chemicals and beakers, you're using marketing materials.

Through A/B testing, you can see how small changes in your content or design can have a big impact on your audience's behaviour. This is achieved by randomly showing one half of your audience version A and the other half version B. The variations could be anything from different headlines to completely different layouts.

The main goal of A/B testing is to compare and contrast the performance of two or more variants to determine the best option. This is done through statistical analysis, which helps to identify which variation is more effective in achieving a desired effect or goal.

By employing A/B testing, you can make decisions based on hard data and fine-tune your campaigns for optimal impact. It's like having a constantly available virtual focus group, guiding you towards the most effective marketing strategies. This powerful tool allows you to continuously optimize and improve your approach, leading to even better results. With A/B testing, your marketing decisions are backed by concrete data rather than guesses or assumptions, giving you unparalleled insights into what truly works for your audience.

What is the A/B testing process?

The following is an A/B testing process/framework you can use to start running tests:

Collect Data: Utilise analytics tools or conduct surveys and interviews with users to gain understanding of the issue. Identify processes with high drop-off rates for further examination.
Identify goals: To measure success, a conversion metric is chosen based on the business objective. These metrics, also known as conversion goals, determine the effectiveness of the variation compared to the original version. For instance, the number of visitors who register for a free trial can be used as a conversion goal.
Generate & Prioritise Hypothesis: By closely observing user behaviour and brainstorming potential hypotheses, you can then start developing ideas for A/B testing. These testable hypotheses should aim to improve upon the current version, and you should have a clear goal in mind before beginning the testing phase.
Create Variations: Choose which feature you want to modify and determine the specific change, such as altering the colour, labels, form, or size of the call-to-action button, layout, etc.
Run Experiments: Randomly divide users into a test group and a control group and gather necessary data for analysis. Begin the experiment and await participants to engage.
Experiment Wait Time: Be patient for the test results: The amount of time it takes to get a satisfactory result depends on the size of your sample (the intended audience).
Analyse Results: At the end of the experiment, we will analyse the % result to know which performs better.
Experiment Results: If we want to further our experimentation and explore different possibilities, we can simply repeat the process described above based on our current findings.

Genty Sneakers Use Case in Real Life

Genty Sneakers, a European-based sneaker company, has experienced a decrease in sales for one of their best-selling shoe models.

Sneakers 1 - The Control or Master Sneakers

Top Selling Sneakers

In the real world, we can examine the A/B Testing process/framework using this specific use case.

The company is facing a drop in sales, and they are unsure of the cause. Rumours have started to circulate that customers are choosing to purchase their competitors' similar products because they offer more colour options. While there is no solid data from an analytics platform to support this claim, the company has gathered feedback from 200 individuals suggesting this may be true. Additionally, there are rumours that customers want sneakers without a sole and electric sneaker, but again, there is no concrete user feedback data to back up these claims. This Collect Data Phase highlights the need for further research into possible reasons for the decline in sales, such as limited colour variety.

Genty Sneakers had one clear goal in mind: to boost sales of their best-selling sneakers. Every team member was focused on this objective, pouring all their energy and creativity into devising strategies to achieve it. The stakes were high, but the potential rewards were worth it - increased revenue, brand recognition, and satisfied customers. The air buzzed with excitement and determination as they pushed forward towards their shared goal. Sales targets may be daunting, but Genty Ventures was ready to rise to the challenge with confidence and drive.

After carefully analysing data from 200 users and receiving feedback from 3 individuals, the team at Genty Sneakers has made a decision. Based on this information, they have decided to prioritise the colour request over other potential changes. Through their extensive research and analysis, they have been able to generate and prioritise an hypothesis for further development and implementation.

Effort/Impact Table to prioritise Hypothesis.

The proposed theory/hypothesis is that the decline in sales for their top-selling product is due to a limited range of colours, and increasing the variety of colours available will lead to higher demand and sales.

After careful consideration, they decided to conduct another experiment by creating a different version of the sneakers. This time, they focused on changing the colour as the variable. In other words, they created multiple variations of the same sneaker with different colours to test their effectiveness.

The same product with two different design variations

As the grand opening of our shoe store approached, we decided to host an open day for 5000 lucky customers. The buzz and excitement surrounding the event could be felt in the air. Upon arrival, 5000 people were greeted with a stunning display of our two new sneakers. As they eagerly browsed and tried on pairs, 3000 customers ultimately chose to purchase Sneakers 1 (70%), while 1000 opted for Sneakers 2 (20%). The launch was a huge success, with customers only having to wait a few hours for their desired pair.

And while it may not definitively tell us that one sneaker is better than the other, it does indicate that a majority of the 5000 buyers prefer Sneakers 1. Interestingly, there were still some who did not make a purchase (10%), but it's clear that Sneakers 1 has emerged as the clear winner.

With this data and customer feedback, we are confident in investing in more designs of Sneakers 1 as it has the potential to generate more revenue for our store. This validation of our hypothesis is a promising start to our new venture.

Conducting an A/B test, in which a variation is directly compared to the current experience, allows for targeted inquiries into potential changes on your website or app. The resulting data provides valuable insights into the impact of those changes.

A/B testing eliminates guesswork and empowers data-driven decisions, shifting conversations about website optimisation from speculation to certainty. By tracking how alterations affect key metrics, you can ensure that each change yields positive results.

Despite individuals' beliefs that they know what's best, biases often lead to erroneous assumptions. However, by implementing scientific experiments like A/B testing, one can truly determine if their idea is superior through unbiased evidence and proof of knowledge.

Before We dive into the Software application, where does Natural Language Processing (AI) come into A/B Testing

Natural language processing (NLP) is a field of computer science, and more specifically, artificial intelligence (AI), that focuses on equipping computers with the capability to understand written and spoken words in a manner similar to humans. NLP can be utilized to generate various versions of our product content; not only limited to colour options but also providing insights for enhancing our workflow. One key aspect where we will employ NLP in this process is content creation.

Deliveroo Homepage

Deliveroo has a landing page content of “Restaurant food, takeaway and groceries. Delivered.”

With the use of Natural Language Processing, ordering will be even simpler for you. See below for examples of how AI can enhance your delivery experience in content creation.

Using NLP for Content Creation

Now, what does A/B Testing Look like in the Software World?

Genesis, a leading recruitment company with over 10,000 monthly visitors, is determined to increase their conversion rate and attract even more job seekers. Despite their success, they are still not seeing enough users clicking the "Join" button to apply for positions within the company. Their ultimate goal is to entice and motivate potential applicants to take action by clicking that crucial button and opening the door to exciting career opportunities.

Career Master or Control Page

The primary page we need to examine is the Control or Master Page. The pages with different versions are referred to as Challenger pages.

Running our Experiment:

Now Let us Follow our A/B Testing process and framework as defined above.

Collect Data: Through Google Analytics, we discovered that our audience was not clicking on the "Join Us" button. This led us to collect data on their behaviour and preferences.

Establishing Objectives: Our primary objective is to increase the number of individuals who click on the "Join Us" button.

Identify goals: We aim to attract a larger audience to click the Join Us button.

Generate & Prioritise Hypothesis:

We believe that by altering the background colour, our conversion rate will increase. In addition, changing the button colour and adjusting our content writing may also lead to a higher conversion rate. Another possible solution is to add three buttons, which could potentially boost our conversion rate.

After carefully evaluating the level of effort and potential impact, it has been determined that our top priority should be focusing on the top three options while deprioritising the last one.

Create Experiment Variations: We choose the Career landing page as the focus of our experiment, along with its owner and current status. Then, we specify the name of the experiment or test, as well as the variations we have made - including changes to the button colour, background colour, and text.

Run Experiments: Our monthly audience/visitors average around 10,000. Before beginning any analysis, we calculate the minimum sample size needed.

Experiment Wait Time: After careful consideration, we have chosen to conduct this experiment for a period of two months. We have selected specific start and end dates and times for the duration of the experiment.

Analyse Results: The data from our experiments is summarised below.

Experiment Results: The winner from our experiments is defined below.

Example of a Single Variant A/B Testing (One Challenger)

Single Variant

Before diving into the creating another variant, let us breakdown how we applied NLP to create a variation below. To do this, we will show how NLP helped us in the content creation process.

Applying Natural Language Processing (AI) to Challenger A below

Here is an instance of how we are utilizing Natural Language Processing, a branch of Artificial Intelligence, to generate unique content.

The Control page featured a landing page with the following text:

Control Page:

Will you like to join us and make global impact?

With NLP (AI): New NLP Modified content.

Would you like to join us and be a part of something bigger? Together, we can make waves that will reach far beyond our borders and have a lasting impact on the world. Will you add your voice and talents to our cause.

Example of a Multi Variant A/B Testing embedding NLP AI (Two or more Challenger)

The Winner is Challenger A after embedding Natural language processing (NLP) into the A/B testing variant creation process.