A/B Testing Mastery: SSM Thoughts Bridge Review (Part 3)

Continuing to the third part of my review on A/B Testing mastery, here’s another part. This article is based on what I have learnt from CXL institute. I am going to cover the planning phase for A/B testing by mentioning the scientific method as well as the interesting 6V model of desk research on successful A/B testing.

The A/B testing mastery course prompts viewers to start with the reason behind the tests and work from there. You cannot conduct a successful A/B test without a specific question in mind or problem to solve. This is where setting up a solid hypothesis comes in. A/B testing can be tricky especially if the proper planning, time, and effort are not invested. Starting with your “why” and work from there will definitely help according to Ton.

Hypothesizing

As mentioned in the intro, the first question to ask ourselves is “why are we running this experiment?” The purpose behind the hypothesis is to get everyone involved with the test on board and in alignment with one another. The hypothesis is composed of three parts:

1.    Problem

2.    Proposed Solution

3.    Predicted Outcome

By creating a clear hypothesis, we will save time by avoiding clarifying questions and discussions around the purpose of the test. The hypothesis may seem simple, but one sentence can hold a lot of pertinent information. 

Don’t create your research plan after the outcome is known… 

In order to set up a concrete hypothesis, use the format:

If ___________, then __________ will happen (among “this group”), because of ___________. 

See the image below for suggestions to fill in the blanks:

Prioritizing

The final phase of the planning pillar is prioritizing your A/B tests. Some well-known prioritizing models mentioned in the course are PIE & ICE.

• PIE = Potential * Importance * Ease
• ICE = Impact * Confidence * Effort

Ton recommended we use his A/B testing success formula:

• Hypothesis * Location

I’ll expand on this more later.

PIE to PIPE

Ton suggested we change the PIE model to PIPE. PIPE still considers potential and ease, but it changes the “I” to “impact” and adding another “P” for “power.” Potential questions the chance of the hypothesis being true. Impact questions which hypothesis is most effective. The main focus of power is finding a significant outcome while ease focuses on the logistics behind testing and implementation.

PIE misleads testers when it comes to prioritizing pages to test. It’s about the customer journey and the right hypothesis, then we pick the page(s) to test on. When prioritizing A/B tests and hypotheses, it is best practice to use a table to do so. Potential ranks the strength of the hypothesis (ex: scale of 1-10) while the impact ranks the page strength (ex: low, medium, high). Power includes multiple metrics like weekly visitors, conversion rates, number of test variations, minimum detectable effects (MDE), estimated runtime, and lift in six months. Lastly, Ease shows an estimated level of effort (ex: low, medium, high). See the table below for a visual example of what this could look like:

PIPE prioritization also considers the set-up of the experiment. See the other example below:

In order to see a higher potential, make sure to focus on the right products/services using the right metrics and the hypotheses that have the largest potential.

The A/B testing success formula 

Ton believes the formula for successful testing is :

Relevant Location x Relevant Hypothesis x Chance on Effect.

Follow the “experiments roadmap.” The customer journey is the first consideration when it comes to experimentation. We should know the typical ins and outs, but strive to learn more through our tests. The next stop on our roadmap is location. What page are they on? How did they get there? Lastly, the hypothesis is our destination by identifying the determinant and testing outcomes.

Scheduling 

When it comes to scheduling our test (or series of tests), consider how much data we have access to. More data should equal more tests on our end. If we are receiving 1,000 conversions a month (or more) we should consider conducting one new experiment every two weeks (26 experiments per year). Keep in mind, this means experiments will overlap.

With more traffic, and multiple testers it is possible to launch one new experiment every week (52 per year). With a full team of dedicated testers and more than 10,000 conversions per month, consider conducting two experiments live each week (104 experiments per year). I realize this sounds overwhelming and seems like a lofty goal, but keep in mind there isn’t just one right way to test or create a testing schedule.

We shouldn’t worry about overlapping experiments. More experiments directly correlates to an increase in the absolute number of real wins (even when we lose, we have the opportunity to win through gaining additional insight). This will ultimately allow us to grow faster. Another thing to be mindful of is the noise impacting the percentage of winners negatively; however, this is typically not as fast as the growth in the absolute number of experiments.

The most important thing to keep in mind when scheduling experiments is the fact that you are testing. Some tests are better than no tests and although setting their frequency is required, it is not the most important factor when it comes to getting results and applying them.

Execution

When it comes to executing A/B tests, we’ll need to know some context behind the following three steps:

1.    Designing an A/B test

2.    Developing an A/B test

3.    Quality Assuring an A/B test

Designing our A/B test

Design starts with choosing just 1 challenger - this functions as the control variable in the experiment. The reason behind this is keeping implementation costs low and results clear when starting out. If we decide to practice multi-variate testing, that is perfectly fine, but it is not recommended on the first go around.

The change behind the challenger should be visible, scannable, and usable. Usability guidelines from big site practices can be helpful in deciding which variable to manipulate. Next, be sure to mirror your design with the hypothesis.

Developing our A/B test

If our test is complex and more technical, we may need to use coding to make it happen. Ton recommends against using the WYSIWYG code editor,

“It’s an experiment, if it works, it works.”

Our code does not need to be perfect the first time. If we can’t make it work over time, we should propose design changes. When using code, adding measurements and additional analytics events can be helpful when tracking the outcomes of the tests. If we decide to use server side coding, we should be sure to note that client-side testing can only manipulate front-end code. This can lead to an increase in time and complexity. 

Sometimes it is necessary to get a JavaScript Developer involved - this type of testing is typically a back-end release. Deployment and back-end developers will increase the time-span before deployment and the level of effort due to the complexity of the test. The course recommends we utilize the back-end and test through the front-end upon launch to ensure a proper user experience. 

Quality Assuring (QA) of A/B testing

More conversions are directly proportional to a higher quality assurance. What does this mean? Low traffic sites may not be worth utilizing A/B tests for, but testing can help spot and troubleshoot any issues before they become a bigger problem. This can often be an afterthought, but it plays a large role in not only the success of our tests, but the success of our website (and additional content) in general.

Quality assurance comes down to testing our website, advertisements, email formats, and any other promotional and/or marketing collateral on various devices and browsers to ensure they are functional. In order to provide QA to our tests we should make sure they apply anywhere and everywhere our target audience(s) might be.

***

Sheikh Shafi Mahmud is an Economics Graduate of Notre Dame, Digital Full Stack Marketer. Author and Founder of SSM Thoughts Bridge. He is also the Co-Founder, Music Composer & Producer of Apeiruss. Can be reached at sheikhshafimahmud@gmail.com