Whether you’re a seasoned entrepreneur or just starting out, you’ve likely seen countless articles and resources about A/B testing. You may even already be A/B testing the subject lines of your emails or social media posts. Despite the amount of talk about A/B testing in the marketing space, many people still get it wrong. The result? People making important business decisions based on inaccurate results from poor testing. A/B testing is often oversimplified, especially in materials qatar b2b leads
written for store owners. Below, you’ll find everything you need to know to get started with the different types of A/B testing for eCommerce explained in the simplest terms possible. A/B testing can be the difference between choosing the right product positioning, increasing landing page conversions , and more.
Contents hide
1 What is A/B testing?
2 How A/B testing works
3 What is A/B/n testing?
4 How long should A/B tests run?
5 Why Should You A/B Test?
5.1 Should you A/B test?
6 What should you A/B test?
7 Prioritizing A/B Testing Ideas
8 A Crash Course in A/B Testing Statistics
8.1 What does “average” mean?
8.2 What is sampling?
8.3 What is dispersion?
8.4 What is statistical significance?
8.5 What is regression to the mean?
8.6 What is statistical power?
8.7 What are threats to external validity?
9 How to Set Up A/B Testing
9.1 Choosing a tool for A/B testing
10 How to Analyze A/B Testing Results
11 How to Archive Past A/B Tests
12 A/B Testing Processes of Professionals
12.1 Krista Seiden
12.2 Alex Birkett, Omniscient Digital
12.3 Ton Wesseling, Online Dialogue
12.4 Julia Starostenko, Pinterest
12.5 Pip Laja, CXL
13 Optimizing A/B Testing for Your Business
14 A/B Testing FAQs
14.1 Related publications:
What is A/B testing?
A/B testing, sometimes called split testing, is the process of comparing two versions of the same web page , email , or other digital asset to determine which performs better based on user behavior. It’s a useful tool for improving the effectiveness of a marketing campaign and better understanding what engages your target audience . The process answers important business questions, helps you generate more revenue from the traffic you already have , and lays the foundation for a data-driven marketing strategy .
How A/B Testing Works
When using A/B testing in a marketing context, you show 50% of your visitors version A of your asset (let’s call it the “control”) and 50% of your visitors version B (let’s call it the “variant”). The variation that produces the highest conversion rate wins . For example, let’s say that a variation (version B) produced the highest conversion rate . You would then declare it the winner and transfer 100% of your visitors to that variation. That variation then becomes the new control, and you would develop a new variation. It’s worth noting that A/B test conversion rates can often be an imperfect measure of success. For example, if you price an item at $50 on one page and have it completely free on another page, that won’t provide any truly valuable information. Like any tool or strategy you use for your business, they should be strategic. That’s why you should track conversion value all the way to the final sale.
What is A/B/n testing?
With A/B/n testing, you can test more than one variation against the control. So, instead of showing 50% of your visitors the control variation and 50% of your visitors the variation, you might show 25% of your visitors the control variation, 25% the first variation, 25% the second variation, and 25% the third variation. Note: This is different from multivariate testing, which also involves multiple variations. With multivariate testing, you’re not just testing multiple variations, but multiple elements, like A/B testing UX or split testing SEO . The goal is to figure out which combination works best.
You will need a lot of traffic to run multivariate tests , so you can ignore them for now.
]
How long should A/B tests run?
Run your A/B test for at least one, and ideally two, full business cycles. Don’t stop testing just because you’ve reached significance. You also need to stick to your pre-determined sample size. Finally, be sure to run all tests in full-week increments. Why two full business cycles? To begin:
You can account for buyers who say "I need to think about it."
You can take into account all the different traffic sources ( Facebook , email , organic search , etc.).
You can account for anomalies. For example, your Friday email newsletter.
Two business cycles are usually enough to gain valuable insight into the user behavior of your target audience . If you’ve used any tool to test landing pages with A/B tests, you’re probably familiar with the little green “Statistically significant” icon. Unfortunately, for many, this is the universal “test done, cancel it” sign. As you’ll learn below, just because an A/B test has reached statistical significance doesn’t mean you should stop testing. And your predetermined sample size? It’s not as scary as it sounds. Open up a sample size calculator like this one from Evan Miller and use it on all your web pages to improve your conversion rates .
