Subscribe
Join 40,000+ sales and marketing pros who receive our 4x a week insights, tips, and best practices.
Thank you! You have been subscribed.

Free Guide: The Ultimate Inbound Marketing Strategy Playbook 2022

Download the Playbook
Fill out the form to get your playbook
Thank you! You have been subscribed.
... Inbound Marketing Marketing Strategy
Close
The Ultimate Inbound Marketing Strategy Playbook 2022

Free Guide:

The Ultimate Inbound Marketing Strategy Playbook 2022
Read the Playbook
The Ultimate Inbound Marketing Strategy Playbook 2022
Free Inbound Marketing Playbook
View The Ultimate Inbound Marketing Strategy Playbook 2022
The Ultimate Inbound Marketing Strategy Playbook 2022

Free Guide:

Take your inbound strategy to the next level

  • Master the 7 principles of highly effective inbound marketing
  • Dramatically improve your inbound sales
  • Get more buy-in at your company

How Long Should I Run An A/B Test For?

By Carly Stec

Carly Stec also recommends this free guide: The Ultimate Inbound Marketing Strategy Playbook 2022.

How Long Should I Run An A/B Test For?

How_Long_Should_I_Run_an_AB_testI'd like to think that I have a great deal of patience. 

(I mean, I taught my mom how to use an iPhone, so that's got to count for something, right?)

However, when it comes to A/B testing, it seems as though my willingness to wait goes out the window. 

Eager to report on my findings, I find myself refreshing the page insistently. But truth be told, the key to effective A/B testing is to give the test the time it needs to run it's course. 

Free Guide: The Ultimate Inbound Marketing Strategy Playbook 2022

In other words, prematurely putting a cap on it could cost you the validity of the experiment. And conversely, waiting too long to report on the results could sway the data just the same.

So when should you conclude an A/B test?

We've dug into our own experiences (and the advice of others) to help answer this question for once and for all. 

The dangers of concluding too early

Before we get into real numbers, it's important to define what's at stake. 

Often times marketers will begin to spot what they think is a trend in the data after just a couple of days and close the test. 

Let's make one thing clear, "just a couple of days" is rarely enough time to draw any significant conclusions about which variation performed best. 

Test results can change very drastically, very quickly. 

To help paint this picture, check out this example from ConversionXL. The following details the results of an A/B test just two days after launch:

How_Long_Should_I_Run_An_AB_Test_For_

As you can tell, the control was crushing the variation. There was a 0% chance that it would outperform the control. 

Here's a look at the results after 10 days:

How_Long_Should_I_Run_An_AB_Test_For_

Looks like that 0% chance turned into a 95% chance really fast. 

Point being, had they concluded the test early on, the results would have been totally invalid. 

How to make the right call

While it's tough to definitively say, "You should run your test for X days", there are a few ways to come about a sound ending point for your test. 

In fact, The Definitive Guide to Conversion Optimization by Neil Patel and Joseph Putnam defines the following guidelines for determining when to end a test:

  • You should run a test for at least 7 days.
  • There should be a 95% (or higher) likelihood of finding a winning version. 
  • You should wait until there are at least 100 conversions. 

However, as we mentioned, there's no "one size fits all" approach when it comes to defining a concluding point. 

In a recent "unwebinar" with our friends at Unbounce, our marketing director, John Bonini revealed that we typically run our tests for anywhere between 30-90 days, depending on the amount of traffic that we're driving to the variants. For example, if you're driving millions of people to your pages, it's likely that the time between launch and conclusion will be far shorter than a website that is only bringing in a couple thousand views a day. 

While there are "statistical significance" calculators out there to help you determine whether or not it's time to call it quits, we urge you to proceed with caution. 

I, too, was excited about the functionality of these tools upon discovery, however, after reading into a bit I found that sources were reporting that these tools often call tests too early, putting you at risk of disrupting the legitimacy of the results. 

The Ultimate Inbound Marketing Strategy Playbook 2022

Free Guide:

The Ultimate Inbound Marketing Strategy Playbook 2022
Read the Playbook
The Ultimate Inbound Marketing Strategy Playbook 2022

Free Guide:

The Ultimate Inbound Marketing Strategy Playbook 2022

Take your inbound strategy to the next level

  • Master the 7 principles of highly effective inbound marketing
  • Dramatically improve your inbound sales
  • Get more buy-in at your company

Topics:

Marketing Strategy
Published on January 30, 2015

Recent Articles

4 Ways To Recession-proof Your Website In 2022
June 9, 2022 • 5 min read
Demand Generation vs. Lead Generation: What’s the Difference?
June 6, 2022 • 5 min read
What Small Business Leaders Need To Know to Thrive During a Recession
May 23, 2022 • 7 min read
Blog Editorial Calendar for 2022: Templates, Examples, and Tips
May 21, 2022 • 6 min read
Green Energy Inbound Marketing Strategy: The DIY Approach to Getting More Customers
May 7, 2022 • 9 min read
HubSpot Pricing: Your Guide to Everything HubSpot Costs for 2022
April 29, 2022 • 13 min read
33 Most Important IMPACT+ Resources To Train Your Marketing Team in 2022
April 26, 2022 • 13 min read
SaaS Inbound Marketing: How to Get Started With Your Strategy (+ Examples)
April 16, 2022 • 9 min read
5 Revenue Metrics You Should Be Measuring
March 29, 2022 • 6 min read
What Short-Term Wins Can We Expect With They Ask, You Answer?
March 15, 2022 • 6 min read
19 Business Blog Topics Your Audience Wants You To Write About
March 14, 2022 • 12 min read
10 Marketing KPIs You Should Be Tracking
March 3, 2022 • 9 min read
Why Does They Ask, You Answer Actually Get Results?
February 24, 2022 • 5 min read
Inbound Marketing for Financial Services Providers and Financial Advisors (+ examples)
February 19, 2022 • 10 min read
Sales vs Marketing in 2022: What’s The Difference?
February 7, 2022 • 7 min read
How To Set Marketing Goals Based on Business Goals
February 1, 2022 • 6 min read
Video Marketing: What Your 2022 Business Video Strategy Must Include
January 17, 2022 • 10 min read
8 Crucial Elements Every Homepage Design Should Have [+Video]
January 15, 2022 • 6 min read
5 Organizational ‘Cleanses’ To Start Your Year Off On the Right Foot
January 1, 2022 • 6 min read
What Should a 2022 Content Marketing Strategy Include?
December 30, 2021 • 9 min read
Brutally Honest Pros and Cons of They Ask, You Answer for Digital Marketing
December 7, 2021 • 10 min read
The Step-by-Step, 12-Month, DIY They Ask, You Answer Implementation Plan
December 2, 2021 • 16 min read
‘The Big 5’ Best Topics For Your Business Blog That Drive Traffic, Leads, and Sales (+ Video)
November 26, 2021 • 9 min read
Why Trust Is the True Currency for Any Business
November 16, 2021 • 5 min read
How To Get Started With They Ask, You Answer: Your First 6 Months
November 11, 2021 • 11 min read