Join the IMPACT coaches for a deep dive on a new topic every month in our free virtual event series.

Register Here
Close

Join the IMPACT coaches for a deep dive on a new topic every month in our free virtual event series.
Register Here
The Ultimate Inbound Marketing Strategy Playbook 2022

Take your inbound strategy to the next level

  • Master the 7 principles of highly effective inbound marketing
  • Dramatically improve your inbound sales
  • Get more buy-in at your company

How Long Should I Run An A/B Test For?

By Carly Stec

How Long Should I Run An A/B Test For?

How_Long_Should_I_Run_an_AB_testI'd like to think that I have a great deal of patience. 

(I mean, I taught my mom how to use an iPhone, so that's got to count for something, right?)

However, when it comes to A/B testing, it seems as though my willingness to wait goes out the window. 

Eager to report on my findings, I find myself refreshing the page insistently. But truth be told, the key to effective A/B testing is to give the test the time it needs to run it's course. 

 Join the IMPACT coaches for a deep dive on a new topic every month in our free virtual event series.

In other words, prematurely putting a cap on it could cost you the validity of the experiment. And conversely, waiting too long to report on the results could sway the data just the same.

So when should you conclude an A/B test?

We've dug into our own experiences (and the advice of others) to help answer this question for once and for all. 

The dangers of concluding too early

Before we get into real numbers, it's important to define what's at stake. 

Often times marketers will begin to spot what they think is a trend in the data after just a couple of days and close the test. 

Let's make one thing clear, "just a couple of days" is rarely enough time to draw any significant conclusions about which variation performed best. 

Test results can change very drastically, very quickly. 

To help paint this picture, check out this example from ConversionXL. The following details the results of an A/B test just two days after launch:

How_Long_Should_I_Run_An_AB_Test_For_

As you can tell, the control was crushing the variation. There was a 0% chance that it would outperform the control. 

Here's a look at the results after 10 days:

How_Long_Should_I_Run_An_AB_Test_For_

Looks like that 0% chance turned into a 95% chance really fast. 

Point being, had they concluded the test early on, the results would have been totally invalid. 

How to make the right call

While it's tough to definitively say, "You should run your test for X days", there are a few ways to come about a sound ending point for your test. 

In fact, The Definitive Guide to Conversion Optimization by Neil Patel and Joseph Putnam defines the following guidelines for determining when to end a test:

  • You should run a test for at least 7 days.
  • There should be a 95% (or higher) likelihood of finding a winning version. 
  • You should wait until there are at least 100 conversions. 

However, as we mentioned, there's no "one size fits all" approach when it comes to defining a concluding point. 

In a recent "unwebinar" with our friends at Unbounce, our marketing director, John Bonini revealed that we typically run our tests for anywhere between 30-90 days, depending on the amount of traffic that we're driving to the variants. For example, if you're driving millions of people to your pages, it's likely that the time between launch and conclusion will be far shorter than a website that is only bringing in a couple thousand views a day. 

While there are "statistical significance" calculators out there to help you determine whether or not it's time to call it quits, we urge you to proceed with caution. 

I, too, was excited about the functionality of these tools upon discovery, however, after reading into a bit I found that sources were reporting that these tools often call tests too early, putting you at risk of disrupting the legitimacy of the results. 

Join the IMPACT coaches for a deep dive on a new topic every month in our free virtual event series.

Topics:

Marketing Strategy
Published on January 30, 2015

Recent Articles

24 Essential Questions for Understanding Your Ideal Customer (+ Infographic)
December 1, 2022 • 5 min read
Take It From an Expert: You Probably Don’t Need a New Website
November 16, 2022 • 4 min read
Want to Win More and Improve Company Culture? Do More Role-playing
November 9, 2022 • 4 min read
Why Trust Is the True Currency for All Business
September 5, 2022 • 7 min read
The Big 5: Best Business Blog Topics to Drive Traffic, Leads, and Sales
August 29, 2022 • 10 min read
What is a Content Strategy? (Definition + Templates)
August 25, 2022 • 12 min read
10 Marketing Objective Examples To Guide and Focus Your Strategy
August 18, 2022 • 6 min read
What Marketers Need To Know About Switching to GA4 [Google Analytics 4]
July 30, 2022 • 5 min read
Inbound Marketing Benefits Explained (Updated for 2022)
July 28, 2022 • 6 min read
How To Inspire Your Marketing Team To Try New Ideas
July 25, 2022 • 5 min read
Do I Need To Invest in Marketing When My Business is Thriving? (+ Video)
July 21, 2022 • 7 min read
How Much Does It Cost To Become a World-class They Ask, You Answer Case Study?
July 16, 2022 • 12 min read
4 Ways To Recession-proof Your Website In 2022
June 9, 2022 • 5 min read
Demand Generation vs. Lead Generation: What’s the Difference?
June 6, 2022 • 5 min read
What Small Business Leaders Need To Know to Thrive During a Recession
May 23, 2022 • 7 min read
Blog Editorial Calendar for 2022: Templates, Examples, and Tips
May 21, 2022 • 6 min read
Green Energy Inbound Marketing Strategy: The DIY Approach to Getting More Customers
May 7, 2022 • 9 min read
HubSpot Pricing: Your Guide to Everything HubSpot Costs for 2022
April 29, 2022 • 13 min read
33 Most Important IMPACT+ Resources To Train Your Marketing Team in 2022
April 26, 2022 • 13 min read
SaaS Inbound Marketing: How to Get Started With Your Strategy (+ Examples)
April 16, 2022 • 9 min read
5 Revenue Metrics You Should Be Measuring
March 29, 2022 • 6 min read
What Short-Term Wins Can We Expect With They Ask, You Answer?
March 15, 2022 • 6 min read
19 Business Blog Topics Your Audience Wants You To Write About
March 14, 2022 • 12 min read
10 Marketing KPIs You Should Be Tracking
March 3, 2022 • 9 min read
Why Does They Ask, You Answer Actually Get Results?
February 24, 2022 • 5 min read