Learn
Learn
Close

They Ask, You Answer

A revolutionary approach to inbound sales, content marketing, and today’s digital consumer.

Learn MoreLearn More

Free Courses in IMPACT+

Dozens of sales, marketing, and content courses inside IMPACT+. Start learning now.

See all coursesSee all courses
Events
Events
Close
IMPACT+ Membership
IMPACT+ Membership
Close
Coaching & Implementation Services
Coaching & Implementation Services
Close

Services Overview

See how you can dramatically increase your inbound leads and sales.

Services Overview

Digital Sales and Marketing Mastery

Fast track your team’s success with IMPACT's most popular service.

Digital Sales and Marketing Mastery

Web Design

Launch a beautiful website that consistently generates leads and revenue.

Web Design

Virtual Sales Training

Equip your sales team with comprehensive training designed to help them close more deals in today's virtual-first world.

Virtual Sales Training

HubSpot Training & Implementation

Train your company to take ownership of HubSpot and get the most out of your investment.

HubSpot Training & Implementation

Case Studies

See dozens of examples of companies succeeding with Digital Sales and Marketing Mastery and They Ask, You Answer.

Case Studies
Become a Coach
Become a Coach
Close

A Breakdown of HubSpot’s Outage Retrospective for the Non-Technical User

A Breakdown of HubSpot’s Outage Retrospective for the Non-Technical User Blog Feature

Carina Duffy

HubSpot Trainer, Co-Host of the Hubcast, IMPACT+ Instructor, INBOUND Speaker, Retains 12 HubSpot Certifications

April 9th, 2019 min read

If you’re a HubSpot user, unless you were on vacation from March 29 until now (if you were, I’m jealous!), you recently experienced one of the most catastrophic outages HubSpot has ever had.

Honestly, it was a mess. Most of the time during product outages you’ll have one or two tools that aren’t working or are having bugs, and it’s resolved within minutes or hours.

This time it was just about everything: emails, form submissions, workflows, lists, sales tools, CRM, imports, analytics. It’s hard to find any pieces of the tool that WEREN’T affected during that time.

What’s even more troubling is that this outage took about 36 hours to be mostly resolved, but the processing of the backlog of data from those 36 hours is still going on at the time this article was released.

Thankfully, during all of this, HubSpot’s crisis communication was solid. Updates on status.hubspot.com were regular and timely (although are they ever frequent enough??) given the scale of the situation.

JD Sherman (HubSpot’s COO) released an article on March 29 with an apology and an outline of next steps for the team -- namely, doing an in-depth retrospective on the cause of the issue and how they’ll make sure it won’t happen again.

That retrospective was delivered on April 4. You can read the full article here. There’s a lot of detail in there about how their systems are structured and what exactly happened. If you’re not into all of the “geek-speak,” we’ve got you covered.

A Quick Review of How HubSpot’s Infrastructure Works

HubSpot uses a combination of software systems -- Kafka and ZooKeeper -- that allow all of the HubSpot tools to talk to each other and all of the data to be processed effectively.

Both these software systems have redundancies and safeguards built into them so that if some servers crash, other servers can pick up the slack, and end users don’t experience any issues.

So What Broke?

It’s a bit difficult to explain without getting super technical, but think about it like a series of unfortunate events.

High strain was put onto ZooKeeper, causing parts of it to crash. Typically, ZooKeeper recovers quickly, but in this case, it took several minutes. The delay in recovery then broke the communication between ZooKeeper and Kafka, causing Kafka to crash.

Even though the team was able to restore ZooKeeper, the damage was done in Kafka and it wasn’t able to recover. What made things worse was a second outage in ZooKeeper accompanied by trying to restart Kafka, which started to cause data corruption.

Why Did It Take So Long to Fix?

Corrupted data? That sounds bad. And well, it is. This is actually why some things took so long to come back online.

When the HubSpot team realized that the server recovery was starting to corrupt data, they had a decision to make: either focus on recovering data (and safeguarding against corrupted data) or focus on restoring the tools.

They decided to focus on recovering data to ensure that there would be no gaps in historical data for customers (which in the long run they believe to be the right decision, and I’d personally agree!). This is the reason that the affected tools took almost 36 hours to be restored.

So, in the name of protecting customer data, HubSpot manually recovered a whoooole bunch of our data, and then was able to restore the affected tools.

This is also why you’re still seeing (at the time this article was published) the “continuing to process data from March 28 & 29” status message from HubSpot.

What Now?

Now that we know exactly what happened, HubSpot’s got a plan to make sure this never happens again. An interesting note in all of this is that HubSpot’s own teams use many of their tools across different parts of the business, so this not only affected their customers but their own business (even more motivation to make sure it never happens again!).

They’re making changes in a few different areas to protect against another outage: technical/infrastructure, reliability, testing, and communication.

Technical / Infrastructure

As is to be expected, HubSpot will be doing some restructuring of their server clusters to make sure it’s not even possible to have an outage this large again. By doing this, any outage that does happen should be restricted to a small piece of the platform, and the recovery time for issues should be significantly quicker.

Reliability

HubSpot does have a team of people who test and upgrade their systems, but it hasn’t been as high of a priority as it should be. Now, they’ll have a dedicated team of people who will “oversee new standards, frequencies, and resources to ensure that we're consistently evaluating our key infrastructure systems for code fixes and critical patches without gaps.”

Testing

Along with investing much more heavily into the reliability of their platform, HubSpot is also increasing the level of frequency and depth to which they’re testing their systems. Again, it’s not that these processes didn’t exist before, but this outage uncovered some gaps in the frequency in which they test for massive failures, as well as how comprehensively they test these systems.

Communication

Lastly, HubSpot is committing to making their communication during any major incident more frequent and helpful, specifically in the minutes and hours immediately following an issue.

Their status updates will now include more detailed explanations of what is going on, as well as when the next update can be expected.

In Conclusion

No one here is pretending that this outage wasn’t bad. Not even HubSpot. But one of the things I appreciate the most about HubSpot as an organization is their transparency and willingness to admit when they’ve messed up.

They know the impact this had on their customers, and on their own business, and they’re actively seeking to make sure it never happens again.

So, even if you’re a little rattled by this outage, know that improvements are being made, fixes are being implemented, and HubSpot will continue to make their product the best it can be. Okay -- HubSpot lovefest over!

Want to learn more about digital sales and marketing?

Master digital sales and marketing when you join IMPACT+ for FREE. Gain instant access to exclusive courses and keynotes taught by Marcus Sheridan, Brian Halligan, Liz Moorehead, Ann Handley, David Cancel, Carina Duffy, Zach Basner, and more.

Sign up for free
Take a peek

Access hundreds of FREE courses and keynotes

Free Course
Digital Sales & Marketing Framework for Today’s Buyers
By: Marcus Sheridan
View Course Preview View Course Preview
Free Keynote
A New Way to Compete & Win
By: Brian Halligan
View Course Preview View Course Preview
Free Virtual Keynote
How To Genuinely Feel Confident, Be Likable, And Build Trust On Camera
By: Liz Moorehead
View Course Preview View Course Preview
Free Keynote
The Future of Marketing is Conversational​
By: David Cancel
View Course Preview View Course Preview
Free Deep Dive
HubSpot On a Budget: Getting Started with HubSpot Starter Hubs
By: Courtney Caldwell
View Course Preview View Course Preview
Free Course
Assignment Selling: Content is Your Greatest Sales Tool
By: Marcus Sheridan
View Course Preview View Course Preview
Free Keynote
The Big, Bold, Brave New World of Marketing​
By: Ann Handley
View Course Preview View Course Preview
Free Course
Investing in Incredible Digital Sales and Marketing Results
By: Chris Duprey
View Course Preview View Course Preview
Free Course
How To Show Your Content Makes Money With HubSpot Marketing and Sales Hubs
By: Liz Moorehead
View Course Preview View Course Preview
Free Virtual Keynote
The Visual Sale
By: Marcus Sheridan & Tyler Lessard
View Course Preview View Course Preview
Free Keynote
Marketing Tech: What’s Now, What’s Near and What’s Not Anytime Soon?​
By: Dharmesh Shah
View Course Preview View Course Preview
Free Course
Inbound Lead Generation & Conversion Optimization
By: Carina Duffy
View Course Preview View Course Preview
Free Virtual Keynote
Fundamentals of Videography: You Can Be a Great Videographer Too
By: Zach Basner
View Course Preview View Course Preview
Free Deep Dive
The Ultimate Customer Service Toolkit for Any Budget
By: Andriti Gulati
View Course Preview View Course Preview
Free Course
6 Topics Subject Matter Experts Need to Address
By: Kevin Phillips
View Course Preview View Course Preview
Free Keynote
Fanocracy: The Power of Making Business Personal​
By: David Meerman Scott
View Course Preview View Course Preview
Free Keynote
The Power of Pillar Content
By: Liz Moorehead
View Course Preview View Course Preview
Free Keynote
SEO for the Future
By: Franco Valentino
View Course Preview View Course Preview
Free Course
Fundamentals of Social Media Marketing
By: Stephanie Baiocchi
View Course Preview View Course Preview
Free Deep Dive
A Practical Guide To Video Graphics
By: Colton Trcic
View Course Preview View Course Preview
Free Session
How To Get Your Community To Know, Like, And Trust You With Video
By: Marki Lemons Rhyal
View Course Preview View Course Preview
Free Course
Video Sales and Marketing Strategy
By: Zach Basner
View Course Preview View Course Preview
Here Are Some Related Articles You May Find Interesting

Want to Contribute Content to impactplus.com? Click Here.

IMPACT+ Sign Up
A FREE online learning community with on-demand courses, hundreds of expert-led sessions, thousands of your peers ready to support you, and much more.
Check it out
Hundreds of courses and recorded keynotes, completely FREE
Check it out
Access hundreds of digital sales and marketing courses and recorded keynotes, completely FREE