Subscribe
Join 40,000+ sales and marketing pros who receive our 4x a week insights, tips, and best practices.
Thank you! You have been subscribed.

Free Guide: Everything You Need to Know About HubSpot Marketing Hub

Read the Guide Subscribe
Join 40,000+ sales and marketing pros who receive our 4x a week insights, tips, and best practices.
Thank you! You have been subscribed.
Learning Center
Learning Center
Close
The IMPACT Learning Center

Free resources to help you master inbound marketing and They Ask, You Answer

Access the Learning Center

Access the Learning Center

Access the Learning Center
learning_center_grey__What is They Ask, You Answer-v2-black

What is They Ask, You Answer?

What is <span>They Ask, You Answer?</span>
Articles, Podcasts, & Updates

Articles, Podcasts, & Updates

Articles, Podcasts, <span>& Updates</span>
learning_center_grey__Free Courses and Certifications.svg

Free Courses & Certifications

Free Courses & <span>Certifications</span>
On-Demand Keynotes & Sessions

On-Demand Keynotes & Sessions

On-Demand <span>Keynotes & Sessions</span>
Events
Events
Close
IMPACT+ Membership
IMPACT+ Membership
Close
Services
Services
Close
Navigation_8_2021_taya

They Ask, You Answer Coaching & Training

They Ask, You Answer Coaching & Training
They Ask, You Answer Workshop

They Ask, You Answer Workshop

They Ask, You Answer Workshop
Navigation_8_2021_workshop

Inbound Marketing Services

Inbound Marketing Services
Navigation_8_2021_website design - monitor

Website Design & Development

Website Design & Development
Navigation_8_2021_hubspot implementation

HubSpot Training & Implementation

HubSpot Training & Implementation
Navigation_8_2021_virtual selling

Virtual Sales
Training

Virtual Sales <br>Training
Navigation_8_2021_swell - paid ads

Paid Search & Social Services

Paid Search & Social Services
Become a Certified Coach
Become a Certified Coach
Close
... Inbound Marketing HubSpot
Close
Everything You Need to Know About HubSpot Marketing Hub

Free Guide:

Everything You Need to Know About HubSpot Marketing Hub
Read the Guide
Everything You Need to Know About HubSpot Marketing Hub
Free HubSpot Marketing Guide
View Everything You Need to Know About HubSpot Marketing Hub
Everything You Need to Know About HubSpot Marketing Hub

Free Guide:

Master the ins and outs of the HubSpot Marketing Hub before you get started

In this free guide, you’ll learn:

  • How to know if HubSpot is right for you
  • How to set up your HubSpot portal
  • How to truly get the most our of your HubSpot investment

A Breakdown of HubSpot’s Outage Retrospective for the Non-Technical User

By Carina Duffy

Carina Duffy also recommends this free guide, Everything You Need to Know About HubSpot Marketing Hub.

A Breakdown of HubSpot’s Outage Retrospective for the Non-Technical User Blog Feature

If you’re a HubSpot user, unless you were on vacation from March 29 until now (if you were, I’m jealous!), you recently experienced one of the most catastrophic outages HubSpot has ever had.

Free Guide: Everything You Need to Know About HubSpot Marketing Hub

Honestly, it was a mess. Most of the time during product outages you’ll have one or two tools that aren’t working or are having bugs, and it’s resolved within minutes or hours.

This time it was just about everything: emails, form submissions, workflows, lists, sales tools, CRM, imports, analytics. It’s hard to find any pieces of the tool that WEREN’T affected during that time.

What’s even more troubling is that this outage took about 36 hours to be mostly resolved, but the processing of the backlog of data from those 36 hours is still going on at the time this article was released.

Thankfully, during all of this, HubSpot’s crisis communication was solid. Updates on status.hubspot.com were regular and timely (although are they ever frequent enough??) given the scale of the situation.

JD Sherman (HubSpot’s COO) released an article on March 29 with an apology and an outline of next steps for the team -- namely, doing an in-depth retrospective on the cause of the issue and how they’ll make sure it won’t happen again.

That retrospective was delivered on April 4. You can read the full article here. There’s a lot of detail in there about how their systems are structured and what exactly happened. If you’re not into all of the “geek-speak,” we’ve got you covered.

A Quick Review of How HubSpot’s Infrastructure Works

HubSpot uses a combination of software systems -- Kafka and ZooKeeper -- that allow all of the HubSpot tools to talk to each other and all of the data to be processed effectively.

Both these software systems have redundancies and safeguards built into them so that if some servers crash, other servers can pick up the slack, and end users don’t experience any issues.

So What Broke?

It’s a bit difficult to explain without getting super technical, but think about it like a series of unfortunate events.

High strain was put onto ZooKeeper, causing parts of it to crash. Typically, ZooKeeper recovers quickly, but in this case, it took several minutes. The delay in recovery then broke the communication between ZooKeeper and Kafka, causing Kafka to crash.

Even though the team was able to restore ZooKeeper, the damage was done in Kafka and it wasn’t able to recover. What made things worse was a second outage in ZooKeeper accompanied by trying to restart Kafka, which started to cause data corruption.

Why Did It Take So Long to Fix?

Corrupted data? That sounds bad. And well, it is. This is actually why some things took so long to come back online.

When the HubSpot team realized that the server recovery was starting to corrupt data, they had a decision to make: either focus on recovering data (and safeguarding against corrupted data) or focus on restoring the tools.

They decided to focus on recovering data to ensure that there would be no gaps in historical data for customers (which in the long run they believe to be the right decision, and I’d personally agree!). This is the reason that the affected tools took almost 36 hours to be restored.

So, in the name of protecting customer data, HubSpot manually recovered a whoooole bunch of our data, and then was able to restore the affected tools.

This is also why you’re still seeing (at the time this article was published) the “continuing to process data from March 28 & 29” status message from HubSpot.

What Now?

Now that we know exactly what happened, HubSpot’s got a plan to make sure this never happens again. An interesting note in all of this is that HubSpot’s own teams use many of their tools across different parts of the business, so this not only affected their customers but their own business (even more motivation to make sure it never happens again!).

They’re making changes in a few different areas to protect against another outage: technical/infrastructure, reliability, testing, and communication.

Technical / Infrastructure

As is to be expected, HubSpot will be doing some restructuring of their server clusters to make sure it’s not even possible to have an outage this large again. By doing this, any outage that does happen should be restricted to a small piece of the platform, and the recovery time for issues should be significantly quicker.

Reliability

HubSpot does have a team of people who test and upgrade their systems, but it hasn’t been as high of a priority as it should be. Now, they’ll have a dedicated team of people who will “oversee new standards, frequencies, and resources to ensure that we're consistently evaluating our key infrastructure systems for code fixes and critical patches without gaps.”

Testing

Along with investing much more heavily into the reliability of their platform, HubSpot is also increasing the level of frequency and depth to which they’re testing their systems. Again, it’s not that these processes didn’t exist before, but this outage uncovered some gaps in the frequency in which they test for massive failures, as well as how comprehensively they test these systems.

Communication

Lastly, HubSpot is committing to making their communication during any major incident more frequent and helpful, specifically in the minutes and hours immediately following an issue.

Their status updates will now include more detailed explanations of what is going on, as well as when the next update can be expected.

In Conclusion

No one here is pretending that this outage wasn’t bad. Not even HubSpot. But one of the things I appreciate the most about HubSpot as an organization is their transparency and willingness to admit when they’ve messed up.

They know the impact this had on their customers, and on their own business, and they’re actively seeking to make sure it never happens again.

So, even if you’re a little rattled by this outage, know that improvements are being made, fixes are being implemented, and HubSpot will continue to make their product the best it can be. Okay -- HubSpot lovefest over!

Everything You Need to Know About HubSpot Marketing Hub
Everything You Need to Know About HubSpot Marketing Hub

Free Guide:

Everything You Need to Know About HubSpot Marketing Hub

Master the ins and outs of the HubSpot Marketing Hub before you get started

In this free guide, you’ll learn:

  • How to know if HubSpot is right for you
  • How to set up your HubSpot portal
  • How to truly get the most our of your HubSpot investment

Topics:

HubSpot
Published on April 9, 2019

Recent Articles

Ultimate List of HubSpot Pros and Cons
January 8, 2022 • 20 min read
How To Optimize Your Marketing Automation Workflows With HubSpot (Tips)
November 12, 2021 • 11 min read
Top 13 Inbound Marketing & HubSpot Solutions Partner Program Agencies for 2022
October 29, 2021 • 8 min read
HubSpot Sales Hub: 18 Things Every Sales Rep Should Know How to Do (+ Videos)
October 28, 2021 • 5 min read
INBOUND 2021 Recap: Takeaways, Speakers, and Lessons Learned
October 25, 2021 • 7 min read
Need a HubSpot Specialist? Here’s How to Find and Hire the Right Candidate
October 22, 2021 • 5 min read
How to Get Sales Reps to Use the HubSpot CRM
October 18, 2021 • 4 min read
HubSpot and Data Privacy: How to Collect Contacts the Right Way
October 4, 2021 • 4 min read
INBOUND is Fast Approaching, Google Leads are Syncing, and Workflow Actions are Placeholding [Hubcast 275]
September 23, 2021 • 4 min read
How to Know When You’ve Outgrown HubSpot Sales Hub Starter
September 23, 2021 • 4 min read
How to Get the Most Out of Your 2-Week HubSpot Free Trial
September 13, 2021 • 5 min read
CMS Hub Starter, business unit add-on, and Stephanie does email validation on a giant database [Hubcast ep. 274]
August 27, 2021 • 3 min read
How much HubSpot do I need?
August 17, 2021 • 4 min read
New HubSpot CMS Hub Starter Tier Released for Growing Businesses
August 6, 2021 • 4 min read
HubSpot CRM review (updated for 2021)
July 27, 2021 • 8 min read
5 HubSpot Sales Hub Tips for Assignment Selling
July 23, 2021 • 7 min read
Custom email nurture reporting, a map hack, and are pop-up forms a thing of the past? [Hubcast 273]
July 22, 2021 • 3 min read
Get your sales team using your CRM platform in 5 easy steps
July 13, 2021 • 6 min read
HubSpot vs. ActiveCampaign: a head-to-head comparison
July 13, 2021 • 5 min read
Close rate reports, custom surveys, and Apple Mail ruining everything for email marketers [Hubcast 272]
June 24, 2021 • 3 min read
Is Your Inbound Marketing Working? Track These 4 Metrics to Evaluate Your Progress
June 15, 2021 • 5 min read
Operations Hub hot takes and college degrees [Hubcast 271]
May 27, 2021 • 2 min read
4 keys to an effective HubSpot strategy in 2021
May 7, 2021 • 6 min read
Think you’re paying too much for HubSpot? Here’s how to get your money’s worth
May 5, 2021 • 5 min read
I’m Hiring a Digital Marketer; Do They Need HubSpot Experience?
May 3, 2021 • 4 min read