Lessons to learn from the massive CrowdStrike outage

Last Friday felt like something out of a Y2K nightmare after the cybersecurity company CrowdStrike, pushed a software update to all its clients — including health care systems, banks and the federal government — that ended up crashing computer systems worldwide.

The fallout is still being felt, particularly in the travel sector, as airliners try to reschedule canceled flights while trying to get everything back to normal.

It’s also become something of a reminder that the internet and a lot of the online services we rely on are delicate.

Marketplace’s Lily Jamali spoke with Kate Conger, a reporter at The New York Times who recently wrote about this with her colleague David Streitfeld.

The following is an edited version of their conversation.

Kate Conger: We tend to think of the internet as just something that’s floating around in the air with us, like oxygen, and it’s not. It’s actually a very complicated system that is full of infrastructure. And I think the infrastructure of that is really interesting and really fragile, because it was built at a time when we had no idea what role the internet would come to play in our lives. So, you know, there’s sort of the physical infrastructure of the internet itself, these undersea cables that carry the internet around the world from place to place. Then we have the [domain name] service, which is what all of our websites live on, and that is a very complicated and fragile system that was not really designed with the modern internet in mind. And then we layer on top of that apps and updates and all these different parts of the web, and it becomes this very complex system that’s built on an underlying infrastructure that we never really see or think about, but is quite fragile, you know? I mean, you think of it as sort of like a little decaying, subway system that’s kind of hidden underneath the earth and we’re just up above.

Lily Jamali: Sure. So it sounds like what you’re saying is that maybe we shouldn’t be all that surprised that the cause of this mishap was so unspectacular, in a way, even though the results were quite spectacular and quite profound. I mean, we talk a lot about the rise of AI, and there’s so much doomsday talk there about how our systems could crash in very dramatic fashion. This was not that.

Conger: Yeah, and I think it’s honestly sort of surprising the internet doesn’t break more, I think, and it’s really due to the fact that there are so many dedicated professionals who are focused on trying to keep these things online and maintaining them.

Jamali: Absolutely. And you write historically, there is precedent for this. This is actually pretty common from what you write, throughout the history of tech. It can often be that these understated problems can have these very big ripple effects. So can you give us an example of that?

Conger: Sure, I think one that we talked about in the story is this kind of mass blackout in the Northeast in 1965 when power went out from one Canadian relay station all throughout Delaware, Maryland, Massachusetts, New Hampshire, New York, kind of all along the upper East Coast. And it’s these little bits of infrastructure that can fumble and break down and have these very massive ripple effects.

Jamali: Here we have an example of one tech company, CrowdStrike, not exactly a household name outside of tech, we should mention. This company provides services to more than half of Fortune 500 companies, plus a lot of government agencies, including here in the U.S., that includes the top agency, by the way, in charge of cybersecurity. So Kate, with that in mind, I mean, what’s the takeaway here?

Conger: Yeah, I mean, so this is something that’s really interesting, I think, about the cybersecurity industry. You know, like you said, CrowdStrike is really not well known outside of tech, but within tech, it’s kind of a mythic company. They’re the people that get called in during the worst of the worst cyberattacks. They were famous for investigating the 2014 hack of Sony Pictures, the 2016 hack of the DNC.

Jamali: That’s the Democratic National Committee, which, because of a leak, we were able to see all of then-presidential candidate Hillary Clinton’s emails.

Conger: That’s right, and CrowdStrike came in and helped the DNC investigate that incident, and find out who was behind that hack. So they have this very outsized reputation within tech as a company that can handle kind of the most complex and worrisome cybersecurity problems. And there’s a lot of consolidation in the industry of security, because you want a company protecting you that sees all the worst of the worst. They’re not just seeing kind of the everyday phishing scams on email, but they’re also seeing the techniques that, you know, Russia and China’s nation-state hackers are using to break into these big companies and to break into power grids. And the people who are protecting you have the most extensive knowledge of all of the online threats that you could face. So it’s pretty common in this industry for companies to rely on just a handful of companies that have that kind of breadth and reach when it comes to security.

Jamali: And that is a problem, at least in the view of someone like Federal Trade Commission Chair Lina Khan, who is saying this is a teachable moment about just how concentrated certain corners of tech have become.

Conger: Yeah, it is a very teachable moment, but I think it’s also a really tricky one, because you have to think through, OK, what are the other options for a company, right? If they’re using CrowdStrike today, they have this outage, they’re frustrated by it, but they still want to stay protected from future threats, they’re going to turn to another big company. I mean, there’s three or four really big players in this space, and they want to turn to those companies that have the resources to protect them.

Jamali: So just going back to the specifics of this incident, and with CrowdStrike in particular, is there consensus about what they could have done to prevent this from taking place?

Conger: I don’t think we know enough yet about the specifics of how this incident took place to say this is exactly what CrowdStrike should have done to prevent it. In general, when we think about these kinds of software updates, companies typically will test them on a variety of different machines and environments to make sure that when that update goes out, it doesn’t cause these kinds of problems. And so there’s a possibility here that CrowdStrike didn’t test thoroughly enough, or this just slipped through the test that they did do, and they somehow didn’t catch that this error would happen. But we don’t know for sure yet that that was the case.

More on this

Last week, the U.S.’ Cybersecurity and Infrastructure Security Agency said to be wary of any phishing emails or suspicious links related to the CrowdStrike incident.

It seems hackers started sending these emails pretty much right after it happened, some saying they could “fix the CrowdStrike apocalypse” just as long as you send a bunch of money to a random account.

Hopefully, you’re all up to date on your cybersecurity training at work.