Lessons learned from the CloudStrike outage

Photo supplied
Leslie MeredithLast Friday, CrowdStrike Falcon, a security platform used by large companies and government organizations for their PC networks, suffered an outage that caused worldwide disruption. The problem stemmed from a faulty update and caused Windows systems to crash and display the blue screen of death. Hackers quickly jumped into the fray and took advantage of the confusion.
There are three important lessons to learn from this debacle. First, big tech companies make mistakes, and only some disruptions are the work of cybercriminals. Second, sometimes an easy fix does the trick – more about that below – as long as those affected remember the basics and are persistent. And third, bad actors are always ready to exploit a chaotic situation. Let’s take these lessons one-by-one and understand how we can apply them in the future.
CrowdStrike is a leading cybersecurity company known for its Falcon platform, a cloud solution designed to protect against a wide range of cyber threats, including malware, ransomware and zero-day exploits. While the outage affected only 8.5 million PCs worldwide (about 29,000 companies) the results were significant.
For instance, Delta, American Airlines and United all canceled flights and delayed others, totaling more than 1,300 flights altogether. Bank of America users experienced difficulties accessing their online accounts and mobile banking apps, while Charles Schwab Investors faced problems accessing trading platforms and account information. Alaska’s 911 services were disrupted, potentially delaying emergency response times. Various government websites and services were impacted, affecting public access as well.
CrowdStrike responded quickly by identifying the problem and ways to fix it. The incident began at 04:09 UTC when CrowdStrike released a faulty sensor configuration update for its Falcon security software on Windows systems. By 05:27, they had reverted the faulty update, and by 09:45, a fix was issued. The security company worked with the big cloud service providers – Microsoft, Google and Amazon – to communicate with customers. Unfortunately, each affected machine required manual intervention to be fully restored. CrowdStrike’s CEO has apologized for the incident, and the company is conducting a thorough root cause analysis to prevent similar issues in the future.
The Fix
How many times have you been told to turn off a device and turn it back on to see if that’s the problem? In this case, that’s what Microsoft advised based on customers’ reports, but not just one time. It took restarting the computer up to 15 times for this fix to work. If that didn’t work, IT administrators were advised to delete a faulty CrowdStrike folder from each machine. It’s certainly been a long few days for IT teams.
Cyberattack Danger
Cybercriminals were quick to leverage the panic generated by the outage. CrowdStrike learned of phone calls to its customers from people posing as CrowdStrike employees. The U.S. Cybersecurity and Infrastructure Security Agency (CISA) issued an alert about phishing emails attempting to capitalize on the chaos that said: “CISA has observed threat actors taking advantage of this incident for phishing and other malicious activity. CISA urges organizations and individuals to remain vigilant and only follow instructions from legitimate sources. CISA recommends organizations to remind their employees to avoid clicking on phishing emails or suspicious links.”
These tactics prey on the panic that ensues when computer systems are disrupted. Companies can virtually hear business being lost, and it’s hard to regain it and their reputations regardless of who is at fault.
So, what have we learned? Big companies are made up of people and even the most competent can make mistakes. However, backup systems and rigorous testing should be in place to avoid problems like the CrowdStrike outage. And when something goes wrong, a quick response, collaboration with partners and full disclosure to the public is essential. Sometimes, the simplest fixes work – try those first, multiple times. And finally, cybercriminals will always exploit panic and fear. That’s why IRS scams are so effective and why Microsoft repair scams have been raking in money for years. This outage was just too good for bad actors to resist, and the simple tools were easy and quick to deploy.
These lessons are not just for IT administrators – you can adapt them for your own use. Before adding a new program to your PC or updating it, make sure you have a backup. If something goes wrong, you’ll be able to revert to an intact copy. Before calling a repairperson, try the on-and-off method, which could save you a hefty repair bill. Never click on links or attachments in an email that appears to come from a company. Instead, reach out to the company directly.
Leslie Meredith has been writing about technology for more than a decade. As a mom of four, value, usefulness, and online safety take priority. Have a question? Email Leslie at asklesliemeredith@gmail.com.