Windows/CrowdStrike Outage: The Most Important Lesson
Photograph Source: Smishra1 – CC BY-SA 4.0
On July 19, users of about 8.5 million Windows users worldwide faced the dreaded “Blue Screen of Death.” As I write this column, many remain down. Microsoft has issued a manual fix for machines that aren’t able to automatically recover, but it’s a black eye for Microsoft and for Crowdstrike, the cybersecurity firm whose fault software update caused the outages.
While 8.5 million may not seem like a lot of machines in the scheme of things (about a billion and a half PCs run Windows 10/11, not counting older versions of the operating system), it wasn’t the number so much as the user identity that mattered.
The victims weren’t, for the most part, kids playing Minecraft. They were corporate customers — airlines, banks, hospitals, hotels. Flights were canceled. Account holders couldn’t access their bank accounts online. Surgeries were postponed.
My knee-jerk reaction, I confess was: Well, yeah …NEVER trust Windows or Crowdstrike (I’m a long-time Linux user and consider Crowdstrike’s close relationship with, and willingness to manufacture cybersecurity scams for, the Democratic Party suspect).
But I quickly realized that WAS just a knee-jerk response. The real lesson is: Widespread and exclusive reliance on single systems is a bad idea.
This outage didn’t affect MacOS, it didn’t affect Linux (and variants such as ChromeOS), and it didn’t affect cybersecurity software other than Crowdstrike’s product.
It did, however, affect the CUSTOMERS of businesses using the Windows/CrowdStrike combo on centralized systems.
For example, four US airlines had to cancel flights.
Why were they all using the same OS/security software combo?
And why didn’t they have backup systems, running different OSes and different security software, that could be quickly brought online to work from the same data sets as the usual systems if something like this happened?
Over the last few years, we’ve seen lots of loud calls for government to impose various top-down, one-size-fits-all “cybersecurity” solutions.
This outage demonstrates the problem with that idea. Various government operations, including 911 call centers, fell victim to the problem. Requiring private sector entities to use government-approved “solutions” would expose even more users to problems hitting those “solutions.”
In the future, we can expect more, not fewer, collapses of computer systems and networks. Putting all our eggs in one operating system / cybersecurity basket is just asking for worse and more widespread disruption.
Unfortunately, as an individual user, you remain continually vulnerable to mistakes and poor decisions made upstream from your home PC desktop.
On Friday, an update to a cybersecurity program took down Microsoft systems across the globe. Microsoft has resisted efforts to regulate a root cause of this chaos: the concentration of digital infrastructure in the hands of a few tech giants.
A little more than a year before Microsoft’s systems crashed on Friday, creating global chaos in the banking, airline, and emergency service industries, the company pushed back against regulators investigating the risks of a handful of cloud services companies controlling the world’s technological infrastructure, according to documents we reviewed.
“Regulators should carefully avoid any intervention that might disturb the competitive offerings that have promoted the explosive innovation and growth attributable to the cloud,” the company wrote in response to the Federal Trade Commission’s 2023 review of cloud computing companies’ security practices and interoperability protocols.
The agency questioned whether these companies “invest sufficient resources in research and development” of systems upon which the economy and government rely.
Microsoft is blaming this week’s global cloud outages on an update from CrowdStrike, a cybersecurity firm whose software protects against hacks. The debacle comes two days after federal agencies released new guidance sounding additional alarms that Big Tech’s consolidation of cloud services could put consumers at serious risk. It also comes one day after Microsoft’s cloud services experienced a separate outage in certain parts of the United States.
“This is a CrowdStrike-caused outage. It would be inaccurate to report this as a Microsoft outage,” the company said in a statement. “A CrowdStrike update was responsible for bringing down a number of IT systems globally. We are actively supporting customers to assist in their recovery.”
CrowdStrike did not respond to a request for comment.
“All too often these days, a single glitch results in a system-wide outage, affecting industries from healthcare and airlines to banks and auto-dealers,” posted Lina Khan, Federal Trade Commission chairwoman, whose agency spearheaded the probe of the cloud computing industry. “Millions of people and businesses pay the price. These incidents reveal how concentration can create fragile systems.”
At the root of the problem, regulators and researchers say, is Big Tech’s consolidation of cloud services, a technology that allows consumers to store computer information in massive data centers rather than storing it on-site. Just three companies — Amazon, Microsoft, and Google — control 65 percent of the cloud market, according to a report released on July 18 by CloudZero, a cost management platform.
Microsoft and CrowdStrike also dominate the end point security market, which ensures cybersecurity for devices like desktops, laptops, and mobile devices. As of 2022, the two companies controlled more than 30 percent of the market.
This consolidation helped allow a simple error to spiral on Friday.
“We had this cascading failure of all of these businesses, banks, the London Stock Exchange, all of these airlines had to be grounded, because of this one mistake,” said Zane Griffin Talley Cooper, a researcher at the University of Pennsylvania studying digital infrastructure. “And it’s because the internet has become so centralized in the hands of four or five big companies.”
“With that model, catastrophic failures like this are going to be increasingly common,” he added.
Regulatory Scrutiny Intensifies
In March 2023, the Federal Trade Commission announced a wide-ranging survey of the business practices of cloud providers. The agency looked at “market power, business practices affecting competition, and potential security risks,” soliciting comments from companies and the public.
In its response to the Federal Trade Commission’s probe, Microsoft claimed the marketplace for cloud services remains robust, and warned that regulations may affect “billions of dollars” in investments.
The company also suggested that the Federal Trade Commission’s intervention would “run the risk of impacting the quality of these solutions and the pace of innovation, and ultimately disadvantaging American companies on the global stage,” Microsoft wrote.
Public Citizen, a consumer advocacy nonprofit, warned the Federal Trade Commission in 2023 that the market dominance of Amazon, Microsoft, and Google over the cloud services sector is a threat to the economy.
“Single point dependency on a cloud provider is a structural weakness for the entire economy with the potential to cause more consumer harm in the future,” the group wrote in June 2023.
On Wednesday — just two days before the global outage — the Department of the Treasury, along with the Consumer Financial Protection Bureau and other federal agencies, cautioned that the industry’s deep reliance on a small handful of cloud service providers left it vulnerable to widespread outages and disruptions.
The Treasury also released a suite of guidance for banks and financial institutions, following its report from last February that raised an alarm about the potential risks of the highly consolidated market. The report advised that a failure like the one on Friday “could impact multiple financial institutions or U.S. consumers,” and recommended additional oversight, like inspecting third-party service providers.
The Consumer Financial Protection Bureau’s chief, Rohit Chopra, said on Friday that the failures are just a glimpse of the havoc that could be wreaked by this kind of outage in the financial sector. His agency has warned that in the future, such events could further “freeze parts of the payments infrastructure or grind other critical services to a halt.”
“There are just a handful of big cloud companies where so much of the economy is now resting on,” Chopra said on CNBC. “We’re getting a taste of some of the potential effects of a real reliance of sectors across the economy relying on a handful of cloud companies and other key systems.”
Friday’s outage was just a preview, he said, of what could go wrong in extreme cases of corporate consolidation and deregulation.
“Break Up This Cloud Consortium”
First reports of the outage surfaced early Friday morning, as computers running on Microsoft’s Windows operating system went down all at once. The issue traced back to a system update that was pushed by a company called CrowdStrike, a cybersecurity provider that is used to protect against hackers in a wide range of sectors, from airlines to banking — and was previously known for its involvement in the 2016 investigation into Russia’s hack of the Democratic National Committee.
CrowdStrike quickly said it had identified the problem with the update and began pushing a solution, but added that the fixes could take hours.
“We are aware of this issue and are working closely with CrowdStrike and across the industry to provide customers technical guidance and support to safely bring their systems back online,” Satya Nadella, Microsoft’s CEO, posted on X.
Microsoft, which was one of the early pioneers of cloud computing software, controls a staggering 85 percent of federal productivity software, and even more of its operating system.
Yet the Big Tech giant has a history of pushing back against cybersecurity measures. In 2016, the Federal Reserve, the Office of the Comptroller of the Currency, and the Federal Deposit Insurance Corporation released a joint rulemaking notice regarding the need for increased regulations on “enhanced cyber risk management standards for large and interconnected entities.”
The proposed rule would have “significant consequences not only for the financial services industry but also for third-parties like Microsoft,” the company wrote in a comment letter. It also raised concerns about the new rules, and added that cloud service providers offer better service and cybersecurity than traditional on-site storage centers.
The rule was withdrawn in March 2019.
Agencies and Congress have repeatedly tried and failed to strengthen cybersecurity regulations. Within the past three years, lawmakers have introduced at least four legislative initiatives to address these concerns, though none have been adopted so far.
This February, the federal Cybersecurity and Infrastructure Security Agency also announced it was renewing a task force charged with managing risks to the global information and communications technology supply chain, crucial for protecting computer hardware, software, and applications.
The companies themselves were seemingly aware of the potential threat caused by an overreliance on cloud-based systems.
In a 2023 comment letter to the Consumer Financial Protection Bureau about a proposed rulemaking to tighten personal data restrictions, CrowdStrike — the cybersecurity company responsible for Friday’s data breach — argued that the biggest risk to cybersecurity was not software supply chain issues, but hackers.
“It is our view that perhaps the most significant threat to data comes from bad actors operating unlawfully, leading to data breaches, cyberattacks, exploits, ransomware attacks and other exposure of consumer data,” CrowdStrike wrote.
CrowdStrike echoed their concerns about the dangers of hackers and resulting system failures in their most recent annual 10-K report. The company told investors that the “consolidation of siloed products” was a concern because “integrating and maintaining numerous products, data and infrastructures across highly distributed enterprise environments” created “blind spots that hackers can exploit.”
Microsoft, in its 2023 annual report to shareholders, also expressed that “providing [their] customers with more services and solutions in the cloud puts a premium on the resilience of [their] systems.”
But the companies have worked hard to keep regulators from taking steps to address these risks.
Microsoft is one of the country’s top spenders on lobbying, ranking in the top one hundred of corporations. So far this year, the company spent more than $5 million on campaign donations and lobbying lawmakers and regulators. Microsoft lobbied Congress, the Federal Trade Commission, the Treasury Department, the Executive Office of the White House, and other regulators on “policy issues in cloud computing,” among other issues, disclosures show.
“What we really need,” said Cooper, the University of Pennsylvania researcher, “is regulators to break up this cloud consortium of four or five companies and help distribute management of the internet backbone through a host of different companies.”
