If I were to sum up the CrowdStrike catastrophe in three words they would: ironic, terrifying, and hopeful. The situation is ironic because a cybersecurity vendor caused (potentially the worst) one of the worst and most expensive outages of all time. It's terrifying because essentially two organizations caused a nationwide halt on travel. In spite of the catastrophic implications, at the end of the tunnel the light truly shone through with the IT community banding together to help resolve the issue.
How many times a week do you hear the word cybersecurity? If it seems like an unusually high amount in the past couple years then you may have the same feelings as I do. I think that there has been a shift in the industry to completely rely on third-parties to take care of all of your security needs. It's one thing to have an antivirus and firewall installed but to insure people are monitoring your network 24/7 for threat actors seems a bit excessive for your pizza shop down the street. This incredible push has come from what I believe to be a plateauing of the industry in other aspects like hardware and development. Machine learning may be the final frontier outside of something more far off such as quantum computing. Now don't get me wrong there are constantly improvements to everything but there haven't been massive jumps in the industry like there were every few years in the past 20 years. We are going to dive into the blame game a bit in this article and figure out what the heck is going wrong in all areas of the pipeline and how this disaster could have been avoided.
We are going to start by talking about Apple. I know, I know, Apple doesn't have anything to do with what happened on July 19th 2024. That may be the point. Apple's impenetrable gated community is an example of what I believe most casual users should be aspiring to be using. I always recommend MacBooks to non-tech people because there is only so much that they can mess up on their devices. Are there cybersecurity vendors who offer products on Apple devices? Sure. Would Apple ever let them get to their equivalent of Ring 0 or the Kernel access? I highly doubt it. The control that Apple puts on their devices is both a blessing and a curse but in this particular circumstance probably would have saved 29,000 customers from having to deal with an outage nightmare. I am not blaming Microsoft. The only thing that Microsoft did wrong in this situation was to give CrowdStrike access to the kernel which really shouldn't have been a problem. You should be able to trust the biggest government cybersecurity vendor right? Right?! The insurance plans that Windows has for recovering from these types of things allowed for quick recovery of most devices and the automatic process of recovery was also destroyed by the CrowdStrike bug. If Microsoft is to continue holding the majority of the PC market, they may need to think about designing more implementations of recovery for catastrophic plans though. It is not Microsoft's job to hand hold every other software organization into creating the best possible product, they should be able to do that themselves. On the other hand, the balance that Microsoft and OS vendors have to offer is that the software that is being implemented on their system will have well thought and easy recovery options if something like this happens.
The main thing that is missing today in bad products is accountability. When an organization creates a bad product or causes some sort of massive issue all that happens is a slap on the wrist and life goes on. CrowdStrike's stock plummeting may not be enough reparation for the estimated 5.4 billion worth of damages that have occurred from their mistake. The halt on air travel on a national level was a very frightening moment for the nation. People's lives came to a screeching halt not due to what was originally thought to be a massive cyber-attack but instead a cyber mistake. 8.5 million devices were affected and the majority of them were government and infrastructure devices. There has to be more diversification in the cybersecurity industry. I also believe that there needs to be diversification of OS but I think that the world will slowly work its way to that point and that the majority of critical industries have learned to use Linux. These massive federal and state contracts have to be looked at with a magnifying glass if you are the technology manager to determine if there are areas where you can use different products to avoid these calamities. On the contrary, cybersecurity is an incredibly hard thing to diversify. You can't exactly install two different types of security products because they often clash and provide the same services. You almost always have to give one of them the overarching permissions which in turn becomes the single point of failure. How much access and how much you do in house for security is between the manager and the team to determine and often companies are pushing to hire less and less which causes teams to have to use cybersecurity products to their full extent. In my opinion, a whole lot of jobs could be created by organizations refuting the cybersecurity norm and having specialists within their organization. I know this isn't feasible for most but for an airport like Atlanta or LAX to completely rely on an outsourced cybersecurity vendor in 2024 seems like the world could do a little bit better. In the end, people need to be better. Organizations need to be better. Especially if you are putting this much trust in them to take care of your products. Anyone with any experience in the industry would be able to tell you that this could have been avoided with more testing. Test, test, test, test, and test again if you don't want to end up like CrowdStrike.
The last point I'll make in this blog is that this incident was a great indication of the community and the faith that people should have in good actors over bad ones. The response was so swift in finding a solution and posting it on social media that the outage was limited in how badly it affected most things. Unfortunately, the fix was something that almost had to be done by hand on each individual device which took a long time for larger organizations that were low staffed. Smaller organizations were back up almost immediately during the work day since the solution was found before people woke up. It was a rough day for the IT world but the good actors of the industry helped mitigate the disaster to the point where it wasn't a scary existential threat anymore. The actions taken by people to help solve the issue gave me faith that despite what the world throws at the technology sector, there will always be good people to help fix the issues fast and effectively.
-Macon Moyer 08/01/2024
Comments