Patching Keeps the Lights On (Except when it Doesn’t)

Published on 21st April 2022 Author: Rob Black

You probably don’t give much thought to lightbulbs. I wouldn’t either, except that somehow, we own a house with lots of them.

We have so many that in any given week, one, two, sometimes three of them burn out. And (of course), they are not all of the same type.

Some are floodlight bulbs. Some are halogen. Some are decorative. Some are heat bulbs. Some go in the dining room chandelier. The list goes on and on.

As a result, not only do I have to stock a gigantic supply in the basement, but I also have to dedicate time each week to lightbulb replacement duty.

On the suggestion of my brother, I am now in the process of switching over to longer-lasting LEDs.

But even so, it’s complicated enough that I’ve had to create a room-by-room scheme of which types of LEDs go where. The first time I accidentally put a “daylight” bulb in a family room light socket, it reached into my eyeballs and tried to forcibly remove my retinas (I might be exaggerating).

Most Software Has Vulnerabilities

When it comes to the lightbulbs in my house, I confess to being a bit obsessive. When it comes to managing the software patches in your company, however, this same degree (or more) of vigilance is required.

That’s because software almost always has vulnerabilities – bugs that could allow an attacker to gain access to your systems. Why?

First, because most popular software programs have hundreds of thousands (if not millions) of lines of code. Even with only a 0.0001 error rate, a million lines of code will have 100 errors!

Second, because the software you use is part of a very complicated system. What may have worked every time before fails because something changed: Maybe your computer has too many programs open. Maybe there is a weird registry setting that has not been cleaned up. Maybe a lightbulb blew at the exact moment something was being downloaded.

Whatever the reason, a computer “reboot” will often restore things to an earlier starting point and the problem goes away. However, sometimes there are persistent problems that just aren’t caught.

If the functional problem relates to someone being able to compromise confidentiality, integrity, or availability, then it is a security problem. When these bugs are discovered, the software vendor will often issue a patch.

Problem fixed … provided the patch is installed before an attacker exploits the vulnerability. When that happens, it’s too late.

Not convinced? Consider what happened to Equifax in 2017 when its failure to install a patch led to a massive data breach: $1.4 billion in clean-up costs, a $425 million settlement with the FTC, and the firing of its CEO, CIO, and CISO. Not to mention years of horrible publicity!

You Need a System

For many small organizations, the patch management “system” is Frank in IT – a hardworking, but very busy specialist who does his best to keep things current. Unfortunately, this type of ad hoc, single point of failure approach is bound to crash.

A good patch management process needs to come from senior management. There needs to be an organizational commitment to…

First – understanding how big a deal a given vulnerability is.

Second – tracking which key systems have that software on it.

Third – establishing a methodology for testing and implementing the patch.

Fourth – dedicating an appropriate level of time and resources for getting the work done.

Automate What You Can

Even in smaller companies, it is important to develop repeatable processes for managing technology, including patching. Much of this can be automated. Some examples:

Forcing Auto-update of laptops
Using Mobile Device Management (MDM) tools
Having cloud infrastructure that rebuilds itself periodically from a known, good configuration
Automated scanning of infrastructure to alert you to critical issues, such as out-of-date components or other vulnerabilities

When Patching Takes the Lights Out…

Patching is obviously important, but new patches have the potential to break things or cause downtime. There are a couple of recent, real-world examples we can learn from:

1. Testing patches can be important.

Earlier this year, Microsoft released a patch that caused some Windows servers to get stuck in a boot loop and break VPN clients. Companies that were affected by this bug had a really bad day!

So, make sure to test your patches before rolling them out universally. Test them on a few machines first or, if it’s in production, set up a staging/test environment to try them out.

2. If you don’t patch it, someone else just might.

In April, The FBI obtained many court orders and logged onto company/people’s systems that had a Russian botnet on them.

They removed the botnet and made changes to the infrastructure. With this new model of intervention by the government, it is even MORE important to patch.

For example, imagine if your infrastructure were accidentally taken down by a “good guy” while trying to fix it. Mistakes happen. Plus, even if done properly, you don’t want the fix to occur during your busy time, something the FBI is probably unconcerned with!

Prioritize Your Risk

On any given day, there are literally thousands of patches that a company could apply. But none of this comes for free – you can’t patch everything. So the key to smartly using scarce resources is to understand the organizational risk involved.

My rule of thumb is to patch all Internet-facing infrastructure with serious vulnerabilities immediately. The bad guys spend all day scanning the Internet looking for things that are easily exploitable – you don’t want to be caught up in any of that. Log4j, for example, had a vulnerability detected at the end of last year that required immediate attention.

Once the super-critical items are handled, it’s a question of further prioritization. If you have 100 things that need patching, the top five are probably essential. The bottom five won’t make much difference.

As with most things cybersecurity-related, effective patching is all about managing risk. And while installing patches may not be necessary in many cases, neither is a seatbelt – it only takes one really bad incident to shut the lights off for good.

Want to get great cybersecurity content delivered to your inbox? Click here to sign up for our monthly newsletter, Tales from the Click.

Rob Black, CISSP

Rob founded Fractional CISO in 2017 and has helped dozens of companies improve their security posture as a vCISO. He consults, speaks, and writes on IoT and security. Rob has held product security and corporate security leadership positions at PTC ThingWorx, Axeda and RSA Security. He received his MBA from the Kellogg School of Management and holds two Bachelor of Science degrees from Washington University in St. Louis in Computer Science and System Science and Engineering. He is also a Certified Information Systems Security Professional (CISSP).