Don’t press that panic button!

Author: Rob Black

You may assume that I am a fully functioning human. I assure you, you are mistaken.

There are certain things – things that most people are quite capable of – that I am dazzlingly terrible at.

I cannot sing. Literally. Once, when I joined in on “happy birthday” at my then four-year-old daughter’s party, two little girls started crying and another ran screaming out of the room.

I have no ability to properly match clothes. My fashion knowledge comes to a hard stop at “don’t wear a checkered shirt with striped pants.”

And I am absolutely terrible at administrative paperwork. I will do just about anything to avoid it.

And so, when I flew to Toronto for the day recently to visit the team at our new Fractional CISO office (yay!), I had no idea what I was getting into. The problem, as with many things these days, was COVID.

Apparently, they will let you into Canada no questions asked. Leaving, however, is another story. Because when I went to check in for my flight, I was rejected.

The problem was that while I had my vaccination card in hand, I had not taken a COVID test since my arrival that same morning – a requirement that I was unaware of.

Now what? My plane (the last of the night) was leaving in 45 minutes; the airport testing site was a train ride away; I didn’t have an appointment; and all of this required filling out a bunch of (horrors!) administrative paperwork.

So I evaluated my options and made a decision: I rented a car (there’s no testing or paperwork required to drive across the border). Five hundred and fifty miles later, punctuated by a short stay at the Red Roof Inn somewhere between Syracuse and Albany, I arrived home at 8:30 the next morning.

I’ve since shared this experience with several people and the response is always the same: “You drove?” I understand, for most people, this isn’t a reasonable choice.

But my priority was to be back in business the next day (and, okay, I prefer the certainty, control, and lack of bureaucracy that comes with renting a car).

When Disaster Strikes

Having a predetermined disaster recovery plan in place for a one-day business trip probably isn’t necessary, even for a cover-all-bases, cybersecurity guy like me. It’s fine to respond on the fly.

But having a well thought out plan in place is absolutely essential when it comes to the ongoing health and performance of your business. Unless, of course, you’re okay with being down for hours, days, or longer.

For example, we know of many companies that run their business-critical operations out of a single data center or SaaS region. Often, even if there is a backup site established, it is in geographic proximity to the primary site. That might help if the failure is tied to a specific location, but will be of little use if the outage is due to a natural disaster or similarly regionwide event.

Other companies operating out of a single site intend to spin up resources in a SaaS data center if things go terribly wrong.

But how many days or weeks could that take? Are there contracts in place? Is the data backed up outside of the affected area? Is the configuration scripted? Is a copy of all software available? Is the cutover plan well understood?

In other words, it can take a long time!

“The Cloud” is Not Fool-Proof

As a business leader, you may be comforted by the knowledge that “everything is in the cloud.” But the cloud is not some magical place.

Your cloud provider (e.g., AWS, Azure, GCP) can experience a disaster as well. If that happens, what are the chances that they are going to prioritize your company over their much larger customers? Answer: zero.

So make sure you insist on understanding the plan for what your organization would do if there were a total disaster (and George in IT was on vacation).

Building a Plan

If you don’t have a disaster recovery plan in place, here are suggestions for getting started:

1. Any plan (even a bad one) is better than nothing.

Start putting pen to paper. The plan may be, “If a disaster happens, sign a deal with AWS/Azure/GCP/etc. Spin up some server instances. Put software on them. Restore database. Hope for the best.”

That’s a terrible plan … but it’s a plan.

2. Make improvements.

Work on the riskiest parts of the plan first: “Okay, we should probably back up all of our configuration data.” Or, “ We should sign that agreement with the cloud provider.”

3. Perform some tests.

You don’t need to change over everything from your current production systems to the disaster recovery – you can test aspects of your plan.

At the very least, you can perform a “table top” exercise to talk through the issues. This way, when someone says, “What about X?” you can think through the options.

4. Keep iterating.

With a plan in place, maybe your recovery time is 20 hours. Could you cut that in half if you found a way to migrate the customer data over faster? What if some things were done in parallel?

The point is, just as my priority was to be back home and working by the next morning, your business also has priorities. Determine what matters most and decide where and to what degree you are willing to invest resources.

The key is to avoid surprises and last-minute decisions.

Everything is a Business Decision

Like most things of a disastrous nature, a devasting failure or outage is unlikely. But when it comes to your business, I’m guessing that’s a chance you’d rather not take.

Make sure you’ve got a cutover plan that is well thought out, well understood, and well documented. Then keep making improvements from there.

In the meantime, let me know if you need any musical entertainment for your next birthday party.

Want to get great cybersecurity content delivered to your inbox? Click here to sign up for our monthly newsletter, Tales from the Click.

Rob Black, CISSP

Rob founded Fractional CISO in 2017 and has helped dozens of companies improve their security posture as a vCISO. He consults, speaks, and writes on IoT and security. Rob has held product security and corporate security leadership positions at PTC ThingWorx, Axeda and RSA Security. He received his MBA from the Kellogg School of Management and holds two Bachelor of Science degrees from Washington University in St. Louis in Computer Science and System Science and Engineering. He is also a Certified Information Systems Security Professional (CISSP).