What To Do If Your Database Catches Fire

I need to deep fry a turbaconducken.

This isn’t a want, no. This is a primal need of mine.

I feel so strongly about this that it’s on my bucket list. It is positioned right below hiring two private investigators to follow each other, and right above building an igloo with the Inuit.

Deep frying a turkey is a dangerous task. You can burn your house down if you are not careful. Why take the risk? Because the end result, a crispy-juicy turkey bathed in hot oil for 45 minutes, is worth the effort. Or so I’ve been told. Like I said, it’s on my bucket list.

Being the good data professional that I am I started planning out how to prepare for the day that I do, indeed, deep fry my own turkey. As I laid out my plans it struck me that there was a lot of similarity between both an exploding turkey and the typical “database is on fire” emergency many of us know all too well.

So here’s my list for you to follow for any emergency, from exploding turkeys to databases catching fire and everything in between. You’re welcome.

Don’t Panic

People who panic are the same people who are not prepared. A little bit of planning and preparation go a long way to helping you avoid “panic mode” in any emergency situation. Whenever I see someone panicking (like ripping out all their network cables just because their mouse isn’t working) it is a sure sign that they have little to no practical experience with the situation at hand.

Planning will help you from feeling the need to panic. If your database is on fire you can recover from backups, because you prepared for such a need. And if your turkey explodes you can always go to a restaurant for a meal.

Rely on all your practice and training (you have practiced this before, right)? Emergency response people train in close to real life situations, often. In fact, firefighters even pay people to burn down their spare barns. Really. In fact I bet firefighters would make *great* DBAs: https://thomaslarock.com/2012/10/i-bet-firefighters-would-make-great-dbas/

Go to your checklist…you do have a checklist, right? And a process to follow? If not you may find yourself in a pile of rubble, covered in stripper dust.

Assess the Situation

Since you aren’t panicking you should be able to calmly assess the situation. A turkey on fire inside your oven would require a different response than a turkey that explodes in a fireball on your deck and is currently burning the side of your house. Likewise, an issue with your database that affects all users will require a different set of troubleshooting steps than an issue affecting only some users or queries.

In order to do a proper assessment of the situation you will be actively gathering data. For database servers you are likely employing some type of monitoring and logging tools. For turkeys, it’s likely a thermometer to make certain it has completely thawed before you drop it into the hot oil.

You also need to know your final goal. Perhaps your goal is to stop your house from being engulfed in flames. Perhaps your goal is to get the systems back up and running, even if it means you may have some data loss.

Not every situation is the same. That’s why a proper assessment is necessary when dealing with emergencies…and you can’t do that while in a panic.

Know Your Options

Your turkey just exploded after you dropped it into a deep fryer. Do you pour water on the fire quickly? Or do you use a fire extinguisher?

Likewise, if you are having an issue with a database server should you just start rebooting it in the hopes that it clears itself up?

After your initial assessment is done you should have a handful of viable options to explore at that point. You need to know the pros and cons for each of these options. That’s where the initial planning comes handy, too. Proper planning will reduce panic, allow you to assess the situation, and then you can understand all your viable options along with the pros and cons. See how all this works together?

It may help for you to phone a friend here. Sometimes talking through things can help, especially when the other person has been practicing and helping all along.

Don’t Make Things Worse

Pouring water on the grease fire on your deck is going to make the fire spread more quickly. And running 17 different DBCC commands isn’t likely to make your database issue any better, either.

Don’t be the person that makes things worse. If you are able to calmly assess the situation, and you know your options well, then you should be able to make an informed decision that doesn’t make things worse. Also, don’t focus on blame. Now isn’t the time to worry about blame. That will come later. If you focus on fault, you aren’t working on putting out the fire right now. You might as well grab a stick and some marshmallows for making s’mores while your house burns to the ground.

Also, a common mistake here is done by people who try to do many things at once, specifically for database issues. If you make multiple changes then you may never know what worked, or the changes you make may cancel each other out leaving you still with a system offline. Know the order of the actions you want to take and do them one at a time.

And it wouldn’t hurt you to take a backup now, before you start making changes, if you can.

Learn From Your Mistakes

Everyone makes mistakes, I don’t care what their marketing department may tell you. Making mistakes isn’t as big of a concern as not learning from your mistakes. If you burned your house down the past two Thanksgivings, don’t expect a lot of people showing up for dinner this year.

Document what you’ve done, even if it is just a voice recording.  You might not remember all the details afterwards, so take time to document events while they are still fresh in your memory.

Review the events with others and gather feedback along the way as to how things could have been better or avoided. Be open to criticism, too. There’s a chance the blame could be yours. If that’s the case, accept that you are human and lay out a training plan that will help you to avoid making the same mistake in the future.

I hope everyone has a wonderful and safe Thanksgiving this year, surrounded by family and friends. It’s good to come together at least once a year and give thanks for the fact that we are all allowed to walk this Earth at this very moment in time.

I’m thankful that my turkey, or any database server, isn’t on fire.

But if it was, I know I’d be prepared.