It sounded like thunder, but not thunder, at the same time.
Something was different about this sound. Also the lack of a flash of lightning made me think that it was not a clap of thunder. Not at all.
I had just built a fire in our fireplace, the first of the year. I went outside onto our deck to look up and see if the chimney had crashed down onto the house. It hadn’t. By then I could hear neighbors outside asking each other “what was that?”, and no one had any answers.
Having been through a few disasters in the past 17 months here I looked to the sky for signs of an explosion. I guess I was looking for a mushroom cloud, plague of locusts, or something distinctly Mayan. I didn’t see anything unusual.
Wanting to know what just happened I did what anyone else would do: I turned to social media. Nothing on Twitter but on Facebook a neighbor posted a question about the noise and we were quickly informed that there was a gas leak in downtown Springfield that resulted in an explosion of the Scores Gentleman’s Club. I know what you are thinking: it will take weeks to clean up all that glitter.
Over the course of the next two days we started to get details about what had happened. Apparently there had been reports of a gas smell in the area for a few days prior. Utility crews had been on the scene to investigate but could never locate the source. On Saturday they were back, still trying to find the source, when a worker accidentally struck the service line running to one of the buildings in the area.
Now, here are the lessons for you to take from this incident. What that person did in the next few minutes likely saved many, many lives.
1. He Admitted He Made A Mistake
He didn’t try to hide the error. He didn’t worry about blaming someone else’s mistake (the gas line markings in the street were possibly incorrect) first, instead he focused on what needed to be done now. What had already been done didn’t matter. What mattered most was getting the gas turned off and evacuating buildings immediately.
2. He Had a Process To Follow
Just admitting you made a mistake, if only to yourself, isn’t going to be enough. When things go suddenly wrong you need to have a process in place. Just like with pilots and surgeons this man knew exactly what needed to be done once he realized he had punctured the service line. He didn’t need to guess as to what had to happen next. The clock was ticking at that point and there was no time to sit around and throw possible solutions up on a whiteboard to see what might be best. He didn’t need to go and buy special tools or hardware, he didn’t need to Google for a solution. He probably had a card with a few important phone numbers on it that reads: IN CASE OF EMERGENCY.
The process doesn’t need to be overly complex, but it needs to exist.
3. He Took Immediate Action To Mitigate Damages
Having a process in place is good, but what is better is taking action to follow the process. The service line was punctured about 4:20PM. The worker immediately called to have the line shut down and it took 30 minutes to get the gas turned off to the building. During that time the worker also notified the fire department and they started evacuating buildings about 4:35PM. The building exploded about 5:25PM. Imagine if that worker had spent time trying to pretend they didn’t do anything wrong? Every minute spent either pretending nothing was wrong, or arguing about who to blame, could be the difference between life and death.
Now chances are you don’t deal with such things that lead to life and death choices as part of your daily tasks (of course, those working in Healthcare are likely to have such pressures). But I bet there’s a damn good chance that at some point in your life you’ve screwed up something on the job and have tried to pretend it wouldn’t get noticed.
And that’s not the way you should ever treat a mistake you make.
If you make a mistake, you admit that you made the mistake. I’ve written before about incidents where I’ve screwed something up, like not being able to easily recover the master database in the middle of the night.
I admitted I wasn’t perfect, that I was not prepared and as a result it took me longer than necessary to recover the master database. I explained to my managers what happened and what steps I was going to take to fix things. Then I built a small training scenario for my team to practice recovering master so that they wouldn’t ever get stuck like I was that night.
If you make a mistake, admit what happened. Don’t spend your time trying to pretend it didn’t happen, or figure out a way to blame someone else. Notify the people that are likely to be affected, and take steps to mitigate any possible damage.
Otherwise you may find yourself in a pile of rubble, covered in stripper dust.
You Should Start Planning For Disasters
Start now, before it is too late. You should already have a list of critical systems in your shop. Start thinking about what process you will follow when things go wrong. If your company is already doing regular DR exercises, that’s great. But most DR plans are for widespread outages. What will you do if just one server is offline? What about one database becoming offline? Now what?
Document what steps you need to take. Share that documentation with others. Make your own IN CASE OF EMERGENCY cards and distribute them to your team.
And then practice. You need to be prepared, right up to the point you earn yourself the nickname “master of disaster”, but in a good way.
Also, if you ever do smell gas in or around your home, here’s the information you need to know.
Great lessons, Tom. We always need to be ready for disasters and when they do strike we need to deal with them. Especially if we caused them – quickly owning them and working to make them better is a great solution. I blogged about this awhile ago and my “device” wasn’t so different than yours, come to think of it.. http://www.straightpathsql.com/archives/2010/03/bill-clinton-was-no-impeached-for/ 😉
Thanks Mike!
I thought about your posts when reading this…
Very apt comparison
Thanks Andy!