“Signs point to yes.” – Magic 8-Ball prophecy
You have probably heard this line at one point, the one that says “the best DBAs are the ones you never see or hear?” I used to think that was a result of some sketchy personal hygiene choices and later on understood it to mean that if everything is running smoothly then there isn’t a need to see your DBA. It’s kinda like how you never see your plumber. Oh sure, you could call him (or her) and chat, have them over for drinks or dinner one night, but you never bother talking to them until your toilet backs up and there’s crap everywhere. That’s how most DBAs are treated as well, we don’t get called until after the crap is on the floor.
Unlike your plumber most DBAs are performing regular work on your databases. Much of this work will go unnoticed. The reason for that is simple: you’ll freak out if you knew what we were doing. And if you ask us about something we are likely to tell you half-truths because (1) we don’t want you to freak out and (2) you never understand us when we talk to you anyway.
Here’s my list of things that I am willing to bet your DBA hasn’t always been truthful about.
1. Your database server is still on physical hardware
Probably the number one misconception that end users have these days centers around the use of virtualization. People fear change, and when they are told that their servers are about to be virtualized they freak out. And for the handful of users that don’t freak out immediately their reaction is just delayed until the moment that something goes wrong and then they blame the fact that the server is on virtual hardware. What’s that? The bank didn’t send the files last night, so the import never started? It must be due to the fact that we were virtualized, because before then nothing like this ever happened.
Wrong. Bad things happened before you were virtual, too. It’s just that you don’t remember. So we don’t bother telling you about the switch. No, we wait until it leaks out months later, just so we can say things like “look, it’s been on a VM for six months and you never once complained about performance, so don’t try blaming virtualization for your issues now.”
2. Everything is fine, nothing to see here
Chances are everything is not fine. In fact, chances are the data center is on fire and your server melted last night (good thing they were virtualized so we could failover immediately to a different data center without you knowing) but we aren’t going to say a word to you about it. And why not?
Because the minute most users hear about something having gone wrong they like to blame EVERYTHING on whatever went wrong. Take this example:
Me: “Hi Craig, just want you to know that last night at 11PM the differential backup failed. We’re not sure why yet, but we re-ran them at 11:30PM and those worked. So, if you ever need something from Monday night and ask for the 11PM backup and you’re told that we only have one for 11:30, you don’t need to panic.”
Craig: “The backups failed?”
Me: “Yes, but we re-ran them right away, so we have a good one, just from 11:30PM, and not 11PM”
Craig: “OK then, that explains why those reports at midnight didn’t work.”
Me: “No, it doesn’t, the backups have no bearing on those reports. It’s two different servers. Your report server melted in the fire. That’s why they didn’t run. This is a different server, and a different time. There is no relation between the two.”
Craig: “But the reports use data from that server. Chances are the data they retrieved caused the fire.”
Me: “Is it too early to start drinking? I guess not, because clearly you have already.”
That’s why we don’t tell you anything. It’s because no matter what happens you will latch onto it and start blaming it for anything else no matter how unrelated or crazy. You’ll send emails to managers stating “facts” that are just not true, creating more chaos and confusion than if we had said nothing at all.
So we don’t say anything. No matter how big or small of an event.
3. It’s the network
Your DBA is likely the best looking and smartest person in your IT department. But that doesn’t mean that they are omnipotent. From time to time they are not going to have an immediate answer for you no matter how long you stand in their cube tapping your feet and reminding them how much money is being lost for each passing minute that they haven’t solved whatever imaginary problem you are having.
It’s during times like these that the most seasoned of DBAs will reach into for the modern-day equivalent of a “get out of jail free” card by turning their heads, looking deep into your eyes and saying those magical words:
“Looks like it could be a problem with the network.”
This one line is pure gold. It buys any DBA the time they need to investigate the real problem as most folks will walk out of the cube once they believe the DBA is on the trail and closer to finding a solution. And once those people leave you alone to focus on the issue you are likely to be able to find a solution.
And don’t worry about blaming the network. The network admins don’t mind, because they just blame management for not buying the right hardware the first time around. And then management tells the end users they will need to live with the bad performance until the network issues are worked out. It’s the circle of life in IT, and it works.
4. Of course your server is dedicated
Not likely, no it is not. Not only does your database server also run all sorts of other things like anti-virus software and software for taking server tape backups, but chances are it is running other services like SSAS or SSIS. And let’s not forget all those vendors that insist their application be installed on the same server as the database “for performance reasons” which I always found amusing every time we found the performance bottleneck to be the application itself. Good times.
And let’s not forget the price of all that hardware and software. As much as I enjoy having my overall CPU utilization to be less than ten percent if the guys who write the checks find out that we are wasting ninety percent of our capacity then there is a good chance we won’t ever be able to buy anything new and then you’ll have to hear your DBA say things like “this is why we can’t have nice things” each time you ask if your server is dedicated. So, just don’t bother. Your server is a resource and is likely being used by others. Get used to the idea.
5. Yes I gave you the permissions you asked for
Not only do vendors like to insist that their applications are installed on the same server as the database, but they also insist on having full administrative rights. Why are such rights necessary? Quite often, they aren’t, but vendors ask for it because they can. Any good DBA will ask for a list of actions that are needed so that they can grant the permissions necessary for those actions. I once had a vendor tell me they needed full system administrative rights for the database instance AND the O/S because they “had to create tables and stuff”.
DBAs, especially the good ones, know this isn’t true. And so we give you the permissions that you need. Quite often that is different from the permissions that you asked for. And honestly we’re too tired to explain to you the difference because you usually don’t care to understand it anyway.
There you go, five things your DBA has probably been less than truthful with you about. That doesn’t make us bad people. In fact, it makes us more like an adult…say a parent…someone who knows the difference between “want” and “need” and has to make the hard choices that others don’t want to make.
Personally I never lied about permissions granted. I prefer going into excruciating details as to why you only need those permissions and make enemies then lie. In actuality I’m a developer, not a DBA, so I write my code to require a minimal set of permissions, and document what those permissions are. I’ve had to argue with the network admins to to not be lazy and make my code run as domain admin, and DBAs to create the roles I specified.
Justin,
Too bad there aren’t more of you out there. It seemed to be that every time we brought in a vendor app we would get asked for sysadmin rights. And when we would ask for details we would essentially be told to “stop being so difficult”. But my favorite had to be the time the vendor said they absolutely needed sysadmin rights so the could “create tables and stuff”.
I probably should have just given them the rights and saved that email for the auditors.
Oooh, I forgot about #2. I remember that. Every little hiccup from the point you told people about something would be blamed on it… for weeks.
Also, the other problem I used to report when I need time was sun spots. I’d start explaining how solar activity generated radio waves and how those could interfere with data in the wires. Managers eyes would slowly roll back in their head and they’d leave me alone to fix the problem.
Yeah, if you ever said ONE LITTLE THING about what happened under the hood, EVERYTHING would get blamed on that for weeks…months even.
I love the sun spots. Wish I had thought of that one. Will be “borrowing” it in the future.
Lying is wrong, but some DBA are guilty of “lying by omission”, which is terrible. Thing are not being done, just because people do not talk about them, so, they do not have to do this.
Where you can see this a lot – automation. I have seen so many shops that just doing everything by hand and DBA team growing and becoming unmanageable – where couple dozen scripts would make everyone’s life so much easier and happier.
Lying by omission is a terrible thing.
My 2c,
Ron Warshawsky
http://www.enteros.com
http://www.linkedin.com/in/ronwarshawsky
Lie 2 I think is a bit of a tricky one. I know I’ve certainly been guilty of this one a few times. When possible I like to try to educate the customer/user but there are some times where that’s just not going to happen. So I like to think of it more as a white lie that is told in the best interests of the customer, helps me sleep better at night and all that.
“Yep…” says Matt with a sigh, “these are definitely some of the lines needed to get some joker out of your face while you attempt to fix a problem that they knew about months ago but have conveniently forgotten.”
I have found very few vendors who do not demand ‘SA’. The usual result is that they get it but I make sure to mention in a crowd (and with the high-ups of the vendor company) that using the ‘SA’ authority is regarded in the DBA world as one of the World’s Worst Practices for database management. Often I will also mention in passing to execs that if a vendor cannot perform such a task correctly, what else is going to be lacking in the product. The intention is to plant that seed of doubt so that when things go wrong and the vendor starts blaming the people who want to do the best for their own organisation, their complaints will carry less weight and the heavies will be less inclined to blame me 🙂
Matt,
I’ve tried those tactics as well. In fact, I think most folks in IT try those tactics, especially when products are purchased before they are given a thorough trial.
This is just…sorry to say so… rubbish. Yes, DBA’s see themself as “parent”, but to be honest, to provide the best service to the endusers, DBA’s and other related people should act a colleagues and not as partent teeling child, what’s good and useful.
Dba,
Yes, acting like colleagues would be a nice change of pace but that is a two way street. If the end-users behave like children, then they should expect to be treated like children.
Number 3 is actually quite often true for me.
Number 5 is a regular one. I had a vendor in a week ago saying they needed to login as SA on a live production server that had several other key 3rd party apps. All they actually needed to do was create a new database and add some users. I told them i didnt know the SA password and could only grant them the basic rights required against a login i would create for them. Needless to say they got everything done they needed to!
Paul,
It’s amazing how often they can get things done without having SA rights.
Really, a DBA should grant an application account only what he deams is necessary. If a 3rd party tool needs to create tables, then adding them to the DBO role on their database should be sufficient, however, creating tables should be necessary only for the initial installation. Maybe it also needs SQLAgentUserRole to create or kick off jobs. If it’s something like a performance dashboard tool, then it probably needs VIEW SERVER STATE or ALTER TRACE permission. However, there is no need for full SYSADMIN permission; that role is for people who administrate the server. Just like a contractor, a 3rd party tool should be granted permission based on the scope of it’s work. Imagine if you hired a contractor, and when he shows up on the first day he says:”I’m sorry, but this keycard you gave me just won’t cut it. I need access to every floor in the building, including the executive offices and the server room. You know… so I can come and go as I please, fool around with your hardware and stuff.”
Hilarious! I had to send this link to our network admin; we have a long-standing joke about this.
I have practiced lie #3 to perfection. 🙂
Thomas – I had some serious misgivings about your article, so I wrote up a rebuttal here:
http://www.saintchad.org/a-rebuttal-to-is-your-dba-lying-to-you/
Philo,
Just read your post, it is a great response and thank you for contributing to the discussion.
I’ll leave the rest of my comments over on your blog.