As my children get older I start to play more and more advanced games with them. First, we started with rattles and things that made noise. Then we moved to chasing around the room and the well-known “Tickle Monster”. Recently it has been games that involve communication skills and reasoning. The latest game I have started playing is to ask my children to compare two items and tell me what they like best.
“Do you like Dora, or Diego,” I will ask my daughter. If she answers “Dora”, I then compare Dora to some other character and the game continues. What is more exciting is when she will start asking me to compare two items of similar nature. For example, the other night she asked me “What do DBA’s despise more, vendor applications or when a decision is made to buy one without considering what it will mean to support the applications?”
Well, how am I to answer that one? I went with “vendor applications” and then got thinking about why. Why do vendor applications create such a source of frustration for my team? What is it about them as opposed to other applications and systems that we support? And most importantly, what, if anything, can be done to alleviate our frustrations?
I would say that the biggest problem we face is the lack of control. At least with our in-house systems and applications we can control the effect they have on our environment. If a programmer came to me and said “I need the sa password for the server because I hard coded the application to login as sa and it cannot be changed,” I would simply laugh, tell them no, and refer them to my manager if they did not like my answer. But with vendor apps, that is just not possible.
So, here is the point of this blog: Where are all the DBA’s? I have been told that there are tens of thousands of DBA’s in the workforce. If so, why do none of them seem to be working at the companies that make the software our business decides to purchase? Who allows their company to build applications that connect as ‘sa’, and only as ‘sa’? Who allows their company to distribute documentation saying that the application must be running with a sys admin account, when all it really needs is to be a member of the db_owner role?
I just do not understand how all of this is allowed to happen. The only thing I can think of is that these companies are building their apps with little or no help from actual Database Administrators. Well, at least not from any DBA that has an idea about security, or stability, or transaction logs, or the fact that other companies might employ a DBA of their own.
Allow me to put together two different examples of why this can be so frustrating. Each example will draw upon many real-life examples, so as to protect the innocent as much as possible.
The first example draws upon from the group of applications that I feel were written for companies that may not have a DBA on staff and they need something up and running as quickly as possible. The application will either require the ‘sa’ account in order for the install to succeed, and then will either require the ‘sa’ account (or an account with sys admin rights) in order for the application to fully function. Part of the function of the application is to run it’s own backups and maintenance by creating jobs in SQL Agent. While the jobs can be disabled in the agent, the idea that they exist means that something inside the app could trigger them to be run at some point, which could cause some issues for your own backup design for the server.
Ready for the best part? Well, you explain to the vendor that you have your own way of conducting backups, say to a dedicated backup drive, or using Litespeed, and the vendor responds by saying that their jobs must remain in place, and running, no questions. Knowing me, I always ask questions, and the first one was Why? And I get the response that I not only heard from my parents, but I give to my own children: Because, that’s why. In other words, they really have no idea, but they don’t want you to fool with anything because it makes it more difficult for them should they be called upon for support.
Hey! What about when we are called for support! Did anyone think of that?
The other example draws upon the idea of support. The vendor gives you an application and things are running fine for a while. No need to run as ‘sa’, no extra jobs loaded to your server, no need for special trace flags for your queries to run correctly because you don’t want to code them correctly, no need for anything special to be done to the server. Now, I like these applications just fine. Up until one of two things go wrong. Either some custom code has been placed into the vendor app (or alongside) and it is causing issues with the applications performance, or the application itself starts to have performance issues over time, issues that are not a result of simple fragmentation or query plans, but issues related to design.
You may see an error message that the transaction log is full, and get a phone call to increase the size of the log. Again, I ask questions. In this case, I asked: What are you trying to do? The answer? “Just tell your DBA to increase the size of the log by 1Gb, this is not an unusual request.” Really? For a 6Gb database that already has a 3Gb log file, I do find it somewhat unusual. If I knew exactly what you are trying to do, I may be able to size the log properly the first time around and not have to keep increasing it each time you press a button.
Please forgive me for trying to do my job efficiently, or for trying to help understand more about your application, or your needs. Perhaps I should just sit here and blindly do everything I am told? I think not.
So, which do I despise more? Neither, it’s a tie.
And how to fix things? Well, getting invited to the table early on in the decision making process would be crucial. But getting there is only half the battle, at most. After being there you may still find out all your fears are going to come true. At that moment you need to do your best sales job to those making the decisions to not make the purchase. You need to detail all the reasons why, and make certain they understand fully the hidden costs for everything. In the end, they may still decide to make the purchase, which means you will be stuck with something, but at least you will have a better idea of what you are up against from the start.
I completely agree with your article and share many of your frustrations. In my recent experience with just such a vendor, I was told that they can’t give me more information because the information was ‘bordering on proprietary’ information.
In another discussion with one of their support staff I discovered that many of them there, at this small company, were underpaid and were not allowed any training at all.
I suppose a lot of small upstart companies feel that they simply cannot attract a DBA with their limited resources.
They might have a great business application, but in todays competitive environment, you really need to plan on providing a very well-rounded and robust application. There are more and more choices every month, and consumers are getting more discerning and educated.
The small software companies need to decide early on how to craft their business philosophy, or whether to do that at all. It’s always tempting to take the easy road, but the successful companies typically are not willing to compromise on the integrity of their vision.
Bill~