You can use the following code to help get you those details. I used a CTE, but feel free to modify it however you want. I also included a parameter for the minimum number of seconds you would want included in your evaluation. For example, you may not care about backups that run in two seconds, have a standard deviation of one-half a second, and suddenly one day it takes five seconds. That choice is yours, really, so that is why I included the parameter.

Also worth noting is that the backupset table has a column for backup size. In theory you could modify this code to focus on the size of the backup as opposed to the length of time it takes to run. I didn’t go down that route because I was asked to focus on time, not size (and I have always been told that size doesn’t matter anyway).

I think the most important lesson here is that you could use code like this to find other outliers. Anything that has an average that can be calculated, really. I’m thinking about things like query run times, of course…or even the amount of logical I/O for a query. Yep, just about anything.

declare @MinAvgSecsDuration int = 2
;
WITH BackupHistData AS
(
  SELECT database_guid, type, MAX(backup_set_id) AS [MAX_BSID]
	,AVG(CAST(DATEDIFF(s, backup_start_date, backup_finish_date) AS int))
		AS [AVG]
	,STDEVP(CAST(DATEDIFF(s, backup_start_date, backup_finish_date) AS int))
		AS [SIGMA]
	FROM msdb.dbo.backupset
	GROUP BY database_guid, type
)
SELECT bup.database_name, bup.backup_set_id, bup.type
	,CAST(DATEDIFF(s, bup.backup_start_date, bup.backup_finish_date) AS int)
		AS [backup_time_sec]
	,bhd.[AVG] as [avg_sec]
	,(1.0*bhd.[AVG]+2.0*bhd.SIGMA) as [max_duration_sec]
FROM BackupHistData bhd
INNER JOIN msdb.dbo.backupset bup ON bhd.database_guid = bup.database_guid
/*Filter for the outliers*/
WHERE CAST(DATEDIFF(s, bup.backup_start_date, bup.backup_finish_date) AS int) >
			(1.0*bhd.[AVG]+2.0*bhd.SIGMA)
/*Filter for only the most recent backup, if desired*/
AND bup.backup_set_id = bhd.MAX_BSID
/*Filter for backups with an average duration time, if desired*/
AND bhd.[AVG] >= @MinAvgSecsDuration
/*Filter for specific backup types, if desired*/
AND bhd.type IN ('D', 'I', 'L')

 
DISCLAIMER: Do not run code you find on the internet in your production environment without testing it first. Do not use this code if your vision becomes blurred. Seek medical attention if this code runs longer than four hours. On rare occasions this code has been known to cause one or more of the following: nausea, headaches, high blood pressure, popcorn cravings, and the impulse to reformat tabs into spaces. If this code causes your servers to smoke, seek shelter. Do not taunt this code. 

Good luck.

The post How To Find Long Running Backups in SQL Server appeared first on Thomas LaRock.

http://en.wikipedia.org/wiki/User_story

Apparently a user story is nothing more than using words to describe the requirements for someone else to use when building software. I also noticed the mention of using user stories with Agile software development. And I had to stop and think to myself, “The Hell? Do these people think this is new?” If you judge everything just from this wikipedia entry, you might think that a user story is some revolutionary new idea that has evolved from Agile within the past ten years. Well, it isn’t.

Go back to Binoogle and look up “Voice of the Customer” or “VOC”, and check on any Six Sigma references. See, getting feedback from customers, or users, has been around for a long, long time. Probably for centuries. Yet it would appear that only recently have some in the software sector suddenly found out that getting such feedback could actually streamline the development process. Talk to the user before I build something? Genius!

Not sure why I am surprised by this fact but apparently everything old is new again. Even in software development.

The post User Stories appeared first on Thomas LaRock.

“Papa, why [whatever]?”

“I don’t know.”

“Why?”

“Because I don’t know everything.”

“Why?”

“Because no one person can know everything.”

“Why?”

“I don’t know.”

And we would continue that loop a few times until they grew tired of getting the same answers about how I don’t know everything and no one person can know everything. Turns out that my five and six year old were only just showing that they have incredible skills when it comes to Root Cause Analysis (RCA). Asking “why?” is a fundamental part of an RCA as it can help to identify potential root causes.

5 Whys

The ‘5 Whys’ is a technique used in RCA and Six Sigma and one we reviewed yesterday during training. I believe that all too often we find ourselves affixing band-aid solutions to symptoms, especially in IT, as opposed to finding and fixing the root cause. The process is quite simple, you just keep asking “why?” until you get to the root cause. So, you may find something like this scenario:

My dedicated log disk filled up last night. Why?

Because someone tried to load ten million rows all at once instead of batching the load into smaller chunks. Why?

Because the package they use for their ETL process does not include logic to batch a transaction size. Why?

Because they do not understand where the option is, or why it is important. Why?

Because no one has explained it to them. Why?

Okay, fine, I’ll go over there after I finish my coffee if you promise to leave me alone. Why?

Stop it. Leave me alone. Why?

And so on until your head will explode. Typically after asking “why?” for the fifth time you might be at the root cause but asking five times is only a rule of thumb. Sometimes you need to ask more, sometimes you can ask less. The key here is to drill down into discovering a potential root cause. You should not consider this tool to be able to easily identify the root cause each and every time. In fact…

Why 5 Why May Not be Effective

Look, no tool is perfect. And the ‘5 Why’ tool is not all unicorns and rainbows either. As much as I can appreciate the simplicity of the technique the fact remains that it has a couple of glaring issues. First, it is not repeatable. That means that two different people may have two very different results. Consider the example above, but this time from the viewpoint of application support as opposed to server support.

My dedicated log disk filled up last night. Why?

Because someone tried to load ten million rows all at once instead of batching the load into smaller chunks. Why?

Because the bank sent us a huge ass file instead of the normal file. Why?

Because their process created a huge extract by mistake. Why?

I don’t know, but they said it would not happen again. 

So, a different perspective can lead to a different conclusion altogether. To me that demonstrates that the tool itself is subjective to the user, which means it may not be ideal. However, there is something positive from that as well, and that is if you get the different people together in a room you can start to list out all potential root causes and work together on a solution rather than in silos.

Another issue with the tool is that people are only able to find causes they already are familiar with. In the two examples above we reached two different conclusions, but each conclusion was the result of that persons viewpoint and experience. In other words, you know what you know and you don’t know what you don’t know, and the ‘5 Why’ tool will not give you insight into things you don’t know. This also helps to explain why the tool is not a repeatable tool.

A third drawback would be that as a result of the tool only allowing people to find causes they are familiar with means that they will often times stop at symptoms instead of continuing on to the actual cause. All three of this drawbacks I have listed are really just variants of each other, but they are all valid considerations to make when using this technique.

I think the most important aspect to all of this would be to remember that you are only looking for potential root causes. At best, that is what you will find with this tool. And if you keep in mind that the tool is only going to help you get closer to your desired goal, and not the tool that definitively gets you there, then you should be fine. Just consider it as one piece of the puzzle, and not the whole picture.

The post Root Cause Analysis – Asking Why? appeared first on Thomas LaRock.

Fortunately for me I was reminded by Louis Davidson that it is not the feature I dislike, it is the abuse (or misuse) of that feature. This would not be the first time that Louis has reminded me that I was (or about to) make an ass of myself, so I took note of his words very carefully. Indeed, it is not necessarily database triggers that I hate, but rather the fact that they are abused…misused…actually, I hate the fact that they are used and it is so difficult to troubleshoot.

But NULLs, well, they are a different animal. Why do I say that I hate NULLs? I mean, there is nothing to them, right? They do not perform any actions like triggers, they just sit there, in your tables, doing nothing. Some of them are waiting to be discovered and some hope to be updated to have real value. It makes me think of that old Schoolhouse Rock video about that little scrap of paper. So, why so much dislike for them? What harm do they cause?

Why NULLs Can Be Harmful

The harm they cause is usually the result of a person not knowing they are there. It’s that simple, and that is why I hate them. OK, not hate, but I would rather design a database with as few NULL values as possible.

Oh sure, I understand that there are times when you do not know the value of something, and it is easy to simply define that particular column to allow for NULL values and you will come back later to update that value. Sure, sounds great. I always hear about people that plan on coming back later to fix the things they didn’t do right the first time around. I only wish I could say that I actually see that happening as often as I hear it is going to happen. So, you put a column into a table, allow it to have NULL values, and are asking yourself right now “WTF? What’s your problem?”

Again, it is because the end user is not always aware that NULL values exist. And sometimes, the end user is a developer (*ahem*) who may be brought into the middle of a project and be building some widget only to have the widget crap out once a month because their code never accounted for NULL values that get inserted (or updated) after a monthly batch load. Not that I know anyone that has ever experienced such things, but I hear about these people (*ahem*). What’s that? Oh, so my hatred for NULLs is really a hatred for the fact that I am not  someone is not good enough at coding or T-SQL or database design to begin with? Maybe, but consider this…

Query Tools

What tool do you use to write a majority of your T-SQL queries? OK, do you know what the default connection properties are? What I am looking for is: Do you know what your ANSI_NULLS is set to? How about this question: Do you even know what ANSI_NULLS setting you want to use? Did you know you even had that option? Do you know the difference in behavior if ANSI_NULLS is set to ON or OFF? I have a handful of tools at my disposal and surprisingly they have different default settings when making connections to an instance. Some of those differences can be with ANSI_NULLS, which makes no sense to me because the whole idea around ANSI is to have a standard so you would think the rocket surgeons in charge would have thought about mandating a default ANSI_NULLS setting whenever someone makes an application that wants to connect to an database server in order to avoid crap like I have had to deal with. Sure, you want to be able to change the option, but the behavior out of the box should be a standard, in my opinion.

Now, how many DBA’s know everything there is to know about ANSI_NULLS, connection strings, the various applications being used to connect to data these days, and each and every database design in their shop? I have no idea how many of you just raised your hand, but I know mine stayed on my keyboard. Now, let’s go one level away from the DBA, and ask: how many customers understand all of this? I supposed that you could only have your users connect to pristine cubes or data marts, right? Good luck with that, because I know most people want to connect to the raw data so that they can go through it themselves. And guess who will get blamed if it turns out that data is “missing” because the end user missed the fact that a NULL value altered their result set?

So, if database designs minimized the use of NULLs, then we can minimize the amount of issues faced downstream. Ever since I took part in some Six Sigma Black Belt training I have come to understand that what I really hate are inefficient processes. The use of and troubleshooting triggers is inefficient. And the same case can be made for NULL values. Their use can lead to a lot of bad result sets being accepted as a “version of the truth”, which is something we will ultimately be on the hook for at some point. Why not save people some time by looking to minimize their use? That would seem to be more efficient for everyone.

So, maybe I don’t hate NULLs. I just hate how so many groups are all operating under the assumption that everyone knows what they are, why they are there, and how best to use them. Seems to me that with all of those assumptions chances are lots of bad business decisions are being made every day.

Wait a minute…maybe we could blame the economic downturn on the use of NULL values? Hey, maybe I found a reason to love the little buggers after all.

The post SQL Server and NULL Values appeared first on Thomas LaRock.

Naysayers will always exist, no matter what.

The post Six Sigma Blogs appeared first on Thomas LaRock.

By now your first question is most likely: why Congress? Well, it is simple. Have you ever been invited to go somewhere, say golfing or a ballgame, and found that you need to check to see if you can get the time off? Not only from work, but from your family as well? My group of coworkers started to refer to our better halves as “Congress”. That way, we could say things like “I wanted to go to the ballgame, but I couldn’t push that bill through Congress”. Or, “I was hoping to buy that new [whatever], but Congress squashed the funding. Next time I will need to hire some lobbyists”. There, I hope that clears everything up for you. Back to the topic that matters.

So Congress asks me for some help putting together a brochure of sorts to be used at an upcoming trade show. She starts by explaining that she wants the customer to feel a certain way after reading the brochure. With my recent Six Sigma training, I translated her words directly into a few acronyms and settled on VOC (Voice Of Customer). At that instant I got up, went to my desk, grabbed a stack of post-its, and came back to the kitchen table.

“We need to gather as many one-word descriptions that you want to convey to your customers in this brochure.” Congress immediately wanted wanted to know what the hell was going on, since she had never seen me so focused on slapping post-its to the kitchen table before. So we spent about twenty minutes coming up with roughly fourteen descriptions that we wanted to convey, things like “professional”, “easy”, “beauty”, “service”, etc.

The next step was to see if the descriptions had a natural affinity. In other words, to group the descriptions together. Once that was done, we went about putting together some actual sentences. We got two or three done fairly quickly, and then decided to benchmark against similar companies by reading their brochures. In about two hours we had all of the meat there, and then spent the rest of the night haggling about order, flow, etc.

The whole experience was eerily similar to what I had read about in the Jelly Effect. In that book, Andy Bounds describes a process for putting together better presentations, and the methods he uses are similar in nature to what we did last night. I have been trying to follow the advice in the book in order to help me put together my own talks, and I think it has helped. To have his suggestions reinforced by techniques learned in Six Sigma training is wonderful. To find myself actually putting them into practice is amazing.

A long time ago I was asked if I would rather be a brick or a sponge. Bricks are hard to change, but sponges can be altered by simply putting them into a bucket of water. Sure, it is still a sponge, but it is also something more than it once was.

I have often been told that I am a sponge, that I absorb a lot around me, and that is one of the reasons I made a good Database Administrator. It is also one of the reasons why I can find myself getting thrown into difficult situations in life but manage to make my way out. To me, the experience last night is just further evidence of my being a sponge, and one of the reasons why I hope to be able to apply everything I learn in Six Sigma training to my everyday lives.

The post Six Sigma, Affinity, Sponges, and Congress appeared first on Thomas LaRock.

Now, before I go any further, I want to make it clear that I try my hardest to avoid using acronyms whenever possible. Six Sigma has acronyms for everything. There is DMAIC, SIPOC, VOC, VOP, CCR, CTQ, and the list goes on. Many of these terms are only found in a Six Sigma context and it can be hard to follow your first time through. That being said, I will do my best to make certain I explain the acronym before just assuming that the reader knows what it means.

DMAIC stands for Define, Measure, Analyze, Improve, and Control. Those five elements comprise most every type of Six Sigma project (I actually do not know of any that would not have them, but anything is possible). At some point during the week I started thinking about how most DBAs I know already apply these five principles to most of their everyday tasks.

For example, let’s assume that someone (call it a customer), contacts you (as a DBA) and says something akin to “The system is slow. It needs to be faster. Please fix.” Now, being the top-level DBA that you are, you get to work. And what do you do?

First step is to define the problem. You start asking questions, trying to get as many details as possible. What system? Which server(s)? Which databases? What do you mean by faster? When did you want/need this done?

Then, you measure the problem by seeing for yourself how “slow” the system is responding. Again, you ask more questions, looking to further define the issue. Most DBAs just lump this into “Performance Tuning” for the example we are discussing, but in general you would look to gather your own sets of measures for whatever the problem may be. Without any measurements, how would you ever know if things got better, right?

Then, you analyze your results. In most cases the analyze phase takes place during the measure phase, but the best DBAs (in my opinion) would gather as much information as possible, take a step back, and then analyze the data in front of them. Your analysis would lead you to an action that you would take in order to do what?

Well, improve the system, which is the very next step. You would apply your analysis in the previous step, formulate a plan of action, and implement the action in an effort to improve the situation. Perhaps an index is missing, or needs to be rebuilt, for example.

Lastly, you would put a control in place to help identify if the system acts slowly again in the future. Perhaps the controls involve configuring a new set of metrics to help you with any additional measures and analysis that needs to be done if things break again. Or your control could be as simple as a report that shows how long the system is running. That way if the customer calls again to say it is slow you can respond back with “our control report says it took five seconds less than the average run time, so any slowness  is either due to a different issue or your (lack of) perception”.

I could go on, but the point here is that many of the Six Sigma processes are built upon common sense (which is not always common), and many of the DBAs I interact with already act in a similar fashion to those processes. Many of us follow a DMAIC process without knowing anything about Six Sigma. And while not every DBA could simply jump into a Black Belt program (although if I can do it then I am 99 and 44/100% certain anyone could), chances are most DBAs would find the transition quite easy. If nothing else, you would be able to follow the logical nature of the processes and probably not be scared by the math involved.

If there is one thing that almost every Black Belt has that most DBAs need, it is a good set of soft skills. If I can take away tips and tricks for a better set of soft skills by the time this is all done, that will be as valuable as anything else I may learn along the way.

The post DMAIC Is For Everyone appeared first on Thomas LaRock.