Quick question: Where would you go to find some basil in your local grocery store? Did you say “by the other herbs?” Or did you say “by the tomatoes?” If you are like me, then you would go looking for basil by the other herbs because, well, that is a natural, logical grouping of products. However, since many people tend to associate basil and tomatoes together, like for a pasta sauce, then some stores may locate basil there instead. Many stores will do both, but some stores that are short on shelf space could decide one over the other.

So, I was “that guy”. Yeah, the one in a million that might actually need basil for a recipe that did not call for tomatoes. So, I went to the section where all the herbs are located and did not find any basil. I then had to report back to Congress that I had failed, that the mission was not accomplished, and we went ahead with dinner plans even though we knew that our meal was not going to be complete.

Why am I telling you this? Good question. Any other questions?

You will hear parts of business intelligence (or business analytics) that are called “Predictive Analysis”. An example of this is the business of product placement in grocery stores, such as basil and tomatoes. When enough people buy things together, stores will start to associate products and group them in displays. Hey, it makes sense, and I doubt the idea of basil and tomatoes being located together is anything new that required a 1.7TB cube to be processed in Excel for some grocery executive. But what is interesting is the idea that you can examine (what they call “data mining”) your data to look for trends so that you can predict the likelihood of future events.

Which is where I call bullshit.

See, I have worked off and on in the financial world for over fifteen years now. And there is one thing you learn right at the start, and that is this:

Past performance does not guarantee future results.

In other words, you cannot predict the future. Yet the title “Predictive Analysis” would lead to you to believe otherwise. There are thousands of business leaders out there right now, many of them in the financial world, that think if they have enough data they can predict the future. And that is just not true. Take my case for example, I just wanted basil. Sure, perhaps you could argue that I fell outside of your 95% confidence interval, and that outliers will always exist, and that your models work a majority of the time. But the end result is the same, you cannot predict events no matter if you have a crapload of data at your disposal or not. [I think some folks like to just stop and call that "free will".]

When I was participating in Six Sigma training we spent considerable time on a section called, quite simply, statistical analysis. As near as I can tell, there is little difference between regular statistical analysis and predictive analysis except for (1) the names and (2) the idea that someone wants you to believe that you can predict the future. In fairness, I can provide you two examples of where I see what could be a reasonable case made for the use of predictive analysis. One is Amazon, and the other is Netflix. Both services do a great job in trying to sell you additional products based upon either your recent selections or the selections of other people that made similar choices. If you want to call this predictive analysis, fine. I call it suggestive selling, based upon statistical analysis, but perhaps that is because I understand you cannot predict the future.

Need another example? Well, check out my library bookshelf. I started placing reference books onto my site, and yes they link back to Amazon. If someone stops by and orders a book I get roughly a dollar for providing the link. Not big money, and no I am not looking to get rich, I just wanted to buy a cup of coffee every now and then. So, in less than a week, no books have been sold, which is to be expected. But what was not expected was the fact that someone decided to browse Amazon a little bit and ended up buying this.

Yeah. And I want to see the analysis that predicted that correlation between that and a book on SQL 2008.

Want to go one step further? How about the fact that most companies compare results to previous quarters and years? How ridiculous is it that you would compare your Q1 results to the previous year, without any other information? Or to compare the results from one region to another? You simply cannot make important decisions on such information without a consideration of all the external factors.

“There are three kinds of lies: lies, damned lies, and statistics.” -Benjamin Disraeli

I’ve written before about how much I love looking for outliers. One example of this can be found in the post I did on the sysjobhistory table. Recently I was asked to put together some code to find out which backups were “running long”. Of course my first question was trying to define what was meant by “long”. That’s when I decided to simply define “long” as any time that is more than two standard deviations away from the mean.

You can use the following code to help get you those details. I used a CTE, but feel free to modify it however you want. I also included a parameter for the minimum number of seconds you would want included in your evaluation. For example, you may not care about backups that run in two seconds, have a standard deviation of one-half a second, and suddenly one day it takes five seconds. That choice is yours, really, so that is why I included the parameter.

Also worth noting is that the backupset table has a column for backup size. In theory you could modify this code to focus on the size of the backup as opposed to the length of time it takes to run. I didn’t go down that route because I was asked to focus on time, not size (and I have always been told that size doesn’t matter anyway).

I think the most important lesson here is that you could use code like this to find other outliers. Anything that has an average that can be calculated, really. I’m thinking about things like query run times, of course…or even the amount of logical I/O for a query. Yep, just about anything.

declare @MinAvgSecsDuration int = 2
;
WITH BackupHistData AS
(
  SELECT database_guid, type, MAX(backup_set_id) AS [MAX_BSID]
	,AVG(CAST(DATEDIFF(s, backup_start_date, backup_finish_date) AS int))
		AS [AVG]
	,STDEVP(CAST(DATEDIFF(s, backup_start_date, backup_finish_date) AS int))
		AS [SIGMA]
	FROM msdb.dbo.backupset
	GROUP BY database_guid, type
)
SELECT bup.database_name, bup.backup_set_id, bup.type
	,CAST(DATEDIFF(s, bup.backup_start_date, bup.backup_finish_date) AS int)
		AS [backup_time_sec]
	,bhd.[AVG] as [avg_sec]
	,(1.0*bhd.[AVG]+2.0*bhd.SIGMA) as [max_duration_sec]
FROM BackupHistData bhd
INNER JOIN msdb.dbo.backupset bup ON bhd.database_guid = bup.database_guid
/*Filter for the outliers*/
WHERE CAST(DATEDIFF(s, bup.backup_start_date, bup.backup_finish_date) AS int) >
			(1.0*bhd.[AVG]+2.0*bhd.SIGMA)
/*Filter for only the most recent backup, if desired*/
AND bup.backup_set_id = bhd.MAX_BSID
/*Filter for backups with an average duration time, if desired*/
AND bhd.[AVG] >= @MinAvgSecsDuration
/*Filter for specific backup types, if desired*/
AND bhd.type IN ('D', 'I', 'L')

 
DISCLAIMER: Do not run code you find on the internet in your production environment without testing it first. Do not use this code if your vision becomes blurred. Seek medical attention if this code runs longer than four hours. On rare occasions this code has been known to cause one or more of the following: nausea, headaches, high blood pressure, popcorn cravings, and the impulse to reformat tabs into spaces. If this code causes your servers to smoke, seek shelter. Do not taunt this code. 

Good luck.

You have no idea that what you are not doing is causing you pain.

Then again, how could you know this? I mean, if you are not doing it, why would you ever think that your inaction could be the very thing that is causing you heartache every time your phone rings in the middle of the night?

I see DBAs make these three mistakes repeatedly. But they are not a mistake as you might think of a mistake: a misstep along a specified process. No, these mistakes are brought about because the person simply didn’t know any better.

But they should.

I’m hoping by listing them here, it will help.

1. You Don’t Have A Recovery Plan

The number one reason for a DBA to exist is this: recovery. It should be your primary focus each and every day. You need to be able to recover from any event, no matter how big or small. Lose an entire data center? You had better be able to recover that data. Lose a handful of LUNs? You had better be able to recover that data. Someone deletes a table? You had better be able to recover that data.

So why then do I see many DBAs spend time focusing on a good backup strategy but never spend an equal amount of time on a recovery strategy? Of course we know that every good recovery strategy must start with a good backup strategy, but so many DBAs (especially novice ones) never bother to test their restore process. Are your backups written to disk, and then saved off to tape? OK, that’s fine, now…quick…how long will it take to recover that tape? If your end user calls and says they need to recover a database from last week, how long before that request can be completed. Here is a better question: how long before your end user become very annoyed that they have to wait any longer?

There is only one way to be certain that the backups you have can even be used for recovery: you need to actually recover them without error to be 100% certain. That’s why I wrote this article for Simple-Talk two years ago on how to use statistical sampling in order to be 95% confident that all your backups are valid.

Taking backups is not even half of what you need to do as a DBA. You need to recover, from a variety of failures, and you need to be able to do it as quickly as possible. Having a plan is good, but practicing that plan is even better.

2. You Pretend That Virtualization Doesn’t Matter

I know that VMWare will tell you that running in a virtual environment can be up to 99% of the performance you have in a physical environment. I also know that when people see that 99% number they decide that they can shove as many guests onto a VMWare host as space will allow for and they still believe they will have 99% performance for each and every instance. This is especially true if those instances are believed to be “lightly used”.

The beauty of running your servers in a virtual environment is that you get to share resources and easily migrate guests between hosts. The ugly truth about running your servers in a virtual environment is that you get to share resources and easily migrate guests between hosts.

As a DBA you are now managing a piece of software that runs inside of a guest O/S that is running on a host, that has a hypervisor, and that likely shares its disks with other hosts as well. What that means is that when someone runs a query, that query has to travel down that stack and back again in order to return a result. Now, if someone comes to you and says that they are having a performance issue what would you do first? Would you look at the query? Or the O/S metrics for that guest? Or something else?

Or would you prefer to spend five minutes examining the virtual layer to see if any bottlenecks exist. For example, if the host is seeing a spike in disk latency, then it is likely the guests are seeing disk latency as well. So, how much time would you want to spend trying to make a query faster when the issue has to do with an overloaded disk subsystem? I know what my answer is: none. What if the bottleneck is host CPU saturation? How long will you want to spend trying to tune a query when you already know that the issue is at the host layer?

Pretending that virtualization doesn’t matter for your database servers is a mistake I see being made by both DBAs and server admins. Those counters hold the key to being efficient at performance tuning in a virtual world. But they are only useful if someone knows to be looking at them.

3. You Don’t Configure Memory Settings Properly

This one drives me crazy.

I still see database servers with the default memory settings. When a customer comes to me and complains about having memory issues one of my first questions is: what are your default settings?  At this point I am shocked when someone doesn’t say “we left those alone”. Configuring your default memory setting is one of the easiest thing any DBA can do and should be part of a standard checklist when rolling out a new instance. Yet it still is not happening.

Things get even trickier with memory in a virtual world, as you are competing with other guests for host memory. You simply must know what your memory requirements are for your database server and then work on securing that memory. The easiest way to do this is to configure those memory settings. Even I can’t remember everything, so that’s why I just reference this post when I need general guidelines regarding the configuration of the min and max memory settings.

There you go, three things that you are likely not doing today and are also equally likely to be causing you and your end users lots of pain.

Which one of these orders more bacon?

No matter what our profession, we all have our favorite set of tools to help us get the job done.

Database Administrators will have a set of scripts that we use in order to gain insight into the server instances under our care. I often see beginner DBAs look to the counters provided by Perfmon (AKA, the Resource and Performance Monitor). The trouble here is the counters themselves don’t necessarily give you enough details to make an informed decision about what actions you should take next. You need to look at a handful of the counters at any given time (or over a period of time) in order to better ascertain where the bottlenecks exist.

Here is a list of five metrics that I find myself frequently using when troubleshooting performance issues. They help me get insight quickly into some of the deep recesses of SQL Server so that I can easily corroborate with standard Perfmon counters in order to troubleshoot issues for customers. Ideally these are all being collected over a period of time so that you can do some proper trend analysis.

These are the metrics, in no particular order, that you should be using during the first few minutes of analysis. They will help you save time.

1. Signal Waits Percentage

This metric has provided the most value personally to me over the years. I started using it four years ago when troubleshooting CPU issues. The instance itself was showing signs of “slowness”, but the overall CPU utilization numbers were nominal. I came across this article by Tom Davidson and found out that I could calculate the percentage of signal waits to the overall waits in order to understand internal CPU pressure exists. [A "signal" wait means that a session is in the "runnable" queue, waiting for a signal from an available processor.]

I filter out some background tasks for this query as well. The value you want returned here is for no more than 20-25% of your total waits to be signal waits. If you are consistently seeing numbers greater than 20% then you are having internal CPU pressure. You can remedy the situation by reducing the number of sessions (not always likely), increasing the number of available CPUs (also not likely), or reducing the amount of time the queries need to execute (often very likely, and sometimes easily done).

SELECT (100.0 * SUM(signal_wait_time_ms)/SUM (wait_time_ms)) AS [SignalWaitPct]
FROM sys.dm_os_wait_stats
WHERE wait_type NOT IN (
'CLR_SEMAPHORE', 'LAZYWRITER_SLEEP', 'RESOURCE_QUEUE', 'SLEEP_TASK',
'SLEEP_SYSTEMTASK', 'SQLTRACE_BUFFER_FLUSH', 'WAITFOR', 'LOGMGR_QUEUE',
'CHECKPOINT_QUEUE', 'REQUEST_FOR_DEADLOCK_SEARCH', 'XE_TIMER_EVENT',
'BROKER_TO_FLUSH', 'BROKER_TASK_STOP', 'CLR_MANUAL_EVENT', 'CLR_AUTO_EVENT',
'DISPATCHER_QUEUE_SEMAPHORE', 'FT_IFTS_SCHEDULER_IDLE_WAIT',
'XE_DISPATCHER_WAIT', 'XE_DISPATCHER_JOIN', 'BROKER_EVENTHANDLER',
'TRACEWRITE', 'FT_IFTSHC_MUTEX', 'SQLTRACE_INCREMENTAL_FLUSH_SLEEP',
'BROKER_RECEIVE_WAITFOR', 'ONDEMAND_TASK_QUEUE', 'DBMIRROR_EVENTS_QUEUE',
'DBMIRRORING_CMD', 'BROKER_TRANSMITTER', 'SQLTRACE_WAIT_ENTRIES',
'SLEEP_BPOOL_FLUSH', 'SQLTRACE_LOCK')
AND wait_time_ms <> 0

2. SQL Compilation Percentage

Another sneaky performance issue is having a high number of statements being compiled. There happens to be a performance counter for compiles but the counter itself does not have much meaning unless you also know the overall number of batch requests. Rather than look at either of those counters separately what you want to do is look at their ratio.

The recommended percentage for compilations is roughly 10% of the total number of batch requests. The following code will show you the current percentage of compilations to the number of batch requests:

SELECT 1.0*cntr_value /
(SELECT 1.0*cntr_value
FROM sys.dm_os_performance_counters
WHERE counter_name = 'Batch Requests/sec')
AS [SQLCompilationPct]
FROM sys.dm_os_performance_counters
WHERE counter_name = 'SQL Compilations/sec'

At this point someone usually asks “what about re-compilations”? Well, re-compilations should only be roughly 1% of the total number of batch requests, so that code would be like this:

SELECT 1.0*cntr_value /
(SELECT 1.0*cntr_value
FROM sys.dm_os_performance_counters
WHERE counter_name = 'Batch Requests/sec')
AS [SQLReCompilationPct]
FROM sys.dm_os_performance_counters
WHERE counter_name = 'SQL Re-Compilations/sec'

3. Page Lookups Percentage

What I like most about this metric is that it often gives me a hint regarding plan cache bloating without me having to dig through the plan cache itself. I also get insight on the queries hitting the instance, to see if they could use some tuning/indexing as there are likely some inefficient query plans being utilized.

What this metric really shows me is if I have one of two extremes. Either I have a few rather large queries hitting my server, or I have a lot of little queries (probably ad-hoc) hitting my instance and likely leading to some plan cache bloating.

What I want to see here is a value that is less than 100 on average:

SELECT 1.0*cntr_value /
(SELECT 1.0*cntr_value
FROM sys.dm_os_performance_counters
WHERE counter_name = 'Batch Requests/sec') 
AS [PageLookupPct]
FROM sys.dm_os_performance_counters
WHERE counter_name = 'Page lookups/sec'

4. Average Task Counts

I got this metric from Glenn Berry (blog | @GlennAlanBerry), and I use this one to determine what my system has been busy doing. I like how Glenn always has a brief description for what you want (or don’t want) to see. In this case it is as follows:

• Sustained values above 10 suggest further investigation in that area
• High Avg Task Counts are often caused by blocking or other resource contention
• High Avg Runnable Task Counts are a good sign of CPU pressure
• High Avg Pending DiskIO Counts are a sign of disk pressure

I can use this one in conjunction with others to get some corroborating evidence as to what is happening under the covers:

SELECT AVG(current_tasks_count) AS [Avg Task Count],
AVG(runnable_tasks_count) AS [Avg Runnable Task Count],
AVG(pending_disk_io_count) AS [Avg Pending DiskIO Count]
FROM sys.dm_os_schedulers WITH (NOLOCK)
WHERE scheduler_id < 255 OPTION (RECOMPILE)

5. Buffer Pool I/O Rate

The most misunderstood counter I see is Page Life Expectancy. I know I used to pay attention as if it meant something. Over time I have come to realize that it is a throwback to days of yore when having 4GB of memory was a really big deal. In other words, its intended purpose no longer has the same value as it once did. It is not totally worthless, however. You just need to know what to do with it.

What you really want to see is the rate at which your pages are being cycled through the buffer pool. The following code returns the “Buffer Pool Rate” in MB/sec. I usually look for rates around 20MB/sec as a baseline. Why that number? Because if I have a “typical” server with 56GB of RAM available for the buffer pool (thank you, locked pages in memory) and I want to keep my pages around for an hour or so (3600 seconds), then I come up with 56000MB/3600sec, or about 15.5 MB/sec. That’s why I look for a sustained rate of about 20 on average, and if I have a spike upward from there then I know I am having memory pressure (pressure that might otherwise fail to be seen if I only examine the PLE counter).

Here is the code I use:

SELECT (1.0*cntr_value/128) /
(SELECT 1.0*cntr_value
FROM sys.dm_os_performance_counters
WHERE object_name like '%Buffer Manager%'
AND lower(counter_name) = 'Page life expectancy')
AS [BufferPoolRate]
FROM sys.dm_os_performance_counters
WHERE object_name like '%Buffer Manager%'
AND counter_name = 'total pages'

You should also be aware that if your server is NUMA aware then you will want to take that into consideration whenever you try to use PLE as a performance metric.

There you go, the five metrics I utilize for customers whenever they need my help in solving performance problems. I’m always looking to add to my toolbox, so if you have a favorite metric you want to share just leave it in the comments below.

As a system admin, DBA, or a developer it is our job to root out performance problems quickly. The trouble is that there are so many things that can cause performance issues it can be difficult to know every possible root cause. This is especially true when the root cause lies somewhere outside your area of expertise or control.

It is also true when the root cause is right under our noses the whole time.

Here are six things that I have seen cause performance issues for end users. None of them are complex by nature. Each is something that is simple and easily addressed when they are known.

So that’s why I wrote this post, in an effort to get these into the mainstream. If you are not taking steps to to mitigate these possible performance issues, start now. If you find yourself one day sitting at your screen and scratching your head wondering “what’s wrong”, then take a chance that one of these six things could be the culprit.

1. Configure your memory settings

I see this scenario all the time: Customer notices they are low on memory. They call me. I ask if they have configure the max memory setting to a non-default value. They say no.

Lather, rinse, repeat. I used to count the number of times this would happen to me but I stopped after the first hundred or so. It is one of the reasons I put together a talk on SQL Server memory management last year, because it just seemed to be the easiest thing for people to be doing and yet they weren’t doing it.

I wrote a post a while back to help explain this fact as well. And then there is this post by Glenn Berry (blog | @GlennAlanBerry) and one done by Jonathan Kehayias (blog | @SQLPoolboy), and yet still people don’t know to configure these settings. Take the few minutes you need to adjust your settings and you will reduce the chances of having performance problems due to memory settings.

2. Optimize tempdb

This one is often overlooked as most folks wait around for a problem with tempdb before thinking about how best to optimize it for performance right from the start. This SQLCAT link mentions this as well, and there is info over at MSDN that you should review before making changes. And yes, you should consider moving tempdb to it’s own dedicated set of drives.

In fact, I would tell you that the placing of data files, log files, backup files, and tempdb onto separate disks should be a part of any standard database server build. I emphasize the ‘should’ because for some odd reason I still see shops where this is not true. If that is the case for you, and you need to boost performance, then roll up your sleeves and get started on the work that should have been done from the onset.

3. Increase transaction log throughput

Another area that will affect your database performance has to do with the configuration of your transaction logs. Kimberly Tripp (blog | @KimberlyLTripp) has a great post on the 8 Steps to Better Transaction Log Throughput. Many folks don’t think about how something as simple as managing log file growth properly can help with overall performance.

One of the hidden parts of transaction logs has to do with virtual log files (VLFs), the “chunks” that are strung together to form the transaction log. I wrote about VLFs a while back when I witnessed firsthand the havoc they could cause. Until I knew about VLFs I had no idea why my log files were “misbehaving”. After I found out about them I decided to come up with a way to identify databases that had too many VLFs and were causing us issues (both for performance and administrative activities) and took the necessary corrective actions. Those actions also led to a review of overall transaction log throughput, as I wanted to make certain we were deploying servers to our end users that were configured for performance right from the start because it is a real pain to go back after the fact and say things like “Oh, sorry, I didn’t realize you wanted that server to perform well, so we just gave you all the default stuff which is usually crappy, we can go back and fix all of that for you and have things ready in about two months.”

Save yourself that headache. Take care of your transaction logs from the start. You can thank me later.

4. Build a proper reporting database

I see a lot of “mixed-use” databases in every type of industry. At some point someone gets the idea that they need a database in order to store information, usually customer information at first, then things like products, inventory, or sales. You know, standard business stuff. Most of those systems are built along guidelines for typical transactional based processing. After some time, perhaps months or years, other people in the office think to themselves “hey, I need some info, and don’t we have it already in that database”? So they go about building some reports.

And before you can say “writers block readers” you soon have yourself an OLTP system that is being used more often for OLAP purposes. I used to have a job that would alert me whenever a spid was blocked for 300 seconds. That’s five minutes. Of doing nothing but waiting for someone else. When I would call the person and tell them something like “hey, you stuff is blocked and just sitting there, and it is Joe doing the blocking” I would get responses that would range from “OK, his stuff needs to run, so I can wait” to “OMG WTF IS JOE DOING IN MY DATABASE RIGHT NOW WHILE I AM TRYING TO FINISH MY WORK!”

Building a proper reporting solution can help you improve performance for everyone, and it is often never talked about. Usually people focus on trying to coexist when they should really talk about signing the divorce papers.

5. Choosing the correct datatypes

More often than not, database performance (good and bad) can be traced all the way back to the design phase. During that phase there is (or, there should) be a discussion around the choice of proper data types for the underlying data. Unfortunately I see that such discussions typically don’t happen, thanks to tools like Visual Studio that will create tables with columns that are nvarchar(50) by default.

Most of the time I try to talk to customer about the selection of data types and the impact on performance I get back one of two responses: “I didn’t know that” or “It’s vendor code, we can’t touch it”.  Clearly we need to raise awareness in either case. Also, we need to stop blaming the database for the problems that have been created at design time. The database is only doing what you have asked, and you have likely asked it to work inefficiently and slowly. Check out this article by Greg Larsen regarding performance and data types. And here is a list of bad habits to kick by Aaron Bertrand (blog | @aaronbertrand).

6. Configure Your Power Settings

I know that “being green” is all in vogue these days. As a result we have things like power savings options for servers that allow for the CPU to be throttled when not in use. The net result of this is that you get the chance to explain to your end users that while database performance is not very good they can feel better knowing that they are helping to keep polar bears colder at night. Then again, you can only make such a comment to them if you are aware about these power savings options. Since most people have no idea about these power settings they are likely to spend many fruitless hours trying to track down the root cause of performance issues. In an incredible twist of irony, the extra power they consume while trying to research the issue brought about by power saving results in those polar bears needing more and more suntan lotion with each passing year.

The above six items are all ways that you could be suffering from poor database performance. The problem I see with many DBAs is that they try to fix the one thing that is on their plate at the moment, without seeing a bigger picture. Similar to saying “you can’t see the forest through the trees”, many DBAs are so focused on getting one query at a time tuned that they never bother to look up and see things like memory settings, or transaction log configurations, or optimizing tempdb for performance as ways to improve performance. Rarely will I ever see a discussion about changing data types or building a proper report solution and yet those are also ways your performance could be slowly draining away.

These are all items that can be addressed early on in any project, or even a rollout of a new instance of SQL Server. It doesn’t take much time and the benefits are worth every minute.

I found a diagram recently that helped to explain the different types of JOIN syntax. Being the type person that enjoys the visualization of abstract concepts such as set theory this diagram spoke to me. Despite the diagram being well done something seemed slightly wrong about it. I thought about it for a while and then it hit me: the diagram does not accurately reflect the complexity of joins for queries I see every day. The diagram seems better suited for a classroom, not the real world. So I decided to put together my own diagrams for you to show you the things I see frequently. You’re welcome.

1. The Three Table Join

This first diagram that I thought about was a simple three table join. In fact, I was going to put together a new diagram for all the possible three table joins but decided that I really did not have the desire to put together 127 different diagrams (I am leaving it as an exercise for the reader to explain why I could need 127 different diagrams, leave your answer in the comments below). So, here is the first type of join I thought about:

What I find worth mentioning here is that you can change your result set by altering that second join. Instead of A.c1 = C.c1, change it to B.c1 = C.c1 and you will find that you may not return the expected data in your select list (another exercise for the reader there, go ahead and try that for yourself or you can just download this quick sample script).

Another fascinating point here is that it occurred to me that this same T-SQL would also apply to the following:

This diagram seems slightly more realistic to me, as I am likely to start with one table and need to go out to two (or more) other tables with joins.

2. The Table Join For Disjointed Tables

That idea got me thinking even more (always a dangerous thing) and I then thought about other types of joins I see frequently:

Yeah, that’s right…multiple tables with no relation between them. The end user just selects everything possible and then filters the rows based upon the WHERE clause or uses features in Excel or PowerPivot to do the filtering. In a way this is what I visualize “Big Data” to be, the grabbing of as much data as possible, mashing it together, and filtering the results until you find something that helps you get one step closer to wherever it is you wanted to go today.

3. The One Big Table Join

After that graph it suddenly occurred to me that there are a lot of people who create OLTP databases with really wide tables. I mean, REALLY wide tables, hundreds of columns. So, their syntax and graph looks like this: 

Everything in one table, what a perfect database design, right? What could be simpler than that? In fact most data warehouses are built this way, where they have a few tables that are really wide and you may not need to join to any other tables. All the more reason to point out how the original diagram was not very indicative of the queries I see. In fact, the last picture I have for you is the EXACT type of query I see all the time.

4. The “What Were They Thinking” Table Join

How many times have you seen code similar to the following? Ever try to draw the picture? Here is my visualization: 

Yes, that’s right, I did include the infamous =’NULL’, don’t pretend like you haven’t seen that one before! I also included column names of ‘pid’ and ‘p_id’ to give you an indication about how we often see column names that aren’t very descriptive and leave you wondering if the columns are truly representing the same value type.

I think the above five diagrams are a lot closer in reality to the types of queries we see every day. And the types of queries we are expected to tune, often immediately.

I often find it easier to help with performance problems once I am able to visualize what the joins are representing. I bet your end users would appreciate seeing similar diagrams as well, it may help them to understand what they are asking for, and it would likely help lead to a healthy discussion about design considerations. If nothing else, it should help to level set the expectations for performance.

This video is an hour-long but well worth sharing with you today. One of the main takeaways from this video I want you to have is the fact that troubleshooting performance is not always rocket surgery. When you watch this video and listen to Conor Cunningham talk about some of the real world examples he has faced you will see what I mean.

I also liked how they help you to understand the concept of how to “bucketize” performance issues. It goes something like this:

Are all queries affected, or just a subset of queries affected?

If all queries are having performance issues then you will want to examine settings that affect the entire instance, such as memory settings, or high CPU utilization. You will want to do this first before trying to examine any one particular query.

If it is only a subset of queries (or users, or a particular application) then you will want to focus your efforts on those queries first. Otherwise you will be wasting time trying to fix one query without addressing the root cause of the performance issue affecting all queries. [A great example of this for my customers has to do with virtualization: why waste time trying to tune one query when the reason for the slowness is because your memory settings have been dynamically changed? Better to spend five minutes talking to your server admins about the memory for your guest than to waste time trying to get a query to suddenly run faster with less memory available.]

Just being able to diagnose “all versus some” in the first five minutes of triage in a production down situation can save you a lot of time as you begin to form your action plan to correct problem and bring performance back to within acceptable limits. Set aside some time this week to watch this video and learn more.

We’ve all had those moments.

It’s late in the day. You are about to head home a few minutes early so you can bring your daughter to soccer practice on time. Your phone rings. The voice on the other end of the line mutters something about the server “being slow”, and needs you to fix it so the company stops losing more and more money with each passing minute. You hang up the phone and see that the voice on the other end of the line has already sent an email to everyone in the company that blames the database server as the source of the problem and that they have spoken to you and you will have it fixed in the next five minutes.

You need to fix this, and fast. You company, and your family, is counting on you being there for them. You open up SQL Server Management Studio. You want to take action. Everyone is counting on you to get this done, and done right.

I am here to tell you about the six mistakes that many novice DBAs make when it comes to fixing performance issues. These mistakes may get you through the next few minutes but they are often band-aid solutions at best and sometimes they can make performance even worse.

1. Throwing Indexes at the Problem

Wait, don’t indexes help make performance better? So, wouldn’t more indexes mean more performance? No, not really. When it comes to indexes you can have too much of a good thing. And yet many novice DBAs look to add indexes whenever they come across a query that is not performing as well as expected. With tools such as the Database Engine Tuning Advisor at their fingertips they go about tuning individual statements and accepting all the default recommendations. And they end up fixing one issue and causing ten others.

Adding extra indexes is not always the right thing to do. Be wary of the easy path, for it is often the wrong path. What you want to do is research. Find out more about the entire workload hitting that table. Find out more about the database design and see if improvements can be made to existing structures before just adding new indexes. Remember this phrase: when it comes to adding indexes, just because you can doesn’t mean you should.

2. Using Index Hints Everywhere

After creating so many additional indexes some novice DBAs will then be confused as to why some indexes are not being used when expected. That is when they find out about index hints and so they go about forcing the optimizer to choose a specific index. Or maybe they will find out about plan guides and try those instead. This method works once and with great success and before too long it becomes the de facto standard to use index hints everywhere. In short, they go through a phase where they feel they are smarter than the optimizer. They will even proudly show you the plan guides they have that prove they are smarter than the optimizer. Apparently query algebra is a hobby for them, right?

If you think you are smarter than the optimizer I am here to tell you that you are not. Let the optimizer do the job it was designed to do and try to understand more about why it made the choice that it did. You think it should have used an index, or a different plan? Then you need to understand more about how and why the optimizer made the choice that it did.

3. Updating Statistics Every Hour

We’ve all been there. A query suddenly takes longer to perform and the users are upset. In an effort to see if you can help get things running well again you update the statistics for the table and suddenly everything is better. A few hours later performance is bad again and you update the stats to make it go away. You sense a pattern and before you can say “Bad Idea jeans” you have a job configured inside of SQL Agent to update the stats every hour in order to keep performance running smooth.

What could be wrong with such an approach? You have jumped ahead to a solution without understanding the root cause for the need to have frequent updates of statistics. Is the table poorly designed? Is the application deciding to do massive deletes and inserts because they once heard “updates are too costly”? You need to find the root cause before deciding upon a solution, and often that root cause is due to bad parameter sniffing.

4. Running Profiler in Production

This is never a good idea and usually the sign of a last resort for a seasoned DBA. For the novice DBA, it is often a tool they use frequently, mostly because they haven’t seen the use of Profiler bring down a production server. Isn’t it ironic that in an effort to help improve database performance they end up using a tool that can cause performance to become worse, or even cause an outage?

How can it cause an outage? Did you know that Profiler uses the default temp space for the O/S, which is often on the C: drive, which is where your O/S is stored. Ever see what happens when that drive gets filled? You can also store the results of your trace into a table and if you end up letting the trace run for so long you fill up a disk you will cause all other database activity to stop. Believe me, it happens, and when it does you don’t want to be sitting in a meeting the next day trying to explain that you were responsible for the production server being unusable yesterday. They hired you to keep things up and available, not to cause outages because you didn’t understand the effects of the tools you were using.

5. Flushing the Procedure Cache

When faced with symptoms of memory pressure and slow query performance many novice DBAs will look to find ways to reduce that pressure as quickly as possible in an effort to restore performance. For example, when faced with a low counter for the procedure cache hit ratio a novice may assume that many of the plans are now invalid and need to be flushed. So they reach for the DBCC FREEPROCCACHE statement in order to eject all plans from cache in an effort to restore performance. Again, they are jumping ahead to a solution without understanding the real problem.

Similar to the issue with updating statistics frequently, the idea of putting a job in place to flush your plan cache without understanding what is causing your plan cache to become bloated in the first place is not the best idea. Take the time to find out why you are having issues with your plan cache before deciding that flushing it every hour is a viable solution. The end result may be you need to change the method in which calls are being made to your server.

6. Not Thinking About Tomorrow

Think of a ship, a plane, or a rocket. Each of them can only go so far, and so fast. Each has what is called structural integrity, and has performance thresholds that when crossed the structural integrity will fall apart. Guess what else has performance thresholds that when crossed will cause everything to fall apart? Your database.

You need to know what your thresholds are. How many simultaneous connections and inserts/updates/deletes can you sustain before performance starts to degrade? Just how fast can your ship travel before it breaks apart? If you don’t know this number then you don’t know when the wall is approaching and you could find yourself running right into it at full steam and knock yourself out.

Think of it this way: you are given a problem to solve, say a query is running slow. You make adjustments and it runs faster. But you never bother to check to see how well it can scale. A month later, the user workload has doubled, and the query is slow again. You dive back in and make some adjustments. Now people are wondering “why can’t they just make things work correctly”? A month later things are bad again and you need to make adjustments but you have no adjustments left to make. You have hit your threshold. There is nowhere else to go. But your end users need more.

Wouldn’t it have been good for you to have seen this coming sooner? Of course it would. Proper testing and bench marking is a forgotten art these days. Pick up a brush and learn to paint, it will pay off for you.

This past Monday I listed out the five trends that data professionals need to be aware of and how it will impact their careers. I’m here to tell you something else: Dont just love your data, that isn’t enough. You need to make love to your data using tools like PowerPivot. Yesterday Rob Collie (blog | @powerpivotpro) provided a shining example.

A few weeks ago I came across this article and I had an idea: could I find out how many computations per second I would get for $1? And could I show how, over time, we are getting more for our money? And could I adjust for inflation? And if I did chart it all out, what would the data tell me, besides the fact that apparently the PS3 is faster per dollar than any other machine on Earth right now?

I had lots of questions. I had lots of tools at my disposal. What I didn’t have was time. I wanted an answer, quickly, but no time to get it done. So I turned to someone I knew could help get it done. Rob is one of the leading experts in Business Analytics. If you haven’t heard that term before, get used to it. Because it’s our future.

Rob took my idea and ran with it. He subscribed to ShadowStats.com in order to get more data to use to correlate my thoughts. He then started applying his own thoughts regarding Moore’s Law. His graphs help to show that Moore’s Law is becoming stagnant, and that in 18 months we shouldn’t expect things to be twice as fast anymore. In other words, he took my idea and was able to quickly gather data, perform analysis, postulate his own theory, and return the entire results back in a very short amount of time.

This is the future for data professionals. It isn’t about racking servers. It’s about getting people access to the data they want, in a meaningful way, and quickly. As a DBA you are already very familiar with data. How it moves, how it’s stored, how it should look. Take those skills and start learning how to apply them in a different manner.

Soon you will find your skills in demand in a similar manner to Rob. The guy is flat out busy for a reason.

It’s Sunday evening. You are sitting down to have dinner with your family. Suddenly your phone rings. You want to ignore it because, let’s face it, your phone rings a lot. But you see the person calling and knowing they don’t call very often you decide to answer. On the other end of the line a harried voice tells you the words that every DBA dreads hearing:

“The server is slow.”

Many thoughts run through your head: Well of course it is, otherwise you wouldn’t be calling, right? And why doesn’t anyone ever call me to tell me that the server is running fast? And what, exactly, is meant by “slow”? 

“OK, do you have any other details for me?” you ask.

“Yeah, I sent you an email an hour ago with some charts that PAL returned. Don’t you check your email all day on Sundays?”

You resist the urge to hang up and go to your laptop to examine the images that were sent. You see the following:

Page Life Expectancy

Lazy Writes per Second

Page Reads per Second

Full Scans per Second

Batch Requests per Second

So, why is the server running slow?

Next week Jason Strate (blog | @stratesql) and I will be doing a presentation for the Minnesota SQL Server User Group (PASSMN). The title of the talk is “Choose Your Own Adventure: Performance Tuning”. We will walk you through a scenario not unlike the one I have outlined above. We will talk about the various techniques you have at your disposal when it comes to performance tuning and help you understand which techniques are most useful.

And you, the audience, will decide how we troubleshoot and solve the scenario presented. Life is an adventure, and at times so is performance monitoring and troubleshooting.

Think about what you would do first for the scenario above, or what a new DBA might do, or what an MCM might do. Think about how long you have before your manager decides to hire a DBA that has all the answers already. Think about what you can do to fix this issue immediately. What steps would you take? What questions would you ask? Leave your comments below.