You can pre-order here: https://amzn.to/3fQr7hz

I thoroughly enjoyed this project, and I want to thank Apress and Jonathan Gennick for giving me the opportunity to update the previous edition. It felt good to be writing again, something I have not been doing enough of lately. And many thanks to Enrico van de Laar (@evdlaar) for giving me amazing content to start with.

The book is an effort to help explain how, why, and when wait events happen. Of course, I also want to show how to solve issues when they arise. Specific wait events are broken down into parts: definition, remediation, and an example. There are plenty of code examples, allowing the reader to duplicate the scenarios to help understand the wait events better.

It is my understanding we will have a GitHub repository for the sample code. This will make it easy for a reader to access the code for their use. I am hoping to keep the repo up to date and expand upon the example as I look towards the next version.

Pro SQL Server 2022 Wait Statistics at Live 360!

I will be presenting material from the book at SQL Server Live! this November where I have the following sessions, panel discussion, and workshop:

• Fast Focus: SQL Server Data Types and Performance
• Locking, Blocking, and Deadlocks
• Performance Tuning SQL Server using Wait Statistics
• SQL Server Live! Panel Discussion: Azure Cloud Migration Discussion
• Workshop: Introduction to Azure Data Platform for Data Professionals

The workshop is a full day training session delivered with Karen Lopez (@DataChick), and you can register for Live 360 here: Live 360 Orlando 2022 – Choose Registration

I am hopeful to have copies of Pro SQL Server 2022 Wait Statistics at SQL Server Live!. At the time of this post, I do not know of a publish date. Amazon shows the book as pre-order right now.

The post Pro SQL Server 2022 Wait Statistics Book appeared first on Thomas LaRock.

They were designed to store data efficiently, protecting the quality of the data written and stored to disk. I’ve written before about relational engines favoring data quality and integrity, and how relational databases were not designed for the reading of data.

Of course, if you are going through the trouble of writing the data into a relational database, it makes sense that you would want to retrieve the data at some point. Otherwise, why go through the exercise of storing the data inside the database?

The trouble with reading data from a relational database is due to the data not being stored in a format that is friendly for viewing, reading, or retrieving. That’s why we have data professionals, like me, to help you write queries that return the correct data, in the correct format, for you to analyze.

I’m here today to tell you we’ve been doing data wrong the whole damn time.

Let me show you what I mean.

Traditional Data Flow Patterns

Here’s what companies do, every day:

Step 1 – Identify useful data

Step 2 – Import that data into a database

Step 3 – Analyze the data

Step 4 – Export the data into dashboards

Step 5 – Profit (maybe)

The trouble with this process is Step 3, the analyzing of the data. Relational databases were not designed for analytical processing. Relational databases do not store data in a way that is readable, or friendly, for human analysis.

That’s not to say you can’t do analytics inside of a relational database. What I am saying is that it could be better for you not to spin the CPU cycles there, and instead do the analytics somewhere else.

For example, data warehouses help with data storage, retrieval, and analytics. But even a data warehouse can fall short when it comes to the use of unstructured data sources. As a result, we’ve spent decades building ETL processes to curate, collate, consolidate, and consume data.

And we’ve been doing it wrong.

Data Mining for the Data Expert

So, we understand that people find data, store it, and try to use it later. They are engaging in the process of data mining, hoping to find gold in the form of insights, leading to better business decisions.

But as I mentioned before, the data isn’t in a readable format. Let’s look at an example.

Following the common data flow pattern, I found some useful data at NFL Savant https://nflsavant.com/about.php, and imported the data into a SQL Server database:

It looks like any other table in a relational database. In this case, it is a table containing time series data pertaining to play by play records for the 2018 NFL season. Each row represents an entity (a play at a point in time of an NFL game), and the columns represent attributes of the play (down, distance, yards to go, etc.)

Nothing out of place here, this is how data is written to a relational database. In an orderly fashion. As a DBA, I love this type of orderly storage. It’s efficient, and efficient is good.

As a data analyst, I’m not a fan. At least, not yet. I have a bunch of data, but what I want are some answers. So, it’s up to me to ask some questions of the data, find some answers, and use that to help make better business decisions.

For this data, here’s an example of a simple question: What are the average yards to go for NFL teams in 2018? I can get that answer with some simple T-SQL:

This is great! I was able to take my data, ask a question, and get an answer. What could be better, right?

Of course, now I have more questions about my data. And here’s the first issue you will discover when trying to analyze data stored in a traditional relational database.

T-SQL is excellent at answering one question at a time, but not as great when you need more than one question answered.

So, if we have more questions, we will need to write more queries.

Here’s a good follow-up question that we might want to be answered: Can we examine this data broken down by each quarter?

Fortunately, the answer is yes, because T-SQL comes with a bunch of statements and functions that will help. In this case, I am going to use the PIVOT operator, as follows:

Easy, right?

No, not easy.

And not readable, either. What’s with that row saying NULL? Why do I not have a result for some teams in that last column?

As it turns out, you need a lot of experience writing T-SQL to get to that query. And you need more experience understanding the result set, too. You don’t start on Day 0 as a data professional writing PIVOT queries against a SQL Server database.

Here’s the good news: You don’t need to write PIVOT queries, ever.

Data Mining for the Masses

The data import from NFL Savant was in the form of a CSV file, which I then imported into my database. Because that’s how ETL is done (see above for the common data flow process).

What if…now hear me out…we skipped step 2? Forget about doing the import process. Instead, let’s open that CSV file in Excel.

Here’s what it would look like:

Back to our football questions. We’ve seen examples in T-SQL, let’s look at how to do this in Excel using a Pivot table.

I click on one cell in Excel, insert a pivot table, drag the offense teams as a row, and the downs to go as a value, change it to an average, and we are done. Have a look:

It took but a few seconds to get this magic to happen. Here’s what I want you to know:

1. No T-SQL is necessary. None. Not one line of code.

2. I have the entire table as a pivot table, allowing me to answer more questions WITHOUT needing to write more T-SQL.

3. There is no code. None.

Let’s say that I want to know the yards to go broken down by quarter. With T-SQL, I would need to write a new query. With the pivot table, it’s a simple drag and drop, like this:

Fin.

There is no need to rewrite code to get this result. Because there is no code, it’s drag and drop, and then I have my answer.

And that’s why I believe the inclusion of pivot table inside Excel is the greatest advancement in the 21^st century for data professionals.

Fight me.

Summary

I did not come here to bury relational databases. I came here to help you understand relational databases may not be the right place to do analytical processing.

When it comes to curating and consuming data, I have three simple rules for you to follow:

Rule #1 – Only collect data that you need. Don’t collect data “just in case you may need it later.” The data you collect must be relevant for your needs right now.

Rule #2 – Understand that all data is dirty. You could build a perfect analytical solution but based on inaccurate data. Know the risks involved in making business decisions based on dirty data.

Rule #3 – Before you collect any data, consider where the data will be processed. Don’t just assume that your database will do everything you need. Take time to list out all the available tools and systems at your disposal. The result may be a simpler solution than first imagined.

I wrote this post to help you understand Rule #3. Analysis of NFL play by play data is best done in a tool such as Excel, or PowerBI, and not (necessarily) inside of SQL Server.

SQL Server is a robust relational database engine, containing integrations with data science-y stuff such as R and Python. Just because you could do your analysis inside the SQL Server engine doesn’t mean you should.

This post originally appeared on PowerPivotPro and I was reminded about its existence while talking with Rob Collie during our Raw Data podcast. I asked Rob if I could repost here. He said yes. True story.

The post You Can’t Marry Your Database, But You Can Have Relations appeared first on Thomas LaRock.

Of course, it depends! But on what does performance tuning depend?

Context.

Most performance tuning methodologies focus on tuning one or more queries. This is the wrong way of thinking. It is an antiquated way of problem solving.

Let me explain.

The Problem with Traditional Database Monitoring

Traditional database monitoring platforms were built from the point of view of the engine-observer. These tools focus on metrics inside the database engine, and may collect some O/S level metrics. They often assume the database is running on a single server node, and not a collection of nodes. And they are reactive in nature, notifying you after an issue has happened.

But the reality is your database engine is but a process running on top of an operating system, for a server that is likely virtualized, and may be running in your data center or in the cloud. In other words, there are many layers between users and their data. And in a world of globally distributed systems, chances are your database is not on a single node.

This means your in-house legacy accounting application requires different monitoring and performance tuning methods than your on-line ordering system. When you focus on one query, or even a top ten list of queries, you have little to no information regarding the entire application stack. And those engine metrics we know and love will not help you understand the overall end user experience.

But when it comes to database performance tuning methods, there is a heavy focus on tuning activity inside the engine. This makes sense, because that’s what DBAs (and developers) know. That’s the silo in which they operate. They need to prove the issue is not inside the database.

Stop focusing on the database engine and open your mind to the world that exists outside of that database.

Once you turn that corner, the mean time to resolution shrinks. The result is a better end user experience.

Tune Workloads, Not Queries

The Heisenberg Uncertainty principle states that the position and velocity of a particle cannot be measured exactly at the same time. The more you know about position, the less you know about velocity, and vice-versa.

The same theory applies to database performance tuning methods. The more you know about activity happening inside of a database engine, the less you know about the entire system. Nowhere in an execution plan is there a metric for ‘user happiness’, for example.

Therefore, troubleshooting modern distributed systems requires a different approach. Enter the four golden signals: latency, traffic, errors, and saturation. These signals combine to help provide a measure of overall user experience. From there, if you need to dive into a database, you’ll have context necessary to start tuning at the server, instance, or query level. Over time you can shift to thinking about how to scale out, or up, as necessary.

Put another way, you would not expect your mechanic to tune your Jeep the same way she would tune a Ferrari. Both are vehicles but built for different purposes. The tools and methods are distinct for both. And so are the metrics and dashboards you want for your legacy applications versus a distributed one.

Summary

Slow is the new broke. But things don’t have to be slow to be broke. A poor user experience with your online ordering system will hurt your bottom line. Traditional database monitoring systems are not focused on the user experience. Instead, they focus on the database engine itself. But those engine metrics won’t tell you that Brad in Idaho got frustrated and left his shopping cart with $2,000 worth of potato seeds.

Your performance tuning methodology should include an understanding of the entire system and workload first, before you start looking at any specific query.

The post Tune Workloads, Not Queries appeared first on Thomas LaRock.

That’s why last year our company (SolarWinds) launched a free tool called SQL Plan Warnings. Often mining the plan cache involves custom scripts and forcing you to work with text output only. We wanted to make things easier by providing a graphical interface. A GUI will allow for the user to have basic application functionality. Things like connecting to more than one instance at a time, or filtering results with a few clicks.

Let me give a quick tour of SQL Plan Warnings.

Connect to an instance

The first thing noteworthy here is how SQL Plan Warnings supports connecting to a variety of flavors of SQL Server. There’s the Earthed version, Azure QL Database, Azure SQL Database Manage Instance, and Amazon RDS for SQL Server as shown here:

From there you fill in your connection details. The login you choose will need either the VIEW SERVER STATE or SELECT permission for the following DMVs: dm_exec_query_stats, dm_exec_sql_text, and dm_exec_text_query_plan. I’ve provided links to the Microsoft docs for each, so you can review the permissions defined there.

Being able to easily connect to instance of SQL Server, no matter where they are located, is a must-have these days.

SQL Plan Warnings Settings

After you connect to your instance, SQL Plan Warnings will return the top 100 plans, with a default sort by CPU time. However, it is possible after connecting you may see no results. This is likely due to the default settings for SQL Plan Warnings. You get to the settings by clicking on the gear icon in the upper-right corner. Here is what the default settings look like:

If you are not seeing any results, change the default settings and refresh plan analysis. For me, I simply made the change to filter by executions, with 1 as the minimum. This returns a lot of noise, so you need to discover what makes the most sense for your particular instance.

Please note these default settings apply to each connected instance. Think of these settings as the highest level filter for all your connected sources. It may be possible you spend time adjusting these settings frequently, depending on the instance, the workload, and your query tuning goals.

Reviewing the SQL Plan Warnings Results

After plan analysis is complete, you will see a list of warnings found. It should look like this:

Note that a plan can have multiple warnings. So this list could be generated by one or more plans found.

From here we are able to filter on a specific warning type with a simple click. This allows us to narrow our focus. Perhaps today we want to focus on Key and RID lookups. We select that filter, then open the plan:

From here we can zoom and scroll, and view the node that has the lookup warning:

If we select the node a properties dialogue that opens to the right. We also see other warnings are included in this plan, if we want or need to investigate those at this time. We also have the ability to download the plan, if desired.

Summary

The SQL Plan Warnings tool is easy to use and allows for you to be proactive in optimizing your environment. The use of a GUI allows for quick filtering at the plan cache level as well as plan warnings themselves. This allows you to focus on the plan warnings with the most impact.

One thing to note is the size of the plan cache you choose to analyze. Instances with larger plan cache sizes (1GB or greater) may require a larger number of plans to parse for warnings.

You can download the SQL Plan Warnings tool here.

The post SQL Plan Warnings appeared first on Thomas LaRock.

This latest report, well, I didn’t bother promoting it, or writing a review. I felt this report was incomplete and suggested to Microsoft they have at least one more round of revisions. They did not agree.

Turns out AWS had similar concerns. Earlier this week I found this tweet from Corey Quinn (@QuinnyPig), linking to a blog post from AWS titled “Fact-checking GigaOm’s Microsoft-sponsored benchmark claims“.

With all love and respect to @azure: Are you folks out of your goddamned minds? https://t.co/A7qos0XZgC

Go read that and then come back.— Corey Quinn (@QuinnyPig) January 30, 2020

You can read for yourself the response from AWS, it details many of the same concerns I had with the report.

But one comment from AWS stood out to me, it was “Publishing misleading benchmarks is just one more old-guard tactic by Microsoft”.

OK, I’ve got some opinions to share.

Grow Up

I’m not going to defend Microsoft or the quality of the report. Microsoft may not be perfect, but right now AWS looks more “old-guard” than Microsoft does. I watch Andy Jassy spread misinformation from the re:Invent keynote. Jassy did similar tricks at the last keynote, and it was awful to watch. There’s a lot to love about AWS, but the idea that they don’t use similar poor marketing tactics as other companies is laughable.

As for these AWS and Azure benchmark reports, I find them to be a fairly useless disk-measuring contest. The cloud technology changes fast. These reports are out of date just after the ink dries. I do not believe there is a company today making a choice for being “all-in” on AWS or Azure and basing their decision on such marketing propaganda. “Oh, look, this article says they are the fastest, let’s use them!”

Look, anyone can build a contrived edge case that will show their hardware outperforms someone else. Watching AWS and Azure bicker over these reports is like listening to two rich kids argue during lunch who has the nicest sports car.

No one cares, Richie Rich.

Summary

As a user of cloud services what I want is a reliable, secure, stable, and affordable solution. That’s it. I expect you to be updating your hardware and configurations to make things better. I expect you are making your services easier to consume, administer, and monitor.

We don’t need these AWS and Azure benchmark reports to see whose disk is bigger. We need guidance on what servers to select, how to configure my workload, how to monitor, and adjust as necessary.

Give us more of that content, please.

Focus on building great services, creating happy customers, and less on poking holes in each other. </rant>

The post Why AWS and Azure Benchmarks Don’t Matter to Me appeared first on Thomas LaRock.

I found this latest effort to be a good start, but it fell short of the effort the authors put forth in their previous benchmark for data warehousing.

Before I go any further, I want to thank the authors for putting together their results. I recognize that these are humans, working hard, putting forth their best efforts at being fair and thorough. Comparing cloud services is not an easy task. I found this latest effort to be good, but not great. If they were students of mine I would grade this latest paper from them a solid B-.

Let’s break it down.

The Good Stuff

First, the good stuff. I love how they drove everything towards a formula, price/performance, where performance is tracked in transactions per second. The downside to price/performance is that not every workload is focused on transactions per second. Still, I’d like to see this formula adopted as a standard way of comparing services.

In the past I’ve focused only on total price as shown by the online pricing calculators. This is because (1) you aren’t supposed to publish benchmarks without permission from the company (Microsoft, AWS) and (2) I can’t bankroll this level of test AND maintain my scotch and bacon addictions. By using price/performance you level the playing field somewhat. A service may cost more, but if it runs your query in half the time, the cost may be worth it.

I also liked the choice of using TPC-E as their test, I believe that to be a fair way to compare how the services will handle a workload. And I liked how they explained the difficulties in comparing services and the associated hardware. That’s something I’ve written about previously. Many times, really.

It is frustrating to compare the data services being offered between Azure and AWS. Part of me thinks this is done on purpose by both companies in an effort to win our favor without giving away more information than is necessary. This is a common practice, and I’m not bashing either company for doing what has been done for centuries. I’m here to help others figure out how to make the right choice for their needs. At the end of the day, I believe both Amazon and Microsoft want the same thing: happy customers.

But it is not in their best interest to make it easy for anyone to compare costs. This is how utilities operate. Make no mistake, AWS and Azure are the new electric company.

Now, for the items that I didn’t like as much. I’ll capture the quote from the article and explain my concern.

The Not As Good Stuff

“There are no exact matches in processors or memory.” – This is a bit of nitpicking, but I took issue here with the use of the word “or”. As someone who charges (and receives) top dollar for performing technical reviews of books, it bugged me. The authors are correct in saying that it is hard to find exact matches. However, I can certainly find a match for vCPU, but not for memory. Azure publishes memory as weird increments, starting at 10.2 GB while AWS shows traditional increments of 8, 16, etc. So, yeah, it’s a nitpick. But it was this exact item what caught my eye and made me dig deeper to fact check everything. Warrants mentioning.

“Thus, R4 seemed a suitable instance class to use for AWS RDS.” – The authors explain why they chose R4 (memory optimized instance) versus the M4 (general purpose). I have no issue with this except that neither M5 or R5 was considered. This study just came out, why were those instances not considered? And since the authors went out of their way to tell us what AWS says about the R4, let me tell you what AWS also says about the R5:

“R5 instances deliver 5% additional memory per vCPU than R4 and the largest size provides 768 GiB of memory. In addition, R5 instances deliver a 10% price per GiB improvement and a ~20% increased CPU performance over R4.”

I can’t think of any reason why the authors chose R4 here. But let’s move past this, because now is time for the hard part: finding a suitable match for Azure SQL Database.

“On the Azure side, we expect customers to gravitate towards SQL Database Business Critical (BC) offerings…” – Well, Azure doesn’t offer a memory optimized version of SQL Database, so I guess using BC is fine. But the question I have now is why not consider using Managed Instance? In the data warehouse benchmark study they tried a variety of sizes against the workload. This study focused ONLY on one size machine. This is part of the reason they got a B-, they weren’t thorough enough for my liking. I’d send them back and tell them to run more tests against different sized machines and include Managed Instance. At the very least they could have made an effort to simply use general purpose, it would have been closer to an apples-to-apples comparison.

“Therefore, we chose the BC_Gen5_80 instance, which has more CPUs than R4.16xlarge, but less memory at 408 GB.” – Yes, finding an exact match is difficult. Here’s a breakdown of what they chose:

But this image shows AWS at 64,000 provisioned IOPS, and further in the study they say they tested against 32,000 provisioned IOPS. So, which is it? I’ve no idea. Neither do you. But I do know that provisioning 32,000 IOPS added about $6k to the monthly bill.

“…the monthly cost of Microsoft Azure comes to $40,043.71. The monthly cost for AWS comes to $65,239.43.” – Verified, I can get the same prices using the AWS and Azure calculators. But the small detail that is glossed over here is single versus multi-zone. The AWS calculator is clear, if you deploy multi-zone, the price doubles. The Azure calculator doesn’t have this option, it only exists when you create your SQL Database. I’d be shocked to find out that deploying multi-zone in Azure didn’t bump the price as well. But the chart above clearly states “in a single availability zone”. So, which is it?

I’ve no idea. Neither do you.

Summary

Some quick math tells me that if we drop the multi-zone from AWS RDS the price/performance result comes in at $1,269.85, slightly cheaper than the $1,410.04 for SQL Database. And this is why I like price/performance as a metric. A database service may have a slightly higher price, but offers greater throughput.

This was the exact conclusion from the data warehouse study, too. The cost for Azure SQL Data Warehouse was just a tad more than AWS Redshift, but the performance with Azure was better. I wanted to see a similar conclusion in this study.

Instead, we have a report with a handful of inaccuracies. Perhaps in an effort to rush to publish ahead of Ignite, they simply used a wrong graph, or missed doing one final round of edits. When you are doing this work it is easy to have such things fall through the cracks.

I’d love to see this study republished without these errors. I’d also love for AWS and Azure to find a way to make it easier to compare costs and services.

REFERENCES:

Azure vs. AWS Data Services Comparison
Updated Data Services Comparison: AWS vs. Azure
Azure vs AWS Analytics and Big Data Services Comparison
Updated Analytics and Big Data Comparison: AWS vs. Azure
Azure SQL Data Warehouse Costs vs AWS Redshift
Azure pricing calculator
AWS pricing calculator
Amazon EC2 Instance Types
Sizes for Windows virtual machines in Azure
Azure SQL Database pricing
Data Warehouse in the Cloud Benchmark
SQL Transactional Processing Price-Performance Testing

The post Reviewing the GigaOM SQL Transactional Processing Price-Performance Testing appeared first on Thomas LaRock.

You can see I have navigated to the SQL Server Plan Cache counter, selected Cache Hit Ratio, and an instance of Extended Stored Procedures. You will also note in the lower left I have enabled “Show description”. This results in the text at the bottom, “Ratio between cache hits and lookups”.

That text is referring to the counter itself, and not to the instance. If I toggle to another instance, such as SQL plans, the text doesn’t change. I have an idea what SQL plans means, but I’m also smart enough to know I don’t know everything. So, where would one find information about the instances?

Those details can be found here: Use SQL Server Objects. From there we can go to the SQL Server, Plan Cache Object page, where we will find the following details:

That’s a lot more detail than I was expecting! Now I know exactly what this counter will consider to be a plan. These details provide more context to the metric, helping users understand what they are measuring.

Having SQL Server performance objects documented is important, and you should review them. Otherwise you run a risk of collecting the wrong metrics.

The post Using SQL Server Performance Objects appeared first on Thomas LaRock.