I was asked last week if  knew a query that would return the size of the databases that have been discovered by Operations Manager. I went about digging through some old queries I put together and found the information that was requested.

Inside the OperationsManager database look for a view named MTV_Database. All discovered databases will be there. There should be a column for DatabaseSize, but the column will have a GUID at the end of the name.

Now, I forget where I found it, but a while back I came across one of the view definitions that helped me to return a lot of other information, such as SQL version. The view that has SQL version is named MTV_DBEngine, but in order to join the MTV_Database to the MTV_DBEngine I needed to add in a handful of relationship tables and filter on the BaseManagedTypeID. I was going mad trying to make everything work until I found that specific view definition and I don’t recall which one it is, but what I do have is the end result which I can share (your GUIDs will be different, I imagine):

SELECT *
FROM [OperationsManager].[dbo].MT_Database T1
LEFT JOIN [OperationsManager].[dbo].MT_Database_2 T2
ON T1.BaseManagedEntityId = T2.BaseManagedEntityId
LEFT JOIN [OperationsManager].[dbo].MT_Database_1 T3
ON T1.BaseManagedEntityId = T3.BaseManagedEntityId
LEFT JOIN [OperationsManager].[dbo].MT_Database_0 T4
ON T1.BaseManagedEntityId = T4.BaseManagedEntityId
LEFT JOIN [OperationsManager].[dbo].[Relationship] RT5
ON (RT5.RelationshipTypeId = ’328C1DCD-4AA5-9964-F3B6-0E607AE58416′
AND RT5.TargetEntityId = T1.BaseManagedEntityId)
LEFT JOIN [OperationsManager].[dbo].MTV_DBEngine T5
ON RT5.SourceEntityId = T5.BaseManagedEntityId
LEFT JOIN [OperationsManager].[dbo].[Relationship] RT6
ON (RT6.RelationshipTypeId = ’6284518A-90CC-FADD-E56B-26669E2A0A8F’
AND RT6.TargetEntityId = T5.BaseManagedEntityId)
LEFT JOIN [OperationsManager].[dbo].MTV_Computer T6
ON RT6.SourceEntityId = T6.BaseManagedEntityId
JOIN [OperationsManager].[dbo].BaseManagedEntity BME
ON BME.BaseManagedEntityId = T1.BaseManagedEntityId
WHERE BME.IsDeleted = 0
AND BME.BaseManagedTypeId = ’10C1C7F7-BA0F-5F9B-C74A-79A891170934′

I think what you want to do is first find the BaseManagedTypeId for the ‘Microsoft.SQLServer.Database’ entity (that is the one in my where clause). Not sure how I got those relationship GUIDs, I must have hunted for them manually at some point.

At any rate, the simplest answer is to query the MTV_Database view for the database size. Or you can modify some of the database state views to include the information, but I always like to know the T-SQL that is being done behind the scenes. It just makes me feel better knowing everything that is happening under the hood.

When you want to get more details about the instances, like version information, then you have to really dive into the relationships. The OpsMgr console can provide you with the information, but not always in the format you want, which means you may need to build your own T-SQL to get all the right details.

Joe the DBA sent me the usually morning email, detailing the failed custom collections in Operations Manager. I had asked for Joe to do this routinely so that we could monitor our monitoring system, as any good DBA with OCD would do. With our recent push for server consolidation I have been spending more time on Operations Manager as I plan to use it for any and all necessary metrics in our project.

“Tokyo failed again,” Joe said over my cube wall. Sure enough, there it was, in the midst of a handful of failures one stood out. It was the SQL box that resides in Tokyo, Japan. I started to dig through the VB script, hoping I could reproduce the error, to no avail. The process is quite simple. Every morning at 2:20AM we have a script that selects all the syslogins on the server, inserts them into our central repository, then for each syslogin row the code takes the login name and then executes sp_helplogins and pushes the result also to our central repository.

Or not. It would seem that Tokyo was not pushing the second result set. Ever, in fact. The logins were coming through, but nothing else.

“Weird,” I thought to myself, and put down my bacon sandwich to focus on the matter.

I rooted around and found that, as expected, the process ran last night. I saw the entries for August 29th, at 2:20 AM, as expected. Only I did not see any entries for August 29th for the second set of records. I did, however, see entries for August 28th at 1:20PM.

“I must have inserted those records myself when I was trying to debug,” I muttered and took another look at the data. Then another. Then I looked at my phone. Then my Blackberry. Then my computer’s time. Then I did a SELECT getdate() on the Tokyo server. It came back with August 29th, 2:45 AM.

Or roughly thirteen hours into the future.

Somehow, I was getting data from the future inserted into our central repository. Well, not somehow, because Tokyo is literally and figuratively on the other side of the planet. The Operations Manager agent was working perfectly, the script was running at the right time, only since it was already tomorrow it was running during our afternoon. My code was doing a getdate() when selecting the syslogins and picking up Tokyo time, but when it was inserting the results of the sp_helplogins back to the central repository it was also using a getdate() and would then insert the local time (in this case, EDT).

I had clearly opened up a rift in the fabric of space time, but the effect was only temporary.

But what to do to solve the issue? Do I change the code to always insert according to local time? Perhaps. But we also have a purge process that runs in EDT, so we would want the time to be in EDT, otherwise we may purge something incorrectly. And what about our ad-hoc scripts or reports? We use datediff() frequently, so shouldn’t we agree upon a standard?

With three servers (Operations Manager, our central repository, and Tokyo) we have the potential for three different timezones. How do you keep it straight? Who wins?

I have no idea right now on what the solution should be, but I am leaning towards having two columns in the tables. One column for the local time on the server, the other for when the data gets inserted. Of course, we know the local time on the server, because that is when the Operations Manager agent will start the script, and that would keep everything consistent. Well, if you think having data from the future as being consistent, then sure.

When designing a system that spans multiple timezones you simply must consider what is meant by “time”.

I had a tip published over at MSSQLTips yesterday and wanted to drop a quick post to brag tell people about it. The tip is very straightforward, it uses xp_fileexist to verify if your database dumps are still on disk or not.

Keep in mind that xp_fileexist is one of those undocumented stored procedures, which is my way of telling you that if you decide to build something using my script it could suddenly stop working one day. Of course, all my scripts are like that but since this one is using an undocumented stored procedure I have the luxury of a scapegoat. Which is nice. Not “total-consciousness-on-my-deathbed” nice, but nice nonetheless.

The whole idea came about because there are a handful of times when files may get moved or deleted accidentally. With this script you can quickly verify a possible issue before you go looking for the file because you need to do a restore. Given the choice between needing to recover a file from tape because you might need to do a restore versus needing it NOW because you MUST do a restore, I will take the former. This script can help, you could put it into a SQL Agent job, or perhaps a local policy, or even Operations Manager. Your choice, really.

I am actually thinking I will take this script and build it into a monitor inside of Operations Manager, giving me the ability to quickly see if there is an issue at periodic times throughout the day.

On more than one occasion I have seen or heard a DBA ask a common question about Operations Manager 2007: “How do I know it is working?” Last week I spent time with Brent Ozar to put together a video on how to navigate your way around OpsMgr. Brent was a perfect choice to have the discussion with because he admittedly knows very little about OpsMgr and asked a lot of the same questions I hear from any other beginner. I did my best to demonstrate how to navigate the OpsMgr GUI but I am sure I did not touch everything. So, if you have any specific questions, let me know and I will see if I can put together additional podcasts to answer your questions. And if you are interested in participating in a podcast with me, let me know that as well.

Well, at least that is the impression I have from Operations Manager. Let me explain why.

It all started around the time we were making some adjustments to one of our servers that was not performing as well as our end users would have liked. The short end of that story is that we ended up making some changes to our tempdb. We split the tempdb into separate data files, one per physical processor. So far, so good. We also went one step further and we set the size of tempdb to be fixed, turning off the Autogrow option.

I bounced this question around Twitter recently, asking people if they consider fixing the size of their database and disable Autogrow. The majority of results are that you should size your database as best as possible upon initial creation, but let the data files grow automatically if necessary just to be safe.

Putting that debate aside for now, let me focus my ire back on OpsMgr.

So, one of the monitors you get out of the box with OpsMgr is the ‘DB Space Free (%)’ monitor, which is a wonderful idea because if the percentage of free space dips below a certain threshold you would want to be notified. I mean, who wouldn’t want to know if there was only five percent of free space left in their database? Sure, if it was a 10Tb database then that five percent would be 500Gb (which is a lot of flippin’ space if you ask me), but still, the idea is sound. And we have never seen any alerts for this monitor, despite occasionally filling up a data drive every now and then, so our shop must be humming along quite nicely.

And we were. Until there was an incident. And by ‘incident’ I mean that a member of my team (let’s call him Joe the DBA) decided that when it comes to striping files it is not necessarily the best thing to ever let them autogrow. And he has a valid point to consider (see above), so he created the multiple data files for the tempdb and turned off autogrow.

And that’s when OpsMgr started beeping at me. Well, not beeping, but it sent me a nasty email saying that the tempdb was running out of space. Actually, it said that there was 0% free space left. The database state view was critical, which means I had a red circle staring at me next to the tempdb for that server instance. Naturally I went and checked on the space and found that out of 24Gb of space allocated to tempdb about 23.999Gb were free.

Dumbfounded, I reset the health of the monitor. It went to green, then five minutes later back to red. Another quick check of the database shows that nothing has changed, it is still very much empty, and not full. So, what is going on? I decided to dive into the details and picked apart the SQL 2005 Monitoring management pack. I found the section of code that was collecting the information for the free space and stepped through in order to find the problem.

After a few iterations of the code I found a curious section:

if ((currentDB.Name == “tempdb”) || (currentDB.Name == “master”) || (currentDB.Name == “msdb”))
{
propertyBag.AddValue(“DBSize”, dbSize);
propertyBag.AddValue(“DBFree”, 0);
propertyBag.AddValue(“DBPercentFree”, 0);

Um…ok…so, if the database is master, msdb, or tempdb, the code sets the “DBPercentFree” variable to zero, regardless of what the actual amount would be. And why no love for model?

So, now I knew where the calculation was coming from. For some reason OpsMgr has decided to simply set the free space to be zero, but why am I not getting alerts for master and msdb? Well, I stumbled across this little tidbit of information, something I had known before but forgotten:

Databases that have autogrow set will never generate an alert regardless of the reported free space.

Wow.

Let me see if I can understand what you are doing.

If I have a database set to autogrow then you won’t bother to tell me if it is filling up? I mean, not even if I want to know if it is 50% full? You just won’t even entertain the notion that I could possibly want to know that my database is getting full, simply because I enabled the data files to autogrow? Really? That makes sense to someone? Because to me, an actual production DBA, it doesn’t. In fact, it is silly to think that because I have a database set to autogrow that I would not care to get an alert. I haven’t seen anything this stupid since I found out BizTalk doesn’t bother to clean up the database backups they force you to enable through their jobs.

Now, on top of that, you have decided that if I did disable autogrow on a database, say, oh I don’t know, how about tempdb, if I did disable autogrow on tempdb that I should be spammed with alerts. Because that is what you have done. I guess I hit the sweet spot with tempdb and this monitor, seems like a perfect storm of issues.

So, to recap:

1. Operations Manager will not monitor the free space available in a database that is set to autogrow.
2. If you want to know your free space available, you need to disable autogrow.
3. Disabling autogrow on tempdb will always generate an alert, regardless of the actual free space available.

Brilliant. Unfortunately, not very helpful. My current workaround is to disable the monitor for the tempdb database, which makes no sense since it is a fixed size and is EXACTLY the type of database I should be able to monitor with Operations Manager, but can’t.

Or I can go back and tell Joe the DBA to set the tempdb to autogrow. And then get another lesson in why allowing striped files to autogrow essentially has the potential to erase the benefits of striping.

Can’t we all just get along?

Just thought I would share some info from Operations Manager that shows the amount of uptime on our servers here since 1/1/2008. Why? Well, since I am the official DBA of Uptime, a distinction that many have questioned, it felt right to provide some numbers.

I am sure many others will be able to provide better numbers than what I have. The numbers are based upon sixty-five (65) production servers here. I set up a custom group in Operations Manager for our production servers and ran the canned ‘Availability’ report against this group. The report worked well, it took a minute or two for it to run. One thing lacking is that there was no overall percentage of uptime, so I had to export the raw data to Excel and do some manual calculations.

I ran the report for the time period starting on 1/1/2008 until about 10:50 AM EST today, which was an opportunity of roughly 8,568 total minutes. Now, one caveat here…Operations Manager will compose this report based upon the state that is defined in the Management Packs. So, if I place a box into ‘Maintenance Mode’, I can avoid having it be considered as unplanned downtime. However, if we have a maintenance window on Sunday, and we do work on the box as part of a planned window, but do not put the box into maintenance mode, then it gets marked as unplanned downtime. Make sense?

Well, most of the time we do not bother with maintenance mode, so I am comfortable that this report shows a very close representation of our production servers. Well, except that some of the downtime on the server could mean the OpsMgr agent wasn’t running, so there would be no way to capture the data on that box without the agent running. So, if the agent is down, but the box is up, the report shows that as ‘monitoring unavailable’. Now my head hurts.

Okay, here is what you want to know. We had 99.999397% uptime for the current year. That uptime is a simple calculation of the number of hours the OpsMgr agents were up and running and detected the MSSQL service was running, divided by the total number of hours possible. So, it is possible that the service was up and running but users could not connect, which you would (rightfully) want to consider as a problem. But hey, no report is perfect, right?

I will keep trying to sift through these OpsMgr reports because I know that many people are interested in them and truthfully I have not been paying much attention to them because they seem to be somewhat nebulous in their details. But before I dig any further, let me make certain I tell my boss about my uptime, and hope he does not ask too many questions.

If you are judging this post by the title, you are probably going to be surprised it is about Operations Manager 2007, and not about personal hygiene. But I came across something very useful in the OpsMgr GUI the other day and I thought I should share with those that may care.

Often times I find myself looking to either recalculate the health state of a particular monitor. And more often than not I find myself pressing a lot of buttons in an effort to get things reset. It finally dawned on me the other day that there must be an easier way to reset the status, and I came across a way to get the job done using the GUI.

First, go to the monitoring tab, and select the ‘Discovered Inventory’ view.

Then, go to the menu item named ‘Actions’, and select ‘State Actions’, then ‘Change Target Type’.

You should see the ‘Select a Target Type’ screen, and from there select the ‘Health Service’.

Next, select the server in the Discovered Inventory state view, and go to the ‘Actions’ menu again. This time, select ‘Health Service Tasks’, then ‘Flush Health Service State and Cache’.

One caveat here. This task will reset the health state of ALL rules, monitors, etc. for the selected server. So, if you were looking to just reset one or two particualr monitors, this is not the tool for you. However, if you are looking to clean out everything, then this is the easiest way to get the job done. After the task has been started you can check on the status by simply going to the ‘Task Status’ state view in the Monitoring tab, it is located just below the ‘Discovered Inventory’ we selected at the start.

Hope this helps.

I got excited the other day. Very excited. “Big Idea” excited. That’s right, I had a Big Idea, and in that moment of Kramer-esque Zen I could clearly see what I was destined to build with Operations Manager. Alas, in order to build it, I would need to make customizations that I was not sure were possible. So, I hit up the Operations Manager SQL newsgroup and asked:

How do I create a custom discovery?

See, Operations Manager, while it is built on the idea that you can monitor anything, also has a similar concept, and that is you can discover anything. It makes no sense for you to be able to monitor something that cannot be discovered, does it? So, I wanted to discover something that I was not able to “see” anywhere.

I received a kind response in a short amount of time and was directed to the Operations Manager Authoring console. Authoring console? Wonderful! This should solve everything! Or make my life miserable for two days. Really, at that moment, I figured it could go either way. Shame that it went the way of the latter option. And here I am, blogging about it, in the hope that someone reading this will be able to point me in the right direction.

Honestly, the Authoring console looks great, it really does. The problem with it is that you seemingly have to start from scratch and build up your Management Pack (MP). Now, contrast that with the custom MP’s you build in the Operations console. When building there you essentially are extending some of the sealed MP’s, right? Well, that is what I really want. I want to export the MP I have already spent a year building, open that in the Authoring console, and extend it further by doing things the Operations console does not allow. Say, custom object discoveries.

Yeah, I know, it was a pipe dream. When I tried to export and open with the Authoring console I was greeted with a nasty error, telling me that a referenced MP could not be found. Well, not a problem, I suppose, except that I cannot get the *.MP file that it is looking for, because I cannot export a sealed MP. Usability fail.

Okay, let the hack job begin.

I decide that I could look to build another monitor using property bags and piggy back off the other monitors I have been building. It should work, but it is not my preferred choice. What about custom attributes? Well, If i try to create a custom attribute to the SQL DB Engine, it wants to look for a registry key or use a WMI query. Anyone know how to use WMI to get at database objects such as a list of stored procedures or tables? If you do, kindly shoot me an email. Thanks.

So, I have a couple of options but dammit, I want to do this correctly! I should be able to simply extend the current SQL MP’s to include the discovery of objects such as tables and save that to the current custom MP I have been building. The way things look right now, it is as if I will need to rebuild my MP from scratch, which is not something I am looking forward to. Mostly because I have no idea how to get started. I found relatively few examples on how to get started authoring MP’s. What I need is a guide that walks through customizing something that already exists, such as MS SQL Server. Sure, I could try to build that myself, starting from a base entity class and working my way up (or is it down?), but I do not believe that is the correct approach. Surely there is someone else out there that has started authoring their own SQL MP using the Authoring console.

Right?

Anyone?

Is the caller there?

I feel…so…alone. Which is very weird, since I usually feel so connected.

<heavy sigh>

About a month ago I came across a wonderful idea. I thought about how I could use Operations Manager to monitor my databases to ensure they have had a backup done within a predefined amount of time. I know lots of DBAs out there that have similar checks in place, most of which involve reports that may or may not be tied to jobs within SQL Agent. And while many of the solutions are adequate, I like solutions that offer pretty lights.

So, I dived into Operations Manager to see what I could see. I found the right target (database engine), and then put together some t-sql that would return the following three columns for each database: database name, days since full dump, days since differential. The code simply goes against the sysdatabases and msdb.backupset tables. I actually have two different versions of the monitor in place, one for sql 2000 and another for sql 2005. And I have plans to deploy to sql 2008, once we start rolling those boxes out.

The end result is that we have real time alerts that send an email notification as soon as a database is found to have a missing backup. I think that is better than waiting until the next day, but I am spoiled because we have a wonderful support team that works during our night shift. We also have the Database State view inside of Operations Manager that will give me a visual interpretation regarding the health of my enterprise.

I gave a demo of this monitor at PASS last week, and it seemed to be well received. I am looking to write up an article about this monitor as well, for those that enjoy looking at the gory details.

Sorry I have been so quiet lately, but it was hard finding time to write about everything that is happening to me this week. I took some notes and plan on writing individual blog entries. Most of them should be done before the end of next week, so expect a handful (or more) of entries coming your way soon.

No word on the election results. They showed the video today, but I was not able to see then on the big screen because i was part of a breakfast panel in an adjoining room and we ran late on time.

I promise to tweet the result as soon as I know, win or lose.

My talk went well. And by “well”, I mean I did not vomit or fall off the stage. The best part was answering questions. Perhaps next year I should just offer my services for an hour and show people how to get something done, and then save it to a custom management pack that I could hand them on a USB stick.