[This was first posted to Technorati on February 6th, 2012. You can read the original here.]

I keep hearing the term “Big Data”. My old-school IT instincts always makes me think of inefficiently storing two-letter state abbreviation codes inside of an nvarchar(50) field because that’s the default data type you get when creating a database table from inside Visual Studio. But it turns out that Big Data means something much more than that.

Big Data means big profits. It means predictive analysis. It means a shift in how we process data. It means that as database professionals our skills are going to be needed for years to come.

It means everything.

Check out this article from Popular Science magazine that discusses how the city of Santa Cruz is using data in order to better prevent crime from happening (yes, almost like “Minority Report”). This is not your ordinary “business intelligence”, this is the next stage of development. This is “Business Analytics“, and people everywhere are going to want to have trusted experts that can help them reliably mine their data.

Want to know when and where the next crime will be committed? How about how the spread of a disease from city to city? Or finding a way to help traffic flow? Don’t believe that Big Data can help us predict such things? Then check out this book: Linked: How Everything Is Connected to Everything Else and What It Means. Or take a few minutes to learn about the “3 Vs” of Big Data (Volume, Velocity, Variety) from Edd Dumbill and how more companies are using these concepts to gain a competitive edge.

Turns out the concepts behind Big Data is not so new. Researchers have been trying to find ways to connect data for decades. Even the idea for hyperlinks goes back to 1945, a full ten years before Tim Berners-Lee was even born.

Think about all this for a moment. We have been trying to find ways to link data together for over 60 years. Big Data is not new. What is new are the shiny tools we are being handed each year to mine the data. And why are companies investing in building us shiny new tools? Because they know the market for business analytics, and Big Data, is quite vast.

And there will also be a market for folks who know how to use the tools. Start investing in some training now, it will pay off for you later. Those folks who shun such concepts as Big Data, or the “cloud”, are going to be left far behind. And at the rate changes happen these days, you may find yourself left behind sooner than you think.

It’s the end of the year and already I am seeing the usual blog posts that either reflect on the year that was, or are pondering the year to come. I thought about writing another post covering all of the things that didn’t happen to me this past year but since I’ve already done that before I figured I needed to do something different. So I decided on this post instead. You’re welcome.

Here are the 12 things you should stop doing in 2012.

1. Using antiquated technology

The line in the sand for me would be the ages of my children. If your software was written before they were born, I don’t want to be using it. But that measuring stick won’t work forever, so a better one is to make certain you are only at most one full version behind. If you are still running SQL2000 or Oracle9i, you are more than one version behind, and you should really stop. It’s time to move on. Seriously.

2. Responding immediately to every email

Stop doing that. Trust me on this, your response to that email can wait. Don’t believe me? Try an experiment. For one month do not reply immediately to your emails. At the end of the month add up the number of emails you received, and the number of emails that required an immediate response. I’m willing to bet that the number is quite low, much lower than you realize. And once you realize just how few emails require an immediate reply, you’ll never look at email in the same way again.

3. Over-thinking everything

Some things in your life require analysis and thought, no question. But you don’t need to spend 30 minutes ordering coffee at Starbucks. Lots of things in your life are like that coffee, you should be able to just make a decision and move on. But for whatever reason we tend to analyze lots of decisions in our lives that don’t really need any extra analysis.

4. Doing unproductive work

Start tracking your daily tasks and see which ones are the most productive. Chances are you have a few tasks that don’t align with your highest level of goals. If you want to become the world’s best widget-maker then you need to focus on tasks that help you achieve that goal, and spend less time on tasks that don’t help you get there.

5. Pretending that work-life balance exists

It doesn’t. Sorry to be the bearer of bad news, but the idea of work-life balance is a myth perpetuated by HR departments that are trying to recruit you. This is especially true the higher up the corporate ladder you find yourself. And yes, that means if you are self-employed, you are essentially your own CEO, and you don’t just work at home but you also live at work. So stop pretending that any such work-life balance exists, it doesn’t. What you really have is work-life management, and we all know how hard it is to find a good manager these days.

6. Multitasking every possible moment of the day

Would it hurt you to focus on the task at hand for once? You don’t always have to be doing three things at once. Like texting and driving, for example. Remember I told you how that email can wait? Yeah, that is especially true if you are driving. Try living in the moment a few times a day and you may be surprised to see the quality of your output rising as a result of your increased focus on one specific task.

7. Trying to keep up with the Kardashians

…or watching Jersey Shore, or those shows about “real” housewives. If we all just stop watching, there is a chance the shows may disappear altogether. That’s the world I’d like to leave for my children: a world free from crappy reality TV, and reality TV “stars”.

8. Stop arguing with people on the internet

It’s just not worth it, even when they are wrong, and even when they attack you. The internet is a big place, and it is full of opinions. Let people express their opinion. It’s OK. If you really feel the need to inject yourself into a conversation, don’t do it. You won’t win. You can’t win. It’s the internet, and every time you try to get the last word in about something, or try to demean someone because you feel you have been wronged, all it does is make you look a little more foolish than if you had written nothing at all.

9. Resisting the Cloud

Look people, stop making excuses for why you can’t move to the Cloud. It’s here, and people are using it already. That means they are learning lessons about Cloud technology. They are learning what works best for them, and they are ahead of you by doing so. How much longer do you want to wait at the starting line while everyone passes you by? Get in the game now. Find something that you can deploy to the Cloud and put the learning curve behind you, and do it quickly. Otherwise you are going to find your skills are out of date, and quickly.

10. Comparing yourself to others

You’ll never be as good as that other person. It’s true. You will only ever become second-best, because you will always feel they are better than you anyway. It’s a losing effort, so don’t bother. Instead, find out what one or two things you like most about them and see if you can adapt those things to fit your personality or lifestyle. At the end of the day you have to be yourself, but it’s OK if you get some influence and inspiration from others. Just don’t try to be an exact replica because that’s just creepy.

11. Thinking that you need to go it alone

I see this a lot. In fact, I lived it. I thought I was all alone in my role as a DBA. But I wasn’t, because I had a network of colleagues out there willing to help. You don’t need to do anything alone in life, you can always find someone willing to help. Even misery loves company. And whether you are a DBA or the CEO of your company, you don’t have to work in a vacuum, there is a network of your peers willing to lend you a hand. It’s OK to ask for help, to admit that you may have been able to do something better. If your end goal for life is to have “I did it myself” on your tombstone then you should expect a lot of lonely Holiday dinners.

12. Reading blog posts titled “12 Things in 2012″

They’re a waste of your time. Go kiss your children goodnight and read them a story instead.

I figure that eliminating the above items will save me a lot of time in 2012. I have no idea what I will do with the extra time yet, but I’m sure I will think of something before the Mayans attack in late December.

Had a quick chat with Buck today regarding a recent bill (for $1.38 USD) for my Azure subscription.

If you are using Azure for testing (like me), and you want to minimize your chances of being billed, then just drop the user databases you have created. You don’t need to drop your Azure subscription, so please do not think that is the case.

Now, if you want to be extra safe, you can also go ahead and drop the SQL Azure database server, but that step is not necessary at this time as you do not get billed for a server that is only running the master database. You only get billed for user databases that exist.

And as always, the web moves fast. Things change. Don’t blow by the agreement you’re signing up for – it’s a new model, so understand what Microsoft is saying. More here: http://msdn.microsoft.com/en-us/library/ee621788.aspx

Thanks Buck!

I decided to drive over to Waltham tonight to listen to Brent Ozar (blog | twitter) talk out of his ass about virtualization. The place was packed and @BrentO put on a good show, as expected. He gave us the four hour slide deck but condensed it down to about 60 minutes.

I chatted with Brent a bit before the talk about some of the issues I had come across with virtualization but it was during the talk that I was struck by a sense of déjà vu. Brent was talking about various ways that certain configurations could affect a host which will then affect all the guests. In my mind I stopped to ponder where I had come across that situation before.

It was with a SAN.

See, when you build yourself a SAN, the idea is to have a crapload (that’s the technical term) of disks carved out into logical units (LUNs). Your servers end up getting assigned to the LUNs which are presented to the server as a disk. This makes it easy for your SAN admin to expand disks quickly when more storage is needed. It also makes it easy for your SAN admin to choke every last bit of performance from your database server, especially if they treat it like it is a file server. But that’s a post for a different day. The point here is this: your stuff is shared.

So, what does that mean? It means that if you have a performance issue on one server, which can cause issues with your SAN, then you could see the effects manifested on other servers. You could roll into work one morning and see that there were problems on 20 servers and not think it was related in any way. Worse yet, you could see a problem on just one server and the culprit was a different server sharing the same spindles as yours.

But with virtualization this problem goes away, right? WRONG! In fact, you not only still have the SAN issue but now you have a possible issue with guests and hosts. And we can throw vMotion into the discussion just because we haven’t had enough things to possibly go wrong. There are so many possible reasons for you to see a performance degradation on your guest that it will really make the forensics challenging.

For those folks that like checklists, you should add in items for checking to see if servers are sharing the same SAN storage, if guests are tied to the same hosts, and if your guest was migrated during the night. It will make your troubleshooting easier. When you go virtual your servers are more co-dependent on each other than ever before. Awareness of this fact will be crucial in how quickly you can isolate problems.

After the talk was done I had Brent call my cell phone to verify that his ringtone would come up (“Bad Romance”) and it did.

I heart technology.

or…not…

Well played, Microsoft

I wanted to cancel my account before I got hit up for “hidden charges” or something ridonkulous like that and could not find any way to cancel my subscription short of calling someone on the phone which could be my least favorite thing to do.

From what I could tell, I should be allowed to “opt-out” of an automatic renewal, except I don’t appear to have that option. I am wondering if I am simply not getting a proper user experience because I am part of some trial program, but it is rather annoying to not be able to simply click a button and say “stop charging me at the end of this month”.

As of right now I owe $1.06 for usage charges over the past month. I would be fine with that except for the fact that I was under the impression I was not going to be charged anything at all during my trial period. I find it hard to believe that Microsoft needs my $1.06 that badly but apparently they do. And they also need it again next month, so that may be why they won’t let me cancel my account so easily.

I suppose I could just drop the server I am using, essentially ending any chance of them charging me for SQL Azure altogether, but something tells me they would find a way to keep billing my credit card. So instead I will just open a new credit card, change my billing information to use the new card, and then just cancel the card.

That should take care of them trying to charge me for a trial service they don’t want me to cancel easily. And yeah, I understand that opening the card and closing it is a lot more effort than just calling someone at Azure support, but I am annoyed to the point that I would go out of my way to circumvent the “process” they have configured.

Since I have been railing on SQL Azure lately, I thought I could share with you some of the videos you can watch for free over at MSDN. In case you were not aware, there is a LOT of free content over at MSDN and I am trying to utilize it more and more these days. Right now I am using it as a way to understand more about SQL Azure and I thought I could share three videos that you may also find useful.

I recently presented a session on SQL Azure (you can download the slide deck here). During the session we talked a lot about some of the little things I have found by simply using SQL Azure. The need for a clustered index on every table was a surprise to me when I first found out about it, and it seemed surprising to everyone in the room as well. One person mentioned how it is a common practice to drop indexes before loading data to a table, for example, so they would need to rethink the steps involved in pushing data to the cloud.

I still think the biggest hurdle will be how Microsoft approaches the application of service packs and hotfixes to SQL Azure. One person at my session suggested that they would assume that Microsoft would have done thorough testing of the patch and also that with such limited features available right now that the testing would be able to be streamlined. I tended to agree but decided to point out something I found with their documentation:

Click to see full image

So, they went out of their way to link to four DMVs that they DO NOT support, and the one DMV they do support in this category is not even a hyperlink? Why, exactly, would you hyperlink to things I cannot use? With such a huge usability FAIL, why is it that I would trust them to do thorough testing of any patch when they cannot even handle the usability of their documentation?

I want to love SQL Azure, I really do. I intend to find more and more ways to utilize it whenever possible. But when I see things like above it really makes me question how some of the plumbing is really being handled. I strongly believe that if you take care of the little things then you will inevitably also take care of the big things. And if you can’t do the little things right then you might as well “use the button”.

Use the button, please.

In the past four weeks I have come across a lot of issues with the documentation being borderline unusable. When I was trying to push data to the cloud a few weeks back I talked to Buck Woody (blog | twitter) about a handful of options to which he replied “Why don’t you just use Sync?”

“Well,” I replied, “because I don’t see that mentioned anywhere as an option. Don’t your departments talk to each other? Why is this option not in the documentation?”

I know this is really v1.0 of a product, and that things will get better in time. I truly hope that is the case and we are not going to be subject to having things half-done tossed over the fence at us to use. But the more time I spend with SQL Azure the less comfortable I am that they are taking care of the little things.

And I consider documentation to be more than just a little thing.

Tomorrow I will be presenting at the first ever Hartford SQL Server Users Group. The title of the talk is Cloud Computing with SQL Azure, but only because I couldn’t pull together my first choice which was “Dora the Cloud Explorer”. Sorry, but I ran out of time on that because I’ve been a little busy with the upcoming 24 Hours of PASS event. By the way, I am going to adopt Brent Ozar’s moniker of 24iHop for the event. In fact, if I can find an iHop near me that has wireless, I may just UStream myself from there during the event.

The talk tomorrow will start around 1-ish. Our meeting is schedule to go from 12-4PM EDT and I expect that we will start broadcasting around 1PM but don’t get upset if we are a few minutes behind because Rob and I need to spend some time taking care of business and I doubt anyone really wants to watch that. Oh, before I forget, here is the LiveMeeting link for tomorrow:

https://www.livemeeting.com/cc/microsoft/join?id=TJB64W&role=attend&pw=4BMMRD

We have over fifty people registered for the event tomorrow, which would be an incredible number for Rob and I to have for our first event. We are really hoping we can continue to meet here at least once a quarter. Oh, how about a calendar invite? Yeah, that could be helpful. I cannot save .ics files to my current web host but what I can do is configure an event in Google Calendar and share it with you. Just click on the friendly button:

I mentioned that our meeting is scheduled from 12-4PM EDT, and I will start talking at 1PM EDT or so, but don’t expect a three hour talk on SQL Azure. My talk will only last about an hour, including questions. So, if you are planning on watching during your lunch hour, that’s fine, you don’t have to worry about missing the last two hours of the presentation because you won’t miss anything except fifty people eating cold pizza and milling about before heading home.

I hope to see you there, or online, tomorrow!

Today I had the need to find tables in a database that did not have a clustered index defined. Why would I have such a need? That’s a great question.

Anyway, I set about with a few basic questions for myself such as “Self, how many tables are there?” So I did what I normally do whenever I have questions of MS SQL, I took a nose dive into the system information. First up would be to count the objects defined as a user table which I got with this (I was connected to AdventureWorks but feel free to run this against whatever database you are curious about):

SELECT COUNT(*)
FROM sys.objects
WHERE type = ‘U’

Running against AdventureWorks gives me a count of seventy (70) user defined tables. The next question I had for myself was “Self, how many clustered indexes exist for those seventy tables?” This required a little more work. I needed to keep swimming in the system information but now I had to swim in a very particular direction. That direction was towards the sys.indexes view where the index_id was equal to 1. Why a 1? Because that is how Microsoft decided to tag each and every clustered index.

So, the following query would give me a count of the number of tables with a clustered index defined, answering my second question:

SELECT COUNT(*)
FROM sys.indexes
WHERE object_id IN
(SELECT object_id FROM sys.objects WHERE type = ‘U’)
AND index_id = 1

Running this I promptly found that AdventureWorks has sixty-eight (68) tables that have a clustered index.Good news? Not really, no, I was hoping that every table had a clustered index. Why? Great question.

And to top it off, I have no idea which tables out of the 70 are missing a clustered index. All this talking to myself has distracted me from what I really need, which is a script that answers the real question: “Self, which tables are missing a clustered index?” I kept swimming in the system information, doing a loop around the same spot in the ocean I was already at, take the two answers above to arrive at:

SELECT name
FROM sys.objects
WHERE type = ‘U’
AND object_id NOT IN
(SELECT object_id FROM sys.indexes WHERE index_id = 1)

And a-bing, a-bang, a-bung there you go, the results are the following:

ProductProductPhoto
DatabaseLog

What’s that? You want to know why I would even care about this? Well, it turns out that this information is going to be important for anyone that is interested in SQL Azure. Why? That’s a great question. Let me introduce you to my new friend, Error Message 40054:

Msg 40054, Level 16, State 1, Line 3
Tables without a clustered index are not supported in this version of SQL Server. Please create a clustered index and try again.

Yeah, that’s right, tables in SQL Azure *must* have a clustered index defined before you are allowed to insert any data. So, if you are looking to push your database to the cloud, you need to start thinking about your entire design, because there are a lot more factors that need to be considered other than just clustered indexes on each table.

And as luck would have it, I will be touching upon this topic (and others) as part of my talk next week at the Hartford SQL Server User Group meeting. We are looking to have the meeting broadcast live and I will post the event details once I have them.

I took part in a one day conference today on cloud computing. It took place at the Harvard Club in Boston, which was an interesting choice to have the event. I had never been to the Harvard Club before, and I enjoyed the history displayed all around me. I did not wear a coat and tie, and when I saw people in the parking lot dressed up I thought it might be the type of place where they were going to hand me a clip-on when I walked through the door. Thankfully they didn’t.

There was quite a lot of valuable information shared today. Each time I attend events pertaining to Microsoft’s cloud computing services I come away with a deeper understanding of where they are taking us. I think this is partly the result of the fact that more times you hear something the more it becomes familiar and understandable. The other reason is that as the months go by it becomes easier for others to understand what Microsoft is doing as well, including Microsoft themselves. In fact, I would say over half of the audience today were Microsoft employees. I bumped into Rob Walters, for example, as well as a handful of other Microsoft employees looking for a deeper understanding of what services their company is looking to provide.

So what did I learn today? Well, I’m glad you asked. Here are some of the items.

So Many Layers

There are a lot of layers at play here, and a lot of acronyms to go with them. You have software as a service (SaaS), platforms as a service (PaaS, which makes me think of Easter eggs), and infrastructure as a service (IaaS). In there you find Azure (Windows and SQL), Sharepoint, Exchange, Office Communicator, and LiveMeeting. You can be in the cloud, host your own cloud, or have a hybrid model. Microsoft has said that every product they have will be offered as a cloud service eventually, which warrants mentioning here as that is going to dramatically changed the future state of IT over the next 5-7 years.

Time and again the speakers today did a very good job at describing all of these layers and the various ways that we could use existing cloud services to enhance our shops as well as lower overall IT costs. Not just the costs of hardware, mind you, but also lowering the costs associated with development of software. We heard lots of stories about how a traditional project that would take six to nine months could be done in six to eight weeks because you were streamlining the ability to get the right hardware in place in a timely manner.

And anyone that has trouble getting hardware delivered from a vendor can certainly understand the value in that. If cloud services were only about a data center it could be valuable itself, but toss in some software and you really have something to take notice. With platforms and software thrown in you get the ability to quickly change your needs to shifting market trends, not to mention the occasional need for you to handle spikes in usage without needing to purchase expensive hardware.

Point of No Return

The funniest moment of the day was when a question was asked regarding a “reverse-migration”. See, Microsoft will do whatever they can in order to help migrate you to their services. But they really don’t have a lot of interest in helping people migrate off of the cloud. To be fair, I certainly understand why they wouldn’t want to spend much time in helping people migrate away, and that’s fine. The point that was driven home was that once you decide to start using their services it is not easy to decouple yourself from them. So, if you do decide to go into the cloud, you might as well understand that your decision is final.

Think of going to the cloud as if you were having children. Once you have them you can’t put them back.

Corporate Tech Lags Behind Households

One of the sessions was on social computing, which was an (almost) clever way to get me to attend a session on Sharepoint 2010. I love the collaboration tools that they are shoving into Sharepoint, I really do, but I’m not certain that I will get to use them anytime soon. Why is that? Because companies are slow to adopt new technologies.

Remember when going to the office meant that you got to play with “cutting edge” hardware and software? I sure do. At some point in the past fifteen years that has changed. These days, our phones are usually more cutting edge and capable than our desktops at work. And the younger people entering the workforce each year don’t think of social computing as a novelty. No, for them, it is an expected behavior. Facebook account? Expected. Use Twitter? Expected. Have or read a blog? Expected. So for Microsoft to be including those tools in Sharepoint makes sense, but it will be five years before I see them adopted and used in mainstream corporate America.

Ask the (Male) Experts and Get Answers Even If They Are Not an Expert

At the end of the day there was an ‘Ask the Experts’ panel discussion. I looked up from my tiramisu and saw nine men sitting in chairs. That’s right, the first thing I noticed is that there was no woman on this panel. Not sure why, but that stood out quite a bit, especially since there were at least a dozen women from Microsoft in attendance (out of maybe 100 people total), I would like to think one of them had some expertise in cloud computing that they could have been invited onto the panel. Anyway, as the questions went along, someone asked a question on SQL Azure. And then I noticed something else.

Everyone sitting in a chair started turning their heads to look at each other. I went down the line and noticed that there was no SQL expert on the panel! I immediately turned to Rob and said “You’re the only SQL expert in the room, perhaps you should go up and answer the question.” One of the experts started to answer the question and I had a hard time keeping my mouth shut. The question? “How do you connect to, and develop against, SQL Azure?” The answer?

“You simply use the same tools you use right now.”

Um, well, technically that could be considered correct, with a salesman-esque type of leniency, I suppose. When I heard the answer I turned to Rob and said “Wow. He really glossed over a LOT of details with that answer.” I really didn’t want to let this go. My mind was telling my mouth to start asking other SQL Azure questions. Things like:

“Is there a limit to the size of a database?” (Yep.)

“Can I take a backup of my database?” (Nope.)

“You guarantee 99.9% uptime, but will you guarantee me a specified I/O per second?” (Nope.)

“Will you roll back a SQL patch if I need it?” (Probably not.)

And so on. I refrained from asking any questions because I knew I would just be a PITA. Yeah, that’s right, I held my tongue. I know. I can’t believe it either. But I didn’t want to scare anyone and I didn’t want to upset my hosts and I didn’t want to be escorted from the Harvard Club until after they raffled off the xBox.

Overall this was a very good day filled with a lot of valuable information, and I am thankful for the opportunity to attend. I hope to attend 3-4 other events this year that are dedicated to Microsoft’s cloud services, as things tend to change frequently and I want to keep informed on everything Azure has to offer. I do believe that it is where our industry is heading.