The Santa Cruz Experiment: Why You Should Care About Big Data

[This was first posted to Technorati on February 6th, 2012. You can read the original here.]

I keep hearing the term “Big Data”. My old-school IT instincts always makes me think of inefficiently storing two-letter state abbreviation codes inside of an nvarchar(50) field because that’s the default data type you get when creating a database table from inside Visual Studio. But it turns out that Big Data means something much more than that.

Big Data means big profits. It means predictive analysis. It means a shift in how we process data. It means that as database professionals our skills are going to be needed for years to come.

It means everything.

Check out this article from Popular Science magazine that discusses how the city of Santa Cruz is using data in order to better prevent crime from happening (yes, almost like “Minority Report”). This is not your ordinary “business intelligence”, this is the next stage of development. This is “Business Analytics“, and people everywhere are going to want to have trusted experts that can help them reliably mine their data.

Want to know when and where the next crime will be committed? How about how the spread of a disease from city to city? Or finding a way to help traffic flow? Don’t believe that Big Data can help us predict such things? Then check out this book: Linked: How Everything Is Connected to Everything Else and What It Means. Or take a few minutes to learn about the “3 Vs” of Big Data (Volume, Velocity, Variety) from Edd Dumbill and how more companies are using these concepts to gain a competitive edge.

Turns out the concepts behind Big Data is not so new. Researchers have been trying to find ways to connect data for decades. Even the idea for hyperlinks goes back to 1945, a full ten years before Tim Berners-Lee was even born.

Think about all this for a moment. We have been trying to find ways to link data together for over 60 years. Big Data is not new. What is new are the shiny tools we are being handed each year to mine the data. And why are companies investing in building us shiny new tools? Because they know the market for business analytics, and Big Data, is quite vast.

And there will also be a market for folks who know how to use the tools. Start investing in some training now, it will pay off for you later. Those folks who shun such concepts as Big Data, or the “cloud”, are going to be left far behind. And at the rate changes happen these days, you may find yourself left behind sooner than you think.