Big Data Brings Big Opportunities for Data Analytic Professionals

Do you have a knack for numbers? Do you find data as fascinating to read as a spy novel?

Then America needs you!

Well, to be more specific, America needs data analysts, a group of people whose vocational calling is to make heads and tails of data and use them to help businesses make lower costs, increase sales, and make sound decisions in general.

They call it “big data” (presumably because of the actual immensity of its magnitude). But where does it come from?

Big data comes from people visiting websites and joining and commenting on social networks. It comes from sensors and software that monitor where shipments go, what they contain, who is sending and receiving them, and what the environmental conditions are around them.

It comes from gadgets we carry around, such as mobile phones and other specialized devices and appliances equipped with mobile capabilities, like Amazon’s Kindle and Barnes and Noble eBook readers. It comes from the photos we upload using these phones, the moods and status updates we share, the tweets we tweet, the highlights and bookmarks we make public on Amazon Kindle. It comes from our location and who we are with – the very personal information we were cajoled into “opting in” for sharing by our social network(s).

It comes from search engines, generated every time we type something in the search bar and click on one of the results.

As a result, we have been creating 2.5 quintillion bytes of data every day – so much that 90% of all the data in the world today was created in the last two years alone.

There’s so much data that needs to be analyzed out there, a 2011 McKinsey Global Institute report stated a projection that the United States needs up to 190,000 additional workers with “deep analytical” expertise, not to mention an additional 1.5 million data-literate managers.

Incidentally, this need for data analysts also goes beyond the traditional business world. Take, for instance, Justin Grimmer, a young assistant professor at Stanford, whose work revolves around using the computer to analyze news articles, press releases, Congressional speeches, and blog postings, all in the interest of understanding how political ideas are spread.

In sports, data analysis is being used to spot undervalued players. Shipping companies use data on traffic patterns and delivery times to fine-tune their routing. Online dating systems use algorithms to find better matches for their members. Police departments use data on holidays, weather, sports events, pay days, and arrest patterns to identify criminal hot spots.

Data analysts have found that a few weeks before a certain region’s hospital emergency rooms started getting flooded with patients, there was a spike in online searches for “flu treatments” and “flu symptoms.”

Similarly, researches have also shown that you can get a more accurate picture of how real estate sales will be in the next quarter by looking at the number of housing-related searches than by asking the opinion of experts.

This scene is repeated in many other fields. Discovery and decision making are all moving under the influence of data analysis. It makes no difference whether we’re looking at the business world, the government, or the academe. As Harvard’s Institute for Quantitative Social Science director Gary King succinctly puts it, “There is no area that is going to be untouched.”

And as the data grows, it also helps the technology for collecting and analyzing it grow as well. The more data they take in, the more intelligent the machines get. Today, not only numbers and words can be analyzed. Even the so-called unstructured data – videos, audio tracks and images – are fair game.

It is a virtuous cycle, so to speak.

Of course, while it is easy to be impressed at our current ability to harness and analyze data, it still helps to remember the old-fashioned and time-tested belief that more is not necessarily better.

As Stanford statistics professor Trevor Hastie puts it, “The trouble with seeking a meaningful needle in massive haystacks of data is that many bits of straw look like needles.”

In addition, with such a huge amount of data, it’s far too easy for people delieberately seeking to skew opinions to make the conclusion first and come up with the supporting “facts” later.

Nonetheless, despite these setbacks, it is undeniable that data and the opportunities it brings are here to stay. We will be using it in different ways, towards different ends – but use it, we will, and if we do not have the ability to do so, we will need people who do to help us out.


Yash Talreja, Vice President, Engineering, The Technology Gurus.