Do you have a knack for numbers? Do you find data as fascinating to
read as a spy novel?
Then America needs you!
Well, to be more specific, America needs data analysts, a group of
people whose vocational calling is to make heads and tails of data and use them to help businesses make lower costs, increase
sales, and make sound decisions in general.
They call it “big data” (presumably because of the actual immensity
of its magnitude). But where does it come from?
Big data comes from people visiting websites and joining and
commenting on social networks. It comes from sensors and software that monitor
where shipments go, what they contain, who is sending and receiving them, and
what the environmental conditions are around them.
It comes from gadgets we carry around, such as mobile phones and
other specialized devices and appliances equipped with mobile capabilities, like Amazon’s Kindle and Barnes and Noble
eBook readers. It comes from the photos we upload using these phones, the moods
and status updates we share, the tweets we tweet, the highlights and bookmarks
we make public on Amazon Kindle. It comes from our location and who we are with
– the very personal information we were cajoled into “opting in” for sharing by
our social network(s).
It comes from search engines, generated every time we type
something in the search bar and click on one of the results.
As a
result, we have been creating 2.5 quintillion bytes of data every day – so much
that 90% of all the data in the world today was created in the last two years
alone.
There’s so much data that needs to be analyzed out there, a 2011
McKinsey Global Institute report stated a projection that the United States
needs up to 190,000 additional workers with “deep analytical” expertise, not to
mention an additional 1.5 million data-literate managers.
Incidentally, this need for data analysts also goes beyond the traditional business world. Take, for instance, Justin Grimmer, a young assistant professor
at Stanford, whose work revolves around using the computer to analyze news
articles, press releases, Congressional speeches, and blog postings, all in the
interest of understanding how political ideas are spread.
In sports, data analysis is being used to spot undervalued
players. Shipping companies use data on traffic patterns and delivery times to
fine-tune their routing. Online dating systems use algorithms to find better
matches for their members. Police departments use data on holidays, weather,
sports events, pay days, and arrest patterns to identify criminal hot spots.
Data analysts have found that a few weeks before a certain region’s hospital emergency rooms started getting flooded with patients, there
was a spike in online searches for “flu treatments” and “flu symptoms.”
Similarly, researches have also shown that you can get a more
accurate picture of how real estate sales will be in the next quarter by
looking at the number of housing-related searches than by asking the opinion of
experts.
This scene is repeated in many other fields. Discovery and
decision making are all moving under the influence of data analysis. It makes
no difference whether we’re looking at the business world, the government, or
the academe. As Harvard’s Institute for Quantitative Social Science director
Gary King succinctly puts it, “There is no area that is going to be untouched.”
And as the data grows, it also helps the technology for collecting
and analyzing it grow as well. The more data they take in, the more intelligent
the machines get. Today, not only numbers and words can be analyzed. Even the
so-called unstructured data – videos, audio tracks and images – are fair game.
It is a virtuous cycle, so to speak.
Of course, while it is easy to be impressed at
our current ability to harness and analyze data, it still helps to remember the
old-fashioned and time-tested belief that more is not necessarily better.
As Stanford statistics professor Trevor Hastie puts it, “The
trouble with seeking a meaningful needle in massive haystacks of data is that
many bits of straw look like needles.”
In addition, with such a huge amount of data, it’s far too easy
for people delieberately seeking to skew opinions to make the conclusion first
and come up with the supporting “facts” later.
Nonetheless, despite these setbacks, it is undeniable that data
and the opportunities it brings are here to stay. We will be using it in
different ways, towards different ends – but use it, we will, and if we do not
have the ability to do so, we will need people who do to help us out.
Yash Talreja, Vice President, Engineering, The Technology Gurus.
