What is ‘big data’?


Big Data: Friend or Foe?
Your ATM transactions and credit and debit card purchases are part of the data profile that helps companies predict your spending habits.
Your ATM transactions and credit and debit card purchases are part of the data profile that helps companies predict your spending habits.
© Erik Tham/Corbis

The idea of big data makes a lot of us uneasy. It sounds a lot like Orwell's Big Brother, and with ads from companies that seem to know what we're doing and the recent NSA domestic spying revelations, it is understandable that some people find the massive amount of information out there about all of us disturbing.

People can tell lots about you from this data, including your age, gender, sexual orientation, marital status, income level, health status, tastes, hobbies, habits and a whole host of other things that you may or may not want to be public knowledge. They need only have the means and the will to gather and analyze it. And whether they mean well or ill, it can have unintended consequences.

We give up more information than we realize to companies with whom we do business, especially if we use loyalty cards or pay with credit or debit cards. Someone can learn a lot about you just from analyzing your purchases. Target received some press when it was discovered that they could pinpoint which customers were pregnant and even how close they were to their due dates from things like the types of supplements and lotions they were buying. In one case, Target began mailing coupons for baby products directly to a teenage girl, sparking her father's ire against the company for sending her what he considered age-inappropriate ads -- until he found out about her pregnancy [sources: Datoo, Duhigg, Economist].

Governments and privacy advocates have made attempts to regulate the way people's personally identifiable information (PII) is used or disclosed in order to give individuals some amount of control over what becomes public knowledge. But predictive analytics can bypass many existing laws (which mainly deal with specific types of data like your financial, medical or educational records) by letting companies conclude things about you indirectly, and likely without your knowledge, using disparate pieces of information gathered from digital sources. Some companies are using the information to do things like check potential customers' credit worthiness using data other than the typical credit score, which can be good or bad for you, depending upon what they find and how they interpret it. One worry, though, is that this type of personal information can lead to hard-to-detect employment, housing or lending discrimination. And worse yet, it may not always be entirely accurate.

It's also possible for patterns seen in big data to be misinterpreted and lead to bad decisions. Like any tool, the results all depend upon how well it is used. Even though math is involved, big data analytics is not an exact science, and human planning and decision-making has to come in somewhere. With huge data sets, judgment calls need to be made about what is important and what can be be ignored. But performing big data analytics well can give companies a competitive advantage.

Such analysis can be used for things that are obviously good, such as fighting fraud. Banks, credit card providers and other companies that deal in money now increasingly use big data analytics to spot unusual patterns that point to criminal activity. On an individual account, they can quickly be alerted to red flags like purchases of unusual items, amounts the customer normally wouldn't spend, an odd geographical location or a small test purchase followed by a very large purchase. Patterns across multiple accounts, like similar charges on different cards from the same area, can also alert a company to possible fraudulent behavior.

Huge data sets can aid in scientific and sociological research, election predictions, weather forecasts and other worthwhile pursuits. Social media posts and Google searches have even been used to quickly find out where disease outbreaks are occurring. So it's not all bad news. It'll just take a while to work out all the potential problems and to implement laws that would protect us from potential harm. Until then, if you're worried, you might want to revert to cash purchases and watch what you put out there about yourself. Still, we're probably too far down the rabbit hole for any of us to be entirely off the radar.

Author's Note: What is 'big data'?

Like anything, big data can be used for good, for ill, and for lots of stuff in between. Having ads and coupons targeted at us can be a convenience or a major annoyance. And it's more than a little unnerving the amount strangers can learn about us just because we're swiping plastic in their stores or using their cards.

Loyalty cards I'd always figured were ways to gather data on our purchases, but I hadn't really appreciated how much similar data was being tied to us individually through debit/credit purchases until now, or the incredible detail about our lives that could be discerned from it. And this isn't even including all the other information about us out there on the Internet.

The thought of my every move being analyzed makes me want to go off the grid somewhat, stop posting online and use cash for everything. Although most of us, including me, will probably continue on as we are for convenience purposes. I just might post and buy as though I'm being watched.

Related Articles

Sources

  • Apache. "Hadoop." (Nov. 30, 2013) http://hadoop.apache.org/
  • Arthur, Lisa. "What Is Big Data?" Forbes. Aug. 15, 2013. (Dec. 1, 2013) http://www.forbes.com/sites/lisaarthur/2013/08/15/what-is-big-data/
  • Brooks, David. "What Data Can't Do." New York Times. Feb. 18, 2013. (Dec. 4, 2013) http://www.nytimes.com/2013/02/19/opinion/brooks-what-data-cant-do.html?_r=1&
  • Brooks, David. "What You'll Do Next." New York Times. April 15, 2013. (Dec. 4, 2013) http://www.nytimes.com/2013/04/16/opinion/brooks-what-youll-do-next.html
  • Brust, Andrew. "MapReduce and MPP: Two sides of the Big Data coin?" ZDNet. March 2, 2012. (Dec. 5, 2013) http://www.zdnet.com/blog/big-data/mapreduce-and-mpp-two-sides-of-the-big-data-coin/121
  • Butler, Brandon. "Lessons From Numbers Guru Nate Silver About Working With Big Data." Network World. Sept. 11, 2013. (Dec. 4, 2013) http://www.networkworld.com/news/2013/091113-nate-silver-big-data-273740.html
  • Cox, Ryan. "Nate Silver Skeptical of Big Data Trends, Keys in on Culture." Silicon Angle. Sept. 12, 2013. (Dec. 4, 2013) http://siliconangle.com/blog/2013/09/12/nate-silver-skeptical-of-big-data-trends-keys-in-on-culture/
  • Crawford, Kate and Jason Schultz. "Big Data and Due Process: Toward a Framework to Redress Predictive Privacy Harms." New York University School of Law. October 1, 2013. (Dec. 4, 2013) http://papers.ssrn.com/sol3/papers.cfm?abstract_id=2325784
  • Datoo, Siraj. "Rapid Development in Big Data Analytics Has Led to Increased Investment." Guardian. Nov. 22, 2013. (Nov. 29, 2013) http://www.theguardian.com/news/2013/nov/22/rapid-development-in-big-data-analytics-has-led-to-increased-investment
  • Duhigg, Charles. "How Companies Learn Your Secrets." New York Times. Feb. 16, 2012. (Dec. 2, 2013) http://www.nytimes.com/2012/02/19/magazine/shopping-habits.html?pagewanted=6&_r=3&hp&pagewanted=all&
  • Economist. "Big Data - Crunching the Numbers." May 19, 2012. (Dec. 1, 2013) http://www.economist.com/node/21554743
  • EMC. "EMC: Behind the Big Data Curtain." 2012. (Dec. 1, 2013) http://www.emc.com/campaign/global/big-data/hfbd-infographic-4web-1500.jpg?cmp=micro-big_data-general-emc
  • Fitzgerald, Michael. "Big Data: Big Threat Or Big Lie?" InformationWeek. Nov. 21, 2013. (Dec. 4, 2013) http://www.informationweek.com/big-data-big-threat-or-big-lie/d/d-id/1112668?
  • Gartner. "Big Data." (Nov. 29, 2013) http://www.gartner.com/it-glossary/big-data/
  • Gnau, Scott. "Putting Big Data in Context." Wired. Sept. 10, 2013. (Dec. 4, 2013) http://www.wired.com/insights/2013/09/putting-big-data-in-context/
  • Henschen, Doug. "Big Data Reshapes Weather Channel Predictions." InformationWeek. Nov. 25, 2013. (Dec. 4, 2013) http://www.informationweek.com/big-data/software-platforms/big-data-reshapes-weather-channel-predictions/d/d-id/1112776?
  • IBM. "What is big data?" (Dec. 4, 2013) http://www-01.ibm.com/software/data/bigdata/what-is-big-data.html
  • Intel. "Big Data 101: How Big Data Makes Big Impacts." (Nov. 29, 2013) http://www.intel.com/content/www/us/en/big-data/big-data-101-animation.html
  • Intel. "Combat Credit Card Fraud with Big Data." (Nov. 30, 2013) http://www.intel.com/content/dam/www/public/us/en/documents/white-papers/combat-credit-card-fraud-with-big-data-whitepaper.pdf
  • Intel. "What is Big Data?" (Nov. 30, 2013) http://www.intel.com/content/www/us/en/big-data/big-data-what-is-big-data-landing.html
  • Laney, Doug. "Deja VVVu: Others Claiming Gartner's Construct for Big Data." Gartner. Jan. 14, 2012. (Dec. 1, 2013) http://blogs.gartner.com/doug-laney/deja-vvvue-others-claiming-gartners-volume-velocity-variety-construct-for-big-data/
  • Lund, Susan, James Manyika, Scott Nyquist, Lenny Mendonca, and Sreenivas Ramaswamy. "Game Changers: Five Opportunities for US Growth and Renewal." McKinsey Global Institute. July 2013. (Dec. 3, 2013) http://www.mckinsey.com/insights/americas/us_game_changers
  • MongoDB. "Big Data Explained." (Dec. 5, 2013) http://www.mongodb.com/learn/big-data
  • Naughton, John. "Why Big Data Has Made Your Privacy a Thing of the Past." Guardian. Oct. 5, 2013. (Nov. 29, 2013) http://www.theguardian.com/technology/2013/oct/06/big-data-predictive-analytics-privacy
  • Novet, Jordan. "Here's Why 2014 Will be the Year of the 'Internet of Things.'" Venturebeat. Nov. 25, 2013. (Dec. 1, 2013) http://venturebeat.com/2013/11/25/heres-why-2014-will-be-the-year-of-the-internet-of-things/
  • Romanov, Alex. "Putting a Dollar Value on Big Data Insights." Wired. July 17, 2013. (Dec. 4, 2013) http://www.wired.com/insights/2013/07/putting-a-dollar-value-on-big-data-insights/
  • SAS. "What is Big Data?" (Dec. 1, 2013) http://www.sas.com/big-data/
  • Sicular, Svetlana. "Gartner's Big Data Definition Consists of Three Parts, Not to Be Confused with Three 'V's." Forbes. March 27, 2013. (Dec. 1, 2013) http://www.forbes.com/sites/gartnergroup/2013/03/27/gartners-big-data-definition-consists-of-three-parts-not-to-be-confused-with-three-vs/
  • Zettaset. "What is Big Data and Hadoop?" (November 29, 2013) http://www.zettaset.com/info-center/what-is-big-data-and-hadoop.php

More to Explore