Big data: Have computers wiped out humans?

Hadoop, Hadoop as a Service, NoSQL, data warehouses, coding, dashboards, the list goes on. Big data technology is exciting, and we love to talk about it. However, focusing so much on the technology or even the data itself can make the process feel very inhuman, and we can be forgiven for wondering just how much of our lives is going to be controlled by mindless robots.

Are IT professionals being replaced by super-smart supercomputers and consumers being reduced down to a profile of coded data sets? Certainly not.

Not if businesses want to avoid the kind of mistakes that get them in the news, such as Target's automated ads that sent coupons for baby products to a pregnant teen. The truth is that for big data to truly be valuable, it has to be treated as a tool used by humans who also have the qualitative evidence to help them interpret and act on the quantitative. The very nature of big data requires a human element. Still not convinced? Let's dive deeper.

Data is not the endgame

The first thing we need to remember is that data isn't the endgame. Collecting or even analysing huge amounts of information has no real purpose without a meaningful end goal. Creating better relationships with customers, improving products, reacting faster to disasters and solving world problems are those end goals, and all of these goals are decidedly human. No computer can define these goals, come up with the questions to find data to help reach these goals or decide how to act on the data to reach these goals. The computer can't replace the IT professional, the marketer or the product developer because they are the ones making and reaching goals. Big data is simply a tool in their hands.

Making connections

Human intelligence has an innate ability to make judgement calls and connections. Just think of how many years of technological development and testing has been required to create a car that can drive itself. Then think about how many humans can drive at the young age of 15 or 16. Yes, some things have to be learned, but many things, such as steering the wheel or responding to a car that stops in front of you doesn't require processing a complex equation in your head - you just know how to do it.

Similarly, big data tools may be able to process millions of data points much faster than humans can, but that innate ability to draw conclusions from those data points is still left to the analyst.

Recognising error

Data is often skewed. Whether its customers not being truthful on a survey or a relationship that is actually correlated (not a cause and effect relationship), computers can't judge whether data has errors or not, but people can. A business manager knows that sales are down because a competitor had a big sell, but a computer might point to a new advertisement as the cause of the downturn. Human judgement is required to recognise the errors that technology can't.

Smart application of insights

Returning to the Target debacle. Businesses are quickly learning that automated responses without some thought into the intricacies of human behaviour can have some negative consequences. If someone at Target had stepped in and recognised that customers under age 18 probably shouldn't be targeted with baby product ads (even if they do show signs of pregnancy) due to the inherent delicacy of that situation, Target's idea would probably be used as a case study for effective marketing today. Of course, that kind of thinking is beyond anything technology can do.

So what is there to take away from all of this? Training employees on how to catch errors, make connections and apply insights to business decisions in an intelligent manner is critical to ensuring that a big data project is successful. Since humans are so central to the data analysis process, business leaders should put just as much thought into selecting their data analysts as they select the tool that they use.

Gil Allouche is the vice president of marketing at Qubole. Gil began his marketing career as a product strategist at SAP while earning his MBA at Babson College and is a former software engineer.