'Big data' is probably the most commonly-used term in IT today. Is is also perhaps the most commonly-misused term. Despite its emergence as a megatrend impacting virtually all aspects in the modern IT landscape, big data remains largely misunderstood, with a handful of widely accepted myths exerting significant influence on the way organisations perceive and attempt to address this disruptive trend.
There is a big difference between these myths and reality and, as with any megatrend, there are so many voices talking all at once that it's easy to perpetuate the wrong information. Here's a look at the truth behind the five most common big data myths, including the misguided but almost universally accepted notion that big data applies only to large organisations dealing with great volumes of data.
Huge volumes of information
'Big' is a relative term. Big data is most often mentioned in connection with large companies that have a sea of information but companies of every size are experiencing data growth in some form today, even if it's just moving from five to 10 terabytes of data. Twice as much information is still twice as much information. Data volume doesn't have to be a certain size to be relevant. Nor is it imperative to analyse all your data at the same time anyway, making the total volume of data in your world irrelevant. Often, smaller subsets of your existing data can be used on its own or combined with external data to produce the desired results. The big data trend applies to all companies, regardless of how high their pile is.
Only large firms are challenged by big data
Small organisations should be as data-driven as large enterprises. Regardless of the size of the company, it's better to make decisions based on information than to simply rely on intuition or gut feelings. Smaller companies may depend on (or only be able to make) big data-driven decisions less frequently than their larger counterparts, but course corrections can be done faster based on the decision. So, while the trend is 'big' data and some in turn assume 'big' company, smaller companies can use best practices to be more data-driven and actually outpace or outmanoeuvre bigger, slower competitors.
It's true that the initial work being done around big data is based on complicated algorithms written by data scientists but, in reality, taking advantage of big data is really about being more data-driven. The mindset and commitment to being data-driven does not require you to be on the bleeding edge of algorithms and sophisticated analysis. You can start with just a desire to better understand the data you have, and to improve the analysis of that information.
With the tooling that is now available, you don't have to have a data scientist. Vendors are producing packaged software that creates the algorithms for you, so you don't need custom work by a data scientist with expertise in writing complex algorithms. Many of these new solutions are designed specifically for smaller organisations that don't have those resources, staff, or huge budgets to support enormous expenditures. The capability is there for you to start small and relatively simple. Don't think otherwise.
Being great at traditional business intelligence
The traditional goals of BI were really about reporting on historical data, but you don't have to know what happened in the past to be able to turn your eye to what's happening right now, and what you can make happen in the future. BI reporting of the past was more static and structured, but now it's more about patterns and relationships between data. Looking at the patterns will point you forward, so you don't necessarily have to know what you did in the past. Practically speaking, you don't need a good dashboarding and standard reporting framework in place to take on analysis of your data.
The misconception is that you have to get really good at doing something the old way before you can do it the new way. That's just not true. (Did my three-year-old daughter need to know how to use a landline before using my smartphone? Not at all.) You can be data-driven and look for the patterns in your information without first mastering the traditional ways of BI. You don't have to have a single massive data warehouse, or standard reporting that you process and look at once a month. This is just a different way of looking at your data, and the key is to think about it from the business and IT perspectives together. Start the way that big data requires you to start - with a business question that you need to answer.
Hadoop guarantees big data success
Hadoop is simply a technology framework. It's powerful and is changing many things in our world today. But, it's not the place to start. To be successful with big data, you have to start with the business objectives. You may or may not need Hadoop to get the answers you need. You may not need new technologies at all. Standing up Hadoop and loading up 50 terabytes of data doesn't get you anything by itself because you still need to have a business objective to move forward. We have many customers who load their data into Hadoop and then say, "now what?" That's backwards. Start with the business objectives instead of the technology and you likely will find yourself travelling along a different road.
We have a number of customers who have shattered all of these myths. A major retailer started with the basic business question of how to increase in-store sales. There were a number of approaches to consider, most of which revolved around fairly expensive ways of getting more people into the stores. But the firm started with how to get people to buy more once they were already in the store. So, they did some in-store analysis. We gave them the ability to mine their security video data to see what customers were doing in the store, and we narrowed down a relationship between actual purchases and customers going into a dressing room. The pattern showed that if they went into the dressing room, customers were 50 per cent more likely to make a purchase. So the company's goal focused on getting more people in the dressing room. Basically, they had no need for an expensive global marketing plan. They just needed sales associates to encourage customers to try things on.
One other point worth noting is that, like many technology trends, big data tends to start with the IT department, from the bottom up, as people start playing with the technology associated with the trend. This is often the accepted way, but it's wrong. To achieve success with big data, you need to start at the top of the organisation. You need to be data-driven from the top, and trusting what the data tells you must start at the top. In the case of the retailer, for example, traditional thinking would have included a marketing campaign to get more people into the store, or spending money to improve the product. Because management was willing to simply look at patterns and relationships in the data, the company found a much more cost-effective and impactful way to answer its business question.
So take a step back and take a breath - big data is here to stay, but it doesn't have to be daunting. Organisations of all sizes can take advantage of what the big data trend has to offer. Instead of buying into the myths, start with a simple business question, look at the data you have, and, with the right tools and the right approach, you can turn your data into the basis for informed decisions and improved business performance.
Darin Bartik is executive director of product management for Dell Software's Information Management solutions.