We recently had the chance to speak with with Rick Farnell, co-founder of data science/Hadoop company Think Big, who gave us the low-down on big data in business.
The full interview can be found below:
What are Think Big up to at the moment?
We are busy scaling our open-source big data solutions business across our initial target countries in Europe and APAC. We are aggressively hiring big data engineers, project managers, data scientists and sales executives in those target territories.
We're currently going through a big data boom. Is there any danger of this slowing down any time soon?
It is not even remotely possible to slow down. Our entire economy is driven by data, personalisation, predictive services and a future that promises to create many more companies that will link humans to machines and data like we have never seen before.
Every industry is undergoing a complete makeover to outfit itself with strategies, processes and technology so it will be relevant in our new world where data is king. The Internet era saw signs of promise for some of these models and companies (fear of being Amazoned) but the infrastructure was not there to support it, which is why it fell apart and had to be rebuilt over a decade.
Now there are arguably more connected devices (laptops, mobile phones, tablets, watches, etc) and machines on the internet than humans on the planet.
What's the single biggest benefit big data can offer to companies?
Innovation for the future, hiring talented young professionals and relevance to stay a viable company.
Why are some companies struggling to get to grips with big data?
Many companies think that big data is about the size of their data sitting in a relational database or a Teradata-style warehouse. It is not even close to that and that is a major misconception.
Big Data is a business model for using relevant open-source technology integrated with your business and changing at the speed of light to create new capabilities and services for your internal workers and your external customers and partners. It is not consolidating a warehouse for reporting.
The open-source software market is moving and innovating at the speed of light and if companies cannot set themselves up to take advantage of these capabilities they will be left behind.
Having all this data brings with it some serious security concerns. What should firms be doing to stay secure?
Companies need to work with services-providers that live, eat, drink and sleep in the big data space i.e. Think Big. Many services and solution-providers are trying to morph into big data relevance by having groups within their traditional keep-the-lights-on legacy data warehousing, BI or systems integration businesses.
Security (especially in the big data space) is a massive deal which is why Think Big has done so well across the five years we have been a company focused 100 per cent on this big data solutions space.
There are hundreds of companies that had traditional legacy data warehouse services business that think they can just start doing Hadoop, NoSQL and streaming platforms and solutions in the cloud and on-premises. It is a very different business, a very different skill set and a very different way of working with technology.
It would be like being back in the late 1990s, when a firm that has a bunch of mainframe resources wakes up one day and says that it builds highly-secure distributed web-based platforms and solutions using modern-day application servers, web servers, identity management, and the internet protocos.
In which industries can big data have the biggest effect?
There is not an industry on the planet that cannot have major breakthroughs and incredible value created as a result of getting on the big data innovation wave.
On the one hand, I would give the example of companies that are trying to predict where their customers will be in the future, what they will want and are having a service ready for them. Companies are sensor-enabling their business in some way to better predict machine failure and to be ready with a service. They may even intervene to prevent the failure occurring.
On the other hand it could be a healthcare or disease research company that utilises big data and enhanced low-cost access to a wealth of data and incredible performance to come up with major breakthrough.
They will process the data at tiny fractions of the cost of what they thought they could do on their own legacy infrastructure. At the end of the day, from among our clients we have seen incredible value across many industries and the common themes of these companies are passionate visionary leaders and a hand-in-glove partnership with Think Big to make the vision a reality through the use of big data technologies.
Is there such a thing as too much data?
Only if you don’t know how to use it or how to get value from it.
What trends do you think will come to the fore in the industry in the next few months?
Over the next year-to-two-years, Hadoop Platforms will become a must have capability present in every client, regardless of size or the industry.
In the next few months we will see many more clients using technologies like Apache Spark for streaming solutions related to big data and data science use cases.