Skip to main content

Big data or die

(Image credit: Image Credit: Nexis Solutions)

The world knows what the deal is with big data: Accenture’s ‘Big Success with Big Data (opens in new tab)” study found that 79 per cent of enterprise execs say that companies who don't embrace big data will lose market strength and may face extinction. Big data is widely seen as the lifeblood of all organisations who are connected to the internet (and by now, who isn’t?). Furthermore, 89 per cent of respondents believe big data will revolutionise business operations in the same way the Internet did. Early adopters see a competitive advantage in big data and are rapidly moving to disrupt their own data practices.

But the crux of this is that many organisations are simply struggling to operationalise big data. Here at Unravel, we recently conducted a piece of research into organisations’ opinions and expectations of their big data stacks, and it was revealed that only 17 per cent of respondents rated the performance of their big data stack as ‘optimal’, meeting almost all KPIs and targets. This is largely due to hurdles such as lack of the right skills, cost and the time taken to derive valuable and actionable insights.

So how are we going to get through the door to data nirvana if the key lies just beyond our grasp? The answer is optimisation. But first, we must evaluate the misgivings business that chiefs gave with their current data stacks in order to fully understand why we as a DataOps community need optimisation so desperately.

As skills remain scarce, will the big data engine grind to a halt?

Our research revealed a broad range of pain points for those working across IT operations. However, a lack of skills kept coming back as a constant hindrance in the pursuit of data stack synergy, with 36 per cent of respondents listing it as a major pain point. Within this skills gap, the most pressing need is for big data architects - an issue for almost half of organisations (45 per cent).

As big data continues to explode and flood our day to day lives with more and more information we need big data architects to handle databases on a large scale and make it possible for data scientists and analysts to comb through this data deluge to pull out actionable insights to make life better for the stakeholders that need insights to make business decisions. They are crucial to achieving what business leaders want for their organisations, especially as they have their eyes set on improved data analysis, transformation and visualisation. Architects will be incredibly integral to enabling the enterprise to reach these goals.

Clouds are high in the sky and we are still stuck on the ground

One of the other principal issues which are likely holding businesses back in their venture for data stack harmony and business bliss is that so many organisations do not yet host their big data applications in the cloud. Many have intentions to: 82 per cent of respondents noted that they have a strategy to move existing big data applications to the cloud. The inference here is that a lot of them do not already have their applications sitting within the cloud and therefore face the challenges of scaling up and down at will - with all the preparation and maintenance of infrastructure that this entails.

The benefits of hosting in the cloud are well known: More businesses are waking up to the possibilities that the cloud offers. The scalability of the cloud opens up the possibility for business infrastructure to encompass multiple servers and provide unprecedented levels of capacity. Hosting in the cloud also reduces the cost and improved the performance of big data applications. Moving to the cloud will likely unlock a lot of potential from the big data stack that businesses across the UK are yet to realise.

Dodge an early end to your organisation’s life with APM

At present big data seems to be most profitable or effective when it’s used defensively. The top four reported use cases were:

  • Cybersecurity intelligence (42 per cent)
  • Risk, regulatory, compliance reporting (41 per cent)
  • Predictive analytics for preventative maintenance (35 per cent)
  • Fraud detection and prevention (35 per cent)

To move beyond the tried and tested into projects that promise a greater impact to the business application performance management (APM) solutions are the golden ticket to fine-tuning, caretaking, and supercharging the complex software and hardware collision that is the big data stack.

APMs, though comparatively new to the big data stack, are a class of technology well-known to the DevOps teams, used to being tasked to manage the tools and technologies of varied project groups within the enterprise.

APM is one technology that can support both sides of the divide, and aide the enterprise in finding common ground. Whether it is missed SLAs, failed jobs or workflows, slow jobs or queries, or computing resources unwisely allocated and causing delays or end-user frustrations... Preventing or fixing these problems cannot be done by just monitoring the big data platform and trying to fix issues using logs and graphs. In a typical big data deployment, that approach could not scale. Metaphorically the traditional approach of monitoring and debugging would be like trying to unravel the intertwined wires from holiday lights. It just cannot scale. There are just too many potential problems across too many different systems for DevOps to troubleshoot issues through trial-and-error and stay on time.

This technology promises to bring new ways of using data to businesses, however, the DevOps team will likely be managing hybrid platforms for the foreseeable future as this is not an overnight transition. Leveraging the power of APMs and optimising processes within businesses will reveal the true possibilities of the big data stack, and more business leaders will begin to see this tech meeting its KPIs, aiding the reduction of costs and time management across the business.

In a ‘big data or die’ world, it’s time to get serious about solving the challenges that come with complex, fast, evolving big data stacks. The main challenge now is to ensure the big data stack performs reliably and efficiently, and that big data teams have the tools and expertise to deliver the next generation of applications, analytics, AI and Machine Learning.

Kunal Agarwal, CEO, Unravel Data (opens in new tab)  
Image Credit: Nexis Solutions

Kunal Agarwal co-founded Unravel Data in 2013 and serves as CEO. Mr. Agarwal has led sales and implementation of Oracle products at several Fortune 100 companies. He co-founded, a pioneer in personalised shopping and what-to-wear recommendations. Before, he helped Sun Microsystems run Big Data infrastructure such as Sun's Grid Computing Engine. Mr. Agarwal holds a bachelors in Computer Engineering from Valparaiso University and an M.B.A from The Fuqua School of Business, Duke University.