DataOps – Reaching big data nirvana

DataOps re-organises the organisation so that the data teams is decoupled from the day-to-day demands of analysts.

Big data and business analytics are finally hitting mainstream. Enterprises are expanding efforts, and there is a wave of adoption as the value of big data finally seems within reach. According to International Data Corporation’s (IDC) latest Worldwide Semiannual Big Data and Analytics Spending Guide, worldwide revenues for big data and business analytics (BDA) will reach $150.8 billion in 2017. While encouraging, businesses trying to scale their efforts to take full advantage their data are still invariably hitting a wall.   

A recent survey conducted by Dimensional Research, polled 406 IT and data professionals globally to find 78 percent of companies are handling big data as a case by case basis. Similarly, the same survey found that many IT organisations are looking to adopt a self-service model as they struggle to keep up with business demand. The survey makes it clear that despite the fervour, big data efforts have largely failed to operationalise and are instead struggling piece-by-piece efforts.

For a growing enterprise, the overwhelming queue of big data uses cases can only be remedied by the transition to a self-service model. Making that transition requires a new way to think about running a data organisation, and approach that takes a page from DevOps called DataOps. 

Re-organising the organisation

DataOps re-organises the organisation so that the data teams is decoupled from the day-to-day demands of analysts to enable more agility and ultimately, faster time to insight. When the data teams are turning the crank on the service organisation then bottlenecks emerge. Instead, much like DevOps, data organisations should decouple data teams from users, which ultimately enables both teams to work more efficiently. DataOps allows organisations to eliminate silos while promoting a culture that understands the importance of data. In a true data-driven organisation, no one comes to a meeting armed only with hunches or intuition. The person with the superior title or largest salary doesn’t win the discussion. Facts do. Numbers. Quantitative analyses. 

Similar to DevOps, DataOps promotes communication between siloed data, teams and systems to connect the people who collect and prepare the data, those who analyse the data, and those who put the findings from those analyses to good business use. You need to eliminate silos of data while seeking out new sources to inform your decision-making. When mining data for insights, demanding data from different and independent sources leads to much better decisions. Today, both the sources and the amount of data you can collect has increased by orders of magnitude. It’s a connected world, given all the transactions, interactions, and, increasingly, sensors that are generating data. Combining multiple independent sources results in better insights.. The companies that do this are in much better shape, financially and operationally.

A DataOps approach to achieve a self-service data model transforms an organisation to the data-insights driven organisation with the advantage in this big-data era. A marriage between cloud services and big data takes away the worry of scalability and allows for the value of big data to be actualised. “An organisation’s commitment to becoming data-driven is only as good as its data strategy and data platform”, said Kash Iftikhar, Vice President of Engineering Oracle Cloud Platform. Creating a self-service data model through DataOps holds the key for data teams to lead their companies through the transformation to becoming a data insights driven organisation.

To achieve a data-driven self-service model via DataOps, there are five stages a company goes through in a data maturity model:

  • Stage 1: Aspiration – The data team acts as a service organisation for the rest of the company through production reporting and data warehouse research. In this stage, the use cases are sparsely spread and completed manually without taking the full capacity of the data team.
  • Stage 2: Experimentation – As this stage, a company’s use cases are taking more time to fulfill. The data team starts to experiment with big data deployments and targeted use cases of analytics operations. These are the newest and growing companies adopting big data for the first time.
  • Stage 3: Expansion – When a company grows larger and scales, the multiple departments are working with a number of analytics engines operating in a top-down method for business functions. Each department is siloed off in its own a hierarchy of needs working within their own department.
  • Stage 4: Inversion – At this point, enterprise transformation occurs, with teams now focusing on bottoms-up uses cases based on collected analytics insights. Those in direct contact and working with the data are able to provide valuable feedback and insights to influence business decisions at the top level.
  • Stage 5: Nirvana – The final stage, organisations becomes truly “digital” with universal insights available to all teams and guiding decisions. A company in this final stage is providing data insights throughout the whole company to utilise across all business decisions.

Facebook is a great example of a company that reached self-service model and data nirvana. When I worked there in 2011, there were 10s of petabytes representing the whole internet. Fast forward to 2015, and suddenly Facebook is processing 10 petabytes each day. Thanks to automation and almost all the employees at Facebook use or interact with data at some form or shape. For example, Facebook collected a lot of details about users and the performance of ads. 

The data team would run it through a lot of data crunching and out comes the most relevant ads with higher success rates. With Facebook’s news feed, personalised stories were driven through data analytics. The vast amounts rankings, recommendations of people you may know, pages you may like, comes from the massive amount of data crunching and data analytics automatically running in the background without heavy manual lifting.

Ashish Thusoo, CEO and co-founder, Qubole
Image source: Shutterstock/wk1003mike