From DevOps to DataOps

Most of us are familiar with the idea of DevOps, the series of slick processes which nowadays combine application DEVelopment with IT OPerations replacing the proven, but slower, methods IT teams once used to build enterprise applications.

It is fair to say DevOps is now becoming ‘business as usual’ at most large organisations. Companies like Facebook and Google would not build apps any other way. Even IT’s traditionalists, such as analyst firm Gartner, have become cautiously optimistic about the new way of working, noting: 'DevOps will evolve from a niche to a mainstream strategy employed by 25 percent of Global 2000 organisations.'

Speed or ‘Time to Value’ and continuous improvement have made application teams fall in love with DevOps making it a no-brainer for most. DevOps has redefined the software lifecycle, changing if not lives, certainly IT careers. Now though it has a rebellious younger sister, DataOps, who promises the same improvements in terms of Time to Value but with a view to continuously improving not development, but DATA OPerations. Who is this new kid on the block?

DataOps defined

Some DataOps definitions make it sound like an impossible dream, including 'A data management method that emphasizes communication, collaboration, integration, automation and measurement of cooperation between data engineers, data scientists and other data professionals'. The reality is much less long-winded. We believe there are four layers to the DataOps process, which in most cases means four skillsets, if not always four individuals, involved in a DataOps cycle. As with DevOps they involve IT and non-IT professionals. From the top of the chain, they are:

  • Information Consumers - make business decisions e.g. CFOs, CIOs etc
  • Business Analysts - understand the data and create business information via data models e.g. Business Intelligence/Business Warehouse consultants, report writers and now, Data Scientists
  • Platform Operations - understand how to manage the data platform e.g. Database Administrators and now, DataOps professionals
  • Infrastructure Operations - understand how to manage the IT Infrastructure housing the data e.g. managers of the hardware and Operating Systems, Sysadmin's, and now DataOps

What’s changed?

In our view, DataOps is particularly important if you have rolled out business intelligence (BI) products, like SAP HANA and BW, which rely on effective collaboration between teams to make the best use of the data. It is this blending of teams, or skill sets, which make DataOps sound similar to DevOps. There are however significant differences.

Large datasets and the real-time nature of modern data-intensive applications require specific ‘Big Data’ skills. In addition, the business’s swelling ranks of information consumers are demanding combined intelligence from Internet of Things devices or vehicles, from public data sources, to use unstructured data, and deliver real-time results from in-memory data, the term data scientist has started to take hold. This is where DevOps and DataOps diverge.

Why DataOps matters for you

Along with new job descriptions and the changing demands of the job, the analytical tools have become more complex. Instead of ‘merely’ blending the role of DBAs and IT operations professionals, the key skills required from the DataOps team are even more varied.

Amongst other things, DataOps gurus require a detailed understanding of the processes, likely timing and the multi-tiered infrastructure required to extract business value from systems like HANA. For example, questions DataOps professionals need to answer at any time may include:

  • Where do I store the data needed to achieve results on time?
  • How will I manage data growth and data retention to stay within budget?
  • How do I report the cost of valuable data vs 'dark data'?
  • During a systems refresh, how do I manage test data sets so they return indicative results which approximate to the live production data set?

Given the scope of DataOps, it may be tempting to stick to ‘Business As Usual’. In our view this is not an option. If an organisation is already using Business Intelligence, this will normally drive an appetite for DataOps because today’s Information Consumers are hungry to learn more and will demand more value from the data which is coursing through their organisations.

DataOps and DevOps as standard

The growth of DataOps promises many of the benefits which DevOps is delivering from continuous improvement to faster Time To Value from IT initiatives. Its scope though is, if anything, wider and will require a broader range of skillsets and touch even more individuals within an organisation, perhaps even realigning departmental responsibilities.

When correctly deployed DataOps can ensure all data is used to its full capacity by extracting the 'goodness' from every data interaction. Over time as organisations realise its power they will develop new frameworks which will be much more efficient in their extraction of intelligence from raw data. At Centiq, we are ‘all in’ with DataOps and we think most organisations will number both DevOps and DataOps professionals amongst their teams much sooner than IT Traditionalists currently believe.

Robin Webster, director of technology and services, Centiq