Skip to main content

Data of future past: A guide to futureproofing workloads

(Image credit: Image Credit: Billion Photos / Shutterstock)

There is a growing need among modern organisations for solutions that can simplify and yield insights from their ever-more complicated data systems. These systems often create demanding, business-critical challenges that need to be navigated through the intelligent use of various data platforms. However, as the amount of data being generated within organisations increases, the number of platforms that are up to the task is slowly diminishing, making the correct choice more difficult. Recent estimates project that this year alone, 44 zettabytes (44 trillion gigabytes) will be created and copied. To deal with this unprecedented volume of data and the problems it will create within data pipelines, organisations will need to give due consideration to the platforms they’re using and whether they will remain viable moving into the next decade.

In this rapidly expanding data landscape, forward-thinking organisations across industries are already adopting new software to build, operate and monitor their applications. Not only are these integral for the applications themselves but for the supporting systems too. These new software are so fundamental to the future of cloud applications that Morgan Stanley estimates ‘New Stack’ revenue will reach $50 billion by 2022. With this in mind, it is worth thinking about whether the recent poor quarterly performance of Cloudera and MapR is the shape of things to come, or merely a temporary trend. In other words, will applications running on Cloudera and MapR be able to support business needs in the future or is a transition to ‘New Stack’ technologies inevitable?

In response to the poor performance of these data giants, there have been suggestions that interest in data is starting to subside. However, with Morgan Stanley’s January 2019 CIO Survey predicting 50 per cent of application workloads to be located in public cloud environments by 2022 (a 22 per cent increase from today), there is empirical data to suggest an alternate answer. The reality is that interest in data is continuing to increase exponentially; it is fundamental to the operation of almost all organisations. Yet at the same time, cloud services are being adopted by enterprises at a much faster rate than expected by industry experts. This means the incumbent data platforms that were originally built for specific use in on-premise environments are becoming outmoded by cloud platforms that more closely align to the demand for cloud services among modern organisations.

The writing on the wall

In light of the maturity of cloud platforms and services, with its increasing automation and AI capabilities, organisations are overwhelmingly seeing it as the ideal environment for their data pipelines. As such, many companies and institutions are moving from a cloud-first strategy to a cloud-only one and their legacy platforms are gradually being phased out. MapR and Cloudera are becoming more marginalised and out-dated as modern platforms designed specifically for the cloud are needed. Aside from just being better suited for the enormous quantities of data in modern organisations, the cloud also has distinct advantages in terms of elasticity and scalability. These may sound like minor considerations, but with applications that are often subject to unpredictable changes in data volume, velocity and variety, cloud platforms can offer far more reliable performance in the long-term.

If the writing is on the wall then, and the cloud is fundamental in future-proofing workloads, what are the next steps for organisations? The answer depends on what stage of cloud migration they are on, or if they have even begun migrating at all. For organisations at the earlier stages of this process, planning will be their number one priority. While there may be enthusiasm to realise the benefits of the cloud as soon as possible, a poorly planned migration can be detrimental further down the line. Due consideration needs to be given to which data platforms are most suitable, which applications should be migrated and how this will be managed in continuity. Following this, the migration itself is the next step. Having undergone a thorough planning process, this stage should be a straightforward implementation of a step-by-step plan mapped out by data teams. But this isn’t where the process ends. Once the workloads have been migrated, data teams need to closely monitor them for performance in order to maximise their operational efficiency and whether decisions made in the planning phase are still viable. For instance, if the platform in use is matching the financial and operational predictions made in the planning phase.

A proactive approach

While choosing the right data platform is an integral decision in the cloud migration process, another factor for ensuring the longevity of applications is the expertise of the data teams that will be managing them. Regardless of if an organisation is using on-premise platforms like Cloudera or any of the myriad cloud platforms available, there is a necessity to build expertise in data operation. Without knowledge in DataOps, organisations will struggle to future proof their architectural, operational and commercial decisions, in the cloud or on-premise. As these applications become increasingly complex and the need to migrate or update them compounds, this knowledge only becomes more valuable in delivering a reliable application.

In summary, regardless of where an organisation is on the cloud migration journey, there is a pressing need to review what platforms are being used and whether data teams have the required expertise. A proactive approach, rather than a reactive one, will always be fundamental in keeping organisations ahead of the curve. For such a rapidly evolving environment as data, this is only becoming more true. As such, building a strong understanding of what is needed from the data, what platform can best facilitate this and ensuring data teams are the best suited to manage it is a strong foundation for any organisation.

Kunal Agarwal, CEO, Unravel (opens in new tab)

Kunal Agarwal co-founded Unravel Data in 2013 and serves as CEO. Mr. Agarwal has led sales and implementation of Oracle products at several Fortune 100 companies. He co-founded, a pioneer in personalised shopping and what-to-wear recommendations. Before, he helped Sun Microsystems run Big Data infrastructure such as Sun's Grid Computing Engine. Mr. Agarwal holds a bachelors in Computer Engineering from Valparaiso University and an M.B.A from The Fuqua School of Business, Duke University.