Skip to main content

Turning data into opportunity post-pandemic

Best cloud databases
(Image credit: Pixabay)

We all know about the importance of data. Over the years it’s become the lifeblood of every organization, regardless of size or sector. And, as digital continues to take precedence, every action, interaction and reaction produces more of it. In fact, Forbes recently estimated that over 2.5 quintillion bytes of data are being produced each day. A number that is only going to grow as time goes on. 

When harnessed effectively, this data becomes the single most valuable asset that any organization owns. Companies can use it to boost productivity and improve decision-making, enabling them to stand out from the competition and offer real value to their customers.

However, whilst many business leaders are all too aware of data’s value, few of them know how to truly maximize its potential. Data is almost worthless if it is not being analyzed correctly and understood. In our post-pandemic world, it has never been more important for businesses to take control and transition from being ‘data aware’ to ‘insight-driven’. However, it has also never been more challenging. 

Data is the key to survival

The complexities of data integration are nothing new. Often, regardless of industry, important information is dispersed across multiple data sources within an organization. Governments, public sector organizations, and private businesses alike are all challenged by the distribution and storage of data across a network of on-premise, multi-cloud, and third-party environments. Further complications often arise from silooed data, legacy applications (some of them never designed for data sharing) and a range of formats and protocols for communication.

Whilst this was an inconvenience before, in the midst of a global pandemic, it quickly became the difference between a business surviving or failing. This is because crises such as Covid-19 become even harder to manage when critical information is not made readily available to those that need it. Whether its medical practitioners at the point of care or businesses trying to navigate the changing landscape, a lack of critical information presents a never-ending list of serious problems.

In order to overcome the initial throws of the pandemic, businesses had to be more resilient and creative than ever before. And, as we emerge into this new hybrid landscape, the journey is far from over. Businesses around the globe must work out how to digitally connect to both employees and customers. At the same time, they need to be able to forecast what is happening in the market, and process that information almost instantly to make quick decisions. 

Ultimately, data is knowledge, and knowledge is power. But this knowledge needs to be delivered quickly and effectively in order to inform decision making during this time of uncertainty. However, in many cases, traditional, outdated data management tools are getting in the way.

Traditional processes no longer fit for purpose

In the past, probably the most popular method for retrieving data from multiple sources has been to Extract, Transform and Load (ETL). Through this, data files are extracted from an existing source, transformed into a common format, and loaded into a new data store – such as a database server, data store or data warehouse. Once complete, information can be made available to prescribed users, under pre-set access and security protocols. Essentially the data is moved and copied into a curated store for a single point of access for the business users.

This sounds positive, right? The problem however, is that ETL has been a standard method of mass data integration since the 1970s. It’s therefore no surprise that certain limitations are becoming increasingly apparent. The reality is that ETL processes and legacy data storage techniques make detailed data analytics almost impossible because you are always working from historic information. ETL also lacks any form of centralized access, which prevents businesses from utilizing all of their desired data. It creates significant bottlenecks for engineers due to the time and effort required to produce data sets, run queries, and perform other requests from business users. Over the years, data volumes have grown (and continue to do so), making ETL process more costly, cumbersome and error-prone than ever before..

To make matters worse, ETL’s method of duplicating data results in the creation of new data repositories which can quickly multiply into complex, siloed datasets with their own governance and security mechanisms. With the General Data Protection Regulation (GDPR) requiring robust personal data policies, strict record keeping and time limits on how long data can be stored, this could present a very real governance problem with potentially devastating consequences.

Using data virtualization to become insights-driven

Data virtualization avoids the movement and copying typical of ETL. In fact the principle is quite the opposite. Using these technologies, businesses can leave data at the source and do not need to move and copy it to another location. Instead they only abstract it if and when needed, for immediate consumption. 

In our domestic lives many of us now enjoy movies and music through Netflix, Amazon Prime, Spotify and other services where entertainment is streamed from somewhere unknown to us. We don’t have to worry about where the media comes from, yet we get immediate access to a vast choice of entertainment and there’s no need to store a copy of the data locally (in the form of CDs, DVDs and records etc). So why not use the same approach in business? 

This is the function of data virtualization! We see it widely used as a key delivery style in many blue chip organizations - including most of the top banks, retailers and manufacturers - as a key element in their data integration architecture. By automatically integrating disparate data sources, optimizing query requests and building a centralized governance architecture, data virtualization enables businesses to securely access the data they need easier and faster, boosting both the top and bottom line.

Data virtualization is a key technology powering an organization’s data fabric. The benefits of data virtualization include being agnostic of location, format or latency. It delivers information in real-time and in the appropriate format required by each individual user. This means that all business data, no matter where it is stored – whether on premise, in a cloud environment, data warehouse or data lake – can be brought together to create a complete, real-time view much faster than when using traditional processes. In fact, Forrester recently discovered that this type of technology could decrease data delivery time by 65% over ETL, saving $1.7 million.  

Data virtualization helps to reduce the burden on IT and data engineers, whilst enabling data scientists to quickly and intuitively get what they need to build models and develop insights. It can help businesses to improve their overall performance and efficiencies in a strategic manner, reducing costs and project cycle times and helping to enhance business decision-making capabilities with real-time insights. It also has some unquantified benefits, such as organizational flexibility and agility, customer and employee satisfaction and peace of mind when it comes to audits and security matters. 

In today’s landscape, data is everywhere. But its value doesn’t depend on how much you create or own, it depends on how you use it. The journey from ‘data aware’ to ‘insights-driven’ won’t be easy for many. It’ll involve an overhaul in terms of infrastructure and also mindset. However, modern technologies such as data virtualization could provide an answer for businesses looking to capitalize on their most valuable asset.

Charles Southwood, Regional VP, Denodo

Charles Southwood is Regional VP for Denodo Technologies, responsible for the company’s business revenues in Northern Europe and South Africa. Born and raised in Ascot, Berkshire with a degree in engineering, from Imperial College, London, Charles has over 20 years in technology sales and sales leadership where he has been involved in both start-up ventures, and the expansion of existing businesses.