Skip to main content

Digital transformation: Reducing “time to data” with a data catalogue

(Image credit: Image Credit: Chombosan / Shutterstock)

Managing and using data for enterprise digital transformation has, until now, offered employees little, particularly when compared with our typical consumer experience of doing a Google search for information or using Yelp to consider options.

As consumers, we have high expectations for data discovery and choice. The web offers a vast catalogue of information that is easily searched and sorted allowing us to select exactly the right choice that meets our wants and needs. Yet, until now, in our working lives, we simply haven’t had that same breadth of easily digestible information at our fingertips.

So, what has changed? Many organisations are making it easier for employees to discover and access secure, trusted, high-quality enterprise data, they can use to identify new opportunities for problem-solving, innovation, and revenue growth.

Empowering employee access to data

Data is typically scattered across hundreds or even thousands of cloud and on-premises systems, from legacy transactional databases and spreadsheets to cloud-based marketing systems and data lakes. Adding to the complexity is the influx of newer data sources and applications, such as the internet of things (IoT) and artificial intelligence (AI).

If data is an essential element in digital transformation, the difficulty for employees to find the right data when they need it is a major reason why so many organisations fall short in their digital transformation initiatives.

Whether the goal is improving the customer experience, delivering analytical insights for decision-making, or migrating operations to the cloud, success depends on the ability of employees to track down relevant data and understand its quality and provenance. And, with experts forecasting that the amount of enterprise data will double every two years (if not more), this challenge is getting more complex.

The result is that much of the data that could be valuable in launching ambitious digital transformation efforts is vastly underutilised, if it’s used at all.

“Most organisations use only a small percentage of the data they have access to — in my experience less than 5 per cent — even though they continue to collect and store terabytes of data,” says Shervin Khodabandeh, partner and managing director at Boston Consulting Group. 

Data catalogue, the treasure map for the digital era

Ideally, business and IT users could search for enterprise data as easily as running a Google search. And have access to ratings and reviews on the data from other users to guide them just like we use Yelp to guide our personal choices. 

To achieve this, enterprise information needs to be catalogued and classified in a logical fashion — “democratised” for use by business users, data scientists, application developers, and other stakeholders across the organisation. Nontechnical business analysts would have self-service access via semantic search, similar to the way consumers filter retail products by brand, color, and other attributes. They would have the context needed to understand and trust the data — where the data is coming from, who uses it, what other data is it related to, and the quality of the data.

That search effort would deliver relevant results no matter where data resides in the enterprise because it’s powered by an intelligent data catalogue, a technology layer to inventory data across the cloud and on-premises to make it accessible. AI and machine learning capabilities make data catalogues “intelligent,” for auto tagging with extreme accuracy, analysing data similarity and defining lineage, and, above all, empowering speed, scale, automation, and insights enterprise data management needs in the digital era.

“Managing data in today’s world without a data catalogue is ill advised and impractical,” says a report by Eckerson Group, a research and consulting firm.  “We’re moving rapidly to an era where communication, collaboration, and crowdsourcing are the mainstays of data management.”

If data isn’t consistent, comprehensive, and accurate, digital transformation efforts may fall short of objectives in a wide range of areas, such as:

  • Establishing the foundation for advanced analytics. Data scientists often spend 80 per cent of their time searching for data, and just 20 per cent on actual AI/ML and modelling. A data catalogue reverses the equation by providing quick data discoverability and access to relevant information. That lets data scientists and business analysts use trusted data to deliver the insights needed for data-driven decision-making.
  • Creating a holistic customer-centric experience. Because customer data exists in so many corners of the enterprise, it’s essential for organisations to have a holistic 360-degree view across all sources if they are to truly understand customers as individuals. By identifying all key sources of customer data, a data catalogue provides the foundation for more personalised engagement and improved customer experience.
  • Enabling seamless cloud data migration. Now that old-time myths about security and cost are effectively debunked, most organisations — even in hold-out industries such as healthcare and government — have embarked on journeys to the cloud. But moving an on-premises data warehouse to a cloud-based alternative, such as Amazon Redshift, Google Big Query, Microsoft Azure SQL Data Warehouse, or Snowflake, isn’t as simple as flipping a switch. Cataloguing data enables architects to first understand the data landscape, assess data quality, select the right data for migration, understand downstream impacts, and, ultimately, accelerate cloud data warehouse modernisation.
  • Ensuring data governance and privacy. Meeting the demands of existing and looming data security and privacy regulations can’t be accomplished if organisations are in the dark about what data they have, where it exists, and how they are permitted to use it. A data catalogue supplies the data discovery capabilities critical to identifying and managing data under governance controls and establishing digital trust with customers, employees, and other critical stakeholders.

Digital transformation takes many forms. Customer-centric business models, IoT and AI initiatives, workforce empowerment, and other mission-critical, data-dependent projects are now on CEO and board agendas worldwide. And catalogue-based intelligent data management establishes the foundation for digital transformation to survive and thrive in the face of constant change.

Ronen Schwartz, Senior Vice President and General Manager, Data Integration, Data Engineering, and Cloud, Informatica