Skip to main content

Know your dark data to know your business – and its potential!

(Image credit: Image Credit: Shutterstock/Sergey Nivens)

Data has become one of any organisation’s most valuable assets – but data can also be its most dangerous. Organisations continue to create and gather more data, yet they can’t inventory what they have, let alone efficiently keep track of it. Whether it’s spread across multiple data siloes or stored in a data lake, the information users and decision-makers need is right under their noses yet not at their fingertips. Organisations can’t get by stating they “don’t know what they don’t know” as regulatory standards become stricter and competition becomes fiercer. This not only leaves organisations at risk for non-compliance, it also means missing out on valuable insights that could help them improve products, services, productivity, and customer experiences. 

Today, every organisation must be a responsive, information business– and as such, they need to have a comprehensive understanding of all their data. With the massive volume of data being created and collected, this can be a daunting task. The entire digital universe is expected to reach 44 zettabytes this year – and in five years, it’s estimated that 463 exabytes of data will be generated every day globally. Organisations must lay the right data management foundation now so that they neither get overwhelmed by big data challenges nor fumble the opportunity to drive their business forward.  

Bring dark data to life

Despite collecting more data than ever, 60 per cent of organisations say half or more of their data can be considered “dark.” Gartner defines dark data as “the information assets organisations collect, process and store during regular business activities, but generally fail to use for other purposes (for example, analytics, business relationships and direct monetising).” The accumulation of this data can seem inevitable, as companies collect data from multiple sources – server networks, digital transactions, call records and web traffic to name a few. Despite how common and mundane dark data can seem, it’s anything but benign. Gartner states that storing and securing this data typically results in more expenses and risk than actual value.  

Organisations need to get their arms around dark data. From a defensive standpoint, if companies don’t know where they store their sensitive data, they cannot demonstrate compliance with data privacy regulations such as GDPR or CCPA. This exposes to them to auditors, fines and damaged reputations. From an offensive perspective, companies leave money on the table if they don’t use all – or most – of the data they collect to inform their decisions and business strategies. So, where should organisations start?

Start with data discovery 

According to our forthcoming survey, the number one barrier stopping organisations from realising full value from their information is finding and gaining access to the right data. To shed light on dark data, organisations need to lay the foundation for improved data management. They must know what data they have, where it came from, who’s using it and how it has transformed. This process starts with building a data inventory, which collects metadata from both internal and external data sources. This must be completed so organisations know what business assets and data are connected to build and refine a business glossary. Having this business context established drives more pinpoint identification of personal data identification to reduce risk of non-compliance. From there, organisations should invest in capabilities such as metadata management and tagging to better understand data relevancy and value – i.e., who should have access to the data and how it should be leveraged.

Once established, organisations can share this newly discovered metadata repository to business users, data stewards and data specialists. Doing so allows less technical users to add additional business context and shine a light on their dark data. According to Splunk’s “The State of Dark Data” report, 75 per cent of the business leaders surveyed suggested that if organisations implemented tools that allowed less technical employees to analyse large data sets, it would help them alleviate the dark data problem. 

Establish data trust

Simply discovering dark data is not enough. Organisations must establish trust with the data to move forward. Bad data – or data that organisations don’t understand – cannot be used to glean insights, inform business decisions or fuel offensive data strategies. For example, if enterprises want to leverage artificial intelligence (AI) and machine learning (ML) to improve to their customer experiences, they need to feed it trustworthy data. If they use bad, wrong or non-compliant data, the AI and ML programs will glean equally bad or wrong insights. As a result, the organisation is vulnerable to misdirection, poor customer satisfaction, or worse, liability. 

To monetise their dark data, organisations must set up processes around their data inventory that empower business users to quickly discover, understand, access and trust enterprise data. An important piece of this is also leveraging AI and automation for their own data management. After starting with the data inventory to understand the good, bad and ugly data, organisations can use data intelligence tools to automate data lineage and trace data from origin to wherever it delivers value. Traceable data is trusted data, creating confidence in decisions and reducing risk – and automation makes data lineage increasingly possible and cost effective. 

With a better data management foundation established, organisations can then develop a governance strategy to ensure data is trusted and compliant moving forward. It should:

  • Manage and progress data related issues with built in workflows and data stewardship 
  • Create and share a common language of business terms and definitions via a business glossary
  • Link business terms to underlying data definitions in the repository

The rate of data-driven business is only accelerating as organisations seek to become more responsive. It is absolutely critical that today’s businesses lay a foundation that is both solid and scalable. Decision makers in the organisation – from IT team leaders to the C-suite – must make discovering dark data a priority. Otherwise, it will only grow, along with the liability and expenses it brings. Establishing an inventory, building trust and understanding data will not only help organisations know their business more clearly, but it will help protect and empower their businesses to be more responsive moving forward.

Kyle McNabb, SVP of Product Marketing, ASG Technologies