Skip to main content

The evolution of machine learning to command dark data

(Image credit: Image Credit: Pitney Bowes Software)

We have been exhausting time and resources pushing aside the benefits of dark data, quickly dismissing the potential it can offer a business or industry. Dark data is a type of unstructured, untagged and untapped data that has not yet been analysed or processed. It is similar to big data, but differs in how it is often neglected by business and IT administrators in terms of its value. With 80 per cent of data considered as dark data, there is undeniably enough information for companies to utilise to their advantage. Therefore, it is now time to bring light to the unknown gold mine of dark data.

For most businesses, understanding the vast amount of dark data can be an overwhelming challenge. Generally, businesses will use excuses like legality issues, legacy workflows or architectural costs as to why it has been reluctant to maximise its dark data. It may also fear that accessing dark data can occupy valuable time which could be used for other tasks and unsettle employees with new ways of operating. Of course, this disruption can be kept to a minimum when implemented precisely with the correct tools.

The rise of machine learning

For the majority of companies today, modifying unstructured data into readable assets involves lengthy processes that are mostly manual. To generate better value, businesses need to automate these processes and minimise resources spent on mundane tasks, and this is where machine learning can help. Machine learning is an application of artificial intelligence (AI), that provides systems with the ability to automatically learn and accomplish the equivalent of continuously running programmes in a fraction of the time.

Businesses can utilise machine learning to build models that work in a particular business function or industry. In the case of dark data, the process of learning starts with data observations to look for patterns that will help make better decisions in the future based on previous examples. Typically the system alerts business users to exceptions, and remembers when these are addressed so that it can automatically offer a solution the next time a similar event occurs. If users keep accepting the recommended solution, the system will learn accordingly.

Structural changes are necessary when implementing machine learning, which costs time and money. But this can be justified in the long term as the business benefits will guarantee a high return on investment.

Unleashing the benefits

At a first glance, dark data can appear unintelligible but when approached correctly, it can unlock benefits for businesses and boost the bottom line. The key to uncovering dark data’s secrets lies in the ability to understand the relationships between seemingly unrelated pieces of information. Machine learning plays a critical role in helping businesses uncover information and reveals a host of patterns or insights that would have otherwise been overlooked.

In today’s market where data is competitive currency, dark data is critical as it allows businesses to learn more about elements of their operations. In fact a recent survey stated that, 76 per cent agreed that businesses who have the most data is going to “win”. By expanding the amount of information analysed by a business, including customer data insights, new levels of innovation and flexibility can be leveraged to create a competitive advantage. If a business fails to maximise new forms of data in today’s digital age, it risks becoming stagnant compared to its competitors.

Bringing light to dark data

As complex as traditional machine learning may seem, it’s still machine like. It largely requires domain expertise and human intervention, which means it’s only capable of what it's designed for. For businesses looking to go one step further and streamline its data extraction processes, deep learning could hold more promise.

Contrary to traditional approaches focused on text only, deep learning looks to extract relations conveyed jointly via textual, structural, tabular, and even visual expressions by using new techniques to automatically capture the features needed to accurately extract relationships from richly formatted data. The biggest advantage to deep learning is that it tries to learn high-level features from data in an incremental manner. This eliminates the need for domain expertise and hard core feature extraction.

Overall, it's important to look at dark data as unfulfilled value. The key to managing dark data lies not only in gathering it, but in analysing it to discover useful patterns and insights. By adopting new technologies around machine learning, such as deep learning, businesses can combine structured and unstructured data to generate high-value results. When executed correctly, the business benefits will easily outweigh the costs and time involved in mining dark data. By allowing machine learning to unleash dark data, businesses can reveal new insights and knowledge that will yield a greater competitive advantage and boost the bottom line.

Laurent Louvrier, SuccessData (opens in new tab)

Laurent Louvrier, SuccessData.