Skip to main content

Dealing with data all day, every day – how to use continuous intelligence

(Image credit: Future)

Every company today seems to be amassing data at faster and faster rates. Worldwide, IDC predicts that the global total for data will be 175 zettabytes of data by 2025, up from 33 zettabytes in 2018, and spread across data centres, cloud and edge devices. However, having all this information is not useful unless it can support the delivery of better processes, faster decisions or smarter employees. With so much information being created all the time, making use of this data has to be a continuous process too.

Gartner identifies the term “continuous intelligence,” where data is consistently processed as it is created or comes into the company in real time and then used. That use can be for an individual in a specific team or department, or it can be part of a joined-up approach across a company. For something that appears to deliver exactly what businesses want from their data, why is it not everywhere already?

Data, data everywhere

In Samuel Taylor Coleridge’s 1834 poem The Rime of The Ancient Mariner, one of the most famous verses is “Water, water, every where, / And all the boards did shrink; / Water, water, every where, / Nor any drop to drink.” Being surrounded by data that is not easy to manage or understand is very similar – with so much potential data at our disposal, it can be easy to get frustrated. These gaps in data are problematic as they make it harder to get the full and accurate picture of what is taking place; without solving these information gaps, it is difficult for companies to fully succeed in an information economy.

One of the main challenges around data is that it exists in silos across an organisation. In the software development team, data on each application can be gathered in the forms of log data, metrics and tracing. Log data tends to be used after an incident to investigate what happened and how to either fix the issue or prevent future problems. Metric data shows how well – or how badly – an application is performing. Tracing data shows the application’s path and how it executes over the course of a request. This is particularly useful for modern applications that are built on lots of separate components in microservices environments.

Each of these sets of data can be used on their own, but they become more powerful and more useful together in context. Taking these multiple strands and automatically creating an understandable and useful set of insight data can not only reduce the workload for the team; it creates the opportunity to use that data in smarter ways. The information gap here is one of context, as combining each set of data can be used to improve decision making and results over time.

The issue of data silos is also present across multiple teams. IT security departments will have a set of data covering all the activities across a company’s applications so they can spot anomalies and attacks. Most of the attacks will be automated scripts that are looking for vulnerabilities which can be detected and blocked automatically, so the security team can spend more time on valuable analysis and prevention. However, that set of data on application actions is the same as that used by a DevOps or software team and based on logs.

When both DevOps and IT security collect log data at the same time, this leads to twice the amount of data being stored over time. While the analysis tasks and end goals are very different, the data itself is the same. This doubling up on data can quickly get expensive. Other teams may also want copies of the same data, leading to more silos of information that all take up space. The information gaps here are that each team can only see its individual needs, rather than the impact that their actions have.

From a practical point, cloud computing has helped with some of the problems around this data - it offers a way to store and manage much more data than a single company could realistically and economically achieve on their own, but it has also led to more silos of data being created, rather than less. However, cloud is essential to managing the problem of scale around data, as it can be a focal point for making data available to multiple teams for multiple purposes.

Dealing with data continuously

So how can IT teams collaborate more effectively on this data? The first step is to recognise the problem and look for information gaps within the organisation. This involves looking at processes and data sets that are already being created and how they are analysed. By spotting duplications of data for logging, for example, you can reduce the cost associated with running IT as a whole.

The other result from this is to find areas where data is not being shared, or where there are current gaps in the data record. This can help improve collaboration around data and how it is used, while it can also flag opportunities for teams to share their data more effectively or use it in new ways. This is not an overnight process – instead, it relies on teams being curious about each other’s results and how they get put together. This can spur on new thoughts and approaches in turn.

While many company leadership teams want to improve this, it can be hard to mandate from the top down. Telling people to collaborate is not enough on its own – instead, it has to be nurtured from a cultural perspective too. This encourages more cross-team collaboration and data sharing over time. Management support and direction can make these discussions more powerful and more likely to succeed over time.

The last point is how to deal with data continuously – with business processes generating data all the time, the analysis process has to be automated and continuous too. This continuous intelligence approach relies on that constant stream of data coming in, being sorted and managed automatically in the background, and then passed out to the teams involved for them to use in their own ways.

Let’s go back to the logging example – for the software development team, logs and metrics can be looked at for potential problems in applications that might affect performance of the business; security, on the other hand, is looking for potential issues around compliance or attacks. Both teams can look at this data continuously, but the results will be in different dashboards and aligned to their specific goals.

The future for continuous intelligence

According to Gartner, over half of all new business systems will be built with continuous intelligence embedded in their operations by 2022. What makes continuous intelligence useful is that it provides real-time analysis and decision support to the individuals and teams involved. By finding where data already exists in multiple silos, and where information gaps exist for those teams, organisations can improve how well they make use of the data they create.

This involves a mindset change around data and how teams collaborate, but what it leads to is smarter ways of working across the entire business. By finding and solving those information gaps that exist, companies can improve their performance in the information economy using continuous intelligence.

Christian Beedgen, Chief Technology Officer and Co-Founder, Sumo Logic