Back in February, Gartner issued a very timely warning: that organizations using traditional analytics techniques that relied heavily on large amounts of historical data have had to wake up to a significant post-Covid reality; as many of these models are no longer relevant—and that the pandemic has rendered a lot of that data essentially useless.
That’s going to come as quite a shock to any CIO who’s been carefully building up a great conventional BI (Business Intelligence) stack. Many, perhaps all, enterprises now have some sort of BI capability these days, and let’s not forget that the global BI market is vast and indeed growing, set to jump from $23 billion last year to north of $33 billion by 2025.
However, Gartner’s not telling anyone to put their BI cubes into any kind of digital landfill. Instead—and it’s a suggestion I find persuasive—the idea is to revitalize and extend all your OLAP and slice and dice capacity with what it calls “augmented” Business Intelligence, where predefined dashboards and manual data exploration will get democratized out from a handful of data experts to “anyone in the organization”.
I agree, as do many of the customers I am helping do just that. But I think the augmentation is much more machine learning-shaped than Gartner’s initial formulation has indicated, for reasons I will explain. First, let’s start with some definitions. Business Intelligence is a term that has been around for many, many years, but basically denotes for a modern business how it can gain insight from data, primarily through reporting.
BI’s main drawback: it’s only ever about the past
BI involves end-users using technology to visualize data and to understand the patterns within that data, who then make decisions based on those patterns that they see—hence the idea of ‘intelligence’. There is clearly a large market around that, but I think it’s useful to see that it's very much around report generation and interactive exploration of data. BI tools allow you to see what is currently happening and what has happened in the past. But there's nothing in BI, by this definition and instantiation, that allows you to predict what's going to happen in the future: what activity is going to happen to a product, a customer, a service, a demand—a ‘something’.
If you can only look at what has happened to customers in the past, what has happened to those products in the past, then that’s clearly restricting you to just one axis of change. And the intelligence about how you provide insight into the future also has to be done through a human extrapolating the patterns that they're seeing in the past onto the future. And because there are humans involved in that, there's a lack of scale and there's bias and error in there because what a human is very good at doing is picking out a pattern when there isn't really actually one, or if the pattern is there it’s so complicated they can't visualize it.
So visualization is a great tool, but we humans can only really visualize in three dimensions. That means the challenge that you have with BI tools is they're very good at exploring large amounts of data, but across limited dimensions, and you can only give a view of what has happened in the past, not moving forward. Users know this, accept there are certain inherent limitations of how the visualizations are currently working, but so far have kind of had to accept it.
AI changes the picture. Why? Because what AI is about is providing a forward-looking view. Suddenly, we get that other axis back, the future; we’re not just working by looking in the rear mirror. The organization can get a realistic, data-driven prediction or a forecast of what's going to happen with those customers or those products.
In parallel, another advantage you immediately gain from starting to work with not just BI but AI is that you can look at a large amount of data. Plus, you can look at that data across lots and lots of different dimensions. Even better, the algorithms behind your new artificial intelligence help-mate can automatically pick out the key features that would be important to predict the outcome moving forward for a particular use case—whether it be the demand of a product, transaction fraud, whether or not the customer is buying or not buying or churning from a particular service.
So you get the future, and you get granularity. We're creating systems, now, today, that allow the models that underpin that new artificial intelligence to be created more quickly to try and give the consumers of BI tools the additional boost of AI capability. I think you can see where we’re going with this; what would we achieve if we could get BI and AI to work together, getting a real insight into both the business’s past, but also its future?
A way to get around all the limitations you have with regards to traditional BI capability?
Something great is the short answer. But we’re not quite there yet. The challenge that you have there is today’s BI analysts don't understand AI; they don't understand the nuances needed to build a particular model. What they're interested in is the outcome of the AI’s intervention—the forecast. They for sure want to ensure that all of their data is being considered across all of the dimensions, but they mainly want to see what's going to happen to the business moving forward if we don't do anything. They want to know, what happens if interest rates move from 1 percent to 2 percent—what would the impact be to my customer base? What they want to do is interact with a BI tool that has artificial intelligence underneath it to get around all the limitations you have with regards to traditional BI capability, to enable them to then see what's going to happen.
What needs to happen is that a working fusion of BI demand forecasting with AI depth needs to materialize, but in ways that satisfy both BI ninjas and the data science folks. The solution is to create AI-based apps that visually look very similar to a BI tool that hide the complexity, essentially. These apps have dash-boarding capability and the ability to interact with large amounts of data; they're able to scale, but they have nonetheless been designed to have AI automatically and correctly embedded within them. And as soon as an analytical model underpins those BI-style visualizations, a regular non-expert business user can then start to interact with these applications to understand questions like, if I was to tweak certain leavers of a process, what impact is that going to have on my business? And so forth.
Such BI-AI fusions solve a particular business challenge. Take predictive maintenance, where you use AI to scientifically predict the performance of particular sensors. Traditionally, with BI tools, I can have a look to see what has been the values that are coming off that sensor so I may be able to create an early warning system to say that when a particular sensor goes above a particular threshold, maybe I should go and investigate that in more detail. But by embedding AI into BI frameworks, you still have a dashboard-type framework, but now rather than looking at the actual value of the sensor, we can have a statistically valid outcome of what is the probability of that sensor failing in the next two days. And as more data comes into the system, that probability is constantly changing, and your probability calculation is supported and deepened by a machine learning model that's taken all of the vast amounts of data about all of the different sensors and the environment that they sit in to drive that particular prediction.
To be explicit, you're still within a BI framework here; you’re still utilizing visualization tools to see what's happening. But you're also able to use machine learning to pinpoint a specific thing. And if you start to see a particular sense that has a high likelihood of failing, you can then immediately dispatch an engineer to go and investigate why the machine learning model is giving a particular prediction: what was it about the environment that caused that prediction to be high? That helps, a lot, as you are then able to do some root cause analysis to go, OK, It's because of a pressure setting on another sensor that's causing this sensor to then fail; therefore, to prevent this sensor from fading, we have to go and fix the underlying issue over here.
One of the most exciting developments I have seen in my professional career
Melding BI and AI together like this gives you the immediate capability of building predictions but also understanding how those conditions have been made, all of which can then be surfaced to the business user to do something about it. And I can confirm my company, H2O.ai, is working with clients doing just this, in applications like predictive maintenance across manufacturing, healthcare and many other sectors.
This is one of the most exciting developments I have seen in my professional career, I have to say. The use cases are almost uncountable; to take just one, imagine having fraud detection so accurate that through a dashboard you can not just call out transactions identified as fraudulent, but to also be able to see the specific reasons why that transaction has been highlighted, so the business can then investigate those transactions in more detail to accept or dismiss as a false positive.
As we accept that a lot of our pre-Covid business data is invalid, then, the rise of the augmented BI-AI approach to business data seems not just very welcome but highly desirable. The CEO needs to have a strategic view around outcomes and predictions and to help her get them, responsible CIOs need to be looking at how AI and BI can come together in harmony—as only by doing that will her business be able to compete better, create better returns and drive further efficiencies.
John Spooner, head of Artificial Intelligence, EMEA, H2O.ai