Skip to main content

Transforming AIOps with Enterprise Observability

AI
(Image credit: Shutterstock / metamorworks)

Years removed from the first analyst reports about the “new” thing called AIOps, organizations have been using point solutions and elements of more intelligent and more automated IT Operations to create value in the technology delivery chain.

From more intelligent tools and solutions to new technology platforms and organizational changes, a plethora of concepts have made AIOps a little bit better than it was before.

At the same time, the concept of Application Observability has changed and matured. The latest entrant, Enterprise Observability, has the ability to change AIOps completely, going beyond enhanced capabilities to completely transform the AIOps concept into another level of value. 

To understand the synergistic potential of Enterprise Observability and AIOps together, let’s examine the progression of AIOps, Machine Learning and Observability.

The ABC’s of AIOps 

To generate precise results, AIOps artificial intelligence algorithms need large swaths of data on which to operate – ingest, infer, conclude, learn, repeat. The richer the data, the better the inference, conclusion and learning is. AIOps combines big data and machine learning to assist across several IT disciplines – provisioning, monitoring, root cause analysis and remediation. 

Using four primary data sources (metric, event, tracing, and logs), AIOps can proactively detect service faults and issues by identifying anomalies based on service availability, response times and distributed traces of individual requests. Over time, an AIOps platform learns from the known problem symptoms and patterns, aggregated and correlated, to become more predictive, ultimately enabling remedial actions to occur before a service problem exists.

The more distributed an environment is (especially hybrid cloud), the more difficult it is to discover anomalies. 

AIOps, Machine Learning and Predictive Analytics  

The common theme among AI methodologies is learning, which requires rich data. Rich data, embodied by highly granular, high-cardinality complete data sets, enables AIOps to rapidly produce well-correlated conclusions. The computational algorithms of machine learning enable automation of repetitive tasks recognized by the algorithms. Deeper learning compares recognized patterns against each other to predict which learned pattern would work best. 

The use cases of AIOps Predictive Analytics run the gamut from Advisory (which merely provides advice to human users) to Semi and Fully Autonomic, where the platform not only learns from experience, but also takes the most appropriate action(s) on its own.

Automation: Data In, Data Out 

Even though practitioners are reluctant to unleash it, fully autonomous AIOps is the ultimate goal. The biggest concern is that no matter how well trained the AI learning is, there are always anomalies that can invalidate the ultimate outcome. Getting comfortable progressing toward full autonomy requires gaining confidence in the algorithms, the data set and the performance model(s).

No matter what, data drives the machine learning engine. The quality of data directly drives the quality of conclusions. The richer the data, especially pertaining to context and correlation within the data set, helps drive more well-founded decisions and reliable predictions. One way to think about it is to put yourself in the place of the intelligence platform. Sure, you can make a decision with limited data, but the better quality (and more context) in your data, the more confident you feel in your decision. 

Enriching Observability Data 

The standard definition of Observability data is a collection of metrics, traces and logs (sometimes doubled up as “events”). Of course, getting the metrics, while not exactly easy, is only part of the equation. The concept of Enterprise Observability takes the raw data from standard Observability, adds richness to it and ensures that its ready for the enterprise by making always available and always accurate.

There are three essential aspects of Enterprise Observability that contribute to a better data set for AIOps platforms to use – Automation, Context and Intelligence.

Automation

Infrastructure and service discovery and monitoring with real-time change detection, instant feedback, anomaly detection and root cause analysis 

Context

Understanding of relationships between applications, services, infrastructure, code, user requests and configuration --- all correlated and available for deep analytics

Intelligence

Programmatic knowledge and machine learning algorithms to interpret the rich data set to make recommendations and decisions. 

The depth of the learning and the accuracy of analytics predictions is predicated by the data granularity and context. The richer data set from Enterprise Observability is a key ingredient that results in more effective learning and more accurate AIOps predictions.

Feeding Enterprise Observability data into an intelligent AIOps platform offers the following improvements over classical AIOps with normal application performance data:

  •  More Accurate Predictions
    Identify potential issues based upon trending conditions and learned experiences 
  •  Better Recommendations
    Identify the best actions most likely to resolve and/or solve any issues 
  •  More Effective Conclusions
    Minimize the human effort needed to keep the system(s) operating optimally 

Enterprise Observability Makes a Smart Tool Even Smarter

Adding Enterprise Observability to the AIOps data mix adds a rich data set, one that is steeped with context across the data set. This richer data not only allows any AIOps platform to learn patterns more effectively, but also allows better decisions to be made – in some cases by providing expert system knowledge from its own built-in learning algorithms. 

Add Enterprise observability to your data mix to gain optimized system use, achieve maximum system availability and minimize service incidents. Oh, and don’t let it start building its own robots (wink, wink).

Chris Farrell, Observability Strategist for Instana, an IBM Company

Chris Farrell is the Observability Strategist for Instana, an IBM Company.