The initial excitement around big data is translating into real-world progress. Appropriate architectures and processing platforms are now in place, while modern data management technology capable of storing and organising large data sets while taking advantage of the underlying massively parallel processing grids to crunch data at volume with high throughput and low latency, is accessible either from the cloud or as on-premise computing.
Yet, the excitement is contained as despite some fine examples across different industries, success at converting investments in big data technology into real business value eludes many CIOs.
The driver for investing in big data is the insight derived from analytics that improve existing processes or inspire entirely new ways of creating value. Analytic tools are increasingly coming on stream - but a gap exists between businesses’ thirst to consume analytics and their ability to derive real business value.
The core problem is a shortage of data scientists; those rare individuals who combine the business analysis skills needed to become knowledgeable in a specific business domain with the technical skills to codify that knowledge into applications that help organisations achieve their strategic goals.
Businesses urgently need to fill this gap. The next wave of enterprise computing, in the shape of data-driven software applications, offers an opportunity to do just that.
This rapidly emerging new class of applications embeds deep domain knowledge of a business and its challenges in software designed to solve problems. Brian Ascher, a partner at venture capital firm Venrock, neatly encapsulates the benefits in a recent piece for ForbesCIO Central website: “These solutions use algorithmic data mining on your own data and often on external third party data accessible by cloud ecosystems and APIs.
"Data-Driven Solutions make predictions about business functions, prescribe what to do next, and in many cases take action autonomously. Trained analysts are not required to query databases. Instead, business users get answers directly from the software."
So technology fills the skills gap for businesses. These new data-driven applications ingest data from a wide range of internal plus external sources, extract the entities of interest (people, groups, places, things), establish relationships between these entities while applying knowledge of what is and what is not important to the business community being served, and then analyse the emergent situation to present answers, and not merely reports or dashboards.
Importantly, to accelerate the rate at which business users realise value, these applications include virtuous feedback loops, increasingly implemented in machine learning algorithms, that continuously observe their users’ interactions to learn and capture valuable behaviours which can then be published to the entire community accessing this software as a service to improve outcomes for all.
Deep domain knowledge embedded in software combined with this continuous learning capability helps businesses successfully convert data into knowledge. The business can now consume that knowledge in one of two ways:
- It may be channelled directly into a business process with no human intervention, for example as confirmation that all checks are completed successfully and a customer can be on-boarded.
- The knowledge can be presented to a human audience to make a decision, for example a deep analysis of an application for a business loan unearths a history of defaults that demand further consultation with the applicants.
In cases when knowledge is to be communicated to people, data-driven applications employ interactive data visualisation, technology already familiar to everyone who uses a touch screen and navigation software on a smart phone. Such visualisations help businesses quickly and efficiently gain an understanding of complex situations informed by multiple data sources to drive fast time to insight.
In-built domain expertise, ongoing learning capability and interactive data visualisation distinguish what Jake Flomenberg of venture and growth equity firm, Accel Partners describes in a recent O’Reilly Media, Inc. article as “above-the-line” technologies such as data-driven applications from their “below-the-line” counterparts such as data platforms, data infrastructure and data security services, which serve as essential building blocks but lack specific business domain intelligence.
So instead of employing data scientists in every business and government department, software companies will employ these scientists within multi-disciplinary teams to develop applications specifically for vertical industries. Embedding domain knowledge in software brings scale, meaning it can be used to the advantage of hundreds or thousands of businesses.
With this in mind, it is not surprising that it is the above-the-line segment that industry observers expect to really take off. As Flomenberg puts it: “We’re in the early innings for the above-the-line zone and expect to see increasingly rapid growth there.”
It might be counter-intuitive but data is not really the point of data-driven software. Instead this rapidly emerging technology is all about finding solutions. As Ascher points out, “most business customers don’t really care about data. They care about solving business problems.”
That’s what data-driven software is so effective at doing today – and that’s why, at Encompass, we echo Ascher’s confident assertion: “Software-as-a-Service and cloud computing has been transformational for the software industry, but compared to what is coming next, you ain’t seen nothing yet.”
Michael Kearney, product marketing specialist, Encompass