The Spark for real time decision making

“Every second we don’t react to the customer, we’re losing money,” Rupert Steffner, chief BI platform architect for Otto Group told us recently.

His remark sums up the speed at which most successful organisations have to operate these days. They can no longer base their decisions on information that is a week, a day or even hours old. Now response times are measured in seconds and real-time data integration and data analytics have become the aspirational gold standard, against a backdrop of exploding data volumes.

It is no surprise that IDC has forecast that the big data technology and services market will grow at a 26.4 per cent compound annual growth rate – about six times faster than the overall IT market.

The steady growth of the Internet of Things (IoT) means that big data applications in sectors such as healthcare and agriculture are now demanding real-time capability as well as renowned data-focused industries like retail and finance. Converting big data or IoT sensor information into immediately actionable insights means seizing opportunities before competitors on the one hand, but also using proactive management to stop threats as they occur on the other.

But, sometimes in technology this kind of predictive thinking is a little too ‘blue sky’ for what can be achieved in reality. In the case of big data, until now Hadoop has been the platform of choice for analysis and computation and it is still a crucial cog in the drive for data accuracy and increased performance, but as data volumes continue to expand, the need for real-time analytics without significant financial investment has become essential. Many are finding Hadoop alone is not enough.

Thankfully developers invariably manage to be one step ahead of market needs – and in the case of Apache Spark and Spark Streaming, it’s never been clearer. Many data integration firms have been talking about how they will support Spark in the future and we’re taking the plunge with this in our newly launched 6.0 version which has native support for Spark – ultimately enabling organisations to achieve this real-time insight.

But what kind of applications does this real-time capability open up?

A great example is in online retail, where it will enable more accurate behavioural predictions and the automatic delivery of incentives to ensure shoppers complete their purchases. For bricks and mortar stores it will enable marketers to study traffic patterns to decide how best to reach their target customers.

Potential applications are far-reaching. For example, surveillance analysis can be used to ensure public safety. Already the West Japan Railway System has installed cameras to detect tell-tale signs of intoxication. Analysed in real-time, data produced could be used to stop these people falling onto the tracks. In healthcare, providers will be able to continuously monitor at-risk patients by combining real-time personal device tracking with medical record information. Some airports and cities are looking at the possibility of replacing CCTV cameras with 4K systems to conduct real-time searches to identify possible security threats, as well as determine how travellers spend their time during stopovers.

These are just some examples of how real-time data analysis could be used. But potential will only be fully realised when this analysis is both affordable and quick to develop. Real-time analysis is often claimed but so far rarely achieved without exceptional time, effort and cost.

But we’re on the cusp of a breakthrough – the much-needed spark is there to bridge the gap between data and decisions.

Ciaran Dynes, VP of products at Talend

Photo Credit: Adchariyaphoto/Shutterstock