While many companies across a range of industries have placed artificial intelligence (AI) and machine learning (ML) at the heart of their growth strategy, most do not feel they are in a position to successfully harness its power. The major reason for this is because many Big Data projects lack a mature approach to getting the best out of AI and ML deployments.
According to a 2021 Databricks and MIT Technology Review Insights survey, companies’ most important business objectives for their enterprise data strategy over the next two years are expanding sales and service channels ( cited by 45 percent of respondents), better operational efficiency (43 percent) and improving innovation and reducing time to market (42 percent). It’s great to have these objectives, but are businesses equipped to execute them? According to Gartner, 85 percent of big data projects fail, and according to the MIT Report only 13 percent of companies excel at implementing their data strategy with measurable results. When asking “low-achievers” (organizations having difficulties with their data strategy initiatives) what their main barriers are, the feedback highlighted limited scalability of their data management platform, difficulties in facilitating collaboration and slow processing of large data volumes. It’s clear that many organizations face challenges in scale, speed and collaboration in all areas of data exploitation.
To generate actionable insights from data, often in real-time, organizations have to take a joined-up approach that delivers the desired business results from AI and ML deployments. They need staff to be data literate, to tackle the fear of the unknown and to really streamline their processes. Here are three ways that businesses can gain data and AI maturity.
1. Building a solid foundational architecture
When paying attention to the foundations of strong data management and focusing on building an architecture that “democratizes” data, companies are much more likely to succeed and see measurable results. Managing enterprise data is highly complex and organizations need to remove the burden of legacy systems and a variety of tools, as well as data silos - unless they can be integrated or isolated. These combined issues impact organizations’ data platforms and the ML models that they support by reducing the speed and scale required to deliver the expected business results. The right foundational architecture should reduce data duplication, increase ease of access to relevant data, enable the processing of large amounts of data to be processed at high speeds, and improve overall data quality.
UPS’s delivery optimization project aims to shorten each delivery driver's route by one mile per day with an expected annual gain of over £36 million. This is a good example of how companies can use data to break and rebuild the system. It relies on several maturity criteria: a modern and scalable proven infrastructure, the continued support at every level from management, despite failures and delays. Furthermore, it relies on the desire to obtain rich, instantaneous and accurate data via GPS on transactions, points of locations, vehicles and even drivers where necessary.
2. Create a culture where data and AI is everyone’s business
As part of enterprise transformation, data and AI maturity is also about the democratization of analytics, ML and AI capabilities to help business users make informed decisions. This requires a strong data culture through collaboration and cutting-edge technology to enable the use of data to improve decision-making and its respective impact. From an operational point of view, it means the ability to share all these decisions, data and data-driven architecture to provide the necessary resources to business projects. This is only possible with a modern data architecture, where the right users have access to the right data to quickly generate insights that drive business value.
Despite a new generation of data management in systems, leadership, and perceptions of business value, at most organizations a gap still needs to be bridged between data teams and end-users, as well as the front or back-office employees who need data insights to make decisions on a daily basis. A good way of bridging the gap is to embed data scientists directly into business units where they regularly interact with users or to put analytics at the direct disposal of users so that they can draw out insights themselves as needed. In other words, it requires pushing data closer “to the edge”, to where the user is. As people throughout the organization become more familiar with advanced analytics and data science, they gain the ability to run the analytics themselves, rather than just consuming analytics that someone else produces.
3. Evaluate regularly
To succeed in creating value from data-driven projects, an organization must constantly know where it stands on its journey to data and AI maturity. It can do this by adopting a model for assessing the necessary key competencies. A successful evaluation model should help structure the dialogue between teams and should understand the steps required to improve critical business operations that have to be boosted, made more consistent and more stable through large-scale data integration and a true data culture. It enables the use of data to improve, accelerate or monitor decisions and their respective progression and to share all these decisions, data and data-driven architecture to provide everyone the necessary resources for business projects. It’s simply about a model based on collaboration at all times, at all levels, all together.
One of Databricks’ customers, which is a leading coffee retailer, has a set of AI projects which perfectly illustrate the value of this, with use cases based on common data and continuous collaboration for three different departments. The operations team can create AI-based models to determine the location of future stores based on the socio-economic factors of a targeted neighborhood. The customer relations department also benefits directly from AI through the company’s loyalty program. By understanding the beverage preferences and buying patterns of all 19 million subscribers, the team can boost sales whilst still providing an ultra-personalized customer experience. This data is also used to predict which beverages and channels will generate the most profit based on the habits and tastes of a given group. Finally, the store support department relies on AI for predictive machine maintenance and to anticipate supply demands.
In this demanding journey towards data and AI maturity, an organization has already taken a big step forward when various business use cases are integrated into the centralized, secure and scalable services set up especially by IT. The success of a data-driven business transformation or value creation needs global integration from the organization's strategy to training plans and the upgrading of internal resources to continue to spread knowledge. This step is crucial to multiply the deployment of data and AI use cases. The success and value generation of these also could not be achieved without ongoing management support, organizational maturity, and large-scale data science capabilities.
Nicolas Maillard, Senior Director for Field Engineering for Central and Southern Europe, Databricks