Skip to main content

Overcoming the challenges of machine learning model deployment

(Image credit: Image Credit: Zapp2Photo / Shutterstock)

Successful model deployment is fundamental in the AI-driven enterprise

Our societies and economies are in transition to a future shaped by artificial intelligence (AI). To thrive in this coming era companies are transforming themselves by using machine learning, a type of AI that that allows software applications to become more accurate in predicting outcomes without being explicitly programmed. This article investigates the role IT professionals play in enabling the AI-driven enterprise through machine learning model deployment. 

Imagine your company as an AI-driven enterprise. Embedded in core business processes, hundreds of machine learning models ingest streams of data as the company interacts with customers and suppliers. These models, created by people with a detailed understanding of the company’s operations, serve-up predictions which increase the probability of prospects placing orders, of customers having their service queries effectively answered, and of criminals’ intent on fraud being deflected. 

Companies that successfully transform themselves into AI-driven enterprises differentiate themselves from companies that mismanage their embrace of AI in three ways. 

  1. They treat machine learning as a business initiative and not a technical speciality.  
  2. They have higher numbers of machine learning models in production.  
  3. They have mastered simple, robust, fast, and repeatable ways to move models from their development environment into systems that form the operations of their business. 

Commercial payback from AI comes when companies deploy highly-accurate machine learning models that operate robustly within the systems that support business operations. Companies that succeed at becoming AI-driven also find ways to reduce the resources required to develop models while giving sufficient attention to how they are deployed.   

Why companies struggle with model deployment 

While hard data is scarce, anecdotal evidence suggests that it is not uncommon for companies to train many more machine learning models than they ever put into production. Challenges of organisation and technology are in-play here, and success requires that both are addressed. 

From an organisational perspective, many companies see AI-enablement as a technical speciality. This is a mistake. As observed by Joichi Ito, Director of Media Lab at MIT while in a panel discussion on AI at the World Economic Forum in January 2017, “When the people who are actually doing the work actually create the tools, you get much better tools. When VisiCalc came out with a spreadsheet, it was a runaway success not because it solved the accounting problems but because it gave the accountants a tool to express their creativity on a machine.”   

AI is a business initiative. For companies to become AI-driven requires that the people who today successfully operate and understand the business, to create tomorrow’s revenue, be responsible for building and maintaining machine learning models that grow revenues. To succeed, these business drivers will need collaboration and support from specialists, including data scientists and the IT team. Specifically, the successful deployment of machine learning models should be the responsibility of IT. 

Machine learning models must be trained on historic data which demands the creation of a prediction data pipeline, an activity requiring multiple tasks including data processing, feature engineering, and tuning. Each task, down to versions of libraries and handling of missing values, must be exactly duplicated from the development to the production environment, a task with which the IT team is intimately familiar.   

Differences in technology used in development and in production contribute to difficulties in deploying machine learning models. Machine learning models are commonly developed using such languages as Python and R, where IT engineers work in languages such as Java, C and SQL. So, a model reliant on complex tasks with multiple dependencies and developed in one language, such as Python, must be translated to a different language such as Java before it can run in a production system. 

The IT organisations at AI-driven enterprises find means to overcome these differences that otherwise threaten the successful migration of machine learning models from development to production. 

Organising to succeed 

The organising principle in the AI-driven enterprise is that machine learning is a business-led activity which succeeds due to collaboration and support from IT and data scientists. Early attempts at enterprise-scale machine learning attempted to find data scientists with deep domain knowledge of a company’s business - professionals so rare as to earn them the ironic moniker of unicorn. 

The proven approach is to turn this thinking on its head: teach business people how to identify business problems that can be solved with machine learning, and give them the support they need to create their own models. Because business people already own the operation of business processes, they understand what drives specific responses from prospects, from customers, and from partners. This knowledge is fundamental to transforming business with AI. 

It becomes the CIO’s responsibility to ensure that representatives from IT are in the team from the start of each project. By working with the business, IT ensures the context for each machine learning model is analysed and understood, and plans for its deployment within the information systems currently supporting operations. Specifically, IT must ensure that the business provides answers to such questions as: 

  • What usage pattern does the business process demand of the new model: off-line, ad-hoc and on-demand serving ten predictions each week, or on-line and real-time serving ten predictions every second?  
  • What latency period between a request for a prediction and its service is acceptable to the business?  
  • Will the model need to be updated?  
  • Must the model be monitored for latency and for accuracy?  
  • Can the business process afford outages when predictions cannot be served?  
  • Is there a need for interpretability - to understand why a given prediction was made?  
  • Must the company retain, produce, or analyze the history of prediction requests and responses? 

Engaging early and regularly in machine learning projects allows IT to develop sufficient knowledge of the business context for each model so plans can be made for successful deployment into production as part of an end-to-end process. 

Tooling to succeed 

Current approaches of hand-crafting machine learning models are too slow and unproductive for companies’ intent on transforming their operations with AI. Business managers surveyed by Gartner identified time to value as one of their biggest challenges, reporting that it takes an average of 52 business days for their team to build a predictive model, and longer to deploy into production. Even after months of development, which delivers a model based on a single algorithm, the management team have little means of knowing whether their data scientists have created a great model, or one that will soon be outflanked by a competitor, or defeated by a fraudster.   

An automated machine learning platform must be capable of radically transforming the economics of AI, such that models can be produced in minutes and hours and not weeks and months. Further, the platform should support the development and comparison of multiple models, so the business can choose the one model that best fits their requirements for predictive accuracy, latency and compute resources. 

The approach of data scientists being responsible for building machine learning models isolates business professionals from the process of business optimisation. An automated machine learning platform should allow business professionals to build the models they need to transform operations and to collaborate with data scientists and IT when needed. As many as ninety percent of all enterprise machine learning models can be developed automatically. A company’s data scientists or specialist consultants can be engaged to work with business people to develop the small percentage of models currently beyond the reach of automation. 

Handover processes from the team responsible for model deployment to the IT team responsible for putting the model into production should be simplified and automated. Older “passing over the wall” approaches are too slow and introduce risks of model failure and blame games. This progress is achieved when both teams collaborate on the same automated machine learning platform, offering different deployment options that support the needs of the business as identified by the IT team.   

For prototyping or periodic reporting, such as investigating which new customers are most likely to churn, ad-hoc analysis allows the business analyst to be self-sufficient. When network latency introduces unacceptable risks to the success of a model, the platform should offer the option of either creating the exact model in Java, or an approximated, faster model in Python or Java and then deploying the model to edge devices. When the business context demands robust, real-time predictions, with model management features such as interpretation, monitoring and error handling, the platform should offer APIs that let the platform serve as a dedicated prediction server. A broad range of deployment options supports the IT team in ensuring the run-time model fulfils the requirements of its business context. 


AI and machine learning offer companies an opportunity to transform their operations. Success in this initiative requires companies to manage AI as a business initiative, to have hundreds of machine learning models in production, and to move models from development to their production environment in ways that are simple, robust, fast, and repeatable.  

IT professionals play a critical role in ensuring models developed by their business peers and data scientists are suitably deployed so they succeed in serving predictions which optimize business processes. Automated machine learning platforms allow business people to develop the models they need to transform operations while collaborating with specialist, including data scientists and IT professionals. 

Choosing an enterprise-grade automated machine learning platform will certainly make IT’s life easier. By providing guidance on organising for successful model deployment and the choice of appropriate technology, IT executives ensure their teams are recognised for their effective contribution to the company’s success as it transitions to an AI-driven enterprise. 

Yvonne Cook, General Manager, DataRobot UK 

Image Credit: Zapp2Photo / Shutterstock

Yvonne Cook
Yvonne joined DataRobot in 2015, taking responsibility for the development of the business in EMEA.  Before joining DataRobot she held senior roles at SAP, Netezza (acquired by IBM) and Teradata, amongst others.