Machine learning, deep learning and artificial intelligence (AI) are attracting huge amounts of interest from analysts, press and IT teams. We are seeing large corporations apply data science and machine learning to make AI a reality for businesses and consumers. But is there a problem with too few companies holding too much of the market?
What is the “one percent” problem around AI?
Gartner predicts that 80 percent of data scientists will have deep learning skills in place by 2019, according to its research team in September 2017; meanwhile Teradata and Vanson Bourne research found that more than 42 percent of enterprises see opportunities for further implementation and process integration of AI in their operations.
With so many enterprises looking at their approaches to AI and machine learning, the availability of people with the right skills and experience can become a stumbling block. Finding data scientists and analytics experts can be expensive; Glassdoor found that the median base salary for data scientists was $110,000 in the US due to demand levels. In the UK, data scientist roles ranked at £53,000 after six years’ experience based on research from salary benchmarking company Emolument.
While the availability of skills is a challenge in its own right, there’s another problem to consider. What motivates those data scientists, aside from pay? It’s the size and scope of the challenge that they are hired for as well. This leads to those with great data science skills being hired by those companies that can both afford the large salaries associated with data science and provide the right perceived environment for personal development. This situation is normally found in large technology firms that make up one percent of the business population, such as Amazon, Apple, Facebook and Google.
This combination has led to a dangerous situation for other, more traditional businesses. As these leading international technology firms can monopolise the talent side, they can use their advantage in the market to compete more effectively using their leadership in data. This leads to more market dominance and financial success, which feeds into being able to hire more data scientists. This becomes a cycle which other businesses will find it hard to break.
This represents a problem where too much of an available resource has become held by too few people. In 2015, Oxfam published its influential report on the availability of capital and how half of the world’s wealth is held by the top one percent of the population. Similarly, the AI and machine learning market risks its most finite resource – skilled people – becoming controlled in the same way.
How is the “AI one percent” a real risk to businesses?
The challenge around this concentration of talent using data is that the majority of companies are not able to use advances in machine learning or deep learning. The complexity of implementing data lakes, algorithms and analytics can be significant. While it is possible for enterprises to look at those large technology companies to fill these gaps as pre-packaged services, or make use of their research, this does not help over time.
The risk here is that businesses are effectively homogenized – whether they are selling services, specific products, or in particular markets, they will rely on the same subset of companies for their algorithms and approaches to data. At the heart of this approach will be the same sets of open source components, linked up in the same way. However, this does not meet the needs of real world businesses and organizations.
Putting together the right processes around machine learning and AI requires more thought, as each business will be unique in its approach. Even two companies that sell the same products – say shoes, for instance – may have very different internal processes when it comes to targeting customers, buying in stock and managing logistics. While one shoe retailer may work on longer lead times to control costs and improve margin, another may focus on tracking market demand and responding faster to trends. All of these moving parts have to be considered as part of implementing the right approaches to analytics and machine learning. Rather than relying on one size fits all approaches to starting with AI, it’s important to help people get started in ways that will suit them best.
This is where experience with AI and machine learning technologies becomes essential. Rather than trying to follow other industry players and trends in a sector, each business will need to look at how it can apply AI to suit its own requirements. These needs will be based on what the business looks at as its key decisions, what predictions can be made around those decisions, and then how to get insight around those problems. Not all of the problems will be linked specifically to machine learning or AI; instead, these may be more deep-rooted issues that have to be considered in context.
Implementing the infrastructure for AI
Alongside the algorithms that make up AI and machine learning implementations, there is also the infrastructure used to host and run this over time. From setting up traditional servers in a data centre through to using public cloud services, there are lots of options available. However, the top technology companies in the world have an advantage here as they run their own public cloud environments and can use their experience of planning large-scale deployments to their advantage.
For more traditional businesses, looking at DevOps and cloud can help in AI and machine learning strategies too. For example, DevOps puts a lot of emphasis on how IT infrastructure management tasks can be automated to make rapid change easier as well as controlled. For AI deployments, making use of distributed systems that can be scaled up swiftly in response to business requirements can help out massively. This can be used to store the huge volumes of data that can be used for initial algorithm training, and then to store new data as it is created and analysed.
As more companies implement data lakes, there is more demand to get value out of these implementations. However, this should not be viewed on its own or as a one-off process. Instead, this set of data should be the bedrock for ongoing analysis and to inform how AI and machine learning projects are strengthened over time. In practice, this means bringing together the scale and cost benefits of data lakes alongside some of the benefits that traditional data warehouses can provide.
This combination of data warehouse and data lake should help IT teams to support their machine learning and AI projects. By bringing these different approaches together, IT can expand their support relies and develop programmes further to suit the business’ specific requirements.
Without access to the right skills, supported by strong processes and control over infrastructure, this implementation of AI will be more difficult to achieve. Yet helping all companies take advantage of AI – not just those at the apex of the pyramid – can make the whole market more effective for everyone.
David Wyatt, Vice President EMEA at Databricks
Image Credit: Enzozo / Shutterstock