It’s a really exciting time to be in the data analytics space. We’re at the start of a whole new era—the convergence of the Internet of Things (IoT), big data and artificial intelligence (AI). The convergence of these three technologies has the potential to not only automate analytics and decision making, but to actually automate the learning that happens in between. You can think of this as having robots in your data.
Visualise a continuous feedback loop where data from smart devices is collected and analysed, with the insights automatically fed into business processes or even the smart device itself. Every business and every person stands to benefit not only from better information, but from automation that can provide nearly instantaneous response.
Organisations are already gearing up to implement AI into their analytics and apply it to big data, some of that coming from the IoT. Forrester research finds that their investments in AI are expected to increase 300 per cent as compared to last year. It will be interesting to see the different ways in which AI gets incorporated into products and services. However, the thought of this kind of automation has some in the data industry fearing for their jobs.
The future is happening now
We’ve all experienced some form of what I will call “smart data.” Anyone who has ever shopped on eBay or Amazon has been presented with suggested items to buy based on previously viewed items. While those recommendations continue to get smarter, fascinating things are happening in other industries.
The medical industry is ripe with opportunity. Forbes mentions a study that used computer-aided diagnosis to review the early mammograms of women who later developed breast cancer. Through artificial intelligence, specifically machine learning, the data model was able to identify 52 per cent of the cancers as much as a year before the women were diagnosed.
We are already seeing integration in the automotive industry, but we can expect much more. According to IBM’s report Automotive 2025: Industry without borders, the car of the future will be “sophisticated enough to configure itself to a driver and other occupants. It will be able to learn, heal, drive and socialise with other vehicles and its surrounding environment.” I expect we will actually see that happen much sooner.
Natural language processing, or the ability for computers to understand and process spoken and written words, is gaining in sophistication thanks to machine learning and big data. It works on pulling meaning from unstructured text data. The result could be used to understand customer sentiment data or to provide help reviewing large volume legal documents in preparation for a trial.
The catalysts for change
So many things, in so many aspects of our lives, are constantly collecting data. For consumers, smart devices, smart homes, wearables, and even our cars are producing data every day in amounts that are difficult to wrap our minds around. IDC says that by 2020, there will be 44 zettabytes of data, nearly as many bytes as stars in our universe. It also says we have only been able to analyse 5 per cent of our data. So much data goes unused, but that is all about to change.
Data storage technology has improved and prices have come way down, allowing organisations to collect and store massive amounts of all types of data culminating in big data. It used to be that transactional data was the only data typically stored and used, but now unstructured data like music and photos, documents and files, web pages and emails can be used as well. Unstructured data is typically text heavy, but not always.
Until now, statisticians and data scientists were limited to using sample sets of data. However, new technologies for managing big data, like Hadoop and NoSQL data stores, are allowing users to take advantage of all of the data for more accurate modelling, producing better descriptive and predictive results. These beefed up models, arguably using artificial intelligence, will drive faster decisions and much better outcomes.
Machine learning, a subset of artificial intelligence, has the biggest potential to change and improve the way we live. This is where data can take up a life of its own. Machine learning essentially allows computer programs to continually learn and evolve with new data. A rather new term, deep learning, takes machine learning closer to mimicking the human brain and though processes. Instead of us telling the programs what to do, they ultimately tell us what to do.
The impact on data workers
All of this is very exciting, but we need to keep in mind we are still at the early stages. It’s easy to get caught up in the hype and place all trust on various new smart data. It’s human nature that if you look for a pattern in the data, you will surely find one. We must remember correlation is not causation.
Some may have read a frightening headline that, in just 20 years, nearly half of all jobs will be replaced by automation and artificial intelligence. While tools, technology and data can do quite a bit, it is still up to us to have a solid understanding of the business problem we are attempting to solve. Computers aren’t that smart yet. This is why jobs involving data are not going away anytime soon, and why data scientists are in such high demand.
We still need humans to understand the business, build the initial data models, and validate the data. And, the faster technology advances, the more important it will be to have more eyes reviewing the results. This requires that everyone in the business become comfortable working with and understanding data, making self-service analysis a key business strategy for some years to come.
This new era of smart data requires leadership from the top. We are seeing more organisations create and staff new roles, such as chief data officers (CDO) and chief analytics officers (CAO). Additional data-related roles are being created that tend to bridge IT and business, and focusing more on data and analytics, rather than general business intelligence. This validates the necessary shift toward using data and analytics as key drivers of mission critical business priorities.
The next 10 years are going to be pretty exciting, especially in the data space. More of the things we use and interact with every day will begin collecting data. Data storage and processing technologies are advancing almost daily, and artificial intelligence is being deployed in more systems. In fact, many vendors like Adobe, Google, IBM and Salesforce are embedding AI into their solutions.
Someday, in the future, we may be hands off when it comes to the continuous feedback loop of data. Data robots will automate much of what we are doing today. However, the devil is in the details and there is so much more we can and still need to do. Rather than seeing our jobs disappear, they will increase in both numbers and importance. Data analysis is no longer a specialised function left to the geeks but a critical skill that every employee should have, along with self-service analytics. The future will come soon enough.
Avi Perez, CTO, Pyramid Analytics
Image source: Shutterstock/alexskopje