It’s often said that there is no room for sentiment in business. Yet the sentiments of the people involved (for example, customer, salesperson, credit control clerk, support analyst), can have a significant effect on almost any business.
What is sentiment analysis?
The term sentiment analysis (SA) has been coined to describe the use of AI techniques to explore the sentiments conveyed in human language. Sometimes referred to as opinion mining or emotion AI, at its simplest, it might be establishing whether the message conveyed is positive (e.g. “That is a great idea”) or negative (e.g. “That could lose us business”) or at a deeper level, the psychology of the sender - “he doth protest too much.”
Because natural language has evolved over thousands of years, and because humans take its understanding for granted, it is often easy to overlook how complex it is. Simply scoring individual words for their positivity/negativity can result in a machine assessment that correlates well with that of a human, yet there are many situations where this falls down. To name just one, the use of irony and sarcasm where one expresses the opposite view make sentiment difficult to analyze. Similarly, we often use metaphors, slang and expressions in which the individual words bear little relationship to the sentiment conveyed. “This band will knock your socks off” might be difficult for a machine to read as positive. Add to that, foreign languages, dialects, messages used in particular industries - for example, legal, medical, engineering – and it is easy to see how complex it can become.
One way to overcome the need to understand sentiment is to get humans to assess thousands of samples and from the results, and let the computer work out the rules for determining what is positive or not. These so-called neural network techniques have become very fashionable - not least because they do not require the software designer to understand the subject - but they come at the cost of using vast amounts of computing power.
In a sense, the techniques used are academic. The growth in cloud computing means that it is possible to pick and choose SA algorithms as they evolve, and deploy the one that gives the best results at the time. Gone are the days when businesses needed to set up their own R&D departments or commit to some expensive software package that soon became outdated.
So why do it?
There have been many studies that show the relationship between positivity and success but unless you can measure positivity, it is not possible to establish whether it affects your business. Furthermore, one should not necessarily assume that a positive sentiment correlates with a successful business. Some industries, such as insurance, are focused on people’s concerns and fears, and the language may be intentionally negative. However, it is measuring the sentiment that is important.
Many companies, including call centers, rely on training their staff in developing and nurturing positive relationships with customers, but most have no idea whether this training is being heeded, or if it is, whether it works.
For all the potential of SA, it is of no use unless it is measured day-to-day in a working environment. Appointing a firm of consultants to measure sentiment and writing a report might initially result in a shake-up, but it is only a snapshot in time, will cost a lot of money and will not last.
Similarly, taking samples of written documents, such as emails, texts, and reports, - can give some useful insights to the positivity of individual employees, but the lag in obtaining, analyzing and reacting to the results can be short-term and counter-productive.
The key to success is continuously monitoring all forms of communications - this includes existing digital media such as emails, texts and instant messages - but just as importantly, telephone conversations. People say things they would never write - and vice versa - and it is only by measuring both, that the information can be fed back to get benefit.
Undertaking SA on a continuous basis is essential for firms in the financial services sector who need to monitor share prices, and for just about every organization it can give an early warning into potential issues with customers, employees and suppliers. This can enable an organization to take steps to try to remedy and diffuse a potential situation before it becomes a threat.
Trying to do this manually, however, is time-consuming and therefore expensive. You may be regularly requesting feedback from your staff, customers and suppliers, but if there are thousands or even hundreds of feedbacks per month, it is extremely difficult for one person to read, let alone analyze them. By automating the sentiment analysis process, organizations can easily see how all those important to your company are reacting.
The practical implications of SA
The lowest hanging fruit as a source for monitoring sentiment is undoubtedly email. This is because it is composed digitally and written in natural language. Speech, on the other hand, must be converted from an acoustic waveform into a digital waveform and from there into natural language - syllables and words. It is generally much more difficult to transcribe speech than it is to analyze the sentiment contained within it. Another advantage of email, unlike social messaging, is that email uses open standards and as such, can be ingested transparently to the user - i.e. without the need for special widgets, plugins and APIs or requiring users to change their working practices.
The standards or rules for exchanging emails are said to be “open” if they are publicly available, so that anyone who adopts the rules (and, of course, any passwords), can access and process them. Computer programs can read emails just as humans do and perform the sentiment analysis.
The problem with other types of communications such as social media is that their messaging rules are kept secret (i.e. proprietary to the service provider), so it is not possible to monitor them automatically without special agreement with the service provider - if available.
The primary motive for service providers adopting proprietary standards is not security (much as they might claim it), but to maintain full control of the channel and dissuade users from jumping ship to another service provider. This is not the case with open standards where the user is free to choose any email provider and is one reason why email remains the de facto standard in business. Indeed, the pervasiveness of email in business means that most users send and receive many emails every day and so any changes in the sentiment conveyed can be flagged up very quickly and, as discussed, it is the continuous nature of the measurement that makes it valuable.
Last but not least, email also has the further benefit that it is used to exchange and store documents (by way of attachments) - another massive source of sentiment. These attached documents may not be in the same convenient textual format as the emails themselves - but with some additional processing, can be rendered useful. Scanned documents, for example, are from the computer’s viewpoint, just pictures and in order make them readable, the individual letter images must be recognized and converted into words. This process is called optical character recognition (OCR), and while it can yield significant information, it should be borne in mind that the sentiment contained might not be that of the correspondents.
After email, acoustic conversations are the next best source of sentiment. And since most businesses now adopt open digital standards for telephony, it has similar benefits to email in terms of access. As previously mentioned, speech is not text and in order to transcribe acoustic speech into text, we can use the rapidly advancing technique of automatic speech recognition (ASR) - which now costs as little as 1$ per hour to deploy.
ASR has come a long way in the last five years, but user experience is mixed and people tend to either love it or hate it. Some applications, like command and control - e.g. “turn off the lights please” - generally work well because the vocabulary and tasks are relatively limited. Transcription of continuous speech, on the other hand, is a far more difficult task and performance varies enormously depending on the quality of the articulation and conversion to digital.
However, the thing about sentiment analysis is that absolute transcription accuracy is not essential, so even when some words are mis-recognized, it will not significantly affect the sentiment detected.
Further benefits to the value of the sentiment extracted from any medium, occur when the number of samples are large. And this can best be achieved when every available message is analyzed. By processing a large number of samples, individual inaccuracies are smoothed out.
Anyone not already using tools to help with sentiment analysis and is planning to, needs to ensure they select the right one to suit their business needs. The software should be appropriate to the size of a business, including small companies, not just for large call centers. In addition, it should allow users to make any arbitrary selection of messages and measure the sentiment in that specific selection. So, for instance, an organization should be able to select specific staff, specific companies, messages that mention specific words, messages sent after working hours and so on to examine.
But what about privacy?
Although in principle, it is easy to analyze the sentiments of individual employees, the real value is in measuring the organization as a whole and in so-doing the data can be anonymized just as it is with most companies’ sales figures. If a company has a good month, then staff morale is better served by congratulating everyone rather than just the salesperson who took the order. This has no more privacy implications than displaying the average time to pick up the phone in a call center.
It may be that valuable lessons can be learned without focusing on individuals.
Sentiment Analysis is one of an ongoing stream of AI tools that have become viable because of cloud computing. It joins the ranks of automatic speech recognition, optical character recognition and many other tools that were simply unthinkable for small business use 10 years ago.
Not only has cloud computing made it possible, it has also made it economic.
Knowing the sentiment of employees when dealing with third parties is not the panacea for success but it can be just as important as the profit and loss statement and balance sheet in determining the overall health of a company. What is key, however, is to measure it continuously, automatically and from sources that represent real day-to-day interactions.
Dr John Yardley, Founder, Threads Software