Skip to main content

Testing chatbots - quality assurance drives delighted customers

(Image credit: Image source: Shutterstock/Montri Nipitvittaya)

Chatbots have made huge strides since they were first developed nearly a decade ago. They’re now in many websites, performing a multitude of tasks - from executing simple tasks like checking account balances to providing insurance advice. The most savvy businesses realise the game-changing potential of this technology, and some experts have even predicted that conversational interfaces will become ubiquitous and eventually the channel of choice for consumers.  

And, they might just be right. International giants like Google, Microsoft and Apple have already placed huge bets on AI, leveraging big data and machine learning to get as close to human intelligence as possible.  

We know that consumers consistently gravitate towards the easiest available channel for brand interaction - even when that means moving between suppliers - and streamlined interactions with customer services can be make or break for organisations. So perhaps it’s no surprise that conversational interfaces are becoming increasingly important - representing an appealing engagement method for a user to accomplish a task at "any time on any device in the moment of need.”  

With almost every enterprise declaring they are in the midst of a “digital transformation”, Chatbots open the door for implementing new business models or reviving old ones. Early adopters are learning an old truth – innovation must meet user expectations of quality.   

When bots go bad 

According to analyst firm Botanalytics, while a projected 35.6 million people will use voice-activated assistants at least once a month this year, 40 per cent of bot users disengage after just one interaction. This means the stakes are high for the many brands and developers delivering audio experiences.   

And, whilst there are many examples of good uses of this technology, there are also enough bad ones to demonstrate the risks of not taking quality seriously enough. As we’ve see in recent bot blunders, not all chatbots are created equal. A poorly performing chatbot can easily turn a potential customer engagement into a horrible user experience.  

Microsoft’s recent experience was a warning for many. In March this year, the company launched a chatbot named ‘Tay’ designed to have conversations with Twitter users, and learn how to mimic a human by copying their speech patterns. It was supposed to engage with people aged 18–24 but a brush with the dark side of the net, led by users of the notorious 4chan forum, instead taught her to tweet offensive phrases and left Microsoft with a tricky damage limitation exercise. 

Keep it simple, stupid  

So, how do companies assure the quality of service which is so crucial in driving delightful customer experiences?  

The short answer is it isn’t easy – but for many, simplicity is key. While Cortana, Siri and Alexa might eventually develop “‘ask me anything” capabilities, without unlimited budgets and huge development teams, for enterprises, it’s better to deploy task specific bots to engage your audience. The key is to set clear goals and identify high value or high frequency use cases for your chatbot. Don’t attempt to address problems beyond your scope. Instead, manage customer expectations by keeping the conversation within clear parameters.  

But, of course, however simple your objective is, like any other emerging technology, chatbots inevitably add complexity to applications. Adding capabilities through this new channel, together with the difficulty of processing open-ended conversations, means that the development of this type of application can be extremely complicated. Development teams need to strategise on bespoke solution integrating an AI engine (like IBM Watson), together with a Natural Language Processing speech engine (such as Nuance) to power these capabilities or leverage APIs to connect to various specialist bot. But the bottom line is that they must work hard to create a platform where systems of engagement and systems of record can work together to power a seamless customer experience. 

Overcoming challenges with continuous testing  

Testing is the answer to quality assurance. Any developer burned by late night de-bugging appreciates a robust testing suite, and its ability to ensure well-functioning code and developer sanity. Continuous testing allows developers to iterate faster, cheaper, and with confidence that they’re not introducing new bugs as capabilities are exposed to users.  

Chatbots bring their own set of challenges to testing. Voice activated technology and chatbots need to accommodate a broad dictionary, and in the future, add imagery inputs. Languages, voice variation, accents and speech impediments all need to be catered for including the usage of “millennial slang” adds another level of complexity to a burgeoning industry. And, of course, assembling a test environment that resembles real user conditions - integrating third party platforms, like Facebook Messenger or Telegram - is hard.   

For many teams, initial testing efforts are manual – it’s not an oversimplification to imagine several engineers in a room talking to “Kate” using their smartphone, inquiring about insurance options or trying to find out the balance of their current bank account. But, automation is critical in quality assurance. Moving voice and chatbot testing from lengthy manual processes to automated systems is the proven way to increase the effectiveness, efficiency and coverage of testing. And having an automated solution for testing means you’ll test more often.    

Which is why, some vendors have added a new capability that automates testing for voice-enabled chatbots and Facebook Messenger or Siri integrations. Its purpose is to provide developers building virtual assistant capabilities for functional correctness, responsiveness, and voice quality — shrinking the number of testing hours and giving them the resources they need to build voice-activated assistants. The capabilities are fully integrated into cloud-based platforms, which includes testing for web, mobile and IoT devices. 

Embedding test processes throughout the development lifecycle 

By embedding testing throughout the development process, uncertainty is reduced and feedback loops are accelerated, ensuring that development teams’ assumptions are correct and that they’re building in the right direction. With chatbot technology, just like any other new capability, the earlier quality issues are dealt with; the less focus is taken from teams’ primary goal of building awesome services. 

Agile development methods are also a key component in ensuring speed – and in chatbot technology, velocity is critical to introducing the new innovative capabilities which help enterprises stay one step ahead of competition. The need for speed means that agile development, and automated processes, is a must have rather than a nice to have.  Manual testing means slowing down feedback to developers after an issue has been introduced – and makes the development process cumbersome and unnecessarily extended. 

A bright future, with the right testing 

There is no doubt that voice technology is going to fundamentally change the way users interact with applications. Chatbot deployment might be just one element of the wider pursuit of better digital engagement, but it’s a crucial tactic – and one which may well be a key component in determining the ultimate digital winners.   

Chatbots are rapidly adding value to customer interactions. The conversations aim at solving customer problems in the most convenient manner. So, the entire experience, thus, depends deeply on how tactfully the chatbot steers conversations to yield the proper responses. 

So, if conversation is the key skillset of a successful chatbot, businesses need to thoroughly focus of enriching the user experience delivered by bots - and that process starts with robust testing. 

The companies who get it right be those who start from firm foundations – investing in agile development, automated processes – and software that can help to put testing first – right at the heart of innovation. 

Carlo Cadet, Product Marketing Director for Perfecto 

Image Credit: Montri Nipitvittaya / Shutterstock

Carlo Cadet
Carlo joined Perfecto in 2013 and currently leads the product marketing team. In his role, he is responsible for evangelising best practices to customers, analysts and media. His team delivers Sales training, crafts company positioning, messaging and determines go-to-market strategy, for the company’s cloud based solution. Carlo has an MBA from MIT Sloan School of Business and a degree in Computer Science from Tufts University.