Measuring the mobile user experience – an evolution from ‘man with a suitcase’ to ‘mass-sourcing’

null

The global network test and measurement market is a multi-billion dollar industry and plays a key role in the development of new products and technologies and drives ongoing improvements to mobile quality and customer experience. As the world becomes increasingly mobile, testing and measuring mobile network performance has taken centre-stage and the industry has come a long way since the old ways of manually ‘road-testing’.

Network operators continue to throw money at investing in spectrum, improving their networks, and as 5G rolls-out, attempting to map coverage areas and data usage that continues to change and expand.  Meanwhile, methods for testing and measuring these factors are constantly developing and there is little consensus about which method is the most useful. Network operators say one thing, their rivals say another, and third-party data collection companies sit in the middle – ostensibly providing some objectivity.

In the early days of mobile testing, networks began employing someone to drive around the country with a clutch of mobile devices on the passenger seat – or even a man with a suitcase on foot! 

Things have moved on since then and modern drive tests tend to be performed with a customised measuring van, on which an antenna is fitted for each of the various measuring instruments to ensure the instruments have consistent and repeatable conditions. These can also be performed using mobile equipment on trains, for example, and via ‘walk tests’ where using a vehicle isn’t possible or practical.

Even though drive testing has come a long way, it still has limitations. While it remains an important part of fault-finding and ‘field acceptance testing’ – ensuring new antenna are positioned correctly to deliver a signal as expected – the issue with drive testing for more comprehensive measurement is that it tends to focus on major cities and is conducted infrequently. It also measures only network performance rather than end user experience, and drive tests usually take place at night, as the roads are quieter, meaning the results do not give an accurate representation of day-to-day usage. As testing can’t take place everywhere simultaneously, the value and available use-cases for the data is limited.

Drive testing data is now commonly supplemented via crowdsourcing – using the harvesting of real-time data from the phones of actual users. In fact this is nothing new. Speed-testing applications and mobile operator applications have been silently and anonymously measuring the user’s network quality in the background of their applications for years, and transmitting this back to central databases. In this crude form of crowdsourcing, however, users are required to download and use specific applications – meaning that results were more likely to come from more technically minded users or from those who are running speed tests because they’re experiencing issues, skewing the results.

True crowdsourcing is not only more cost effective, but more representative of “real-world” conditions because operators can harvest real-time app, device and connectivity data. At Tutela for example, we run our software in the background of popular mobile applications to crowdsource anonymous data on the phones of real users throughout the day and night to measure typical performance. Users cannot manually start a speed-test for themselves, as doing so would bias data towards when users had a good or bad experience.

With information being shared in ever-greater quantities, however, we are already starting to see crowdsourcing evolve into something much greater. Advances such as the introduction of 5G, for example, will mean increased numbers of antenna and faster, more complex networks needing more sophisticated insights and measurement, with ultra-high resolution quality maps. For network operators to gather a true picture of where their data is being used and how, for example understanding exactly which antenna is causing a problem or the quality of individual cells, they really need to be looking across hundreds of millions of devices – not tens of thousands.

This new form of crowdsourcing – perhaps better termed ‘mass-sourcing’ – allows a truer picture to emerge as it provides a detailed, continuously updated view of mobile quality of experience and usage globally. While advances in smart phone technology have made collecting data on this scale much simpler, it is not without its challenges. It is still a major undertaking to install the software onto hundreds of millions of mobile devices to reach the scale required. 

The benefits of mass-sourcing, however, far outweigh the implementation challenges. By gathering data in this way, and on this scale, network operators can analyse problems and make decisions that are informed by evidence on the ground, not just intuition. This is because the data is representative and shows where people actually use their devices, across a whole range of geographic areas - from big cities to small villages.

For example, Tutela recently carried out some research into mobile usage in Brazil and discovered a significant difference between the download speeds available in the North and South of the country. The South was receiving speeds up to three times faster than those in the north. Meanwhile, looking at different cities, Sao Paulo had the fastest average download speeds across all networks – up to 12.10Mbps – whereas users in Roraima reported speeds of just 4.20Mbps.

Similarly, a study of Argentinian mobile usage in May revealed that users on the Movistar network were getting the fastest 4G download speeds across the country, taking the lead from the previous leader, Telecom.  Drilling in to the data, however, revealed some differences in network performance across the provinces – with Telecom and Movistar tied in Buenos Aires and Telecom taking the lead in Cordoba and Santa Fe for the fastest download speeds, despite Movistar leading on average performance across the whole country.

Armed with this kind of information, mobile network operators are given the confidence to make highly targeted improvements to increase coverage for consumers exactly where and when they most need it. For example, with the huge volumes of data that mass-sourcing offers, it is possible to monitor network quality on specific neighbourhoods, streets - even specific cell towers. This opens new use-cases such as planning for 5G small-cell roll-out, monitoring quality of newly installed infrastructure and even plotting the locations of competitor cell towers.

In an industry experiencing growing demands and shrinking margins, this could be the difference between those that evolve and those that become extinct. 

Tom Luke, VP and GM of Tutela 

Image Credit: Nito / Shutterstock