Feedback

Samsung stops cheating on benchmarks with new Android update: Galaxy S5 provides real test

MobileNews
by Joel Hruska, 05 Mar 2014News
Samsung stops cheating on benchmarks with new Android update: Galaxy S5 provides real test

Last year, Samsung came under heavy fire for its benchmark shenanigans and certain dubious practices. The company deliberately built a whitelist of benchmark applications for products like the Note 3 and Galaxy S4. If any of these applications were detected, the CPU would immediately ramp up to 100 per cent of its possible clock, regardless of the impact on power consumption or battery life.

This kicked Samsung's benchmark scores into overdrive and predictably angered a number of reviewers and users. There was no chance that the behaviour was incidental; investigations demonstrated that Samsung was deliberately detecting specific benchmarks rather than other applications, and handing those benchmarks different operating conditions than the rest of the programs that run on the device.

With Android 4.4 KitKat, Samsung has stopped this behaviour. The whitelists have been removed, the browser no longer detects SunSpider or Rightware's Browsermark, and the phone doesn't display unusual activity or processor clocks when benchmark applications are run.

Ars Technica spoke with John Poole of Geekbench, who shared some pre- and post-patch data for the Galaxy S4 that neatly illustrates how Samsung twisted its performance results. Under Android 4.2.2, the Galaxy S4 scored an 1,812 multicore score. That jumped to a 2,114 using Android 4.3, but fell back to 1,913 with Android 4.4.2 (which is confirmed as behaving normally).

In other words, Samsung inflated its multi-threaded benchmark scores by 11-16 per cent, depending on whether you consider 1,812 or 1,913 as an appropriate baseline. The reason this kind of cheating matters is that it makes it impossible for reviewers or benchmark authors to fairly evaluate the product. Despite occasional claims to the contrary, the goal of a reputable reviewer is to build a suite of test software that collectively paints a fair picture of phone performance in real-world conditions.

Samsung's whitelist watched for all of the most common smartphone benchmarks and deliberately altered the SoC's performance in a way that didn't reflect the experience of running games or other applications. Attempting to capture the performance of a device with a handful of applications and then translate those metrics to the variety of real-world conditions is already difficult; deliberate score inflation makes this nearly impossible.

We're glad to see Samsung moving away from this behaviour on older devices, but the real test will come with the launch of the Galaxy S5. If the company includes the same kind of whitelisted activity or attempts to implement a new type of sneakier obfuscation, the goodwill earned from stopping such practices today will go right out the window.

Topics
blog comments powered by Disqus