How OnePlus and Meizu got busted for faking benchmark tests

The Android smartphone scene always has a handful of devices competing against each other at any given time. For manufacturers, reviews meant everything. They were the primary way of knowing how the public perceived the phone as. Camera reviews were done using comparing shots from different devices, speakers by ears, and it was a wide belief that performance could be accurately measured using benchmark tests. Although it is up to debate whether benchmarks results could be interpreted as real-world usage, there is no denying that they are one of the most popular tests for a smartphone and many buyers still take benchmarks tests into consideration before making a purchase.

The guys over at XDA-Developers, with help from Primate Labs (authors of Geekbench) recently unearthed that OnePlus 3 and OnePlus 3T and Meizu Pro 6 were cheating in the benchmark tests.

What does Benchmark Cheating mean?

Under normal usage conditions, a smartphone adjusts its clock speed according to the current usage scenario. This means that under light usage (say, music and/or checking messages), the smartphone lowers the clock speed in order to conserve battery and be more efficient. In the case of muti-core processors, the extra cores are also shut off, hence “parked”, as they are not required for that particular usage scenario. This is completely normal and this should be put into practice at all times when the device is in use. However, when a well-recognized benchmark app is fired, it was observed that the parked cores were also initialized. In the case of the OnePlus 3/3T, the cores idle at around 0.31GHz. However, when popular benchmarking apps were turned on, the CPU Governor became more aggressive, resulting in practically an artificial show of performance, which would not be visible in real-world use. It wasn’t looking for a change in CPU Load to trigger this, but a specific app name (here, Geekbench and others). A benchmark app disguised as a normal app would not trigger this state, resulting in a bit lower scores, but accurate ones. 

OnePlus recently issued a statement regarding the above events:

‘In order to give users a better user experience in resource intensive apps and games, especially graphically intensive ones, we implemented certain mechanisms in the community and Nougat builds to trigger the processor to run more aggressively. The trigger process for benchmarking apps will not be present in upcoming OxygenOS builds on the OnePlus 3 and OnePlus 3T.’

This is a good sign as OnePlus is admitting to their mistakes and practices and vows to put an end to this.

Meizu’s play in the game

In the case of the Meizu Pro 6, the CPU Governer always stayed in a conservative mode, parking the BIG set of cores at most times. This was first observed and reported by Anandtech. A week later the review, Meizu pushed an update which made the BIG cores much more “awake”. This obviously resulted in better benchmark scores. However, in XDA’s test, it was noticed that the device would prompt the user to switch to “high-performance mode” once it detected a benchmarking app, hence again artificially rigging the scores and not reflecting accurate performance which would be seen in real world usage. 

Meizu’s deviation from the real scores to the rigged ones were MUCH more substantial. This is a very bad sign as this would absolutely not reflect real-world usage performance.

Verdict

Benchmark tests are still a good way of determining a device’s raw prowess but must be taken with a grain of salt. Although the OnePlus 3/3T are great devices nevertheless, rigging benchmark tests is a bad sign, and the company must see that this practice is stopped. Meizu’s case is much more drastic and benchmarks are definitely no way to judge a device in this scenario.

Via: XDA-Developers