The HTC 10 with Nougat (Android 7.0) update appears to be specifically designed to cheat battery life benchmarks. Here's what I've found.
We make Construct 2, a game creator. A user reported a bug involving choppy animations (often called "jank"). I tried out their demo on my phone — a HTC 10 — and sure enough it was pretty choppy. According to an FPS measurement, it was running at 50 FPS instead of syncing to the display refresh rate at 60 FPS.
Initially I suspected a Chrome bug, so I reported it. However upon investigation, Google engineers identified the fundamental cause as the Android OS throttling the v-sync signal. This means the HTC 10's version of Android is pretending the display is running at 50 Hz (with 20ms intervals) instead of the correct 60 Hz (with ~16ms intervals). Since it wasn't a Chrome bug the issue was closed, and one of the Chrome engineers left a comment saying they suspected it was done for benchmark purposes.
It also emerged that the device does actually render at a smooth 60 FPS, until it is left idle for two or three seconds — then it drops down to 50 FPS. This means when you interact with the device, like scrolling, it appears smooth. Then when you leave it idle for a few seconds, it starts throttling. In my opinion this is an attempt at hiding the throttling from users: so long as you keep tapping it stays smooth, but if you wait and watch something like a game just for a moment, it suddenly drops in to a horrible juddering. (It looks bad because 50 does not divide evenly in to 60, so the frame update pattern is irregular, which many people find uncomfortable to look at.)
Our engine uses WebGL for games in the browser, and this made me think of battery life benchmarks. For example in Ars Technica's original HTC 10 review, they specifically use a WebGL battery life test on page 2, which is one of their standard tests. Assuming nobody was interacting with the device during the test (which appears to run for hours), the HTC 10 will be running at 50 FPS, while other devices it is compared to are running at 60 FPS, therefore giving the HTC 10 an unfair advantage. Not only is 50 FPS simply less work per second, but every millisecond counts for saving power, and a few extra milliseconds idle per frame allows the hardware to remain in a low-power state for longer. This all makes the HTC 10 look like it has better power characteristics than it really has.
HTC may try to claim this is some kind of power saving feature, but it doesn't appear it can be switched off anywhere despite obnoxiously affecting performance, and it still happens when the phone is charging. Alternatively they may claim it's a bug, but it appears to be carefully calibrated to avoid it being noticed — it won't happen while scrolling, it's not a massive change which might be too noticable, and it reliably switches between smooth and choppy mode depending on when you last touched the screen. Therefore I would conclude this is deliberately designed to cheat battery life benchmarks. If you run benchmarks you might want to look out for this — it's possible other devices are affected too.
You can check for this yourself by visiting this test on a HTC 10. It runs a very simple animation and shows an FPS counter in the corner. On a HTC 10 the FPS clearly switches between 49-50 and 58-60 depending on when you interact with the device. I should point out the device originally came with Marshmallow (Android 6.0), but I only noticed this when the device eventually updated to Nougat (Android 7.0) — I don't know if Marshmallow was also affected.
This is incredibly frustrating both as a developer and a customer. As a developer I work hard to make sure users get a smooth 60 FPS experience, and Google has made enormous efforts to do the same for Chrome and Android, and then HTC ruin it to cheat a benchmark. And as a customer, I want this expensive high-end phone to run smoothly in all situations, not just when it thinks I'm not running a benchmark.
Guess I should have bought a Pixel after all.