The Great HTML5 Gaming Performance Test: 2014 edition

We've done a big performance test in the past, but that last test is nearly two years old now (the references to the Playbook make it look particularly dated!) and things are moving quickly in the world of web technology. This time around we've tested a range of desktop and mobile devices, browsers and OSs covering a total of 30 combinations of device and software. The overall state of HTML5 performance is considerably ahead of where it was in 2012. Browser technology, Javascript engines, device hardware, drivers and our own HTML5 game engine have seen countless improvements. The future of HTML5 gaming is very promising - but there are still a few lingering issues. Read on to learn more.

The test

Our standard test is the same as before: sbperftest, an automated version of our stock demo game Space Blaster. It's based on a real-world game using our engine, it doesn't need input so runs the same every time, and it automatically measures the framerate producing a final test score which is the average framerate throughout the test. It actually gets pretty intense with the sprite counts and blend modes on the lasers and explosions, and we've found it's a pretty good indicator of a platform's performance - if sbperftest runs well, most other HTML5 games should run decently as well.

Note the CPU utilisation measurement is a slight misnomer - it's measured from Javascript and is based on timers so it should be viewed with some skepticism, especially since it's only measuring the main thread and many browsers like Chrome are parallel in key components like the renderer. A better name would be "percentage of time available to run Javascript that is used up", but that's a bit of a mouthful! It's not taken in to account for the test result anyway, it's just interesting to observe sometimes.

The devices

For testing and development, we've amassed a fair few devices in the Scirra office. This puts us in a good position to test a wide range of devices and OSs. For brevity in the results table we've given them nicknames (in bold below), but the full specs are listed here if you're curious. Note mobile device specs can be hard to come by so some details below are sourced from Wikipedia.

Desktop devices:

• Windows desktop: Windows 8.1 x64, Intel Core i5-2500 @ 3.3 GHz, 8 GB RAM, nVidia GeForce GTX 660
• Mac Mini: Mac OS X 10.9.2, Intel Core i5 @ 2.5 GHz, 2 GB RAM, Intel HD Graphics 3000
• Ubuntu desktop: Ubuntu 14.04 x64, Intel Core i7-3820 @ 3.6 GHz, 16 GM RAM, AMD Radeon HD 6570

Mobile devices:

• Nexus 5: Android 4.4.2, Krait 400 @ 2.3 GHz, 2 GB RAM, Adreno 330
• Nexus 7 (first generation, 2012 model): Android 4.4.2, nVidia Tegra 3 T30L @ 1.2 GHz, 1 GB RAM
• SGS3 (Samsung Galaxy S3 i9300): CyanogenMod 10.2.0 (Android 4.3.1), Cortex-A9 @ 1.4 GHz, 1 GB RAM, Mali-400 MP4
• HTC Sensation: Android 4.0.3, Qualcomm Scorpion @ 1.2 GHz, 768 MB RAM, Adreno 220
• iPad 2: iOS 7.1, Cortex-A9 @ 1 GHz, 512 MB RAM, PowerVR SGX543MP2
• iPad 3: iOS 7.1, Cortex-A9 @ 1 GHz, 1 GB RAM, PowerVR SGX543MP4
• iPhone 4S: iOS 7.1, Cortex-A9 @ 800 MHz, 512 MB RAM, PowerVR SGX543MP2
• iPhone 5: iOS 7.1, A6 @ 1.3 GHz, 1 GB RAM, PowerVR SGX543MP3
• Lumia 520: Windows Phone 8.1 (preview), Krait @ 1 GHz, 512 MB RAM, Adreno 305
• Kindle Fire HD 7" (first generation, 2012): Fire OS (based on Android), 1.2 GHz, 1 GB RAM, PowerVR SGX540
• Blackberry Z10: Blackberry OS 10.2, Krait 1.5 GHz, 2 GB RAM, Adreno 225
• Tizen: Tizen OS 2.2, developer device with unknown specs but looks/feels like a Galaxy S4

The results

Without further ado, here are our measurements!

Analysis

The results are far stronger than our previous test in 2012. Performance is good across a wide range of devices and OSs. Thanks to optimisations to OSs, browsers, and our own engine, many scores were improved. Mobile browsers have benefited particularly well, most of which can now easily keep the framerate above 30 FPS at all times.

The main problem device was the HTC sensation. This ageing Android device never got updated beyond Android 4.0.3, and Chrome has blacklisted its GPU presumably due to stability issues. (Graphics driver issues are back to haunt us!) Stuck with software rendering on a weak mobile CPU, it produces an unplayable framerate averaging at just 7 FPS. The stock browser is not much better: it never supported GPU acceleration in the first place, so similarly gets a lousy average of 18 FPS due to software rendering. Interestingly Firefox for Android makes full use of the GPU even running with WebGL support, and scores a respectable and playable average of 48 FPS. This is better than some of the fully-up-to-date iOS devices. Chrome can actually also score an average 48 FPS on the HTC Sensation if you head in to chrome://flags and enable 'Override software rendering list', so it ignores the GPU blacklist. This illustrates that even on a crusty old Android phone on its last legs, the hardware is still perfectly capable of handling the game. The only problem is that sometimes the browser will not use the GPU.

WebGL support has come on in strides since our last test. Previously WebGL support on mobile was practically unheard of, and now Apple platforms are the only major platforms without support. We've already seen how WebGL support significantly improves performance, so a future version of iOS adding WebGL support could see even the older iOS devices getting their results topped up to 60. Apart from Apple, there are a few spots where it still fell back to canvas2d: Chrome on Ubuntu, where Chrome didn't trust the bundled open-source graphics driver, but the mighty i7 CPU was fast enough to run software rendering at 60 FPS; Chrome on the Galaxy S3 where the GPU was blacklisted for WebGL but could still be used for canvas2d, and the hardware was good enough to still get a great result; the HTC Sensation which suffered badly from blacklisting; and the Kindle Fire HD which appears to lack WebGL support but could get by with canvas2d.

The newly-supported Ejecta wrapper for iOS also performs excellently, even outperforming Safari despite Apple's restrictions that native apps cannot JIT-compile Javascript for top performance. That's the kind of difference WebGL support can make, and also allows you to make use of WebGL shaders on iOS.

Note we didn't test Crosswalk on Android - there is very little point, since it is essentially the same as Chrome for Android. It can be assumed that Crosswalk will perform identically to Chrome.

Conclusion

Desktop devices are generally all sorted. Despite being plagued by performance issues in 2011, with a now-solid selection of browsers, OSs and drivers you can be pretty confident that desktops will be able to blow through HTML5 games just fine.

Mobile devices have for a long time now had sufficient hardware to run HTML5 games just fine. Even the HTC Sensation, now a junk Android phone from 2011, can manage a decent framerate providing the browser makes full use of the GPU. However many mobile manufacturers have the somewhat galling attitude of abandoning their devices a year or two after releasing them. This is frustrating as a developer and offensive as a consumer: it's analogous to buying a brand new Windows 7 desktop, then finding 18 months later it's officially impossible to update it to Windows 8 or newer. Google's line of excellent Nexus devices shows it doesn't have to be this way, but for devices like the Sensation it ends up stuck with an old OS with broken drivers that crash regularly, forcing browser developers to blacklist them so you don't have to keep pulling the battery out of your phone because it locked up while browsing. When you hear about HTML5 performance complaints, it is almost always actually a device with perfectly sufficient hardware that has been blacklisted by a browser maker, so it gets software rendering. (The next most common case is usually someone designing or porting such a complex game that even a native engine would struggle on the limited, battery-powered mobile hardware.) Ironically the stability problems are caused by the native technology in the drivers, not the web technology.

The situation on mobile is very similar to the situation on desktop in 2011. The problem is effectively already solved: the latest crop of devices have great performance and support, but there are some legacy systems still hanging around being left behind by their manufacturers. These can be problematic, but the good news is there can be workarounds (such as switching to Firefox on Android, or skipping the blacklist), and the old broken devices are steadily dying off. Even new budget devices like the Lumia 520 can pull off decent performance even with limited specs thanks to good software. The wait for the devices running abandoned software die out is awkward, but the hardware has been good enough for years, and the software today is already good enough. This would appear to guarantee that the future of mobile HTML5 gaming is high performance everywhere.