Our standard test is the same as before: sbperftest, an automated version of our stock demo game Space Blaster. It's based on a real-world game using our engine, it doesn't need input so runs the same every time, and it automatically measures the framerate producing a final test score which is the average framerate throughout the test. It actually gets pretty intense with the sprite counts and blend modes on the lasers and explosions, and we've found it's a pretty good indicator of a platform's performance - if sbperftest runs well, most other HTML5 games should run decently as well.
For testing and development, we've amassed a fair few devices in the Scirra office. This puts us in a good position to test a wide range of devices and OSs. For brevity in the results table we've given them nicknames (in bold below), but the full specs are listed here if you're curious. Note mobile device specs can be hard to come by so some details below are sourced from Wikipedia.
- Windows desktop: Windows 8.1 x64, Intel Core i5-2500 @ 3.3 GHz, 8 GB RAM, nVidia GeForce GTX 660
- Mac Mini: Mac OS X 10.9.2, Intel Core i5 @ 2.5 GHz, 2 GB RAM, Intel HD Graphics 3000
- Ubuntu desktop: Ubuntu 14.04 x64, Intel Core i7-3820 @ 3.6 GHz, 16 GM RAM, AMD Radeon HD 6570
- Nexus 5: Android 4.4.2, Krait 400 @ 2.3 GHz, 2 GB RAM, Adreno 330
- Nexus 7 (first generation, 2012 model): Android 4.4.2, nVidia Tegra 3 T30L @ 1.2 GHz, 1 GB RAM
- SGS3 (Samsung Galaxy S3 i9300): CyanogenMod 10.2.0 (Android 4.3.1), Cortex-A9 @ 1.4 GHz, 1 GB RAM, Mali-400 MP4
- HTC Sensation: Android 4.0.3, Qualcomm Scorpion @ 1.2 GHz, 768 MB RAM, Adreno 220
- iPad 2: iOS 7.1, Cortex-A9 @ 1 GHz, 512 MB RAM, PowerVR SGX543MP2
- iPad 3: iOS 7.1, Cortex-A9 @ 1 GHz, 1 GB RAM, PowerVR SGX543MP4
- iPhone 4S: iOS 7.1, Cortex-A9 @ 800 MHz, 512 MB RAM, PowerVR SGX543MP2
- iPhone 5: iOS 7.1, A6 @ 1.3 GHz, 1 GB RAM, PowerVR SGX543MP3
- Lumia 520: Windows Phone 8.1 (preview), Krait @ 1 GHz, 512 MB RAM, Adreno 305
- Kindle Fire HD 7" (first generation, 2012): Fire OS (based on Android), 1.2 GHz, 1 GB RAM, PowerVR SGX540
- Blackberry Z10: Blackberry OS 10.2, Krait 1.5 GHz, 2 GB RAM, Adreno 225
- Tizen: Tizen OS 2.2, developer device with unknown specs but looks/feels like a Galaxy S4
Without further ado, here are our measurements!
|Device ||Software ||Notes ||Score|
|Windows desktop ||Chrome 34 ||webgl ||60|
|Windows desktop ||Firefox 28 ||webgl ||60|
|Windows desktop ||IE 11 ||webgl ||60|
|Mac Mini ||Chrome 34 ||webgl ||60|
|Mac Mini ||Firefox 28 ||webgl ||60|
|Mac Mini ||Safari 7 ||canvas2d ||60|
|Ubuntu desktop ||Chromium 34 ||canvas2d, blacklisted gpu ||60|
|Ubuntu desktop ||Firefox 29 ||webgl ||55|
|Nexus 5 ||Chrome 34 ||webgl ||59|
|Nexus 5 ||Firefox 28 ||webgl ||58|
|Nexus 7 ||Chrome 34 ||webgl ||59|
|Nexus 7 ||Firefox 28 ||webgl ||51|
|SGS3 ||Chrome 34 ||canvas2d, blacklisted webgl ||58|
|SGS3 ||Firefox 28 ||webgl ||58|
|SGS3 ||Stock browser ||webgl (enabled by CyanogenMod) ||52|
|HTC ||Chrome 34 ||canvas2d, blacklisted gpu ||7|
|HTC ||Firefox 28 ||webgl ||48|
|HTC ||Stock browser ||canvas2d, no gpu support ||18|
|iPad 2 ||Safari 7.1 ||canvas2d ||46|
|iPad 2 ||Ejecta ||webgl, no JIT ||55|
|iPad 3 ||Safari 7.1 ||canvas2d ||42|
|iPad 3 ||Ejecta ||webgl, no JIT ||52|
|iPhone 4S ||Safari 7.1 ||canvas2d ||40|
|iPhone 4S ||Ejecta ||webgl, no JIT ||51|
|iPhone 5 ||Safari 7.1 ||canvas2d ||60|
|Lumia 520 ||IE 11 ||webgl ||47|
|Kindle ||Silk ||canvas2d ||44|
|Z10 ||Browser ||webgl ||54|
|Tizen ||Browser ||webgl ||57|
The results are far stronger than our previous test in 2012. Performance is good across a wide range of devices and OSs. Thanks to optimisations to OSs, browsers, and our own engine, many scores were improved. Mobile browsers have benefited particularly well, most of which can now easily keep the framerate above 30 FPS at all times.
The main problem device was the HTC sensation. This ageing Android device never got updated beyond Android 4.0.3, and Chrome has blacklisted its GPU presumably due to stability issues. (Graphics driver issues are back to haunt us!) Stuck with software rendering on a weak mobile CPU, it produces an unplayable framerate averaging at just 7 FPS. The stock browser is not much better: it never supported GPU acceleration in the first place, so similarly gets a lousy average of 18 FPS due to software rendering. Interestingly Firefox for Android makes full use of the GPU even running with WebGL support, and scores a respectable and playable average of 48 FPS. This is better than some of the fully-up-to-date iOS devices. Chrome can actually also score an average 48 FPS on the HTC Sensation if you head in to chrome://flags and enable 'Override software rendering list', so it ignores the GPU blacklist. This illustrates that even on a crusty old Android phone on its last legs, the hardware is still perfectly capable of handling the game. The only problem is that sometimes the browser will not use the GPU.
WebGL support has come on in strides since our last test. Previously WebGL support on mobile was practically unheard of, and now Apple platforms are the only major platforms without support. We've already seen how WebGL support significantly improves performance, so a future version of iOS adding WebGL support could see even the older iOS devices getting their results topped up to 60. Apart from Apple, there are a few spots where it still fell back to canvas2d: Chrome on Ubuntu, where Chrome didn't trust the bundled open-source graphics driver, but the mighty i7 CPU was fast enough to run software rendering at 60 FPS; Chrome on the Galaxy S3 where the GPU was blacklisted for WebGL but could still be used for canvas2d, and the hardware was good enough to still get a great result; the HTC Sensation which suffered badly from blacklisting; and the Kindle Fire HD which appears to lack WebGL support but could get by with canvas2d.
Note we didn't test Crosswalk on Android - there is very little point, since it is essentially the same as Chrome for Android. It can be assumed that Crosswalk will perform identically to Chrome.
Desktop devices are generally all sorted. Despite being plagued by performance issues in 2011, with a now-solid selection of browsers, OSs and drivers you can be pretty confident that desktops will be able to blow through HTML5 games just fine.
Mobile devices have for a long time now had sufficient hardware to run HTML5 games just fine. Even the HTC Sensation, now a junk Android phone from 2011, can manage a decent framerate providing the browser makes full use of the GPU. However many mobile manufacturers have the somewhat galling attitude of abandoning their devices a year or two after releasing them. This is frustrating as a developer and offensive as a consumer: it's analogous to buying a brand new Windows 7 desktop, then finding 18 months later it's officially impossible to update it to Windows 8 or newer. Google's line of excellent Nexus devices shows it doesn't have to be this way, but for devices like the Sensation it ends up stuck with an old OS with broken drivers that crash regularly, forcing browser developers to blacklist them so you don't have to keep pulling the battery out of your phone because it locked up while browsing. When you hear about HTML5 performance complaints, it is almost always actually a device with perfectly sufficient hardware that has been blacklisted by a browser maker, so it gets software rendering. (The next most common case is usually someone designing or porting such a complex game that even a native engine would struggle on the limited, battery-powered mobile hardware.) Ironically the stability problems are caused by the native technology in the drivers, not the web technology.
The situation on mobile is very similar to the situation on desktop in 2011. The problem is effectively already solved: the latest crop of devices have great performance and support, but there are some legacy systems still hanging around being left behind by their manufacturers. These can be problematic, but the good news is there can be workarounds (such as switching to Firefox on Android, or skipping the blacklist), and the old broken devices are steadily dying off. Even new budget devices like the Lumia 520 can pull off decent performance even with limited specs thanks to good software. The wait for the devices running abandoned software die out is awkward, but the hardware has been good enough for years, and the software today is already good enough. This would appear to guarantee that the future of mobile HTML5 gaming is high performance everywhere.