The Great HTML5 Mobile Gaming Performance Comparison

Update 11th June 2012: added score for Playbook OS 2.1. It's literally 3x faster than OS 2.0, so the conclusion for Playbook has changed from "unusable" to "great"!

Much has been said about the performance of HTML5 games on mobile, often around the fact it's too slow to be practical. This week we measured the performance of 23 combinations of browser and device with a real-world game test made using Construct 2, our HTML5 canvas based game creator. We've spent some time optimising our engine for best performance, and results are promising: despite the naysayers, it looks like HTML5 mobile gaming is a genuinely viable option already - and it is likely to improve further.

The test is an automated version of Space Blaster, our stock demo game for Construct 2. It's a simple arcade space shooter that, despite occasionally having a lot of on-screen objects, should not cause any problems for any desktop or mobile device using hardware acceleration. Of course, not all platforms use hardware acceleration or they don't implement it efficiently, which is the cause of the well-known performance problems.

To run the test just visit the performance test URL and watch it go. The test scores by taking an average framerate over a period of about a minute. Browsers often spend the first few seconds optimising Javascript, so to prevent JIT compiles and other startup processes affecting the result the first five seconds are not counted. Further, the test has no sound since some platforms don't play audio (like iOS's Safari browser), and devices which can sometimes take a performance hit from having to also process audio, so for fairness sound is excluded completely. Since the test has been adapted from a real game using a real engine, this is probably the best real-world test we can come up with. Hopefully its results provide a good indication of the practical performance you can expect from each platform.

The devices listed below are desktop (Windows 7 Core i5-2500 quad-core @ 3.3 GHz; 8 GB RAM; AMD Radeon HD 6570 graphics), iPad 2, iPad 3, iPhone 4S (all on iOS 5.1.1), Android 4 (Ice Cream Sandwich) (on a HTC Sensation Z710e), Windows Phone 7.5 (Mango) (on a Nokia Lumia 800), and a BlackBerry Playbook. Browsers tested include various desktop and mobile variants of Chrome, Firefox, Internet Explorer, Safari, and Opera, as well as the Android stock browser ("Internet") and the Playbook stock browser ("Browser"). We also tested the non-browser HTML5 game wrappers directCanvas by appMobi (currently only available for iOS, but coming soon to Android) and CocoonJS by Ludei (for both iOS and Android), both designed to boost the performance of HTML5 games and supported by Construct 2. Obviously the total number of combinations of device, platform and browser out there is mind-boggling (which some people cite as part of the problem), but I think we've covered a fairly good cross-section. It's also important to note the test is running identically on all 23 tested platforms, with fluid scaling depending on the device screen size including support for Retina displays, all with the same code base - the only variant is the performance. This is why we don't believe there is a problem with having lots of platforms. Take this as a comparison: Windows runs on a mind-boggling array of hardware, and most Windows apps run just fine everywhere. You can make a Windows app and just trust that it will run on any PC with any hardware running Windows. Likewise we believe while testing is prudent since HTML5 is newer technology, we hope one day you can similarly design a HTML5 game and simply trust it will run everywhere. This test is a good working example of that already beginning to happen.

Another thing essential for HTML5 mobile gaming is multitouch support. Imagine a jump and run game: single touch means you can't jump while holding down to run - game over! All platforms support multitouch, with the exception of Android's stock browser in 2.x (support was added in 4.0) and Windows Phone 7, which shockingly only gives you a "click" event. Frustratingly both actually support multitouch in their hardware, but are held back by archaic software support. Windows Phone 7 is nearly totally useless for HTML5 gaming in its browser because it does not even have proper single-touch support! Luckily the PhoneGap team came to the rescue and added support for single-touch via mouse events (make sure you enable 'use mouse input' in Construct 2's Touch plugin), but as mentioned single-touch still rules out many games. I don't know why anyone would want to develop for a device with a vanishingly small market share and severe device limitations. Still, if you have a game that works with single-touch, you might want to give it a shot with PhoneGap Build. Android 2.x is rescued by directCanvas and CocoonJS - both add support for multitouch even if the stock browser does not support it.

So now on to the results. Remember the average framerate was measured, so higher is better. Much below 20 means it is questionable if the game will be playable; 30 FPS is perfectly playable; 60 FPS is the target. Notes: on iOS the results for Safari are after 'Add to home screen', so the game is rendered as close to fullscreen as possible. Also for technical reasons some technologies don't use the "retina" high-resolution display - they will render at low-resolution and just stretch the game up to fill the screen. These technologies are noted "using retina" or "not using retina". Using retina looks better with crisper graphics but should be expected to be slower since it has more pixels to fill. The iPhone 4S and iPad 3 have hi-res retina displays, but the iPad 2 does not. Finally, PhoneGap for the Playbook was not tested because I couldn't find an obvious way to run the output of PhoneGap Build on it without making a full submission to the Blackberry App World.

The Test Results

Review

Performance is extremely varied: ranging from 3-60 FPS, a difference of a factor of 20!

Desktop performance is generally no longer a problem. The top three browsers barely broke a sweat thanks to their hardware acceleration. Desktop users still need to make sure they have up-to-date graphics card drivers else the browser may switch off hardware acceleration; however even with software rendering (like Safari and Opera) desktop performance is usually in the playable range. Our WebGL renderer also helps Chrome and Firefox run about twice as fast as Canvas 2D, although we currently have disabled WebGL rendering on mobile devices since it has always been slower than Canvas 2D in our tests.

PhoneGap unfortunately appears on the whole to be unsuitable for publishing HTML5 games. This is mainly because it is stuck with the system default browser which is usually too slow. On iOS the system browser is Safari which actually performs very well, but when you wrap it up in PhoneGap it drops three times slower on iPad 2, ten times slower (!!) on iPad 3 (using the retina display), and bizarrely for reasons I really struggle to comprehend, is only marginally slower on the iPhone 4S. So the only utility for PhoneGap appears to be for targeting the iPhone. Windows Phone 7 also seemed to take a performance hit from using PhoneGap, so perhaps there are difficult technical issues around putting a browser in a native app.

Safari on iOS is consistently good - the best performing mobile browser by a clear margin, even when rendering to a high-resolution retina display. If you 'Add to home screen' like was done for this test, the game also gets an icon which makes it look like a native app. Combine that with the fact Construct 2 games still work offline, and no 30% fee if you charge for your game, and adding to the home screen may be a viable way to distribute HTML5 games. Unfortunately you will run in to problems trying to get Safari to play audio - you might have to stick to just a music track. More information in How to make an iOS Web App.

Across the board directCanvas and CocoonJS out-perform all the browsers. This is not surprising since they are optimised to maximise performance and always use hardware-acceleration. They also add multitouch for Android 2.x. These options are definitely the best ways to package up your HTML5 game as a native app for mobile. Both are works-in-progress with Construct 2 integration for the time being though, but they should both be robust and ready for production in the near future. However they prove good performance is definitely possible with HTML5 games - it's only the browser software that is holding anything back.

Android has a few browsers available, but Chrome for Android beta is the only actual browser that provides playable performance. It's a separate download and can't be used with PhoneGap, but it's GPU-accelerated fast and supports multi-touch. Hopefully one day Chrome will become the default browser on Android and that will improve the situation considerably, but it seems that is years off becoming reality given the prevalence of Android 2.x devices. Until then you're stuck with a very slow stock browser that in this test started glitching so badly you couldn't tell what was going on. Use directCanvas or CocoonJS for Android.

The Playbook comes in with a suprise result: OS 2.1 is great! OS 2.0 is too slow to be usable, but OS 2.1 fixes it and is literally a whole 3x faster. Its result is competitive with the Android results, but with the advantage of having good support in the native browser (with multi-touch, too). It's also worth noting the Playbook is considerably cheaper than most other devices - given how much less than an iPad it costs, it puts out a very good framerate indeed. I did notice a few jumps probably due to garbage collection pauses, so hopefully those can be ironed out eventually too. OS 2.1 is still a beta, so let's hope it gets pushed out to all the Playbook owners soon.

Windows Phone 7 comes in a sore last. With poor touch support and an unusable performance result, I cannot see any HTML5 developers seriously targeting this platform. The Playbook stomps all over it with multitouch support, far better performance, and a low one-off cost.

tl;dr: Use directCanvas or CocoonJS to get your games on mobile. The Playbook is surprisingly good, so consider it. Windows Phone 7 is useless. And if you can live with limited audio, consider an iOS web app as well.

Conclusion

Mobile browsers are still largely a big mess. Safari on iOS is good, except when PhoneGap'd. The Blackberry Playbook also has an excellent browser, but the realities of market share make it a small target compared to iOS. However, if your game just works on it, it's definitely worth considering putting in the time to publish to the App World.

For making Android and iOS native apps, directCanvas and CocoonJS are two much-needed technologies that help make HTML5 games really fast. Doubtless mobile browsers will continue to improve in the future. In the long term, I am confident both browser and hardware improvements will make HTML5 games in the browser practical for all platforms. Chrome for Android beta and the Playbook browser are glimmers of this distant hope, as well as the idea that, maybe one day, using PhoneGap won't make Safari totally useless. In the short term, we can work around with these other wrappers which really perform very nicely indeed.

I barely mentioned audio at all, because I could basically write another article this long on the subject. However it should suffice to say that directCanvas and CocoonJS both support polyphonic audio on all platforms. So HTML5 audio on mobile is not the problem it used to be either, if you use those wrappers.

The "write once run anywhere" slogan has been thrown around for decades, but consider this: the game is a single Construct 2 project, and it directly exports and runs successfully on 21 of the 23 platforms tested. And it's early days for HTML5.