Preloading music doesn't work in mobile?

Hey everyone,

I've been prepping Blitz Breaker for mobile release, and have been running into some major issues with audio.

One of the game's biggest selling points is how awesome the music is, so ensuring it plays smoothly is pretty pinnacle. The issues I'm running into kind of muck up that smoothness.

• First off, does preloading music not work on mobile? I've been using it on nw.js and it works just fine, but when I load the game up onto iOS or test it through mobile safari, I can't help but notice delays in the music loading. Also if I use the combination of 'Preload audio' action and the 'All preloads complete' condition, it just hangs on mobile, while working correctly through nw.js

• When playing music via the steaming method, there is a brutal noticeable delay when it loops. Is there anyway around this?

• I can work around the whole 'needs a touch event to start playing' but the delays are agonizing, especially when it loops.

• If all else fails, are there methods of reducing the footprint putting the music in the sound folder would take? I've tried reducing the quality but it didn't make any difference, and still crashes upon preloading it all through sounds.

If I am missing anything or if anyone has any tips please let me know. Thanks!

I thought preloading was only for html5 games, where it needs to download the music, sound, etc.

AnD4D It can be used for nw.js, I know that for sure. It's what I've been doing for my game's music and it works flawlessly on PC. You can use it for sounds too if you have 'preload sounds' set to off in the editor. The issue I'm running into though is that if I do that on mobile, it uses up too much RAM and causes the app to crash.

Preload does not work for iOS for awhile now, ever since iOS9, Apple broke that feature.

I don't know if Ashley is aware of it or not, but there's been lots of audio related bugs with iOS9+.

Sometimes my audio tracks don't play at all, it's messed up actually!

— I just uploaded a sort of sound test app using cocoon.io's canvas plus, using all of blitz breaker's music files. It looks like it uses .ogg files instead of .m4a.

Anyways the files preloaded up perfectly and played without requiring a touch event and without a gap. Going to explore this and see how the game's performance is in general.

Canvas+ is their own engine so it does not have issues like Safari based ones that WebView has.

But WebView+ is heaps faster and more power efficient than Canvas+.

Yes I couldn't help but noticing this after uploading the full game.

Back to the experimenting page, gonna play around and see what results I get with wkwebview, as that's where the performance is at.

May post a bug report here too, to help get ashley's attention.

Mobile browsers have never supported preload for audio/video, ostensibly to save bandwidth on cellular data connections (but they still do this on wifi etc anyway). This exacerbates the fact you can only start playback in a touch, since it will have to start loading the track from scratch in the touch. Most browsers don't support seamless audio playback either. The workaround to all of this is to treat it as sound, but then playback won't begin until it's fully decoded and decompressed the track in memory.

These are the only options available in mobile browsers I'm afraid. I've argued a lot against the browser restrictions for music (played with the <audio> tag) pointing out you can work around every restriction, albeit inefficiently, with the Web Audio API, but they've never changed it...

Mobile browsers have never supported preload for audio/video, ostensibly to save bandwidth on cellular data connections (but they still do this on wifi etc anyway). This exacerbates the fact you can only start playback in a touch, since it will have to start loading the track from scratch in the touch. Most browsers don't support seamless audio playback either. The workaround to all of this is to treat it as sound, but then playback won't begin until it's fully decoded and decompressed the track in memory.

These are the only options available in mobile browsers I'm afraid. I've argued a lot against the browser restrictions for music (played with the <audio> tag) pointing out you can work around every restriction, albeit inefficiently, with the Web Audio API, but they've never changed it...

are you sure ?

cause i did done the preload on mobile (ejecta/cordova cli) on both android and ios by executing the javascript call (browser plugin) using these scripts (for example music.m4a)

to preload

var music1 = new Audio();

music1.preload = 'auto';

music1.src = 'media/music.m4a';

to play

music1.play();

Ashley Yes I'm aware that is the current work around. Unfortunately for me that won't work since the amount of music Blitz has causes the game to crash when it loads roughly half of the tracks, so it's not a viable work around. This will happen regardless whether or not I load them at the start of the game or try to stagger it with loading them throughout the game, since there is no method of unloading the tracks that aren't being used.

I do find it interesting that cocooon.io's canvasplus is able to preload music and play it reliably. I gather that it doesn't rely on apple's webview so it's able to get around some of it's limitations. The unfortunate part is that canvasplus' performance doesn't match wkwebview's, which has been stellar performance from a mobile stand point. I realize I could fix this with some heavy optimizing, but it's super frustrating that one option has the performance, while the other has correctly working music.

matrixreal This is interesting, is there a method to detect if a music file has been successfully preloaded?

matrixreal - that code is pretty much what the Audio plugin is already doing.

Ashley so the only method using webview and to ensure smooth music playback is to load it all into sound and go from there? The only issue with that though is it causes the app to crash after roughly 1/3 of the music is loaded. I gather that it decompresses them as .wav files, as the actual .m4a files only take a total of 20mb after export.

Am I missing any steps to reduce the music footprint, or is there a method of offloading what has been preloaded to prevent crashes? Thank you!

Yeah, music is fully decompressed and it uses a lot of memory. We are also forced to do this in WKWebView (just blogged about it here). I guess we need to add a memory management solution for music playback with the Web Audio API. Maybe we could even make it an option on the Audio object so you can choose to opt in to treating music as sound so you get the preloading and seamless-looping benefits? It could easily use 50mb memory for a single track though.

I think the engine already restricts there to be only one music track playing at a time, so maybe if you only pay the memory overhead for the current track, it might be OK...

Ashley I've tried something similar. This was all done when they were in the sound folder, with preload sounds set to off.

At first I tried loading all the music tracks on boot up. That failed, caused a crash.

Then I tried splitting up the preloading between the first 3 layouts. That failed, caused a crash.

Then I tried only preloading music when it was needed. That worked...for about the first 1/4 tracks. After that you could tell it was starting to bog everything down by having it all loaded into the memory, and eventually yes, caused a crash.

It's worth mentioning that Blitz uses roughly 20 music tracks. 14 of them are rather long, and definitely the ones causing issues. That being said if I could load/unload them 1 at a time I could make it work as these are the tracks that load in game, as opposed to during the menus.

I've tried many many solutions, and will continue to experiment. I don't want to make your life harder, but as it stands it's hindering me releasing Blitz on iOS (probably android too, but one thing at a time). As it stands I may just attempt to optimize the living hell out of it to get it running smoothly on cocoon.io, as the canvas+ webview fully supports preloading the music via the music folder (and uses .ogg as well, for what it's worth), but man it's a tiny bit frustrating to be so close and yet so far. I hope you understand!

Hey Ashley, not to badger but I'm just wondering if you've given any thought into memory management solutions? I'm currently talking to a few publishers for mobile and would like to figure out if I should wait to get wkwebview working properly or if I should go all in on cocoon.io (which has worse performance but better audio atm).

So I want to see if you're considering adding it in a future update, as I would love to see that

Thanks!