a good thing to check is how many objects you have in debug - too many monsters spawning? too many bullets flying off the screen and never getting destroyed? some kind of infinite loop that is creating more or something than you expected?
when you were getting the error, did it happen quickly, or did you have to play for a while? do you have a huge layout, or huge images?