In my experience, performance tends to get worse when I have more layers.
You may want to post a bug report if this is the case, as Ashley specifically states in the manual HERE that:
"Number of layers usually has no effect, unless most of the layers have changed their opacity, have effects applied, have a blend mode other than 'Normal', or use 'force own texture'. However typically there is no performance impact from using multiple layers with default settings."
I've got a test going on at the moment and have 10 layers just for parallax effects. Seems to run fine for me.