Alright, this happened again today and I was able to isolate what causes it.
Turns out that if you have instances through your project that are on different layers in separate layouts, changing their layers to be the same actually moves what layout they're on.
This is the reason why I was able to break some of my levels without ever actually opening them up (thought I was going crazy).
Anyway, I recreated it in a separate project and uploaded it to the bugs github, so maybe it'll get fixed. At least I know not to change what layer an object resides on from the project bar.