There is no right or wrong, it all depends on what you want and/or feel comfortable with..
You already say you want to open a window, which is a choice, because you could also just animate the menu to show the options.
Creating a "pop-up" could be done by revealing a layer, but also by creating all the new elements in events and/or changing the function and look of elements already on the screen.
Layouts are separate, so one layout can not be shown within another layout, global layers are available though..