The objects are loaded from the entire layout yes but only the ones visible in the current screen space are rendered thus manually swapping between visible and invisible isn't going to make a difference (and might even cause an actual additional loading header, but being a binary comparison every tick it's not a real issue).
There are special cases, though, for example disabling shadow casting Light objects disables the shadow rendering too. This became a crucial optimization to a project where we have hundreds of lights but only a couple visible at the time. All of the lights were technically casting shadows (even though they were out of range to even render!) which totally butchered the rendering speed.
Sprites in CC are quads just like in C2. Having thousands of large sprites definitely ups the loading time. To keep it within reasonable boundaries you might want to have several medium sized layouts you switch between. For example the project I mentioned has somewhere between 2000 and 3000 objects per 2560x2560 pixel space (which is the used layout size) and it takes around 20 seconds to change layouts.
Edit: Forgot to answer the actual question: in principle the more objects you have on LAYOUT causes longer loading times. The more objects you have on SCREEN causes lower FPS.