The host will also need to be on the layout for syncing positions to work. The behavior you describe is due to out of sync issues.
You can either move all players to the second layout, alternatively isolate the players in a special area in the current layout, or build your own data syncing mechanics.