I'm currently working on a game which has a very large map, just like you described - millions of tiles, many thousands of objects. Of course, you can't keep all these objects on the layout at all times, even if you disable behaviors and animations, the performance will be terrible.
What I came up with was storing the objects in JSON object and recreating them in a small area around the player. And destroy objects which are outside of this area.
One giant tilemap may be ok for performance, but you can also break it into smaller pieces. For example, make each town on a separate tilemap. Or create a grid of tilemaps. Re-create them around the player as needed.
What I described is definitely not an easy task, took me weeks to do everything right..