I have spent the last day trying to figure it out and reached a hypothesis that Multiplayer.PeerID is broken, I manually replaced each one for a custom version of it.
For example, by sending messages, it sends a separated token of "[Peer ID]|[Other info]" with the Peer ID being picked from MyID (who sent the message) and it worked!
And using Multiplayer.PeerID normally only works with two players because it's easier to not mistake the IDs, as the "on peer connected" trigger is only called once.
To replicate "on peer connected" I made the host a list of IDs and iterate over all players in the room to check if a new ID appeared:
It worked perfectly, but even without any bugs until now, I still think it's unstable for some reason, if anyone have a better idea of how to do it, feel free to say.