Yes, sadly dealing with 4 gamepads is always tricky (on any engine).
I have no problem here on Penelope, but:
1 - users are forced to press a button before being able to play the game (to make sure gamepads are correctly detected)
2 - game rounds are super short, then back to the 4 players select screen > users are forced to press a button again to play > super short rounds, etc...
3 - the game can be played with 4 players on the same keyboard, in case of gamepads problems.
This way, I made sure to not deal with gamepad IDs that much. I avoided all the "player 3 will keep his particular color for many rounds" stuff etc...
Call it lazyness, but 4 gamepads are always a pain in th. do deal with, I picked the easy way on this matter : )