I tried designing a controller abstraction on top of SDL's once.
This bugged me too. SDL joysticks have axis, buttons, balls (yes, balls) and POVs.
Usually the first two axis and buttons are predictable, the rest isn't.
POVs can be considered as buttons, as I did. They're not that different and also makes for stronger controller logic. I merged balls and axis as a float, also made buttons a float that was either 0 or 1.
Thus, unified data from any input (keyboard was in there too).
My solution to the issue of what-is-where was to load an .ini file based on the joystick's reported name (by the driver) and use that to load proper string and image descriptions for each control.
This combined with configurable controls (for which there's already a plugin) makes for a sane and comfy system for the user, as you could show a cute icon with the button to press that matched the joystick you were playing with, like consoles do.
Also, making 'packs' for certain joystick models becomes really easy and one could always have a default set to use when the proper files are not installed.
Not even DirectX has it that good.