I found that I can do this at least in the Play Store "Device catalog" settings by first filtering the supported device list by "device type: phone" and the excluding those manually. It's a bit tedious since there are roughly 14 000 devices and I can only select 500 at a time.