Yeah, the best way to do this is to build your graphics to suit the size of the display you're optimising for.
As soon as you start scaling your images, they will never look "perfect". Unfortunately, unless you are building your game so it doesn't scale at all, you will most likely never build something that looks 100% perfect on every single display size.
I suggest aim for a common resolution (for PC stuff I tend to go straight to 1080p) and roll with that.
You can also build a game that actually scales the canvas around the screen size, so there is no "scaling" of your graphics... but this can cause issues around the players viewable area around the character... larger displays will see more of the level and surroundings, and smaller displays may see less. It can be difficult to design your game around these dynamic factors.