Generally speaking, you don't have to worry about performance/speed when talking about image size. What you do need to worry about though, is available memory. See for more information.
Basically it depends on the memory limitations of the platform you are developing for, and how many assets you have in your game. It is a trade off between fewer high resolution images or many lower resolution images.
Ideally if you are not nearing your limits, you want to design for the highest resolution considered, then downscale.
On the other hand, I haven't had many issues with blurry edges and upscaling. There are some things to try such as pixel rounding and sampling type in your project properties.