Spritesheets (aka texture atlases) are a common technique in computer graphics to combine multiple images on to a single image. However if an image on a spritesheet is resized smaller, things get surprisingly difficult, due to the way it interacts with mipmaps. This manifests as "seams" or fringes of unwanted content along the edges of an image, such as shown below.
I've even spotted it happening on buttons in Google Docs!
Here's what's going on and why it's a tricky problem to deal with.
Spritesheets
Spritesheets just mean pasting multiple images on to a single large image. For example a simple game might use a spritesheet like the one shown below, with different object's graphics all placed on to one "sheet".
Each image can still be rendered separately, since GPUs can efficiently render a smaller area of a source image. Using spritesheets helps improve efficiency since there are fewer images to download and load on to the GPU. It can also save memory and improve rendering performance.
Mipmaps
Mipmaps are a way to improve the performance and quality of resizing images. Resizing an image smaller with good quality is computationally intensive, since you have to consider all the data in the full resolution source image. To make this more efficient, a series of high-quality resizes are calculated ahead of time. The image size is halved each time, at 1/2 scale, 1/4 scale, 1/8 scale, etc. all the way down until the image is a single pixel.
This uses a bit more memory (a third more, to be exact), but improves performance and quality. For example if the image needs to be rendered at 1/8th scale, the GPU can pick a pre-rendered high-quality resized version of the image from the mipmap, and render that. This improves quality and performance since it doesn't have to do a computationally intensive resize on-the-spot.
There's more to mipmaps than that, but that's the basic principle behind them. It's also worth pointing out mipmaps are often used automatically - for example even browsers use them when resizing images on web pages!
Spritesheets and mipmaps
Here's where things start to get trickier. You may think that spritesheets and mipmaps work independently. Actually the problem is when you resize a spritesheet really small, multiple separate images end up combined together. Take a look at our example spritesheet from earlier, and consider rendering just the spaceship sprite in the bottom left. The GPU will cut out the area for just this image from the spritesheet, shown by the blue dashed rectangle below.
To demonstrate the problem, imagine this image is being rendered at a really small size - so small, it picks the mipmap that is just 2x2 pixels big. This will have combined the entire spritesheet in to just four pixels! This mipmap, scaled back up to the original size, looks a bit like this.
Notice that the entire bottom-left quarter of the spritesheet has been combined in to a single color! If we mark out this area on the original spritesheet, we can see it includes other unrelated image content other than the spaceship we want.
All the colors in that area have effectively been mixed together in a pot an then painted over the whole quartile of the image. Now the GPU will render the space ship image area from this mipmap, as shown below.
Now you can see the problem: the color the GPU will draw from the mipmap is the average of the spritesheet plus other images - so will be slightly the wrong color. In this case it'll be too grey, as some of the grey gun sprite above the spaceship has been mixed in to this pixel of the mipmap. This is sometimes called color bleed, as some content from outside the spaceship image area has "bled" in to it.
This is an extreme example to show the principle of color bleed on mipmaps. It can actually happen at any mipmap level, such as at half size - but it's less likely. It also tends to manifest at lower levels as a "seam", or fringe, along the edge of one side of the image. To see why that's the case, take a look at the spritesheet with a 16x16 grid drawn over it.
On the 16x16 mipmap, all image content within each grid cell will be combined in to a single color. Where the image edges don't fit this grid exactly, they will be combined with an adjacent image. As another example, look back at this image I showed earlier:
This is an artifical example I made to demonstrate the problem. The source sprite sheet actually looks like this:
Now you can see where the blue fringe comes from - a big blue sprite directly below it on the spritesheet. I deliberately made the source images large so that displaying them at a small size involved resizing them down a lot, therefore using small mipmaps. The images look like they align to each quarter, but they're not actually perfectly aligned; consequently some mipmap levels have the blue sprite bleeding in to the purple circle's image area. This also happens inconsistently across the mipmap levels, manifesting as a flickering line as the circle sprite resizes smaller.
Looking back at the Google Docs example, you can see icons are rendered from a high-resolution spritesheet of lots of different icons, and an adjacent icon bled through.
Causes of the problem
In practice, this is actually relatively rare. As noted you have to be unlucky enough to have all of the following happen:
- Resize an image considerably smaller than its source size
- Have the image positioned on a spritesheet unaligned to the mipmap pixels
- Have another sprite adjacent that has an obvious and opaque color right up to the edge of the image
- Lack enough spacing or transparent borders on images to overcome the resizing
Our game development software Construct automatically arranges images on to spritesheets. By default all images have 1px of transparent border applied, and then on top of that are also spaced out 1px apart on the spritesheets. This means there is usually at least 1-2px of transparency between images, helping prevent color bleed with modest resizing. Experience shows this is sufficient for most games. However sometimes if sprites are resized a lot smaller the fringing effect still happens.
Avoiding the problem
One way to avoid the problem is to use a source image that is smaller. If it's not downscaled, it won't use a mipmap and so color bleed won't happen. However this is not always possible, since the object may be shown at a range of different sizes.
Another way would be to throw out the whole spritesheeting approach completely and go back to using separate images. However this also throws out the efficiency gains. On top of that, since the problem is relatively rare, it's not always necessary. We want to keep things efficient if possible.
The best solution I've found is to align and space out images by power-of-two sizes. For example if an image is 100x100px, it should have a power-of-two size allocated on the spritesheet (128x128px). Then that space itself should be allocated at a power-of-two position, e.g. with its top-left corner at (0, 0), (0, 256), (128, 1024), etc. Since mipmaps reduce the size by powers of two (1/2, 1/4, 1/8...) this ensures all images are aligned with mipmap pixels as far as possible. Technically if you resize an image smaller than a single pixel it will still color bleed as the image area is smaller than a mipmap pixel - but this is hardly noticeable since the image is so small. In practice it solves the problem of fringes appearing along the edges of images.
However this itself still comes with a big downside: since images have to be padded out so much to fit to power-of-two sizes, the spritesheets can easily end up bigger, and that takes up more memory. For example the spritesheet with the spaceship I showed earlier grows from 256x256 to 512x512 when padded out this way - meaning that spritesheet takes up 4x as much memory once loaded!
Managing the problem in Construct
It's difficult to solve this problem automatically: there is a fundamental trade-off between tightly packed spritesheets, which save memory and generally work fine, and padding spritesheets, which uses more memory but in practice completely prevents fringing.
This problem can happen anywhere images on spritesheets are resized smaller - including in web design with major web apps, as noted with Google Docs! It happens in our game development software Construct as well, and some users have the problem of color fringing along sprites. Since it's difficult to see how it could be handled automatically, we added a user-controlled option for it. We named this Downscaling quality. The default medium tightly packs spritesheets, and generally works. However if there are issues with color fringing, the high mode will pad out spritesheets and avoid the problem - at the cost of increased memory use. To discourage unnecessary use of this option and needlessly wasting memory, "high" mode is marked as "not recommended". The hope is nobody will select it just for the sake of it, but anyone having trouble with fringing can opt in to it to fix rendering issues and accept the cost of increased memory use. It's a tricky tradeoff to communicate in a simple dropdown list - hopefully the negative language will at least prompt people to look it up in the manual, which explains in more detail what it's for, and help users make good decisions about using it.
Conclusion
This is a non-obvious rendering artefact that's probably not widely understood, especially in web design. It's interesting to explore the way computer graphics works and explain how it comes about.
However I guess the wider lesson is that features rarely work in isolation. It would be easy to think spritesheets are an optimisation that doesn't really affect anything else, and also that mipmaps are an optimisation that doesn't really affect anything else. This is naive and in fact the combination of these two optimisations causes some awkward rendering problems with difficult trade-offs - even though both features are working correctly and exactly as designed! It's not even a bug, it's an unintended side-effect of two separate optimisations. Even providing an option to users for this is tricky, since it's hard to explain in a brief sentence what the option is for and why you both might want to use it and not want to use it.
Even more generally, a rule I often think about is "everything is always more complicated than it seems". It's easy to look at each feature and optimisation in isolation, but when you think about the whole system, awkward and non-obvious results can emerge. The same point came up in another blog post I wrote about two years ago: Software development is hard: a collision bug post-mortem - that also involved the intersection of various features and optimisations working in an unexpected way, resulting in an unexpected bug.
And as for users - don't change settings you don't understand, or you might end up with negative consequences! Make sure you understand what settings do and why you are changing them - and if in doubt, read the manual 😉