There are two ways.
One is to do 5bit like you have above
If a tile is blue start with 256
And if the tile is red start with 512
Then add the surrounding tiles.
Blue tile values
0 1 0
2 0 8
0 4 0
Red tile values
00 16 00
32 00 128
00 64 00
If you looked at the tiles around an empty spot too there would be a total of 1024 combinations. Which is a lot to draw.
To knock down the combination count you can put some restrictions.
The simplest would be to have each tile type on its own layer, and do bitwise on it's own layer.
More complex setups could be done depending on how you want it to look.