We don't have a way to render different layers at different resolutions right now. It could be a pretty tricky thing to do as well - currently it just renders the entire game to a small surface and then stretches it up. but if one layer is rendered at a different resolution, how is that supposed to alpha blend or process background effects with a different sized background surface? Other low-res layers on top presumably have to be rendered separately again and then stretched back over the top, all of which takes more GPU bandwidth, and then you'll probably quickly negate any performance benefits of low-quality fullscreen mode - or possibly end up even slower than high-quality fullscreen mode if there's lots of extra surface compositing that needs to be done to handle all of these cases. It's a difficult problem and I'm not clear what a good way to solve it would be. Really we just need a way to draw aliased text to a canvas! But that means requesting a new browser feature. Maybe you could try experimenting with the alpha clamp effect to cut off the low-alpha borders of the text.
The GPU memory measurement includes the surface it is rendering to, which gets bigger when you maximize the window or use high-quality fullscreen mode. So if it increases a bit when changing those things, that's all it is.