I'm less concerned with a way to mask the fluids inside the container than I am to figure out how high the fluid should be to have a certain volume. It ends up simpler to calculate the area of a polygon than it is to calculate the amount of fluid pixels in an image.
If we are just talking masking, then what I ended up doing is using blend modes to mask out an area and draw within it. By controlling the draw order, you can do this multiple times on the same layer as long as you fill in the mask completely before moving to the next object. Destination-out to define the mask shape, and destination-atop to draw in the mask.
Masking with a shader or a clipped polygon have their own sets of quirks and shortcomings.
Anyways, if you want to check out how I made those images here is a modified project you could pick apart to get ideas.
dropbox.com/scl/fi/t1bjw1ybu5gaptqwnwj4s/testtube_fill2.c3p
The algorithm takes about 52 iterations to converge on a near perfect result, but it's good enough with a lot less. I could see it converging faster by interpolating the areas between y's but that would complicate it a bit. The loops could be eliminated altogether by making an approximate area vs y graph formula for multiple angles and interpolating between them.