You’re probably referring to the scissor cut feature that webgl has that was one of the things that c3’s renderer batched. It was never an sdk feature, but a way to use it was found before sdk2 (which walled a lot of stuff away). Anyways, all it did was limit rendering to a bounding box on the screen. I don’t think that really helps solve the masking issue unless all you want to do is mask by a rectangle.
The issue you have is the mask isn’t covering all the objects underneath. Using layers you can effectively first draw the mask to a full screen texture and use the blend mode on that.
You can do the masking by having a layer with two sub layers. You’d put all the objects you want to mask in the first sub-layer, and the mask object in the second sub-layer. Just have normal blend modes on all the objects, but use the “destination in” blend on the second sub-layer itself.
You could likely do it by drawing objects to a drawing canvas as well. But you’d still need two of them since you need to make the mask cover everything you want to mask.
I don’t think it’s really something that’s feasible to do from the sdk from what I can tell.