Thanks Yann! It's still a little confusing, but your explanation really helps! That's a very good point about black being the colour of nothing in the editor. But if that is indeed the case, then why is the editor's colour of nothing typically light blue? If you make all layers invisible or make them (semi)transparent, you see that the background colour of the editor is light blue, not black. Do you think that maybe the Destination In effect is burning through the editor's background colour as well? If so, I'd consider that a small bug in the editor view.
Another potential editor view bug is that I found you could set the light blend mode to XOR and get the same effect as Destination Out, but the editor view would show it opaque (as if blend mode were Normal). There seem to be some other weird things with XOR, such as it still having an effect on the overall image even if you make the layer Transparent. Either I am misunderstanding something (quite likely), or it's behaving incorrectly.
I'll try to paraphrase and expand on your overall explanation:
My understanding is that rendering is done starting with the lowest layer, and each layer renders objects using z-order from lowest to highest. Force Own Texture will render a layer on its own, independent of layers above and below it, and the visual effect of this is that blending modes on objects within that layer will only be applied to objects and the background of that layer, and not to anything rendered on lower layers. Once it's rendered to a texture, the layer will be rendered over the output of the layers below using the layer blend mode. This all seems to make sense and agree with my observations, EXCEPT:
I found another way to do this lighting without Force Own Texture! This time, I have the lighting sprites blend mode Destination Out (same as demo), and the Lighting layer is Black, but instead of enabling Force Own Texture, I set Opacity to 99.999%. It seems that layer opacity has an effect as to whether a sprite's blend mode stops at the current layer or applies to all layers below it. If the layer is opaque, sprite blend modes apply to everything below it, but if the layer is semitransparent, sprite blend modes only apply to the current layer. Since Opacity is at 99.999%, it's enough to trigger this different behaviour, but 99.999% is visually opaque (it might indeed be opaque, although I'd have to screenshot and test pixel values).
I wonder, is this method notably more efficient? From what I understand, it should be considerably more efficient. I also wonder if XOR is more or less efficient than Destination Out, because those two blend modes seem to be visually equivalent in this situation.
I've updated the capx and html with a third layout. The third has settings:
Lighting layer: background black and 99.999% opaque, blend Normal.
Lighting sprites: XOR for mouse, Destination out for 8Direction.
Aside (not really important):
One thing that I could see value in is having a Force Own Texture mode that forces this layer and all above layers to a texture. I think this would allow for implementation of certain effects more easily (or maybe just conceptually more straightforward), but make other effects impossible. I think it's probably fine as it is, and you can probably get whatever effect you want with enough layers and the right settings.
Thanks again for your help!