I think Families can only hold one type of object, so sorting via the System plugin is limited to that. Are you saying that, for example, a Tilemap object can be on top of a Sprite?
If so, the only thing I can think of is to use Rex's ZSorter, which enables you to sort for the whole layer. However, I myself haven't tried this with multiple object types but I think it might work. You also must use the Sort by Function feature, as you are using instance variables as a basis for sorting. However, in my experience, ZSorter is slower than C2's sorter.
You may want to compromise mixing object types and keep it simply to Sprites when it comes to sorting. Let Tilemaps sit on the bottom or at the top, and not in between.
On the other hand, if you had a simple situation where you had a TilemapSortFamily and a SpriteSortFamily, then perhaps you can do a System sort on the SpriteSortFamily first, and then loop through the TilemapSortFamily which checks each SpriteSortFamily's z-index. Then if it finds the proper z-index to insert itself you can 'Move to object > SpriteSortFamily > In front/behind'.