Countmatching expression picks all selected instances. I don't think the ships were filtered at all, so it picked all. You can guess the outcome.
Try this... create a (global) variable named 'allships', that tracks the total number of ships (though that may not be necessary, if you use Ships.Count, but I'm not sure if that is influenced by selection).
+for each formation sorted by formation.UID
+inverted | ship('idformation)=formation.UID //this should select all ships that are NOT in the current formation)
-set "ships" to 'allships'-countmatching(ship) // all ships minus ships not in current formation = number of ships in the formation
Empty formations should have 0. A bit roundabout, but this should work. Perhaps someone else has a more elegant solution. Also, try using Ship.Count instead of 'allships', if it is not influenced by selection.