Now i understand the problem. I apologize for not getting it earlier.
The reason for this is that the force is pushes from the center of mass and the rotation takes the origin.
But lets first go the path that you started. starting from the collision polygon.
Lets explore the center of mass, see if it drifts.
https://www.dropbox.com/s/6s7ay6vwoxtqb ... .capx?dl=0
It does not. It is pretty stable. But it is a little of do inevitable floating point calculations.
Is that enough to cause a torque when applying a force ?
https://www.dropbox.com/s/s2kl9qpgilaoo ... .capx?dl=0
Darn it, yes it does. So in some way, your guts are right.
Same problem when we use an exact imagepoint ?
https://www.dropbox.com/s/92uz1kof9rsp7 ... .capx?dl=0
Nope, not at all. So it kinda a butterfly effect, a little mathematical error that cause big changes.
I have no idea if it possible to do this better. In real life it is not possible, just less noticeable do the proportions of mass, forces, gravity ..... and so on.
Question remains, is this the cause of the problem we face.
https://www.dropbox.com/s/tb85vpfaawclh ... .capx?dl=0
No, no, i still notice drifting. An that is because the rotation gripes on the origin and the force gripes on an imagepoint that is not the origin. Those forces do not balance out.
But, we can not rotate an object with the center of mass as origin. Well, in a way we can, just set a torque towards the mouse coordinates on that layer. But then i dont know how to avoid the wobbling, the torque will always overshoot a little.
So, we have no other chance (but i am not the smartest in here, maybe someone knows how to stop the wobble) then taking the two system apart. The 'system that rotates in tick steps' and the 'physics system'.
https://www.dropbox.com/s/8tmehuf4r61tc ... .capx?dl=0
Hope this explanation satisfies you.