I do feel your pain in trying to balance a physics system, I have been there and it can be frustrating to get right. If your a making a billiards type game like in your capx, some advice would be:
- Start without collisions. Hit a single ball around until the velocity feels right for the scale, by tweaking density, friction and impulse force.
- Then test collisions against other balls. Tweak the elasticity and friction of the balls until that feels right.
- Mass doesn't matter in collisions, it's the relative mass, i.e. the difference in mass between two objects colliding, and mass is calculated by the object size and density, so if the balls are the same this shouldn't really matter.
- Then bring in the table/walls (immovable objects with infinite density) and tweak their elasticity and friction.
- If you're still having problems, Construct uses Box2D which is used in a lot of engines, try their documentation and ask around in other forums and you might get some advice on parameters that work well for you.
In regards to other engines, I find the bullet and solid behavior in Construct better behaved than in other engines. For example, their bullet behavior uses stepping (similar to the physics engine itself) whereas as others will go straight through walls at high velocities.
But they all fall over in the same circumstances, which you have here, a high density of moving solid objects. In your capx you can see most of the problems occur in the corners or when there is a cluster of balls. The engine is calculating the right collision position, normals etc. but when the object wants to move into a position where another object wants to move too - it becomes a very complex problem to solve.
One engine returns collision points and normal (outside the physics engine) which is useful if you are coding collisions yourself - but that comes with being limited to single simple collision shape for an entire sprite - i.e. no complex, animated collision polygons like in Construct.