"Universal gravitation" is the formula you'd want to look at to do the gravity force between two objects.
Force = G*mass1*mass2/distance^2
That's the magnitude of the force, the direction is toward each other.
Now an object will orbit another if it's velocity is perpendicular to the direction of the other object and it's speed is sqrt(massOther/distance).
Here's my test of it:
Now I haven't played solar but considering a good orbit just doesn't happen then I suspect that some additional force is used to steer the smaller objects to get in a good orbit.
i think you should play Solar 2 for few minutes. get a demo of the game
you can find it here
actually what you did is made a real like simulation, but in solar when a steroid is captured it just start orbiting around it in a uniform way, i mean it just start orbiting in an orbit close to the planet, and when you capture another steroid it start orbiting in another orbit, like this they always orbit around planet no matter how much speed you gain after capturing them or no matter how much mass your planet gain, they don't go away (of course its not like actual physics, but games have to have some elements unlike full reality),
even though, your help is very much appreciated, at least now i got a start point
thank you so much for that.