On a technical note performance isn't dead because you using immovable objects. Immovable objects aren't in the physics simulator. They only exist in the collision checks. Also as information Bullet doesn't use brute force collision checking. It uses binary or quad tree collision detection. which means that the collision check can cut out half the objects in one pass. Then another half in a second pass. so in about 4 checks it's cut out all but just a couple of objects to check.
so on the technical note. Your physics should only be doing collision checks on binary space partition detection.
With all this your performance shouldn't be really too bad.