This seems to work well:
Set scale to 0.5*OriginalWindowWidth/min(min(camera.X, layoutwidth-camera.X), min(camera.y, layoutheight-camera.y)*OriginalWindowWidth/OriginalWindowHeight)
are the closest x and y distances to the edge of the layout
using the aspect ratio I can then convert the y distance to a x distance:
So now there are two x distances, and we can get the smaller one with another min.
and to convert the x distance to a scale we multiply by 2 and divide by the window width:
As an additional idea you can also limit the zoom if it gets much too close near edges.
or whatever max scale you want.