In that case, I guess you could compare the actual time when the building start to be built and set the finishing time to that starting time + building time.
That way doesn't matter if the game pauses or not.
But keep in mind that players could cheat changing their own pc clock.
If you could compare to a web clock somewhere, this would be avoided.
Anyway, it's just an idea.