We have already slowed building down by using the rules from Ultimate Rulership where it takes multiple months to build a building, which I think is a good thing, as it should take more than 1 month to build a castle.
He ruled that settlement bonuses from buildings can stack in the same city, like base value, Law, Productivity, magic item slots and the such. So you could build 6 stables to give you a base value of 3,000 gp in a city, but can only get the Loyalty and Economy bonus once.
I went through the UC and totaled the values from each building once and you get a total of 51 to your Loyalty, yet there are 77 hexes in the first map. So your Control DC would be a 97 without counting the districts required to build just the Loyalty bonus buildings. That means you have to get 50+ points from Leadership and landmarks.
I can see no possible way to sustain a kingdom with the first map fully settled, let alone adding hexes from any other map.
You wrote: Personally, if I were to set a limit like this, I'd do it per settlement, and probably say that the modifiers are halved for each building of the same type after the first, except unrest reductions which are always in full.
I will try to use that logic with him as I can see that as more realistic than limiting only 1 bonus per building to the entire kingdom.