This is not a problem if castle and dungeons are properly defended against teleportation (which they would be in any world with teleportation).
As for in game, if you have multiple clerics, you can always offer them the travel domain which grants them access to teleport. Otherwise, you're kind of stuck from the viewpoint of rapid travel. You can buy items to make up for it. But really a cleric with the travel domain would be the ideal.
Yes, this is not a problem in my game. In fact, most lairs will have counter measures for even high level spells. Most of the rulers are CR 20+, and plan accordingly. I've got a scene coming up where the enemy is obviously begging for a scry and teleport sequence, of course it's a trap. Now, it may play out differently.
The travel domain is a good idea, but we're not going to re-write PC's that have been in play for over a year (it might be longer).