Mounts that do not possess combat training (see the Handle Animal skill) are frightened by combat.
The Handle Animal skill entry isn't relevant to my question.
At what point is a mount "frightened by combat?" Is it the second someone makes an attack? Does the GM need to roll perception checks if it's far enough away?
If one group is on mounts and are stopped by a group of bandits asking for a toll, then the party attacks, then that's combat, that's obvious.
But what if there is a fight off in the distance and the party speed up to get there quickly, maybe allies being on one side of that battle (so they're already intending to rush in and help)? At what point are they going to have to start worrying about the mount freaking out?
I think that's left vague because of the number of factors one has to consider, just as you're suggesting. How close is the horse, what kind of fighting is it?
Look up some videos of horses getting spooked maybe? Get some idea of what gets them worked up.
Ol' 3.5 (and Pathfinder as a result) is fond of a few things: +/-2 circumstances, sets of 3, and the 30' radius. Perhaps a simple way of handling it could be that any horse within 30 feet of a hostile creature is "in combat."
Moving 30' a round is pretty fast, and I'd say that the quick movement alone would spook an untrained horse.