In the monster creation section of Bestiary I it states that the DCs for most abilities are based on Constitution or Charisma.
What determines the ability score that the DC is based on?
Mostly designer fiat.
I believe that spells and other magical abilities tend to be Cha based. Physical attacks and abilities tend to be Con based. But those are more guidelines than rules.
Look up a given special ability (e.g. breath weapon) in the Bestiary appendices. It'll tell you what stat to use for calculating the DC. Individual monster entries may also give you the answer. Magical abilities that duplicate spells or nearly do so get the Charisma modifier, as do oddball magical effects. Constitution comes up for things more physical in nature, like poisons.
Yeah, there's not a lotta rules about it, it's mostly what makes sense. If you look at true dragons for example, their breath weapon DC is based on Con (it does physical damage), but their frightful presence is based on Cha (it's not damage and is based more on appearance).