Charisma is nothing to do with looks, it's force of personality and magnetism.
Choose another stat to dump next time.
Decided to go to the rules on this one. Pfsrd defines Charisma:"Charisma measures a character's personality, personal magnetism, ability to lead, and appearance."
And it's not for myself, it's for a barbarian in a game I'm GM'ing. He wrote a backstory that had him captured and tortured and hideously scarred. That's how he became an invulnerable rager. He's also perpetually mistrustful and angry as result. I want to reward him for writing a great backstory that explains his extra tough skin and low charisma.
A person who's forceful and intimidating despite his looks is a high charisma character. The inclusion of appearance in your definition does not throw out or outweigh the other factors of charisma. If your barbarian was beaten down to a low charisma, he wasn't just scarred, he's psychologially broken. Ugly characters of that matrix, aren't intimidating, they're objects of pity, scorn, and mockery.