Bah! I have a 22 STR on my Halfling. You can easily get around penalties. An extra 1, 3, or 5 damage is meh at higher levels. I'm still rolling 9d6+85 on a crit charge. I don't even have the cavalier bonus stuff.
I rode my Giant Wasp around for a level before a GM asked me how the hell I was doing all the crazy things I do... I do Flying cavalier style things, and become the party taxi.