Right now, all races except Humans have a fixed attribute for their racial malus in addition to two fixed attribute bonuses and a free boost.
This is way more flexible than PF1, where most races were very tightly shoehorned into specific classes and vice versa, but it still ends up with odd situations like dwarves not making terribly great paladins due to their penalty in CHA. Even if you spend all possible boosts on CHA, the highest you'll get it at level 1 is 16 and the highest you'll be able to get your ever-important STR is to 14. Or you can swap those around for a 16 in STR and a 14 in CHA. Not great.
I think a reasonable alternative is to let at least some races choose between one of two stats to take a penalty in.
- Dwarf
+CON +WIS +FREE
-CHA or -DEX - Elf
+DEX +INT +FREE
-CON or -STR - Gnome
+CON +CHA +FREE
-STR or -WIS - Goblin
+DEX +CHA +FREE
-WIS or -INT - Halfling
+DEX +WIS +FREE
-STR or -CON - Human
+FREE +FREE
I admit I couldn't always think of two appropriate attributes that could be weaknesses and I imagine some races are probably fine enough being unable to pick their penalty. But the intent is to make it so that there are more viable race/class combinations. Free boosts don't do enough by themselves to counteract penalties and it can lead to weird stuff like a shortage of dwarven paladins.
Alternatively, the option to spend both of your boosts from your background on your racial malus would allow certain races to get an 18 to play against type so long they have a good excuse in their background for being such a charismatic dwarf. They're a diplomat, they had to be good at their job!
I don't have a whole lot to say about balancing humans against all this except that they should not be getting feats or heritages that can be converted into higher tier feats like general or class feats. I feel like if they lose that they might be a little subpar against some races, but I think that can be shored up with something intrinsic that humans get rather than must-pick options. I feel like giving them another free boost and then also letting them pick a penalty is a bit much.
Hmm, I might actually be onboard with the two racial penalty options idea. I'd like to see it tweaked though s that every race has a physical malus option and a mental one to mirror how every race has a physical and mental inherent boost, though I'm not sure if that's logically doable. For instance I can't think offhand of a mental stat Elves would racially fail at. Though I think that's my only hangup.
Yeah, I thought hard about that too. My conclusion was that you already can't pick between either a physical or mental malus and you can only ever have one malus at a time anyways, so it's not terribly important to keep them balanced out. Elves just unavoidably having a penalty to a physical stat I think is OK at the moment, even though it potentially makes them perfect for a Rogue (malus in STR, bonus to DEX, +5 or +10 movement speed and Darkvision based on heritage and feat choice) as it really seems to just put them on par with Halflings, without the option to use teammates as cover to hide and without that free boost to WIS for initiative.
While I'm not strictly opposed to the idea I think giving the ancestries more benefits rather than minimizing their disadvantage would be the way to go for me.
I think there does need to be more in heritages to maybe remix what a race does to address that stat spread issue you brought up, like it is a bit silly that the only non-humans who are really perfectly suited to be paladins are goblins. I think humans could maybe get something other than just the two free boosts to justify themselves to compensate, but I don't know what.
But my focus on changing the racial maluses isn't to make any one race particularly stronger at their most optimal but to make it so more races are a valid option for more classes, since a malus in a primary stat is just a non-starter for most classes. There's (currently) no way to just completely undo a malus no matter how much you invest. For most already strong combinations I don't think changing the malus will have much impact on their power and that's what I'm hoping would stay the case.
It also doesn't help that I pretty much hate all of the heritages and ancestry feat options, I think Paizo did a dreadful job balancing them and for every race there's usually one or maybe two options that are clearly better than everything else. Like humans are in the most embarrassing state where they might as well just delete everything except Natural Ambition but I think that distracts from stuff like elf heritages where a super small and super situational bonus for swinging around in the jungle is up against Darkvision that are also kind of an issue.