Similar to "creating new races" of first edition: https://www.d20pfsrd.com/gamemastering/other-rules/creating-new-races/
I think that new ancestries can be built using similar approach. I'll use "Ancestry Points" (AP) to define costs of different features.
Costs:
* Base package (8 hp, 25 speed, Medium/Small size, a physical ability boost (STR/DEX/CON), a mental ability boost (INT/WIS/CHA), a free ability boost, an ability flaw, common language, their race's language): 0AP
* +5 speed: 2AP
* -5 speed: -2AP
* -2 HP: -1AP
* +2 HP: 1AP
* Darkvision: 4AP, or 2AP if applied as heritage on ancestry that already has low-light vision.
* Keen eyes: 4AP
* Bonus language: 1AP
* low-light vision: 2AP
* free boosts (without mental/physical restrictions, but just 2 boosts, and without flaw) and free language selection (instead of racial lang): 1…2AP (and 3…4AP cost for human heritages)
Some reasons to treat heritages as having "same cost":
* forge dwarf, arctic elf and snow goblin — all have equivalent bonuses, so heritages for these ancestries should be considered equal.
* seer elf is somewhat similar to gnome heritages with cantrips — so gnomes can be considered equal to elves, so they are equal to dwarves and goblins too.
* Gutsy halfling, death warden dwarf — are equal too, so halflings can be added to same list too
* "Half-orc" is similar to "Twilight Halfling", but "half-orc" also gives wider range of ancestry feats, so we can say that human heritages are a bit better than other heritages.
Final AP prices for each ancestry:
* Dwarves: Darkvision, +2HP, -5 speed — (4+1-2) AP=3 AP. (+2AP heritages)
* Elves: low light vision, -2HP, +5speed — (2-1+2) AP=3AP. (+2AP heritages)
* Gnomes: Extra language(Sylvan), low-light vision — 3AP. (+2AP heritages)
* Goblins: Darkvision, -2HP — 3AP. (+2AP heritages)
* Halflings: Keen Eyes, -2HP — 3AP. (+2AP heritages)
* Humans: 1…2AP. (+3…+4AP heritages).
Some restrictions should be in place when designing new ancestry:
* HP value should be in range from 6 to 10. If for some reason GM allows values beyond this range, then it's recommended to double or tripple point-costs of "steps beyond range" (so 4HP should be -3AP or -4AP, and 12HP should be 3AP or 4AP)
* Speed value should be in range from 20 to 30. Other values are not recommended at all even with increased AP costs.
* Only one feature with negative AP cost can be used for new ancestry
Heritages are tricky... Current heritages can be "generalized" to next options:
* Reaction to get +1 circumstance bonus to saves restricted by some condition
* Typed energy resistance equal to half your level, and environmental effects downgrade
* Constant +2 circumstance bonus to saves restricted by some condition (more strict condition than one used for reaction-based heritage)
* Choose magical tradition and gain one cantrip.
* Some other ancestral feature that costs 2AP (normal-sight → low-light vision, low-light → darkvision, etc.)
* Something similar to general or skill feat, but a bit better. For example Nomadic halfling is similar to "Multilingual" feat, but also boost that same feat.
* Gain natural attack (like goblins)
* Some other "custom features" specific to ancestry
Custom ancestral feats are a bit trickier to formalize, but generally:
* They should be equivalent to existing ancestral or general feats.
* 1st level feats might be equivalent to "ancestry feature" with 1AP or 2AP cost, or to some heritage effect (which should generally cost 2AP). For example "Orc Superstition" feat is similar to Dwarven heritage "Ancient Blooded Dwarf".
How to design "powerful ancestries"? This one is still a mystery...
* Maybe such races should be allowed to have more than one feature with negative AP cost, so they would be able to take more "powerful" features.
* Maybe such races should only be allowed for characters that start on third level or greater, with a mandatory requirement to take "Ancestral paragon" as a general feat on third level
What do you think of this? Are there any issues?
P.S. By treating human heritages more powerful than heritages of other races I enforce treating "other halves" (halfling-half-orc) as overpowered. This can be solved in 2 ways:
* "other" base ancestry should already have low-light vision or darkvision to be allowed taking half-elf or half-orc heritage
* or human heritages should be treated as equal to other heritages (and so "free boosts selection and free language selection" cost should be set to 3AP)