You never lose your fame. Changing factions has a set cost of 3PP per the Guide to Pathfinder Society Organized Play, page 19.
"If at any point you become disillusioned with your faction, you can change to a new one. The longer you have spent in service to a faction, the higher the relative cost to change to one that trusts you less and hasn’t seen firsthand your ability to forward their goals. Changing your faction costs a number of Prestige Points equal to 3 times your character level, but does not alter your Fame score. (Prestige Points and Fame are fully explained in Chapter 5). You retain your faction traits, if any, but lose any faction-specific Prestige Awards you have accumulated during your time with your old faction; generic Prestige Awards from Table 5–4 and Vanities (Pathfinder Society Field Guide 60) are unaffected."
