My thoughts if I may. I have thought of this a lot. Even to the point of figuring how to do lvling up. But that's kinda where I get hung up. But now I'm thinking of just using the rule system and using Tamriel for the world. Use the lore and even allow Dwarves even though dwarves are extinct. I'd go as far as teleporting them in from "Sand Point" of one of the other cities so there would be reason for dwarves to exist. Even have npc's act differently around any dwarf characters because of the lore.
But I'd keep it pathilfinder. All rules would remain intact a part from casting for the Wizards. Still not positive how that will work.