If this is a NPC, its all up to the DM honestly as you are allowed to ignore details like that.
For Player wizards it kinda depends on what the players want, but in the end you are basically wrestling control of one of their class features. On the other end if its your own character, sure go ahead.
But at least for the improved familiars the wizard/character need to stay within a step of the alignment of the familiar and kinda suit the personality of the familiar or they could end their service with you early.
What dracoknight said.
Here are some with built in trouble....
The Imp states:
Unlike most devils, imps often find themselves free and alone on the Material Plane, particularly after they've been summoned to serve as familiars and their masters have perished (often indirectly due to the machinations of the imp itself).
The Quasit states:
Newly created quasits are birthed directly into the Material Plane, where they become familiars, and while bonded to their masters' wills, all quasits hate and loathe their lieges, as they can feel the pulse of their lords' souls and know that they could have been more. A quasit serves, yet it watches and waits for mistakes that might cost its master's life, or even better, an error that might let the quasit turn against its master.
/cevah
Note that a wizard who tires of their familiar can just dismiss it and find another but if this is a role playing scenario done as a player, it should not be a real problem. Remember the benefits should outweigh the downsides.
If this is a situation where a GM is controlling one of the player's characters, it ranks as extremely not cool.
Jafar and Iago?