I'm looking for an interesting 3PP caster class to introduce into my game to surprise some of my more veteran players with unique abilities. As much as I love Dreamscarred Press, Psionics doesn't really fit with the flavour of the game.

So far the ones that I've checked out and really thought about using are the Gypsy (Dreadfox Games) and Magister (formerly Magus, SGG).

And secondly, has anybody tried these 3PP classes in their games and how did they fit power-wise?

Onmyoji (wizard archetype) from Way of the Samurai from Rite Publishing. Check out the reviews, both mention this archetype with great interest. This a balanced and non-OP design.

Here's a homebrew Magus archetype with the flavor of the Old West...

Shootist (magus archetype)
The shootist blends mystical arcane powers and masterful prowess with a weapon much like his kin the magus, however the shootist uses a firearm as his chosen weapon and a deck of cards as his spellbook. While he can use all simple and martial weapons, including firearms, pistols or revolvers are his preferred weapon.

Class skills (add these to the Magi's existing skills)
Ride (Dex), Sleight of Hand (Dex), and Use Magic Device (Cha).

Spell Decks: Instead of a spellbook, a shootist uses a spell deck that resembles an ornate playing card deck with cryptic symbols and formula hidden in the numbers, icons and border on each card's face.

Arcane Pool (added details)
At 5th level, these bonuses can be used to add any of the following weapon properties: distance, flaming, flaming burst, frost, ghost touch, icy burst, keen, shock, shocking burst, seeking or speed.

Deadeye (Su): At 1st level, the shootist can resolve an attack against touch AC instead of normal AC when firing beyond her firearm’s first range increment. Performing this costs 1 arcane point per range increment beyond the first. The shootist still takes the –2 penalty on attack rolls for each range increment beyond the first when she performs this deed.

Quick Clear (Su): At 1st level, as a standard action, the shootist can remove the broken condition from a single firearm she is currently wielding, as long as that condition was gained by a firearm misfire. The shootist must have at least 1 arcane point to perform this action. Alternatively, if the shootist spends 1 arcane point to perform this action, she can perform quick clear as a move-equivalent action instead of a standard action.

Ranged Spellstrike (Su) – at 2nd level a shootist can use ranged spellstrike to cast a single target touch attack range spell and deliver it through any firearm attack. Even if the spell normally has multiple targets, only a single missile, ray or effect accompanies the attack. At 11th level whenever a shootist using a multiple target spell with this ability may deliver one ray or line of effect with each attack when using a full round action, up to the maximum allowed by the spell (in the case of ray effects). Any effects used in the round the spell is cast are lost. This replaces spellstrike.

Quickfire (Ex) – At 4th level a shootist gains the ability to draw and fire his firearm as a standard action. This ability replaces spell recall. (Useful with street duels, where you start out flatfooted)

Dual Pistols (Ex) – At 7th level, the shootist learns how to use his magical abilities to effectively blend two one-handed firearms fighting and casting spells. When holding two one-handed firearms in separate hands, the shootist may spend a shootist pool point as a swift action to count one of his hands (shootist's choice) as free for the purposes of casting spells and delivering touch attacks. The shootist may also use this hand to cast spells for the spell combat class ability (but then cannot fire the firearm the hand is holding as a part of spell combat). This ability lasts for a number of rounds equal to the shootist's Charisma modifier. This ability replaces the medium armor ability.

Swerve Bullet (Su) – At 11th level, the shootist gains the ability to alter the trajectory of bullet fired at him by sheer force of will, working as deflect arrow feat, but with magic instead of your hand. This ability replaces improved spell recall.

Lightening Reload (Ex) – At 13th level, as long as the shootist has at least 1 arcane point left in his pool, he can reload a single barrel or chamber of a one-handed or two-handed firearm as a swift action once per round. If he has the rapid reload feat or is using an alchemical cartridge (or both), he can reload a single barrel or chamber as a free action. Further more using this ability does not provoke attacks of opportunity. This ability replaces heavy armor.

Greater Spell Access (Su) – at 19th level, a shootist gains access to an expanded spell list. He learns and places 14 spells from the Bard spell list into his spell deck as magus spells of their bard level. He gains two of each of the spells from the Bard list, not on the magus spell list: 0 level, 1st level, 2nd level, 3rd level, 4th level, 5th level and 6th level.

Suggested Magus Arcana
Arcane Accuracy, Bane Blade, Close Range, Critical Strike, and Ghost Blade. Note: Bane Blade and Ghost Blade can be applied to firearms for the shootist.

I'm looking for an interesting 3PP caster class to introduce into my game to surprise some of my more veteran players with unique abilities. As much as I love Dreamscarred Press, Psionics doesn't really fit with the flavour of the game.

So far the ones that I've checked out and really thought about using are the Gypsy (Dreadfox Games) and Magister (formerly Magus, SGG).

And secondly, has anybody tried these 3PP classes in their games and how did they fit power-wise?

I have used the magister, and it fits rather well. I personally feel its what the sorceror always should have been. It definately steps on the sorcerors toes though. Its not strictly better then a sorceror, but it is a great deal more flexible with the potential to add from other spell lists. And as a caster, flexibility is always a good thing.

I would also recommend from SGG the mosaic mage and the death mage. I've used both and think they work well. You can also use the eldritch godling.

Mosaic Mage looks really interesting, I'll have to give it a buy! I've checked out the death mage before briefly, never really been a fan of necromantic casters (but I may have a use for that one down the road).

Thanks for the tips! Still looking for more if anybody has any :)

Might I suggest ...

The Shaman - a spontaneous casting druid-like caster (Kobold Press)

The White Necromancer - a spontaneous casting, non-evil caster focusing on the cycle of life, death and undeath (Kobold Press)

The Vanguard - a martial / arcane caster hybrid (Super Genius Games)

Genius Guide to Exalted Domains (all the volumes) - not a class, but adds lots of new things to clerics (and other casters) who use domains (Super Genius Games)

I recommend this: Pact Magic Unbound, Volume 1

It's all about binding spirits. There is the occultist class, and archetypes for all the core rulebook classes.

The Gypsy turns spellcasting into a card game. I don't know if the card game mechanics are a selling point or deterrent for you, but I wanted to help you make the most informed decision possible.

We are currently working on several pregen decks for the gypsy (which should benefit GMs and novice card game players the most). The development of these pregens is taking longer than expected (we are consulting with a number of renowned "power gamers" and [busy gamers x labor of love = slow response times]), but I would be happy to advance you a prototype of a build if you tell me the level and general tendency you are going for (generalist, blaster, buffer, healer, etc.).

Onmyoji (wizard archetype) from Way of the Samurai from Rite Publishing. Check out the reviews, both mention this archetype with great interest. This a balanced and non-OP design.

Will second this one. Mosiac Mage (also mentioned above) also gets a nod from me. SGG also has the various godling classes that (if they fit your campaign) would be a good addition.

Also, while not a full caster the Dragonrider is pretty freaking sweet.