So, this was an idea that sprung from inspiration from these forums, where someone mentioned the possibility of a single class that hybridized the Rogue and Wizard/Sorcerer, in the same way that the Magus hybridizes the Fighter and Wizard. This role has traditionally been filled by prestige classes, but the Magus has proved it can be done very well with a single class. I was inspired, and just had to give the class idea a try. After a bit of work, this is my first draft of the Guilemage. Please give it a look!
I may have pitched the power level on the class on the high side with the first draft, (I had way too much fun coming up with Guilemage Knacks, the class equivalent of Magus Arcana or Rogue Talents) however, I was aiming to create a powerful and customizable class on roughly the same level as the Magus or Summoner, including discounting higher leveled spells in theme with the class in the same way the Summoner does. I'm worried that the class overshadows the Rogue it is derived from, but the problem, from how I see it, is that the Rogue is such an underpowered class that it's almost inevitable in a comparison that the Rogue will be overshadowed. There are several improvements I think should be made to the Rogue, including Rogue Talents sucking less and having a full BAB, but that's not the topic of discussion here and I tried not to constrain myself by that in design.
I looked it over quickly last night and will take a look at it again later today but here is some initial thoughts..
Casting: Following all other styles, your class should either use Int and prepared spells, or Charisma and Spells known. Bards, Summoners, Sorcerers, and Oracles all use Charisma and spells known. Guile might mean cunning intelligence but I still think Charisma fits better here.
I would swap Sneak Attack and Guilemage Knack at levels 1 and 2, and the progression following. Rogue, barbarian and ninja all follow the "every even level" progression for their similar class features. Sneak Attack at level 1 makes more sense to me anyways. I'm glad you used the 1/3 SA progression as well. Would be too strong otherwise.
Vanishing trick seems like it may fit better as a Knack instead of a class feature; a 1 round/level ability is not really core or iconic of a class.
I would bump Arcane Dodge up to 5th level with the above changed in mind and the progression to stay the same (9, 13, 17). Have it also require no shield to be worn. Stacking light armor and a mithril buckler (no ASF) with this ability could get to some silly AC.
If you have access to it, check out the 3.5 class Beguiler from PHB2 for some extra ideas (it was a weak class in balance terms so keep that in mind). I'll pop in later for some more feedback
First off, thanks for the feedback.
I'm aware that it is the convention for Int-based casters to be prepared casters, but part of the idea of the class was that it was a take on a spontaneous Int-based caster, something that currently doesn't exist in Pathfinder. I will consider switching it over to Cha based casting, though. I worry that making it prepared would be too powerful.
Switching knack and sneak attack progression is another strong possibility to consider, although I like the idea of being able to create knacks selectable at level 1 that "adjust" the class a bit. It's also somewhat of a relic from when they had full sneak attack progression so they would get each level of sneak attack 1 level after the rogue. But I think a 3/4ths progression for sneak attack is much better.
I felt like invisibility seems like such a core junction of the two archetypes that it made sense for it to be a class feature. I considered having it as level + int modifier in rounds per day but I was worried about it being too strong considering it's already an action economy breaker.
I hadn't thought about the buckler AC, though that would bring up the point of why bother if a mithral buckler gets you a higher AC in the end. Hmmm.
After looking at the beguiler, I liked the fact that the spell lists were somewhat similar, but beyond that, the class does seem a little weak for what you give up from a standard wizard/sorcerer. At one point I had a feature almost identical to cloaked casting (+1 DC and +1 against SR at 4th level, increasing every 4 levels after that) but I realized that the guilemage spell list did not really have a lot of spells where that would be applicable and so I dropped it for arcane dodge.