With 11 levels in swashbuckler it's a little late to change direction entirely. You could maybe dip a few levels in a class or classes which allow you to use and regain panache (/grit/luck) differently (sleuth investigator, some variety of gunslinger, hooded champion ranger, probably some others).
You might go into rogue or the rose warden prestige class for more skills or other roguish abilities which might fit this character.
Or, you might try for the Chernasado warden prestige class if the area fits and you want some outdoorsy stuff.
It's mostly too late to switch to a magical class, but eldritch scion magus wouldn't be completely impossible. Spell combat lets you cast spells and still attack with that starknife.