Weapon Finesse at second level is annoying, as has been pointed out, but the dip problem is a real one.
So why not word it like you get weapon finesse at second level, if you already have the feat then you gain a bonus feat that increases your panache. (I'm assuming the swashbuckler is gonna have a feat similar to Extra Grit)