The way I see it, if you're swapping weapons, then you would still have one light weapon and wouldn't have to face the penalties for doing otherwise. The game never designates what makes the offhand the offhand. Is it less strong for lack of use? Are you more focused on one hand while attacking and the other hand just gets in the way? It never says "Your left hand" or anything, it just says offhand, so hypothetically, swapping your weapon to your offhand would make it your main hand. And for me, that would make it awesome.
I'm currently playing a human fighter specializing in axes, and because of the human weapon familiarity, I'm using a Dwarven Waraxe in my main hand. The damage difference is significant when using my offhand (Handaxe). That and my Weapon Focus/Specialization feats would all be in my main hand.
And while yes, it is titled Weapon Swap, it doesn't say "OMG YOU HAVE TO HAVE A WEAPON IN YOUR OFFHAND." Who says your fist isn't your offhand weapon?