If you intend to use two-weapon fighting to get an extra attack with an off-hand weapon, you must take the appropriate attack penalty for that weapon for that attack. After the attack you may take a move action instead of making your remaining attacks.
If you made your first attack without that penalty, you can't later get an extra attack with your off-hand weapon in that round. You could still use that weapon for one or more of your iterative attacks; the penalties only apply if you want to take an EXTRA attack with the off-hand weapon.
Technically, if you use two-weapon fighting, you only need to specify which weapon counts as your off-hand weapon for this round. Although it's usually obvious there are times when it's not.
You don't need to designate a 'primary' weapon. The 'primary' weapon is simply any weapon you use that round that isn't the weapon you designated as your off-hand weapon. Only the off-hand weapon you designated can make the extra attacks granted by TWF (and the other feats in the TWF tree). If this weapon is light your penalties for TWF are two points less.
Any weapon EXCEPT your designated off-hand weapon can be used, in any combination, to make your other attacks.
Imagine you have a base attack of +11, TWF, Improved TWF and Quick Draw. You designate the short sword in your left hand as your off-hand weapon for this round. This sets your TWF penalties at -2 for the round. It also means that both the extra attacks granted by TWF and ITWF may only be taken by the weapon in that hand. Your other attacks could be with your battle-axe at +11, then you could drop it as a free action, Quick Draw a dart and throw it at the Mage at +6, then Quick Draw a longsword and 5-foot step up to the Orc and hit it at +1.
This usually doesn't come up simply because people use their big sword as their 'primary' and their small sword as their off-hand every time they TWF and there is usually no need to change it.