From the PRD"Somatic (S): A somatic component is a measured and precise movement of the hand. You must have at least one hand free to provide a somatic component.
Material (M): A material component consists of one or more physical substances or objects that are annihilated by the spell energies in the casting process. Unless a cost is given for a material component, the cost is negligible. Don't bother to keep track of material components with negligible cost. Assume you have all you need as long as you have your spell component pouch.
Focus (F): A focus component is a prop of some sort. Unlike a material component, a focus is not consumed when the spell is cast and can be reused. As with material components, the cost for a focus is negligible unless a price is given. Assume that focus components of negligible cost are in your spell component pouch."
It specifically states you need a free hand for somatic components (which the special rule for arcane duelist over-rules) but material components and foci just mentione you having to have them, not hold them. By raw I dont think you need a free hand for material components or foci.
Well, you have come to the same conclusion as myself.
I'm not so sure that this was Paizo's intent, but it's not like this is the weirdest thing in PF. ^_^