Contact typically means you are directly touching it. Wearing gloves would protect you from the damage when holding them. Using a metal knife and fork to eat would cause damage because you are putting it in your mouth. Technically you could avoid it by being extremely careful, but you will probably drop food or accidently touch the utensil when you eat. Using soup spoon would be nearly impossible. You could use wood or silver utensils to avoid this.
Wearing metal armor also prevents you from casting spells, so that will also have to be dealt with. This is more than arcane armor failure; it is a total inability to cast spells while wearing metal armor. Some modern clothes may cause similar problems.
RAW, yeah gloves would work. From a Role play perspective, you would actively avoid using any form of Iron or Steel.
Aside from weapons, this is pretty easy, Leather armor, bone or wood utensils, bone or wood buttons (I don't see Zippers existing on Golarion), Buckles could be made of bass or bone. Etc.
for weapons, you would probably try to stick to bone, wood or stone weapons.
A small amount of metal probably does not count as armor for spell casting, but it would count for the damage. Wearing a pair of pants with a metal zipper would probably not inflict continuous damage, but unzipping them might be a problem. It will not be easy to unzip the zipper while wearing gloves. The gloves will probably need to be fairly thick to prevent damage. You would probably want to avoid any metal in your clothes to make sure you did not hurt yourself.
I was thinking more of clothes with a lot of metal preventing you from casting. Something with lots of metal studs or decorative zippers probably would count as armor for this.