Well, if your character's AC wasn't so high, the GM might not feel compelled to throw so many swarms at you. The GM probably isn't having fun when they can only hit your character when they roll a '20'. If you look at any of the multiple posts by GMs complaining about characters with high AC, one of the easy solutions is to use more swarms.
Once you get good at defeating swarms, I'm guessing the GM will change tactics. Do you want the game to have real challenges? Or do you want to automatically win because you built a well designed character?
Edit: Oops, just noticed this is an old thread that was brought back to life. Moral of the story: if your character has really high AC, expect to see some swarms and plan accordingly, but don't blame the GM for trying to challenge you.