Which characters are you using? It'll be a bit easier to offer advise if we know what your party looks like.
I thought this was about the Korean spammers?
I can't say if you're missing anything, but I can speak to the strategy my group used.
The monsters being on top is actually an advantage. You now know the first few cards of each deck as long as you don't shuffle them. Martials love monsters, let your weapon users hop around taking care of the monsters on top of each location while your spellcasters stay behind hoping to get the swarms. 17 isn't that high when you're getting a free d8 and have around a +6 by this point from skill feats and mythic passives.
Even if your weapon users find a swarm and don't want to banish a weapon, they can still probably make an unarmed 11 through powers and mythic charges. It has never been the end of the world before when you banished a henchman without closing the location, right? Especially when you know that there won't be a single monster in the location afterward. Also remember that Harsk can evade them since they're Vermin. Really only Adowyn should be afraid of finding one, that I can think of.
The barriers, well... If you get a Horde or Blight it's a bad time no matter what scenario it is. But at least in Wrath you have a good chance of getting temptation barriers that (can) do nothing to you.
I hope this helps!
What Fendaso said...I'll add that Adowyn should make liberal use of Leryn's scouting ability to ensure that doesn't run into a swarm unprepared.
Also, if you're carrying a garbage weapon (basic bow/sword/whatever), I'd say go ahead and use it against the swarm. You can always pick it back up at the end of the scenario if your character ends up short on weapons.