Nullpunkt,
It may help to think of the move action taken to move out of the square to include “shaking off” the clinging vermin. As the player has moved out of the main body of the swarm, there are no longer new vermin to replace them (which is why simply shaking them off doesn’t work while still sharing a square). The remaining handful of fine vermin aren’t really a threat until they rejoin the main body of the swarm.
I would argue that any player caught in a swarm would be shaking them off, dancing around, trying to protect their face, etc. by default. In fact, that’s part of what the distraction check embodies – if the player fails the check, they are so busy trying to knock the pesky critters loose that the character isn’t effective at anything else.
And to revisit the swarming bees example, remember that turns are a rules abstraction to simplify games management – technically everybody’s turn happens at the same time. So when you run from the bees on your turn, they can pursue you on their turn and re-surround you, until you reach a destination they cannot follow. It actually parallels the real world quite well – you’ll receive less stings in six seconds of running from bees than you would in six seconds of standing in one spot in the midst of an angry swarm – but ultimately you must either get far enough from the hive the bees cease to feel threatened, or find your way underwater, inside a building, or somewhere else the bulk of the swarm can’t follow.