Yes, per RAW it does affect them. If the Swarm in question is one or more sizes smaller than the Elemental, it would deal its slam attack as damage to it (+50% dmg because it's an Area Effect). Furthermore, if the Swarm in question is diminutive or fine, it can become dispersed or adversely affected by the high winds.
Rules as Written about Swarms:
"A swarm is a collection of Fine, Diminutive, or Tiny creatures that acts as a single creature. A swarm has the characteristics of its type, except as noted here. A swarm has a single pool of Hit Dice and hit points, a single initiative modifier, a single speed, and a single Armor Class. A swarm makes saving throws as a single creature. A single swarm occupies a square (if it is made up of nonflying creatures) or a cube (of flying creatures) 10 feet on a side, but its reach is 0 feet, like its component creatures. In order to attack, it moves into an opponent’s space, which provokes an attack of opportunity. A swarm can occupy the same space as a creature of any size, since it crawls all over its prey. A swarm can move through squares occupied by enemies and vice versa without impediment, although the swarm provokes an attack of opportunity if it does so. A swarm can move through cracks or holes large enough for its component creatures.
A swarm of Tiny creatures consists of 300 nonflying creatures or 1,000 flying creatures. A swarm of Diminutive creatures consists of 1,500 nonflying creatures or 5,000 flying creatures. A swarm of Fine creatures consists of 10,000 creatures, whether they are flying or not. Swarms of nonflying creatures include many more creatures than could normally fit in a 10-foot square based on their normal space, because creatures in a swarm are packed tightly together and generally crawl over each other and their prey when moving or attacking. Larger swarms are represented by multiples of single swarms. The area occupied by a large swarm is completely shapeable, though the swarm usually remains in contiguous squares.
Traits: A swarm has no clear front or back and no discernible anatomy, so it is not subject to critical hits or flanking. A swarm made up of Tiny creatures takes half damage from slashing and piercing weapons. A swarm composed of Fine or Diminutive creatures is immune to all weapon damage. Reducing a swarm to 0 hit points or less causes it to break up, though damage taken until that point does not degrade its ability to attack or resist attack. Swarms are never staggered or reduced to a dying state by damage. Also, they cannot be tripped, grappled, or bull rushed, and they cannot grapple an opponent.
A swarm is immune to any spell or effect that targets a specific number of creatures (including single-target spells such as disintegrate), with the exception of mind-affecting effects (charms, compulsions, morale effects, patterns, and phantasms) if the swarm has an Intelligence score and a hive mind. A swarm takes half again as much damage (+50%) from spells or effects that affect an area, such as splash weapons and many evocation spells.
Swarms made up of Diminutive or Fine creatures are susceptible to high winds, such as those created by a gust of wind spell. For purposes of determining the effects of wind on a swarm, treat the swarm as a creature of the same size as its constituent creatures. A swarm rendered unconscious by means of nonlethal damage becomes disorganized and dispersed, and does not reform until its hit points exceed its nonlethal damage."
I would use a little logic though about how this should work depending on what kind of a swarm you're facing though. For example, if you came across a Spider Swarm (10ft space) and the base of your whirlwind is only a 5ft space, technically 75% of the swarm is not within range of your effect. But, if it were a Bat Swarm, the top of your whirlwind would be much wider, and able to affect the entire swarm normally.
This isn't in the rules, but that's how I would personally tackle this question. RAW the swarm operates as one entire unit with one HP total, and I think that's done to simplify the complexity of dealing with a swarm (10,000 individual HP totals? getouttaheeea...) Anywho, it also doesn't make sense that you have a 5ft area effect (1 square) that can kill every member of a swarm in a 10ft area effect (4 squares) despite 75% of the swarm being not in the area of the effect. /shrug.