I think the OP has it right by RAW, a ray spell has no target, it simply produces rays which are weapon like effects that you then target at enemies so you can target the swarm with a ray just like you can swing a sword at it. As elemental damage is not weapon damage, the damage from the ray would then harm the swarm.
You're the only one in this thread who agrees with me, but I've seen many other threads elsewhere where others take our position. I guess the others in this thread would call it a house rule, but this is why I don't play organized play...
NO, the devs do not have brains the size of planets, and if the rules make a creature that ridiculously powerful, its CR is wrong. This game, like any other, is riddled with holes, inconsistencies, and poor planning. The game would not work in the slightest as a computer program, and requires constant coaxing on the part of a good GM.
A lot of stuff got mixed up and muddled in the port from 3.5 to PF. PF is better in most ways than 3.5, a lot more streamlined and less clunky. However, there are specific points which are messed up, and this is one of them. In 3.5, a weapon with an enchantment does harm swarms with the elemental part (but just not the weapon damage itself), and mechanically there is NO difference between that dynamic and a ray attack.
Perhaps even in 3.5 rays weren't supposed to work, but I never said 3.5 was developed perfectly either, far from it. If a weapon enchant harms the creature, so should an elemental ray attack. Consistency is important, as well as not ramming impossible circumstances down new players throats... and we are talking about magic here after all...
Just my $.02 on why splash weapons work - it's as if they target the square...if you just hit the square a creature was in, it would just do the splash damage to the creature, logically...and each component within a swarm takes no more than 1 hp to kill...seeing as it affects the surrounding squares, it surely affects the whole square for that 1 pt, making it for all practical purposes, an area effect attack.
By RAW, if you accept that rays do not work, no splash damage also does not work on swarms... well, only the splash part... 1 damage to the area. Re-read the splash weapon damage rules. There are two parts to it, one is a direct hit, the other is AoE, similar to a meteor swarm spell... which, stupidly, would also only hurt the swarm with the AoE portion, and not the direct damage portion by RAW with the reading supported by most in this thread.
So, I guess I have a rules answer from other players/GMs of Pathfinder, and I guess that is why I prefer to GM and stay away from organized play, because lots of the rules suck. You can call it houseruling or playing a 3.5/PF hybrid RAW, whatever... but rays, magic weapon elemental damage, and splash weapons all affect swarms normally in my games... whatever.