Hi, GM of the OP here. Running an Extinction Curse campaign and we've hit the 'golem heavy' portion of the campaign.
For me the problem is that the term magical ability is just too wide. "Ability" doesn't really put any limitations on it, it means everything.
This is where I'm at too. I don't know if there ever needs to be a true delineation on what is or is not a magical ability, but better verbiage on what does and doesn't affect this class of monsters would help immensely in fielding them.
One of my players is a Ki Monk and things like Clinging Shadows Stance pose a peculiar question, where the focus spell is obviously magical but if the strikes are magical abilities or are not is less clear. (My inclination is that they are by association)
Another is a Alchemist bomber and as the op mentioned: play-wise it feels bad to say that a Ray of Frost triggers the satisfying Harmed By condition of a Stone Golem's Antimagic, while a Frost Vial does not. I don't have a good answer for my players as for why it wouldn't.
Is that what the difference is? A magical ability that the golem is immune to would be something that directly targets the golem rather than targeting something else and just having an indirect effect on the golem?
I think from everything I've read, both in this forum and others, this is kind of the natural conclusion. Things like weapon runes and magical buffs generally fall under the category of things that 'should affect' the golem. While spells and magical feats fall under 'shouldn't affect'.
Traits go either way with some things that should affect them having the magical or a school trait, while other things that shouldn't lack either. The only commonality appears to be if it's directly targeting the golem or not.
While I'm sure that leaves plenty of room for shenanigans where players could abuse things or specific cases that feel unfair, it at least provides a guideline for a GM to apply some consistency to their rulings and provide some common sense explanations that the players can work with.