I do get that and I absolutely understand the balancing issue. RAW wise I've come round to the idea that the correct interpretation is that Mirror Image has no effect on spells that don't roll an attack.
Like I said it's more from a common sense/describing the action standpoint that I'm now hung up on and the reason I made the ruling in the moment originally.
I'm happy for the examples you give to just inherently 'pick' the correct target from among the images, either because they're magic or because it's a speech/mental effect like demoralize which relies on the target's perception: they can see/hear you intimidating them so they are the target.
But when it actually involves you grabbing or or at least touching the target it's difficult to describe why Mirror Image wouldn't work against that when it works against someone trying to hit you with a sword, without just having to say: "because those are the rules in place to balance the spell".
As I say I think it's a general issue I've got with the way touch range spells with no attack roll are described in action. I suppose you can argue that the same way a ranged spell with no attack roll inherently finds the right target the touch range spell inherently guides your hand to the right target...
It just doesn't seem very satisfying and as I've said, ignoring Mirror Image for a moment, it also feels odd if you have a very agile character who's just dodged a fireball and the 6 attacks from the ninjas around them only to have an octogenarian wizard able to just reach out and touch them and they can't do anything to avoid it...