
Paraxis |

I don't see any problem with it, but to be honest in my games I don't care much about the somatic or material components to most spells. I let my clerics use heavy shields and a weapon without caring about somatic or holy symbol needs. I pay attention to verbal components if silence spells are being used, and material components if they cost 100g.p or more.
Imagine if you were a great sword wielding eldrich knight you could still cast and fight because changing grips is a free action. I don't think letting gish characters dual wield or sword and board is any different.