I don't really see any reason for a DM to deny a straight switch of your casting attribute from INT to CHA. I certainly wouldn't, and I am pretty big on not breaking things that are fine.
Overall, INT is a much more powerful primary stat because it is mechanically powerful because it gives extra skill points. CHA only helps out social skills unless you are a Paladin or Cleric.
That considered, if I allowed such a thing, it would likely be under the caveat that one could not multiclass into Paladin specifically because that could be highly powerful for an Eldritch Knight build.
A level of Rogue is a simple solution, as is being human and using the bonus feat to take Cosmopolitan as suggested earlier.