Generally speaking, do your games have any signs of magic other than the verbal and somatic components? Obviously, something like Lightning Bolt or Shocking Grasp would be rather noticeable. However, I think something like Bestow Curse could be cast without anyone noticing. Of course, the verbal and somatic components would be an obvious sign so you would have to do them before you even see your target and then hold the charge. Am I wrong? Would anyone say that the caster's hand glows, or something along those line, when the spell is delivered?
I don't think the spell Manifests (as in glowing hands), but you have to do gestures that can be recognized as magic for anyone, and you have to speak in verbal components that are obvious too.
Think on DR Strange, doing weird things with his fingers and chanting "Klaata Veerata Nictus" http://4.bp.blogspot.com/-I_LV90SR05g/TbWouhdLAII/AAAAAAAABwk/RVoznZuDZkA/s 1600/dr-strange.jpg
Even without a glow, it's obvious that something is wrong there.
Still Spell and Silent Spell might change that, of course.