Until you hit a scenario where you're suppose to keep your membership in the society a secret, and you pull out a wayfinder every time you cast a spell.
Wouldn't you just shrug, put away the wayfinder, and put an extra hand on your longsword instead?