Ah of course! I do understand that. It's just that for my players, being previously despicable humans won't work at all. And while we do enjoy the lovecraftian touches in other Paizo APs, we're not experts of the Lovecraft mythos by any stretch of the imagination. As an example, my players have no idea that Denizens of Leng or Hounds of Tindalos come from the Cthulhu Mythos. So they're into this AP without any specific expectations.
OTOH, Mind control and losing one's agency is extremely frightnening to them (well, to anyone, really). Knowing this, I wanted to emphasize the otherwordly power the yellow sign has on people's mind. From what I could gather, just by looking to a Yellow Sign you could be influenced by Hastur. Having a simple symbol be a powerful tool to open one's mind is utterly terryfing. I might even allow for cultist's holy symbols to count... not entirely sure yet.
Also, I had this idea of a sociopathic Lowls that through a strong force of personality, but also some suggestion and even Charm spells, would befriend you, and slowly make you do things you didn't want to. Just like a good sociopath would do...