It's also clarified in later books that most of the first group came over while under Ochymua's Domination. One of the NPCs from book 3 talks about it. So Ochymua brought a bunch over and then let the faceless stalkers replace them and take over everything else.