So I'm trying to fill up a first level dungeon with realistic ecology -
Troglodytes would naturally live deeper and further into a dungeon and push the kobolds towards the entrance of the dungeon where it is more dangerous.
But the real question is would these two races even get along and be compatible in the same dungeon? Or would the Kobolds be forced out or eaten by the larger humanoid Troglodytes?
Trogs have incredible cruelty but are intelligent as well so I could see them tolerating the Kobolds by making them live nearer the entrance - sort of cannon fodder for them. But Kobolds are not unintelligent themselves and work well in groups.
What do you guys think? Compatible species and what would their social dynamics be?
They would probably be rivals. They might reach a state of begrudging cooperation -- as you said, kobolds living near the surface, troglodytes further in. There could be links to the darklands at the bottom of the dungeon and the Trogs could hunt and gather fungi and such in those, while the kobolds would get things from the surface, and could trade them. So their relationship would probably be complex.