Yeah Intimidate is a charisma check but I can see where OP is coming from.
From the point of view of a person being intimidated there's also a risk assessment even before any interaction occurs. Context beyond charisma.
Given 2 characters with the same charisma score, which one is going to look more like a threat to your safety? The big muscular guy hodling a weapon or a scrawny and seemingly unarmed teenager in rags?
But I don't know how or even if this should be reflected mechanically somehow.