ECL and LA are just bad when it comes to PCs. They don't really balance anything and were put forward only as a quick and dirty system, and then advanced via chinese whisper into some weird gospel. This fact has been acknowledged in both D&D3.x and Pathfinder by saying the level adjustment of a creature changes depending on the campaign level. That alone demonstrates the half-assed utility of the number.
What should be done is an individual examination of the abilities of each race, and having those abilities only become available at appropriate levels. That's how you balance things against standard races and classes. Similarly, to maintain balance you have to have some form of sacrifice on the part of the character (levels, feats, class abilities, etc) since standard races don't gain "more stuff" later on during character development.
To say something is "balanced" you have to be able to say that, at any given level, it is not significantly better than a character with a standard race and class. Unless the race in question only has better flat numeric bonuses, a simple sacrifice of levels doesn't really accomplish that.
Similarly, to maintain balance you have to have some form of sacrifice on the part of the character (levels, feats, class abilities, etc) since standard races don't gain "more stuff" later on during character development.
Quoted for truth, emphasis added. Number one problem I have seen with monster-as-character adjustment methods is not taking this into account.
Many of the most difficult challenges in dealing with special PC races is that party members, no matter how much they may like to avoid it, are always in a kind of competition with one another. So, rather than balancing PC vs. enemy, the difficulty is in balancing PC vs PC.
Those problems can be sidestepped by giving all of the PCs the same or similar special races. A party of all Noble Drow, all Centaurs, or all Mephits may have to fight tougher enemies, but that's easy to account for in campaign design. Nearly as easy is a party of similarly powerful races, like a party where all the PCs get to choose a CR3 monster as their race.
Unless you're writing for the game company and need to design a system that can be universal, one of these two methods is likely to fit your campaign.
With this idea, the bugbear, with his base race abilities, would be granted 4 hit dice ( (20-10) / 3 = 3.33 rounded up) of humanoid, granting only the base attack, saves, and skill points (2+int modifier per HD for humanoids) associated with that (no class skills).
This still gives it a class all its own, serving the same purpose as level adjustment or ECL. Savage Species gave out racial hit dice on a "Monster as a class" progression for a ton of beings. It still essentially nixes monster casters of any variety, and prevents the beings from getting class abilities appropriate to the level of challenge they are ostensibly facing.
I understand that. But from my stand point, it is essentially multi-classing, so yes, you're right with a fighter-type race is difficult with a caster type. With the idea it wouldn't be any different than a fighter multi-classing into a wizard. It's built more for a less optimized group and more one for flavor and roleplaying, but still keeping it within the context of the rules.
Ah, I misunderstood you, I thought you were trying to put forth a solution that fixed those problems. I'd be keenly interested in that- there are a number of classical "monsters" I'm wanting to make casters and hybrid casters of, both as NPCs (easy in ad-hoc) and PCs.
Also, I'm slightly puzzled by the tone of your last sentence. What is mutually exclusive about optimizing and flavor or roleplaying? When mechanics meet theme, wonderful things happen. It's constantly what I strive for.
I suppose what I'm proposing is to regard the monstrous portion of a race more like a short-leveled class. Just to clarify, the idea is to not have a LA or ECL. Instead, what I am saying is to essentially make it to where the monstrous abilities that make it more than a standard race are the "class" abilities, and then give them starting levels or HD in that monstrous class to balance the hp, saves, base attack of any other class.
That does nothing to address the balance issues that were and are inherent to the LA system or the Savage Species monster levels system. As I said, the issue is that the abilities are not placed at the levels they should be when done with a LA or monster class. Even if you allow multiclassing with a monster class, someone can still frontload character development with the racial class levels and have abilities earlier than they should have.
A player race, regardless of implementation, should not supplant other forms of character development or render any of those forms obsolete.
Classes are the main construct under which character abilities are granted. If you create a race, even if using "racial levels" that makes you a better fighter than a fighter, you've essentially created an unbalanced class that makes the fighter obsolete, in the guise of a race.
Personally I do not like the idea of having a monster character develop abilities in race levels, starting out as a lesser minotaur or something similar seems outright silly in most cases. Game mechanically viable but I cant really have it work with my idea of character growth, though in some cases in campaigns wih significant downtime between leveling it might work better, having an ogre grow from medium to large size for example.