Your best bet is to go ahead and do it anyway.
It only needs to work for your campaign, and as a GM using any house rule at all you have resigned yourself to spot-fix any problems that arise.
Remember, the only definition of balance that matters at your table is "everyone having fun". Proceed with the rule as you just presented it, and if it looks like some people are severely favored, you'll have some data to noodle with. If it ends up working, that's fine.
It doesn't matter if a party with a different makeup would interact with the rule differently, because it is a house rule and you can spot-fix it.
I would say at first blush it looks like this system will get your low level PC's knocked out pretty consistantly. Since armor provides zero AC and they havent earned a lot of BAB yet. Sure they dont die from the initial damage, but non lethal still knocks you out and most creatures arent going to stop just because you are snoozing.
My personal peeve is that, given identical ability scores, it is just as easy to hit a level 10 fighter as it is a level 1 commoner. I know there are some rules in Unearthed Arcane that cover this some, but they, in the long run, make armor almost obsolete at higher levels.
Some players have expressed that armor doesn't make it any harder to hit someone, but harder to cause damage, and they wouldn't mind seeing this somehow. (also don't 100% agree with like rules from UA)
It sounds like you've already looked through the Unearthed Arcana rules, but if you need them, here's a link (no sense doing something that someone else has already done)
http://www.d20srd.org/indexes/variantAdventuring.htm
Now, that said, the 10th level fighter is MUCH harder to hit than the 1st level commoner, at least in a tangible sense.
The hit point system is designed to represent this fact. That's why the 10th level fighter has 19 to 100 hit points before constitution bonuses, and the commoner has one to four (if they still have 1d4 for hit dice). The commoner is far more likely to be injured, slain or knocked unconscious than the fighter, given equal situations and ability scores.
What Hit Points Represent
Hit points mean two things in the game world: the ability to take physical punishment and keep going, and the ability to turn a serious blow into a less serious one.
Hence, the fighter might get jabbed, but using his learned knowledge, he manages to roll with the blow, or turn it into a glancing shot where the commoner would be knocked out and bleeding or slain.
With a lethality reduction, or even with armor damage reduction, the effect that it has will rapidly evaporate at higher levels, when creatures deal more damage (if your fancy plate mail +5 absorbs 9 points of damage (5 for the magic and 4 for the plate), and the creature hits for 30 to 60 points of damage, esp. on a critical hit.
It could also be helpful to look at example cases at different levels.
1st level chain mail, no dodge AC bonus, Armor bonus to AC +3, and the character still takes full damage from every strike (two points of which become nonethal). AC is two points worse than normal.
5th level, full plate +2, +2 dodge bonus to AC, Armor bonus to AC +5 (or six, it wasn't clear how magic armor worked), converting 4 (or five) to nonlethal with a DR of 2/Magic. Six points a strike are reduced (but only two, from the magic would be eliminated), but the character has an effective AC FOUR points lower than by standard rules. The character will go down quicker than usual.
10th level, full plate +5, +5 dodge bonus to AC, armor bonus to AC +7 (including magic)
Converts four points (or seven, if you tally in the magic to the calculation), with a DR/5 magic (and it's likely the opponent will have magic). Reduces nine or 11 points on a hit, though most of that still goes through as nonlethal, and the AC is still lagging behind by two points on what it would be without the conversion.
15th level, full plate +5, +7 dodge bonus to AC, the rest as above.
Finally, the stats equal out.
Now, this system will favor lighter armors, since Dex will be more important (especially at lower levels), and the damage conversion difference is really not that great (two for chain, three for breastplate, four for plate).
As for the other problem that I can see, since you are giving a dodge bonus, it will make rogues a bit meaner (since they can hit flat footed fighters much more easily, since they lose dodge bonuses when flat-footed), and make touch spells and attacks a bit less effective (since touch ACs include dodge bonuses).
All in all, characters, except for non-armor wearing characters, will have generally worse ACs, especially heavily armored types, and lighter armored or non-armored characters (monks, wizards, rogues) will all have better ACs. All of them will still be taking nearly full damage on a hit as well, it's just that part of it is non-lethal.
It's hard to know where to get where you're going. Personally, as I stated at the start, the fact is that higher levels are harder to hit (in a tangible, effective, do serious-injury sense) because of their hit points, so it's already factored in, but semantically unsatisfying.
My suggestions, looking through the Unearthed Arcana variants, would be perhaps to use the defense tables for AC (purely) and then apply the damage reduction or conversion for the armor, without giving armor an AC bonus at all. This means that the character's level (skill) is what determines if they get hit, and the armor reduces the effective injury. The table is balanced fairly well (but comes with drawbacks, again, the fact that touch attacks have a harder time hitting, and flat footed means more). It also factors in the various classes' martial abilities better. On the other side, it heavily favors 'dipping' to get the fighter AC bonus, and gives lightly armored fighters the same chance to be HIT (not counting DR) as heavily armored characters, leaning towards the heavy ACs (and making DEX more effective).
Anyways, whatever you decide on, I wish you luck.
In this thread I showed how to adapt the two armor rules from Unearthed Arcana to accomplish what it appears you want to do.