I always been a bit put out by the various arguments for Improved Unarmed Strike being easier to get or more ubiquitous. Doubly so if they use the reverse guy at the gym argument (which the OP doesn't). They do however seem to feel that Improved Unarmed should be as easy to learn as how to use a sword somewhat effectively.
The ability to reliably do real damage to a standing opponent who is fit and aware is quite rare. Fighting an armed opponent is also ridiculously difficult. Movies and self defense trainers make it look easy, but their technique is usually akin to getting a new initiative roll to be made, winning initiative, then scoring a disarm or K.O in the surprise round.