A big problem is Initiative order + tactical movement speed = crap options for retreat.
Rules for an actual chase help; Con and Dex, Acrobatics and such checks to see who can cover ground, keep going, follow/cover up a path, etc. But then there's the matter of the transition.
I think what you need is layers of defense. Not just traps,then mooks, then boss guy. More like traps, then mooks, then boss guy with mooks, then boss guy falls back and unleashes their pet monster, then they falls back more behind traps, etc.
An exit strategy right from the start is essential. I mean, it's what an actual villian would be most concerned about, isn't it? Unless they're an absolute madman, they care more about survival than victory.
In an example like this one, I'd probably do something like:
1. boss (and one mook with a ranged attack) get behind cover and lay down some Grease or caltrops or something near the closest exist at the first sounds of battle in the other room.
2. Two melee mooks engage PC's while boss and ranged mook attack from cover.
3. The moment the two melee mooks are down or the boss has taken significant damage (probably between 25-50%, depending), they cast obscuring mist and turn tail. Between the last mook, the fog and the trap, the party should have enough to deal with to give the boss a chance.
After that, a chase scene or a "I'll get you next time!" bit.
The PCs could block the BBEG from escaping because they had uninterrupted movement lanes. Fill the room with furniture and it's quite a bit harder; certainly it's easy to stop people charging.
As for adding mooks that weren't already deployed - you can add noncombatants who just get in the way and cause confusion. That woman is a Commoner 1 in reality who will just try to escape (perhaps trying to rescue her jewellery and pet cat first) but to the PCs she looks like a threat. Even the pet cat can be a problem / target if it's in the way and looks like an AC or familiar.
I just don't understand how so many GMs out there have monsters/villains run away successfully. In this fight, as well as many others over the past years, I've tried to have the villain escape and without just being an overbearing GM that handwaves that the villain gets away because I said so how this is so easy to accomplish. What am I missing?
It is very hard to escape if you are in melee. Most of the time that the villain escapes is because they run while minions occupy the party. Other times it involves having an escape route the party cannot quickly follow. Secret doors, and spells are the common form here. Some magic items help here. The best option for low level melee escape is to defensively cast Vanish and then move 30'.
For the rest of the post, you made some mistakes likely due to being new at this. Once you have that encyclopedic knowledge, you won't make these mistakes. It just takes time and experience. :-)
The main villain in this instance is a NE M Ratfolk Alchemist 3/Wizard 1
Ratfolk are size small, not medium.
So essentially the two of them were 15' apart opposite one another, forming a gauntlet through which the villain would have to pass. They were still technically in 5' of mist each however but since the villain would pass between them within 5' of them in the mist, no one had Concealment.
The wizard ends up in the Obscuring Mist spell's area. The Investigator does not, since the door he is next to is 10' outside of the spell. Were the ratfolk to run the gauntlet, the wizard would have to deal with a 20% miss chance due to concealment.
When making a melee attack against an adjacent target, your target has concealment if his space is entirely within an effect that grants concealment.
At the outset of the first round of retreat, the BBEG begins at 10' from the barbarian....
The BBEG went next. He made for the front door with a Run action losing his Dex bonus to AC and moving 80' in the round. This movement put the villain 70' from the barbarian...
To start 10' away and end up 70' away, with only 80' of movement, means the barbarian was in the path the BBEG ran. This is very difficult and requires an tough acrobatics check to accomplish. [It could be easier, by not moving as fast and loosing 5' of distance.] Additionally, the Barbarian would get an AoO unless that acrobatics check was made.
When you run, you can move up to four times your speed in a straight line (or three times your speed if you’re in heavy armor).
You can’t move through a square occupied by an opponent unless the opponent is helpless....
A trained character can attempt to use Acrobatics to move through a square occupied by an opponent (see the Acrobatics skill).
When moving in this way, you move at half speed. You can move at full speed by increasing the DC of the check by 10.
The next round the wizard and investigator ran back after the villain who used Acrobatics, darted 10' away from the Barbarian and drinking a formula for Shield (I didn't have any "escape" type formulae prepped).
This generates an AoO from the barbarian.
Based on random rolls, one giant centipede engaged the barbarian but the other 2 attacked the bad guy.
All should have attack the BBEG, since the Barbarian is not your opponent.
It attacks your opponents to the best of its ability.
/cevah