Level 6 as a brawler gets you the ability to flex two feats at once. Get that before you multiclass out anyway.
As to whether it might be a good idea - possibly? A brawlers' swift action is usually free, an inquisitor uses it all the time, so there's that much synergy anyway. The spellcasting isn't going to amount to much as it's 5-6 levels behind though.
What's your character like at the mo, besides being a level 5 brawler?