I had a lvl 4 party of 5 attempt this: a rouge, a barbarian, a cleric, a sorc, and a ranger.
The rouge bluffed his way into the fort alone and after waiting till night, managed to kill 2 bandits in their sleep. After this he was caught and instead of fleeing for his life, he tumble his way to the owlbear cage and managed to release it before the bandits killed him.
With the bandits distracted with the owl bear, the rest of the party charged the gates and managed to fight their way through in the confusion. The stag lord came out and killed the ranger's pet in one shot before the cleric managed to blind the stag lord and effectively render him useless.
Although he had died, the rouge's sacrifice is what won them that battle.
