
Brainwave |

Normally, undefeated barriers are shuffled back in (unless the card states otherwise).
The "nice" barriers like the loot chests often give you the option to banish them if you like (if you are in a hurry and can't waste a turn encountering it again or if you have no chance to defeat them) but if it didn't say that, then you would have to shuffle it back in.
Also side note - I believe there is errata to the Ambush card saying that if undefeated - after it goes off the card is banished, regardless of whether you defeat the monster (or if there is not a monster left in the deck to ambush you).