I'm in the same party as Damon and I'll add a couple of things.
A) The wizard also failed his save and would have also been confused but it was moot since he was also taken out on the surprise round. Actually, I think he was the first to drop.
B) While it is true that the DM was not giving us the miss chance, it is also true that he was not giving the shadows the total cover they would have been entitled to. As a result, I suspect the outcome would have been about the same, just taking longer to play out. The shadows might have missed us a few times, but the melee characters would have gotten a lot fewer attacks on them and that's where most of the damage we did came from given that the spell casters were almost all neutralized before they got to take any action.