The Performance is both visual and auditory, meaning even if they don't see the Shadowbard, they can still hear it and get the penalties. If you remove both ways to perceive the performance from the Shadowbard (like a silence spell), a creature can't be affected by it.