I'd allow it, but I'd impose a Riding check penalty equal to the off-hand weapon penalty. If one were trying to be at all realistic (not a criticism! I avoid reality myself!) I'd say no. Try riding a horse with a ten foot 2-by-four under each arm.
But wasn't there a King Arthur movie recently where Sir Lancelot did just that?