I agree with knightnday. So long as the knitting doesn't get in the way of playing let it go. Particularly if it's a large group and people have to sit around waiting. Otherwise they -will- talk, check their phones, or something else that -will- get in the way of the game.
The whole point of this is to have fun. If we turn it into a religion by taking it too seriously it isn't fun any longer.