The idea of a dog familiar is awesome. It makes more sense than a wizard walking around with a toad.
Au contraire: a not inconsiderable part of the magic user's power, especially the witch's power (and while wizards might have copied the practise, it originated with witches), lies with being otherworldly, mysterious, and unsettling.
Dogs are not unsettling, they're loyal companions. Those small purse dog variants will rather spoil the image, of course, and while large dogs might seem frightening, they're nothing more than a common bully, and being common will prove the witch's undoing. Unless you manage to snare a genuine hell hound, of course.
No, you really want something generally considered unclean or uncanny, like a rat, toad, or snake. The added bonus with the toad is that you can claim that he used to be human until he crossed you.
Naturally cats are always an option, too. It takes no genius to know that cats are servants of sinister trickster forces.
Finally, you can use an animal that, while not necessarily being creepy in of itself, is usually not a pet. Something that is never tamed. Foxes work well for that. Foxes are shy creatures by nature, and seeing one eating out of that conjurer's hand will put people in a thoughtful mood. And, of course, people equate foxes approaching with rabies, so you can use even such a weak creature to spread panic.
Anyway, poodles, dachshunds and beagles might be great companions, but they don't fit the familiar's job profile!