When the impulse ends your familiar returns unharmed and in its original state, regardless of how the impulse is ended.
So you're familiar is only really in danger from two things: any remaining enemies it might appear next to when the impulse expires, and multiple castings of this spell in the same day, which will overtax and kill your familiar when it returns.
IMO when a summoned creature is banished and your familiar returns:
Immediately when you finish Casting the Spell, the summoned creature uses its 2 actions for that turn. A spawn or other creature generated from a summoned creature returns to its unaltered state (usually a corpse in the case of spawn) once the summoned creature is gone. If it's unclear what this state would be, the GM decides. Summoned creatures can be banished by various spells and effects. They are automatically banished if reduced to 0 Hit Points or if the spell that called them ends.