I'd have a sliding scale of difficulty with spellcasting. At stage 2 give a DC 5 flat check to cast spells, DC 11 at 4 and 16 at 6. Then at stage 8 it removes all spellcasting. All stages before stage 8 would wear off given time and recovery.
It also might be thematic if the swarm has blindsense targeting creatures capable of casting magic without the aid of magic items. I'd probably leave items alone from this, so if a full caster is unlucky enough to hit stage 8 they aren't completely without options.
Finally consider whether a potential cure to this affliction is appropriate for your campaign, and to what extent it should be available. In the long term narrative, the stage 8 effect of removing a character's spellcasting is similar to death in that it removes a full caster's ability to contribute as an adventurer. I'd probably have a cure about as available as Raise Dead, wherever that lies in your campaign/setting.