Just reached my fourth level of Sorceror, but I also have a level of bard.
The wording says at fourth level I can exchange a spell I know for another, but it doesn't specifically say it has to be a Sor/Wiz spell.
So can I exchange one of my Bard spells?
The spellcasting of each class is separate; you can't exchange a bard spell known to gain a sorcerer spell known, just like you can't use a sorcerer spell slot to cast a bard spell.
I gathered that, but can I exchange a bard for a bard?
No. You can do that when you reach your 5th bard level, as detailed in the write-up for the bard class.
Edit: That said, however, I wouldn't see any problems related to house-ruling that you could exchange spells for another class.