If you retrain away from a class those class skills associated with the lost class are no longer class skills so you no longer get that +3 to those skills, you don't lose any ranks in those skills obviously. Now, since you have a new class, you get that +3 to these new class skills.
This isn't reality it is a wargame so yes you somehow forget 'old' class skills because if you didn't you would be getting something for nothing and every sleaze character will have those +3's for every skill that they have any ranks in.
The exact wording there is an inheritance from D&D, where class skills didn't get a +3 and instead cross-class skills cost 2 skill points per rank. If you were a ranger who took a level of fighter, during that level survival would be a cross-class skill rather than a class skill. If said ranger/fighter had a trait which made survival always a class skill (Paizo made campaign traits before they made Pathfinder) then survival would be a class skill for that fighter level.