I though there was a rule that let you Recall Knowledge at a -2 Penalty which stacked. Was I just thinking up this rule up?
You might be thinking of this:
Sometimes a character might want to follow up on a check to Recall Knowledge, rolling another check to discover more information. After a success, further uses of Recall Knowledge can yield more information, but you should adjust the difficulty to be higher for each attempt. Once a character has attempted an incredibly hard check or failed a check, further attempts are fruitless—the character has recalled everything they know about the subject.
That's not quite the same thing since it's changing the DC.
The only -2 I can think of for RK is Diverse Lore, as that imposes a -2 on your check with Esoteric Lore.
Are you thinking of the Additional Kniwledge rule, which specifies that if you make further recall attempts about the same subject (for further information), the DC increases each time? That's the same one that also says after a success on an Incredibly Hard check or after a failed on any of the checks you can't learn more.