If you are in a prone position, when getting up do you provoke an AoO?
pretty sure standing up has the move trait, so yes.
I was performing a trip move on my enemy. I rolled my athletics check (which i had a +3 in) and the DM rolled a reflex DC for the NPC (which he has a +8 in).
I rolled a 9 total and the DM rolled a 10(total) and thus I failed to trip him.
Is this the correct way we roll trip maneuvers when using a weapon that doesn't have the 'trip' weapon quality?
Not exactly. You roll athletics, the DM doesn't roll. The enemy's reflex DC is their reflex modifier+10. So if the NPC had a +8 reflex save you need an 18 or higher.
Also, make sure you keep in mind armor check penalties don't apply to tripping or other athletic uses with the attack trait.
As said before, standing up is a move action, so it does trigger attacks of opportunity from creatures that have them, unless you have a feat that states otherwise (mainly, the skill feat Kip Up).
As for tripping, DCs are static, set to 10+ whatever the bonus is. So in this case, you would have to roll your athletics check, and you would need to beat the DC, which is 10+ the target's Reflex save bonus. Therefore, you would have needed an 18 to trip that NPC. It would work the same if your weapon had the trip ability, with the difference that, in that case, you would have added its magic bonus to the roll. Also, trip weapons can be used to trip (otherwise you need a free hand), and if you crit fail, you can drop the weapon instead of falling.
Edit: I got ninja'd. Good point about the check penalty. I sometimes forget that.