You can't take a weapon they are grabbing with Sleight of Hand or the Steal maneuver, but you could take a weapon they have not yet drawn.
Taking a weapon from someone's hand requires an unarmed disarm (edit: or a whip with that one feat whose name I'm blanking on, but which is probably relevant for a follower of Calistria).
Okay yes you can, however there are a few things you need to consider.
First there are two mechanical ways of doing this as a chassis. Two weapon fighting (or equivalent) or high bab.
To do this with high bab you will need at least 6 bab to get two attacks. One will be at +6(+whatever other modifiers) the second will be at +1(+whatever other modifiers again)
To do this with two weapon fighting you almost absolutely must have the feat two weapon fighting. You can try without, but the penalties are horrendous. Assuming the feat You will have two equal attacks at -2(+bab and whatever else)
Now one of your attacks is normal, but as I'm assuming you will have a dex based build you will probably have weapon finesse for that. However to get your dex on the steal attempt you will need agile maneuvers as well.
Furthermore as steal is a combat maneuver if you do not have improved steal (and combat expertese as it is a prerequisite) you will provoke an attack of opportunity (witch you do not want unless you have a truly exceptional AC, which as a rouge or rouge-like you probably do not).
The least feat intensive way to do this is wait till you have 6 bab, and take combat expertise and improved steal.
The most immersive way to do this would be to take weapon finesse, agile maneuvers, two weapon fighting, combat expertese, and improved steal.
If you are playing a new rouge you can do this relatively quickly, with relatively little pain. Be a human
lvl 1: rouge finesse, two weapon fighting, agile maneuvers.
lvl 2: combat expertese (weapon trick rouge talent)
lvl 3: improved steal.
and you are fully up and running at level 3 worst case scenario.
Oh sorry not steal, but disarm. You take a -4 penalty to the check for using your hands, but improved disarm gives +2, so you have an okay chance of sucess if your dex is good.
First there are two mechanical ways of doing this as a chassis. Two weapon fighting (or equivalent) or high bab.
Er, or you could just be a Cutpurse which literally gets an ability called "Stab and Grab." He was just asking whether or not the free Sleight of Hand check to pick pocket them could be used to steal their weapon (and no, it can't, unless it's still sheathed).