This is not about D20/Pathfinder. It's about a game I have in development called D6 Dramatics.
A roleplaying game is not a simulation. It's a reward mechanism and incentive system for specific kinds of play. Not all games had their 'play style' reward chosen consciously - see D&D for an example. D&D is ostensibly about Robert E Howard-style Conan stories. What it rewards is like Munchkin without the sense of humor.
(This isn't a slam on either D&D or Munchkin. I enjoy D&D and still play Munchkin when nothing else is available...)
Mechanical Rewards For Roleplaying
People who like 'old school' RPGs tend to say "We don't need no rewards for roleplayin', it just happens." I've heard this argument before, and I'm going to quash it here. You will get the roleplaying that exists in the Venn diagram overlap between what your system mechanically rewards and what your game master and fellow players reward. This does mean that for some groups, roleplaying just happens. These people could also probably run a campaign about Pride & Prejudice & Zombies using a rulebook that alternates every even numbered page from Amber Diceless and every odd numbered page from RIFTS.
Not everyone is lucky enough to be in that skilled a play group.
The reasons most old school gamers give for not incentivizing roleplaying is because it means the blabbermouth player gets more XPs than everyone else - it's a reward for scene hogging, and for slowing the game down when we could be killing more things to take their stuff. It also leads into charges of GM favoritism. We'll address all of those concerns in this post.
The Basic Types of Incentive
Boost Average Roll: The first type of incentive is to do something that increases the average result of a die roll - "Yeah, get +2 on the die roll for that cool description" is one example. One of the reasons for doing roll-and-count is that you can make incentives that improve the average die roll without increasing the maximum possible die rolls. (This can still be done with roll and add systems by turning them into roll X, keep Y systems, but it's more cumbersome.)
Boost Maximum Outcome: This type of incentive almost always increases the average die roll, but also changes the maximum possible outcome. The "Yeah, get +2 on the die roll for that cool description" does this in a d20-style game. In D6 Classic, this is using a Character Point or a Fate Point.
Having a system that can do both types of incentive - and knowing that they're different - means you can incentivize different kinds of behavior with specific bonuses.
Rewarding Player Behavior
In D6 Dramatics, a normal skill roll is expressed as a die code - 4D+1. You roll 4 six sided dice and count any dice that come up as a 5 or 6. You add +1 to the lowest die rolled. If you give a more than perfunctory description of what you're doing, your dice succeed on a 4,5 or 6. This is called an "Upshift" and is an example of a bonus that increases the average roll but doesn't increase the maximum output.
One thing people sometimes ask me is "You have Upshifts. Where are the Downshifts?"
There aren't any - Upshifts are meant to encourage players to be more descriptive about the cool things their characters are doing. Upshifts reduce the chance that you'll outright fail, but they won't increase the maximum possible number of successes you can get.
The other thing that Upshifts let me do is avoid special rules. The classic example is the player who has lots of knowledge about guns trying to use it to argue for special case rules in combat. Here, we can let him use that knowledge to describe how he's setting up the shot, and let him show off the knowledge he has...in return for making an Upshift that's entertaining for everyone else at the table.
Rewarding Character Action
In D6 Dramatics, every character has to have three goals defined, and the party will likely have a Mission defined. Goals are rated in dice, and cap out at 4D.
Whenever a character works towards a goal, it goes up by 1D. Whenever a character works against a goal, it goes down by 1D. A player may convert Goal Dice into skill points at 1:1 at any time using the normal West End Games Star Wars D6 rules. Once this is done, the skill points must be spent immediately, and you can't convert them back.
There are a few other things that players can choose to do that reduce Goal Dice, most commonly powering special abilities. So far, they sound a little bit like Character Points from classic D6, and they are - you just get them for doing things important to your character rather than just showing up and getting 2 for being a warm body eating chips.
However, unlike Character Points in D6, when you're rolling dice on a task that pertains to your Goal, you roll those Goal Dice in addition to your Skill dice - make them a different color set of dice. Goals are never Upshifted (it proved to be too powerful in playtesting). The +1 or +2 on Skill dice only applies to the Skill dice - never to a Goal die. This keeps the bonus of the +1 or +2 from being diluted by being used in too large a die pool.
This applies to every single roll in that scene so long as you're working towards your Goal. It's not a case of "OK, I burn a character point for an extra die, once." It's "My name is Inigo Montoya. You killed my father. Prepare to die." However, you only get this massive bonus when doing something matching one of your character's Goals.
This also replaces Fate Points.
This means that someone with 6D+2 and a full up Goal is rolling 6D+2 and 4D and counting successes. Which means that 'if it's important to your character, it's likelier to succeed'. And in the event that someone manages to get all three Goals bearing on a scene (it's likely the climactic thriller with the Fate of the WOOOOORLD at stake...) they're throwing an extra 12D at the scene...
But wait, there's more!
If you complete your Goal, it automatically converts to skill points at 1:2. One Goal Die becomes two skill points. Not only do you get more powerful for pursuing your goals as single mindedly as fictional heroes tend to, but completing them gives you more skill points.
I mentioned Mission Dice. Your group can come up with a Mission. It can go to 6D. It works like a Goal Dice pool set - except it's going to be at the same value for everyone. If at least two people in a scene doing something that furthers the group mission, they both get the Mission Goal dice to roll.
If anyone works against the Mission Goal, it goes down by one die for everyone as well...which tends to curtail the "Well, I'm the party thief. Why did you expect to still have underwear after going asleep near me?" types of ass-hattery.
When a Mission is completed, it converts at 1:1 into Attribute Points for each player. These can be spent to raise Attributes, or converted at 1:2 into Skill Points (which nobody does because the conversion rate is awful). You can't use Skill Points to raise Attributes.
This means that over the course of a campaign, Skills increase, but Attributes tend to not do so as quickly.
Goal and Mission dice also make the game master's life easier. They give the game master a road map of the things that ostensibly interest the players in the game. (If you pick a Goal you're not interested in, convert any dice into Skill Points and re-set it to something you ARE interested in.) Words cannot describe how much easier this makes organizing a plot or an adventure, particularly when combined with another tool I use.
Disadvantages & Points Mongering
Lots of games have players giving their characters Disadvantages to get more points for things they want. This leads to powerful behaviors, like taking "Gets violently space sick in zero G" in campaigns set in 7th century Arabia....
In a lot of those games, the number of points you get for the Disadvantage represents several sessions worth of experience point rewards - and they become something the GM has to track and remember. When you have 6 players each with 7 disadvantages, that's 42 bits of data you have to integrate into your plot. And if you don't remember, they got the points 'for free'.
My solution was to make Disadvantages not pay anything at character creation, but pay out one Goal Die every time they came up during play. And to limit the number of Disadvantages to 3 per player. Suddenly, players started picking Disadvantages that they WANTED to see come up, so they could get the reward for it.
Schadenfreud as Game Mechanic
Once I integrated Disadvantages and the Goal Die mechanic, I thought of something else: Normally, when players roll dice, the game master describes all outcomes. "You hit..." "You missed" "The lock opens..." "The Duke appears to be listening to your proposal..."
What happens if we move some of that descriptive burden on to the players? The GM is already describing the scenery, the NPCs, the results of successes...let's lighten the load a bit. Let's have players describe how they failed.
Even better, if a player described how they failed in an interesting fashion, let's give them a Goal Die to put wherever they want.
Hmm. If the GM decides it's interesting, that can lead to favoritism charges. How about letting the other players decide if that failure was sufficiently entertaining, and allow the GM to second a nomination?
It also allows players who fail on a roll and who are quick on their feet, mentally, come up with a backup plan that integrates into the failure description - and get a second shot at it.
What this does is lighten the GM's load a little bit - he's not doing all the talking. It also means that failure becomes as dramatically interesting in the course of a tabletop roleplaying session as it is in novels - rather than what usually happens, where it's a showstopper, or something glossed over.
Indeed, we can even combine this with the Disadvantages rule - you get one Goal Die every time the GM sicks the Disadvantage on you. If you make your life more interesting in a Chinese sense, other players might nominate you for more of them....
I'll argue the idea that old-school gamers don't like roleplaying. They simply allow it to emerge from play, rather than being the purpose of play. The way to find traps in a dungeon is not to make Search rolls, but to imagine yourself in the room the DM just described and describe yourself examining the room's features so that you might discover the trap. In RPG circles, that's called "immersion" and is one of the goals of many games.
One thing you might do on the Schadenfreude angle is what Prime Time Adventures does. Their mechanic sets up not only who succeeds, but who describes that result. And depending on how things go, it doesn't even have to be any of the people involved in the conflict.
We have a similar mechanic in Untold called "Swapping". Basically, swapping is the mechanic that allows you to put a card into, or take a card out of play. There are four Swap types, but the primary one is a "Story Swap" and it allows you to change (Downgrade or UPgrade) a card based on the story you tell.
This actually dovetails with the card mechanics and can make certain cards more (or less, depending on the change) effective in the game; thus making your character more (or less) effective at what they were trying to do!
An instant and easy in-game, mechanics "reward" for roleplaying! :D