|
This is springing off from my thread about increasing GM recognition; I decided it was too derail-y so I started a new one.
A few reasons against GM feedback that I recall from previous discussions:
A player's bad experience may be tied to not liking the scenario and not to disliking the GM's performance. Nonetheless, the GM gets a bad rating for an unenjoyable experience.
A player's character dies in a scenario, and he takes it out on the GM's rating.
One or more players created a disruption at the table, generally ruining the experience for everyone. The GM may have been able to mitigate it but not stop it completely. The GM gets panned for circumstances that are partially outside her control.
There was no consensus on whether the GM would receive feedback directly or have to just watch as her rating goes up or down without knowing exactly why. Direct feedback risks quantifying what could otherwise be a low-pressure session of "Hey, I had a few thoughts about a recent game you ran."
Not all volunteers are comfortable receiving quantitative feedback. At that point we'd have to consider an opt-in system or the like.
Overall there are still concerns about how such ranking would be implemented and reported. If it's in the hands of the GM doing the reporting, is there the temptation to tinker with the numbers?
[Write your own concerns about jerks here.] GM ranking in this way has proven a very contentious subject. By all means feel free to discuss it—perhaps we'll hit on a good idea—but be aware that folks have debated many takes on it with little that approaches consensus.
Until I'm a tenured professor, my entire life is based on student ratings, which is subject to many of the same vagaries you've listed here, so maybe I'm just jaded. But those all seem like things that should theoretically wash out in the average over time.
As for the rating vs feedback, that's a much more interesting concern. Speaking again from the metaphor of student evaluations, the more questions you ask, the fewer responses you'll get. To the point where ~20% of my students have just waited weeks for their grades to unlock instead of just filling out the eval to find out what they were immediately. So if you don't want a simple "How many stars would you give this GM," you're immediately going to cut out feedback from everyone who doesn't particularly care, and only get answers from people who feel very strongly about something. That might seem a good thing, but it means you'll only get extreme feedback, which can be overly inflating or overly deflating to the ego (depending).
The idea of qualitative feedback--"How do you feel about this GM in 250 sessions or less" seems to open up a whole pile of problems, not the least of which is the ability of players to use that to say horrible things to GMs who, for instance, killed their character. On the other hand, quantitative feedback could be as simple as putting a column in the Sessions list that says "Rate this GM: *****" If you want to make sure people aren't rating the scenario, put a second column next to it: "Rate this Scenario: *****" Even if you just ignored that data, it would hopefully filter out people who were just pissed off about finding themselves playing ... oh ... On Shadow's Door, for instance. :P
|
I did a quick search on the old thread, as I haven't read it, and I couldn't find anything concerning LG or Living Greyhawk. I know that there was a judge rating system there, but it may have only been for Nyrond judges. I also started playing in the waning year or so of the campaign, so I don't consider myself qualified to speak on it.
Perhaps someone could speak about how it did or didn't work?
|
| 2 people marked this as a favorite. |
I don't understand why this needs to be so complicated. It is really very simple. If you don't like a GM don't play with him. The worst thing that can happen is that you don't know anything about a particular GM, you play with him once, realize you don't like him and never play with him again. So in this example at most you have wasted 4 hours of time. Wasting is probably a harsh and exaggerated turn but even if its the case bid deal. There are plenty of GM's I choose not to let run games for me. I just choose not to play with them, I feel no need to nerd rage and rant to anyone who will listen about why I don't play with particular GM's.
Even if there was a GM rating system, what does one propose other than self regulation as I described? Should we ban GM's who get negative ratings?
This is a novel concept but not necessary. It's hard enough to get people to take the time to prep a game and run it without having them worry about people rating them negatively.
No change is needed to what we have now, people just need to make their own choices.
|
|
Back in the days of the RPGA organized play (which included Living Greyhawk among other titles) there was a judge rating system.
I'm another RPGA grognard; excellent summary of the issues with that system.
A few more observations I'd made on the rating system, back in the day:
- If you had a GM who frequently ran tables for a group of his or her friends, that GM would wind up with a very high rating, because his friends would very likely always give him or her a very high score.
- At least IME, it was pretty rare to see GMs get poor ratings. Part of the idea behind the rating system was to give GMs feedback (in the hopes of making them better GMs), but unless the players gave written feedback as well, the rating system didn't give the GM a lot to work with.
- It could be diffcult to tease out the performance of the GM from the adventure itself. Even a good GM can be hard-pressed to make a silk purse out of a sow's ear, particularly in a campaign which stresses GMs run the adventure as-written. One could easily picture a GM who is assigned to run the same adventure throughout a big convention like GenCon, only to discover that the adventure is poorly-written -- and it could then result in the GM receiving a bunch of poor scores for things that are beyond his control.
RPG Superstar 2010 Top 16
|
| 1 person marked this as a favorite. |
Patrick, I'd like to ask, what purpose do you think this would serve? Would it make fairer GMs, or more player-pleasing GMs?
I'm on record that the players are in the worst position to comment objectively on a GM's performance. Or, rather, any player who knows that she's supposed to evaluate the GM after the session, and takes that seriously, never jumps into immersive role-play with both feet.
You have student evaluations, true, but aren't you also evaluated by a dean, or a department head, or some other administrator, who asks to see your lesson plans, sits in your class observing your rapport, the attention you command, and how closely you stick to that plan?
(Yeah, I know: it's academia; that level of professional evaluation is not a given.)
Do you think that the teacher/student relationship is sufficiently analogous to the table judge/player relationship?
I'm also on record that a second judge, standing well off to the side, observing me run a scenario that he himself has already prepped and run, would be a valuable resource. That's the analog to the peer review or dean review.
Now, that observation analysis doesn't come with a quantitative evaluation, and I don't think it should. I don't need to know I'm a "3" on organization. I need someone to look at how I run a table and suggest that I print out individual pages from the bestiaries.
|
|
| 1 person marked this as a favorite. |
I do not trust a rating system becuase I do not trust the people to know how to rate something. Also there is way too many petty people to make something honest
When I look over these boards I see the ignorance, stubborn disagreement, and a hatefull responce to honest differences.
I have seen these things in person too, but it is masked and I feel much more treacherous as people plot against your characters.
This rating system seems just another means to abuse people.
|
|
It's easy enough to tease out when you run it yourself later and realize all the stuff that the GM changed either truly accidentally or "accidentally".
Certainly, but that doesn't help the players at the table who just had a rotten play experience, and now are being asked to rate the person who just GMed that game.
|
|
I don't understand why this needs to be so complicated. It is really very simple. If you don't like a GM don't play with him.
Not much of a choice there. What if it's a small venue and there is no other GM, or all the other tables are full?
-Play with a sub par GM.
-Don't play
-Be the GM you want to see, and provide a great experience for someone else. But you still don't get to play, so that doesn't really solve the problem.
|
On the subject of getting poor feedback not because of how it was GM'd but because of the Scenario.
A rating system that was linked to the Scenario played would actually be giving feedback on the Scenarios as well by looking at the average ratings players are giving all GMs for the same Scenario. This could then be used as a kind of a Curve.
One could also use a Curve system for feedback the same players are giving to different GMs. If a player gives all GMs bad feedback then perhaps it's not the GMs.
My original idea however wasn't a direct rating system like a 5 star system. I was thinking of more of a point system. The lower you get rated the slower you gain points, you still gain points. The purpose behind this was linked directly to the previous threads main topic of rewarding GMs beyond the current Stars for games ran capping at 5 Stars.
|
|
Just out of curiosity, has anyone actually polled the community to see if there are any CSM/survey people around? There are a lot of pitfalls in setting up ratings or feedback systems, and someone with experience in the field would be able to point those out and help work around them. I had some limited experience at a satisfaction measurement firm some decades ago, just enough for me recognize how much I don't know.
The main thing I did learn is that you need to understand what you want to find out and what you want to do with that information. I've had a lot of managers set up content surveys where they just start writing questions without taking the time to decide what they are actually trying to learn. It's really very easy to write a good question with a clear rating scale. It's much harder to write the exact question that provides the specific information you need.
If you're lucky, someone on these boards has enough experience in PSAT/CSAT surveys to work out the issues for you. If you're not lucky enough to find someone who knows what they're doing, you might re-consider instituting a ratings system. (Just for clarity, I don't qualify as "someone who knows what she's doing" in this area. I'm only someone who knows that she doesn't know what she's doing.)
|
|
Just out of curiosity, has anyone actually polled the community to see if there are any CSM/survey people around?
I'm a market researcher -- I've been writing surveys for 25 years. However, that's primarily been in product testing and concept testing, not in customer satisfaction, so while it's not exactly what you're thinking of / looking for, I do have a lot of experience with taking clients' lists of "what we want to learn", and translating that into survey questions which can get at that.
|
|
I asked a sociology professor friend of mine about a feedback system in 2012 when the subject has its last go-around on the boards. Here's a snip of what he said:
Here are some items I thought of while looking through your list of general topics [Preparation =1, Storytelling =2, Role-play =3, Rules =4, Fun =5]:
Opening statement: Using this sheet, please evaluate the GM's performance for this session only. On a scale where "0" represents Strongly Disagree and "4" represents Strongly Agree, please indicate in the grid below how much your agree or disagree with the following statements. Please be honest in your assessment of this session's GM so we can work to improve your PFS experiences in the future. [Note: this section should also include some info about how their responses will be handled and reported once that is decided.]
Response Categories: (0) Strongly Disagree, (1) Disagree, (2) Neither Agree or Disagree, (3) Agree, (4) Disagree (5) Strongly Agree [Note: five response categories work well for evaluations as they are like a GPA/Grade scale: A(4) through E(0). Means are roughly equivalent to GPAs.]
Question Items:
Our GM was well prepared for this game session. (Topic 1)
Our GM appeared to have read and understood the module before running our session. (Topic 1)
Our GM did not seem to have familiarized him/herself with module before running it for us. (Topic 1 - reversal)
Our GM did not appear to be ready for this game session. (Topic 1 - reversal)
Our GM was prepared with adequate resources (maps, minis, props, handouts as needed) for our game session. (Topic 1)
Our GM did not present the plot and/or story of the module in a way that was easy to follow. (Topic 2 - reversal)
Our GM was good at presenting the plot of the module. (Topic 2)
Our GM was able to make the plot of this module easy to follow. (Topic 2)
Our GM was able to make the plot of this module seem interesting. (Topic 2)
Our GM did a good job of role playing during our game. (Topic 3)
Our GM did not role play well during our game. (Topic 3 - reversal)
Our GM made the enemies and NPCs in the module feel real and gave them depth. (Topic 3)
This session felt more like a dungeon grind than a role playing game. (Topic 3 - reversal)
Our GM appears to have a good grasp of the game rules. (Topic 4)
Our GM does not appear to have a very good understanding of the rules of this game. (Topic 4 reversal)
Our GM made significant mistakes regarding the rules and/or mechanics of the game. (Topic 4 - reversal)
I was quite happy with the way our GM applied the rules during our game. (Topic 4)
Our GM made playing this session fun. (Topic 5)
Our GM was able to make playing this module quite enjoyable. (Topic 5)
Playing this session was not at all fun because of the way the GM ran the game. (Topic 5 - reversal)
The GM was great at helping all of us have a good time. (Topic 5)
Overall, I did not have fun playing this module because of the way the GM ran the game. (Topic 5 - reversal)
=======================================================
Multiple items measuring the same topic, if included in the evaluation, can be summed or averaged to create a scale measuring the same thing (i.e. Rules) in somewhat different ways. Multiple measures of the same thing often create a more robust measure and become less subject to influences of specific wording in specific questions.
|
|
I don't understand why this needs to be so complicated. It is really very simple. If you don't like a GM don't play with him.Not much of a choice there. What if it's a small venue and there is no other GM, or all the other tables are full?
-Play with a sub par GM.
-Don't play
-Be the GM you want to see, and provide a great experience for someone else. But you still don't get to play, so that doesn't really solve the problem.
Yeah, but even if you had a rating system and the system led to the extreme of banning bad GM's then what?
-Can't play because there is no GM
-Be the GM you want to see, and provide a great experience for someone else.
It's lost on me as to what benefit said system would provide. If your looking for something a GM can use to make positive changes and this is a sincere motivation I would suggest speaking to the GM about problems so he can make changes like an adult. If he would get mad at a face to face suggestion I am sure an anonymous poor rating would not do much to fix the problem.
|
We could have endless debates about how to implement a good rating system (I like simple ratings like Ebay: positive, negative, neutral), but the real question is "What do you do with this information?"
1) If you're at a convention, it's not like you get to pick and choose your GM. The information is irrelevant.
2) If you're local, everyone knows your local GMs and their strengths and weaknesses. You don't need a rating system to point it out. Also, either you can avoid GM X or you can't. You might not have a choice.
3) Maybe the only thing that the rating system could be used for is to allow Paizo to pick GMs for Gencon with more confidence. But even then we already have a rating system, GM stars. Stars show experience and like anything, the more you do it the better you become. Also, they're very aware of feedback on GMs.
You need an important reason to have this information. Mainly, I can see it used as a bragging tool or perhaps as a way to judge other forum members, and likely the numbers would be completely meaningless (because of concerns other people have already mentioned).
The questionnaire rating system looks good, as a check box rubric grid easier to use. What if there was a gm mentoring program like there is for master gardeners? Build the capability don't just shoot at its lack.
|
|
I think most of you are spot-on with your assessment of the issue.
-GM feedback only helps GMs who care about quality improvement. Of the GMs in your area, how many fit that description?
-In many local areas you don't have much choice about your GM. You can play or you can sit out. Beggars can't be choosers.
-Paizo isn't going to back a GM feedback system. The cost versus benefit of creating, implementing and maintaining such a system isn't worthwhile.
-The feedback that I'd be most interested in would come from my peers, GMs who have run the same scenario multiple times before. Although player feedback would stoke my ego, 5-Star GM feedback would be more insightful and critical.
|
|
It would have been cool if there was a campaign rule that in order to get your 5th Star you would have to GM a table of challenging players in front of a select panel of senior GMs. You could only attempt it once a year. The panel criticizes you afterward then votes on your 5th Star. In the event of a tie you must lipsync for your life.
If you were to be judged by a panel of GMs, who are the five GMs you would choose?
|
RPG Superstar 2010 Top 16
|
If you were to be judged by a panel of GMs, who are the five GMs you would choose?
So, my categories are pretty solid, by my examples are just the first five people I thought of. A half hour from now, I likely would have picked different examples in some categories.
Someone really good at rules: Kyle Baird
Someone really good at story-telling: Bob Jonquet
Someone really good at flavor and characterization: Tracy Windeknecht
Someone with a lot of experience GMing: Doug Miles
Someone with experience playing under a lot of GMs: Pirate Rob
|
|
I think Doug you had a great breakout of questions. Really, it was very comprehensive and I can see its value. I would love to see this feedback and I might implement it personal. I think your right on about the assessment of the issue. Especially about Paizo not having a strong motivation to do it. That got me thinking...
If people want something like this why don't they do it on their own? There are plenty of sites on the internet that rate products and services that have no affiliation with the product or service. I think I remember one similar that did this for college professors... If people want this so badly I would suggest putting the time in to create it, promote it, and use it. Let the free market decide if its a tool worthy of use. If it is then problem will be solved, you will have all the feedback you ever want on a GM. If it doesn't catch on then you have wasted a few hours coding something and get a free lesson on economics.
Win/Win!
|
Pathfinder Society Campaign Coordinator
|
If you were to be judged by a panel of GMs, who are the five GMs you would choose?So, my categories are pretty solid, by my examples are just the first five people I thought of. A half hour from now, I likely would have picked different examples in some categories.
Someone really good at rules: Kyle Baird
Someone really good at story-telling: Bob Jonquet
Someone really good at flavor and characterization: Tracy Windeknecht
Someone with a lot of experience GMing: Doug Miles
Someone with experience playing under a lot of GMs: Pirate Rob
John and I are both 5 star GMs and you didn't choose either of us. You are fired for 24 hours, Christopher! ;-)
|
RPG Superstar 2010 Top 16
|
Lamontius, that's possible; I've videotaped myself when I was a classroom teacher. It can be eye-opening. (I take *how many* minutes to get paperwork filled out?)
But it's hard to make useful suggestions to solve your own problems. If you knew how to do that, you wouldn't have those problems in the first place.
|
|
Kyle has lots of videos of himself, most of which are available at the stores no one ever acknowledges they go into.
Kidding aside, Kyle's a GM from whom I learn something every time. So is Chris Mortika. And that applies even when I drift by a table they're running while mine's on a break or between slots.
|
|
Lamontius, that's possible; I've videotaped myself when I was a classroom teacher. It can be eye-opening. (I take *how many* minutes to get paperwork filled out?)
But it's hard to make useful suggestions to solve your own problems. If you knew how to do that, you wouldn't have those problems in the first place.
Chris, it is pretty apparent to me when I have had a good session and when I have had a bad one. I don't make useful suggestions to myself as if I am some audience of one, but I do go hunting for answers and advice in order to get better.
I listen, I learn and I watch others. I do not need a % score or a rating system. I have far more experienced GMs to draw advice from and regular players who are not afraid to speak their minds without having to cart around a report card for my hobby.
|
RPG Superstar 2009 Top 32, 2010 Top 8
|
*shrug* None of us* are paid to GM. While I might like some feedback on GMs, I don't really worry. When we post events, we list who the GM is, and I'll let that play out. I can think of a couple people in Columbus who would't want to sit at my table for personal reasons (and in return I try not to intrude on their fun) and a couple GMs I really like running tables. A couple not so much. I'd not want to say "X sucks as a GM." unless lots of people have problems with X. I don't know if peer review means anything here. To me peer review is 'how many people want to play at my table?'
|
RPG Superstar 2009 Top 32, 2010 Top 8
|
Matthew,
I agree with everything you say. But I also want as large a toolbox of options as I can cobble together. After playing at my table, Kyle Baird has made some suggestions that have helped my style and made the game more fun for subsequent players.
Oh, I am all for improvement. I am just reading this more as a 'peer review/rating' rather than a 'Ever thought of doing this?' I'm all for David Bowles* saying to me after a game, "Matt, you need to get the alchemist rules down a bit more." Him rating, "Matt was a $&$#&$& because he didn't know the alchemist rules" (or just being a jerk) on some kind of scoreboard, not so much.
*
Someone with experience playing under a lot of GMs: Pirate Rob
Did you just call Rob a ...?!
|
|
Did you just call Rob a ...?!
Someone with experience playing under a lot of GMs: Pirate Rob
I think he's referring to my 164 PFS sessions played (counting modules/APs as 1 session) with very little repeat in GMs, I would guess one or two GMs with above 5 sessions with me, with nobody above 10 and most only GMing 1 session for me. and the following varied locations:
Oakland, CA (Endgame/It's your Move)
San Francisco, CA (YetiZen)
Concord, CA (Black Diamond)
San Rafael, CA (GameScape North)
Castro Valley, CA (Crazy Squirrel)
Santa Clara, CA (Game Kastle)
Tracy, CA (I forget the name of the store)
Manteca, CA (Alluring Treasures)
Fresno, CA (Crazy Squirrel)
Cupertino, CA (Legends)
Alameda, CA (Azmyth's Place)
Vacaville, CA (Solano/Yolo PFS)
Burlingame, CA (KublaCon)
Sacramento, CA (ConQuest SAC)
West Des Moines, IA (Iowa Gamers Association)
Des Moines (Local VC's house)
Johnston, IA (Fields of Honor)
Omaha NE/Council Bluffs IA (Nuke Con)
Iowa City, IA (Fellowship of the Blade/Gamicon)
Davenport, IA (Xeno Con)
Denver, CO (Enchanted Grounds)
Seattle, WA (PaizoCon)
Madison, WI (GameHoleCon)
Google Wave (Run by Netherlands VC Auke)
TTopRPG + Skype (Thanks BlazeJ/Pygon)
Rolld20 + Skype (Road to 20!)
Paizo Forums
That said: I'd totally give Mike or John my seat at your review board. I think their unique inside view is even more valuable.
