RPG Superstar 2014 Top 32
|
BNW, that makes sense.
But then, a Season 1 scenario with a lot of deeper darkness didn't get singled out, and I would imagine that it might have been a prime candidate for that criticism.
That scenario comes loaded with a way to negate the deeper darkness, so it's less of an issue. In that particular one, conventional solutions to deeper darkness don't work, if I recall.
... I absolutely won't review an adventure if I have not GMed it as I don't know whether problems I experienced as a player came from the author or the GM.
Agreed. At least three times my impression of a scenario as a player was significantly different after I had read it. A really great GM made a ho-hum scenario really memorable and twice I thought a scenario was a dog and it was really just the GM not knowing what he was doing.
Thanks again Kyle. I like this. I will try to force myself to write some reviews.
I do find it to be thought provoking that I didn't like some of the highest rated (4-09 The Blakros Matrimony and 3-I1 First Steps—Part I: In Service to Lore) and did like some of the lowest rated scenarios (4-23 Rivalry's End and 3-I3 First Steps—Part III: A Vision of Betrayal).
|
Venture-Captain, Germany–Hannover
|
I treat reviews always carefully, but quite often the language used betrays certain intentions or attitudes.
Writing reviews is something i thought about for a long time now. Especially how to write them and by what criteria to rate them. I´m working on a catalogue there that i can share once i fully decided on it. Suggestions are of course welcome then or now^^
I found that drawing good maps or printing them or having the right flip mats can be very important sometimes, just as communicating special features of the map clearly.
Deeper darkness is a very cool spell and there are more than enough ways to counter it. Two races now not available anymore have ways to counter it, there is a feat to mitigate the consequences and spells.
If you ask me: Deeper Darkness? Yes please, more and often^^
|
|
The only thing worse than no data, is bad data. And with the small number of reviews on most scenarios, this data is BAD. Drawing conclusions from it is pointless (which of course won't stop us from discussing it... ;)
If you have reviewed some scenarios you've played/GM'd but not others that you've played/GM'd, you are skewing the data pool because you are selecting certain data points to influence. This alone means that every review here is unduly influenced by the extreme viewpoints, as has been noted by several.
I have added "review all the scenarios I've GM'd" to my bucket list to do my part in improving the data. Unlikely to be achieved, but it's a goal.
|
|
The only thing worse than no data, is bad data. And with the small number of reviews on most scenarios, this data is BAD. Drawing conclusions from it is pointless (which of course won't stop us from discussing it... ;)
If you have reviewed some scenarios you've played/GM'd but not others that you've played/GM'd, you are skewing the data pool because you are selecting certain data points to influence. This alone means that every review here is unduly influenced by the extreme viewpoints, as has been noted by several.
I have added "review all the scenarios I've GM'd" to my bucket list to do my part in improving the data. Unlikely to be achieved, but it's a goal.
I'd argue that this data is not bad in and of itself, the conclusions some people are trying to draw are the problem. The "Top" lists are heavily influenced by how *I* want to interpret the data and am not trying to proclaim that they are independently the top of anything.
In order for a review to occur, a scenario either has to 1) be experienced by some inclined to write reviews or 2) evoke a strong enough response to get someone to post a review who is not normally inclined to write one.
That said, the reviews do influence the direction of PFS. Reviews got a couple of scenarios changed. Reviews have gotten some authors more or less work. Reviews have changed the style of scenarios. All of these are influenced by other data as well (messageboards, emails, personal discussions, etc).
|
Reviews are important for customers to make purchasing decisions (or in our case, decisions to expend time running or playing a scenario).
Reviews should intrinsically not be "bad data" on a site like Paizo's, because people should take action if they see a review is extremely contradictory to their opinion and then weigh in... a scenario shouldn't be able to stay mis-rated for long as we the community correct the incorrect rating.
I'm inclined to think that for the reviews that Kyle posted in his top lists, there are enough data points to round out the overall data and give us some idea of the relative quality of Night March of Kalkamedes vs Assault on the Wound. Which is to say, if you took 1000 random PFS players and GMs and handed them printed copies of the scenario as they walked into a convention center and then performed exit surveys on each attendee ten hours later, you'd have the bulk of folks agreeing that one scenario was better than the other.
I'm working to make an effort to contribute more reviews on the scenarios that I see have under ten reviews, now. One thing I'm realizing as part of that effort... Season 2 was extremely good.
|
If anything this makes me want to review more scenarios. I think this may have been Kyle's secret goal...
"Secret"???
A big part of why I did this (and have done similar in the past), is to inspire people to post reviews. If you see a scenario that you remember well, whether positively or negatively, write a review! Even just a few constructive sentences can make the difference in future scenario design.
|
|
If anything this makes me want to review more scenarios. I think this may have been Kyle's secret goal..."Secret"???
A big part of why I did this (and have done similar in the past), is to inspire people to post reviews. If you see a scenario that you remember well, whether positively or negatively, write a review! Even just a few constructive sentences can make the difference in future scenario design.
You heard him! Secret.
|
With all the data compiled for every scenario from every season, I present to you this. Thanks to everyone who helped!
Top 20 Scenarios (min 10 reviews)
I see from the raw data that nearly half (80) of the scenarios have less than ten reviews and therefore cannot appear in the top/bottom lists (as they are not considered statistically significant).
I would urge everyone who has played or GMd one of these little reviewed scenarios to go vote, even if it is just a one liner with a score. Then you are improving the significance of the review scores where your vote counts the most.
From a personal point of view I see there are 12 scenarios on 9 reviews, 2 of which I have played or GMd and haven't already reviewed. Reviewing those will almost certainly then put them in the top 20 and bottom 10 respectively (to Scale the Dragon / Halls of Dwarven Lore). I will make those my next 2 reviews.
I note also that Eyes of the Ten part 1 is on 9 reviews and likely a top 20 scenario if one of the veterans could review it. Seems odd the number of reviews are so low for this when it is likely the folk most invested in PFS that play it.
I know ten is a relatively arbitrary number for cut off, but double figures is a psychological milestone for credibility.
I have already reviewed 2 out of every 3 scenarios I have GMd, and have now made it a goal to review all of them to remove any bias that may have introduced.
|
|
Your list needs to be redone and EotT needs to be in the top 20 now that it has 12 reviews.
Considering the time it took to compile, please don't expect it to be updated every time a scenario would move up or down the list.
edit: Unless you go give all 5 of my scenarios a 5-star review...
|
|
| 1 person marked this as a favorite. |
Prompted by this thread, I have now reviewed every PFS scenario that I have ran. I feel like a lot of my reviews ended up a bit samey, but...at least I'm caught up.
Only 4 stars for The Confirmation? Heresy I say! Burn him at the stake!
Eh hem. I mean... Thank you for your review. (*grumble*grumble*grumble*)
|
RPG Superstar 2014 Top 32
|
Prompted by this thread, I have now reviewed every PFS scenario that I have ran. I feel like a lot of my reviews ended up a bit samey, but...at least I'm caught up.Only 4 stars for The Confirmation? Heresy I say! Burn him at the stake!
Eh hem. I mean... Thank you for your review. (*grumble*grumble*grumble*)
*chuckles* It's a good adventure! My only problem with the puzzle is...
|
|
Prompted by this thread, I have now reviewed every PFS scenario that I have ran. I feel like a lot of my reviews ended up a bit samey, but...at least I'm caught up.Only 4 stars for The Confirmation? Heresy I say! Burn him at the stake!
Eh hem. I mean... Thank you for your review. (*grumble*grumble*grumble*)
*chuckles* It's a good adventure! My only problem with the puzzle is...
** spoiler omitted **
That's funny. In my only two runnings at Gencon where that one randomly appeared it was the first (and in one case, only) one they figured out. I was starting to think it was a bit too obvious.
I have run it more than 10 times and I find parties ability to do the puzzles is very random.
|
RPG Superstar Season 9 Top 8
|
Good stuff Baird!
Data is always awesome to see. What Paizo (or we, as participants in PFS) choose to do with this data will also be interesting to see.
I was saddened to find that some of my favorite games weren't in the top 10. Then I realized why: they don't have 10 reviews! So I'd like to ask that anyone that's had an experience (enjoyable or otherwise) with the following scenarios to submit a review. They are among my fondest adventures to run, and I'm curious to see how other people feel.
I'll go ahead and lead by example--my reviews for these scenarios just went up, I look forward to reading yours!
|
Thanks for doing this, Kyle. Writing reviews added to my to-do list. Would be curious to see when all of these reviews were written. I'm guessing that most are written within six months or so of release, but I know there are always some that are outliers. Just saw someone review a season 1 the other day!
Also, the theory is that people are more likely to write a review for an very bad experience over a very good experience.
I thought that this would apply to the reviews I wrote, but looking back at them I wrote two 4-star reviews, one two-star review and one review without a star rating at all (because the scenario was so unmemorable!). Weird.
|
|
Prompted by this thread, I have now reviewed every PFS scenario that I have ran. I feel like a lot of my reviews ended up a bit samey, but...at least I'm caught up.
73 reviews! Wow, you are a 3 star reviewer ;-) Now that is leading by example.
I assume that number also means you don't rerun scenarios that often. (I have run under 40 different scenarios/modules from my 90+ tables GMd)
|
RPG Superstar 2014 Top 32
|
Prompted by this thread, I have now reviewed every PFS scenario that I have ran. I feel like a lot of my reviews ended up a bit samey, but...at least I'm caught up.73 reviews! Wow, you are a 3 star reviewer ;-) Now that is leading by example.
I assume that number also means you don't rerun scenarios that often. (I have run under 40 different scenarios/modules from my 90+ tables GMd)
*chuckles* You'd be sorely mistaken. I'm well over 200 tables GMmed, so I've GMmed the average scenario that I've ran around 3 times. Of course, I haven't went through and done the modules yet...
|
|
| 1 person marked this as a favorite. |
Updated after a week:
Added Ruins of Bonekeep—Level One: The Silent Grave
Fixed a couple of scenarios that had incorrect data (don't expect any of this to be perfect please)
170 new reviews in the last week! Thanks everyone!
New review breakdown: 12/12/29/66/51 – So mostly positive, but follows the cumulative breakdown almost perfectly.
Most New Reviews:
6 6-01: Trial by Machine
6 6-02: The Silver Mount Collection
6 6-03: The Technic Siege
5 5-11: Library of the Lion
5 5-22: Scars of the Third Crusade
5 5-99: The Paths We Choose
4 46: Eyes of the Ten—Part I: Requiem for the Red Raven
4 2-15: Shades of Ice—Part I: Written in Blood
4 5-06: You have What You Hold
Given that it's college football season, I'm moving up to a Top 25 KB Poll. After Week 1, here's the standings:
Top 25 Scenarios (min 10 reviews)
Rnk/Scenario/Change in Rnk
1) #2-03: The Rebel's Ransom (+0)
2) #4-03: The Golemworks Incident (+0)
3) #4-19: The Night March of Kalkamedes (+0)
4) #3-02: Sewer Dragons of Absalom (+0)
5) #3-01: The Frostfur Captives (+0)
6) #4-08: Cultist's Kiss (+1)
7) #3-03: The Ghenett Manor Gauntlet (+3)
8) #46: Eyes of the Ten—Part I: Requiem for the Red Raven (NR)
9) #5-08: The Confirmation (-3)
10) #3-20: The Rats of Round Mountain—Part I: The Sundered Path (+1)
11) #35: Voice in the Void (-3)
12) #2-01: Before the Dawn—Part I: The Bloodcove Disguise (+0)
13) #3-21: The Temple of Empyreal Enlightenment (+0)
14) #16: To Scale the Dragon (NR)
15) #5-13: Weapon in the Rift (-1)
16) #52: The City of Strangers—Part I: The Shadow Gambit (+2)
17) #4-09: The Blakros Matrimony (-2)
18) #5-07: Port Godless (-9)
19) #2-21: The Dalsine Affair (-2)
20) #3-15: The Haunting of Hinojai (NR)
21) #1: Silent Tide (-5)
22) #2-15: Shades of Ice—Part I: Written in Blood (NR)
23) #3-I1: First Steps—Part I: In Service to Lore (-4)
24) #3-18: The God's Market Gamble (-3)
25) #5-02: The Wardstone Patrol (-3)
Top 10 PFS Authors (min 2 credits)
1) Michael Kortes (+0)
2) Jason Bulmahn (NR)
3) Jim Groves (-1)
4) Alex Greenshields (+0)
5) Tom Phillips (+2)
6) Matthew Goodall (-3)
7) Crystal Frasier (-1)
8) Dennis Baker (-3)
9) Adam Daigle (-1)
10) Larry Wilhelm (NR)
Season/Cumulative Avg/PI/Reviews per Scenario
0 - 3.50 (0.6) (14)
1 - 3.66 (0.3) (10)
2 - 3.59 (0.6) (8)
3 - 3.86 (0.5) (15)
4 - 3.64 (1.1) (14)
5 - 3.43 (2.1) (16)
6 - 2.89 (0.7) (12)
|
|
The league table is shaping up well. Thanks Kyle. The big movers interest me most and that is currently confined to Port Godless at -9. Given the closeness of the numbers I am surprised that it is the only big mover. Though any scenario that scores 4 or above is a winner in my eyes.
Sigh. I ate scenarios 2 through 6 and I can't bring myself to play a scenario I have prepped/GM'd, though I understand replay for scenarios where you think the GM did not do it justice. On the plus side I haven't played or GM'd number one yet!
|
|
I still need to break this down by tier. I'm open to ideas on how to handle 1-5 vs. 1-7 vs. 3-7. Keep them separate? I wonder if that would slant the data too much based on season since 3-7 didn't exist and only a few 1-5's existed.
If you're sticking with the current scenario tiers, split the 1-7 ratings between 1-5 and 3-7. (Each 1-7 is 0.5 scenario in 1-5 and 0.5 in 3-7.)
I'd find it more interesting if you did "Everything a level (x) character could play." rather than the tiers. Then the 1-7s and the Specials would fit in naturally. (As would modules and sanctioned APs.)
|
|
Surprising level of Parody, er parity across the playable levels:
Avg
3.61 Levels 1/2
3.57 Levels 3/4
3.59 Level 5
3.55 Level 6
3.53 Level 7
3.56 Levels 8/9
3.39 Levels 10/11
Reviews/Scenario
17.3 Levels 1/2
16.8 Levels 3/4
14.8 Level 5
11.4 Level 6
10.1 Level 7
8.1 Levels 8/9
7.5 Levels 10/11
58-60% of reviews are 4 or 5 star reviews across levels 1-9 (54% at 10-11). The only thing that changes as levels go up, is the replacement of 3-star reviews with 1-star reviews.
|
Kyle,
You've obviously done a lot of work here. When I first started selecting scenarios for events, I looked at reviews to try to pick some good ones. But there are two problems with the system:
1) Reviews are voluntary, which may bias them. People with a really good or really bad experience may be more inclined to write a review, or the bulk of the reviews may just be from folks who have the time and inclination to write a review (which may not be a typical cross-section of players).
2) There are far too few reviews in most cases to really make an assessment with confidence. I can send you some reasoning about this; the statistical theory could be a bit lengthy and technical for posting in this forum. The number of reviews is a vanishingly small percentage of the number of times these scenarios have been played.
Some may cringe at the idea of adding a player review survey to the event reporting, but a rating scale selection against a small number of factors probably would not be hard to implement. A test run could even be done at a convention before promoting such an idea further.
The average alone is also probably not a great indicator. Your idea of the polarizing index is a great way to capture more of the "shape" of the responses, and a step in the right direction. Your next step of doing a count of reviews per month will also help explain the data better. The amount of play has grown each season, and I'd expect to see more reviews of newer scenarios for that reason.
There are also some great tools for pivoting and displaying the data with multiple filter, sort and group criteria. As the standard error of the mean varies with the number of reviews, it might be good to group and compare scenarios that have a similar age and/or number of reviews...
|
|
To be honest, Id much rather see player focused reviews than DMs. Reviews from DMs, due to the perspective that DMs get, knowing the backstory, seeing how the rules and suggestions are suppossed to be, is going to give a very different picture, likely boosting scenarios more than they probably should be.
|
|
I disagree 100%. If you haven't at least taken a look at the scenario yourself, and more preferably GMed it yourself, I'd prefer you not to review a scenario poorly because one GM ran it poorly for you.To be honest, Id much rather see player focused reviews than DMs. Reviews from DMs, due to the perspective that DMs get, knowing the backstory, seeing how the rules and suggestions are suppossed to be, is going to give a very different picture, likely boosting scenarios more than they probably should be.
I have to agree with David on this one. I've seen a number of scenarios come out with newer mechanics built in where review after review was of players complaining about something the mechanic did in direct opposition to what is actually part of the scenario...
I'd rather know the person has had a chance to actually see what is in the product before reviewing it based on another's interpretation.
|
|
To be honest, Id much rather see player focused reviews than DMs. Reviews from DMs, due to the perspective that DMs get, knowing the backstory, seeing how the rules and suggestions are suppossed to be, is going to give a very different picture, likely boosting scenarios more than they probably should be.
Yeah I disagree also; my best playing experiences had more to do with who was GMing and who else was playing at the table more than the scenario itself. When the review comes from the GM at least it takes one of these out of the bias formula.
|
Venture-Lieutenant, Netherlands—Leiden
|
When hastily consulting reviews, I tend to study some reviews with the same number of stars as the average (hoping they're representative), and some of the highest and lowest rating reviews, to see why you would really love or hate a scenario.
When I have more time I parse people's arguments, and try to determine whether the reviewer's preferred play style matches mine.
I think the reviews with a breakdown of how they arrived at the score are most useful.
|
|
The only reason that Hellknight's Feast doesn't feature in these top lists is because a bunch of players gave it a 1 star review after playing under a GM who ran it wrong.
That could be said for several scenarios, and it might even be true for some of them. That statement could go a little deeper though. Maybe the GM wouldn't have "run it wrong" if the style of scenario had been different or the information had been presented in another way.
There are so many interactions going on in any given PFS game, it's impossible for us to know which factors are the most significant.
To be honest, Id much rather see player focused reviews than DMs. Reviews from DMs, due to the perspective that DMs get, knowing the backstory, seeing how the rules and suggestions are suppossed to be, is going to give a very different picture, likely boosting scenarios more than they probably should be.
I certainly agree that, as a player, I'm much more interested in reviews from players. Personally, I try to make it very clear what my role was (player vs. GM vs. both) when I review an adventure.
|
RPG Superstar 2014 Top 32
|
The only reason that Hellknight's Feast doesn't feature in these top lists is because a bunch of players gave it a 1 star review after playing under a GM who ran it wrong.That could be said for several scenarios, and it might even be true for some of them. That statement could go a little deeper though. Maybe the GM wouldn't have "run it wrong" if the style of scenario had been different or the information had been presented in another way.
There are so many interactions going on in any given PFS game, it's impossible for us to know which factors are the most significant.
The reviews to which I refer mostly express disappointment at how Diplomacy doesn't work in the scenario.