|
| 25 people marked this as a favorite. |
With all the data compiled for every scenario from every season, I present to you this. Thanks to everyone who helped!
Top 20 Scenarios (min 10 reviews)
4.75 2-03 The Rebel's Ransom
4.64 4-03 The Golemworks Incident
4.59 4-19 The Night March of Kalkamedes
4.58 3-02 Sewer Dragons of Absalom
4.55 3-01 The Frostfur Captives
4.52 5-08 The Confirmation
4.50 4-08 Cultist's Kiss
4.47 35 Voice in the Void
4.40 5-07 Port Godless
4.38 3-03 The Ghenett Manor Gauntlet
4.35 3-20 The Rats of Round Mountain—Part I: The Sundered Path
4.35 2-01 Before the Dawn—Part I: The Bloodcove Disguise
4.30 3-21 The Temple of Empyreal Enlightenment
4.29 5-13 Weapon in the Rift
4.24 4-09 The Blakros Matrimony
4.23 1 Silent Tide
4.20 52 The City of Strangers—Part I: The Shadow Gambit
4.20 2-21 The Dalsine Affair
4.15 3-I1 First Steps—Part I: In Service to Lore
4.14 39 The Citadel of Flame
Bottom 10 Scenarios (min 10 reviews)
2.61 4-23 Rivalry's End
2.55 5-22 Scars of the Third Crusade
2.55 3-14 Wonders in the Weave—Part II: Snakes in the Fold
2.50 30 The Devil We Know—Part II: Cassomir's Locker
2.47 14 The Many Fortunes of Grandmaster Torch
2.45 15 The Asmodeus Mirage
2.43 6-01 Trial by Machine
2.42 5-20 The Sealed Gate
2.39 5-24 Assault on the Wound
2.20 3-I3 First Steps—Part III: A Vision of Betrayal
2.14 2-19 Shades of Ice—Part III: Keep of the Huscarl King
1.83 2-23 Shadow's Last Stand—Part I: At Shadow's Door
Polarizing Index (PI): Compares what % of reviews are either 1-star or 5-stars, how balanced it is between 1-star and 5-stars, and weights number of reviews (more reviews = more potential to be polarizing/controversial).
Top 10 PFS Authors (min 2 credits/low PI)
4.36 Michael Kortes
4.20 Jim Groves
3.96 Matthew Goodall
3.94 Alex Greenshields
3.93 Dennis Baker
3.93 Crystal Frasier
3.89 Tom Phillips
3.81 Adam Daigle
3.70 Craig Shackleton
3.68 Mike Shel
Season/Cumulative Avg/PI
0 3.49 0.6
1 3.63 0.3
2 3.55 0.6
3 3.85 0.4
4 3.62 1.1
5 3.44 1.8
Top 10 Most Polarizing Scenarios
5-05 The Elven Entanglement
4-26 The Waking Rune
2-13 Murder on the Throaty Mermaid
4-02 In Wrath's Shadow
5-11 Library of the Lion
5-24 Assault on the Wound
47 The Darkest Vengeance
5-19 The Horn of Aroden
11 The Third Riddle
4-07 Severing Ties
Top 10 Most Polarizing Authors
Ryan Costello, Jr.
Kyle Elliot
Nicholas Herold
Kyle Baird
Thurston Hillman
Ron Lundeen
Steven Robert
Russ Taylor
Jonathan H. Keith
Tim Hitchcock
Regional Venture-Coordinator, Central Europe
|
I find it really interesting that in season 4 and even more in season 5 the PI just went thorugh the roof.
Makes me wonder why newer scenarios polarize people so much more.
Is it because of higher difficulty and more people failing at scenarios and therefore feeling overwhelmed? Stuff like the first encounters in 5-05 or 5-20 can really crush people who are not optimized to a certain degree.
Than we have scenarios in season 5 where you had to interact peacefully with NPCs who might also be annoying, while just murderhoboing your way through the scenario might also lead to failure.
And of course they are experimenting more with unusual rules additions (mythic rules, mass combat) which are not to everybody's taste.
|
|
It's interesting to me that Season 5 is the lowest overall rated season and the first 3 scenarios in season 6 are even lower yet. I noticed the tail end of season 5 had mostly low rated scenarios.
I personally have liked them but I have noticed gobs of editing errors from 5-20 forward that may be impacting the reviews.
|
|
2.55 5-22 Scars of the Third Crusade
2.43 6-01 Trial by Machine
2.39 5-24 Assault on the Wound
I really enjoyed playing and running those scenarios!
So did my players actually. They had a lot of fun in all of them.
Probably one needs a GM who is willing to do some extra work and provide a lot of roleplay in a timely manner though.
No idea why first steps is so popular. Was pretty boring to me. Tastes seem to be rather different^^
|
|
5-13 Weapon in the Rift
Was an great scenario, congrats Nathan.
It also cool to see how some scenarios are rated, along with others nice top 20 list.
I also think there is a problem with forcing players to play up in older scenarios. I am not sure how prevalent this is right now. But during this last season I GM'd one of the top 20 from season 3. I believe it is a solid 5 star scenario. I had a perfect party that was doomed to fail, based off their level and their APL. First combat PC deaths, what followed was misery for the party. If they were to review the party now, I think it would reach a lower total.
What is surprising now is the low ratings on season 5. I am not sure this is editing or even negatively biased reviews, or even badly GM'd games. I read some of the longer playing scenarios have got bad reviews due to long game times.
The Sealed Gate man that is reviewed bad. Fun Sponge/Classic Baird reviews go back and forth. At Gen-Con I had some local players that played it with Kyle. I have to say they enjoyed it, they actually told me in depth how much they liked it. They also were pretty excited about the whole scenario.
Now these players will likely never write a review about that game!
Was the fact that Kyle being a 5 star GM made it better, or the fact that he was the author make it better?
Do reviews even matter? As a community or author, editor do you read and reflect on reviews? Do you die in a game and blame the author? Do you GM it yourself to see what it is like or how you can improve on it?
|
Congrats Matt Goodall on making it into the top three authors. Australia represent! Now if we could only get more of those talented Aussies developing scenarios! *hint hint Al Rigg hint hint*
I have to admit, I usually discount 1 star scenarios where it becomes clear that the reviewer died and they feel really badly about their death. Having said that, the sheer weight of bad reviews for the latest expedition into the Bairdlands has me very worried.
I also give reviews more 'weight' if the reviewer has both played and GM'ed the scenario.
|
|
This data is really interesting (and valuable), many thanks to Kyle and everybody who helped to assemble it.
IMO, some of the polarization comes from differing opinions on how to review, i.e. differing opinions on what four stars means versus two stars. This is also why I also read the reviews rather than look at the raw numbers, if I have time when trying to decide what to run.
A few examples:
For almost every scenario, I find that there's usually at least one person who died/suffered a TPK and decided to blame the scenario rather than themselves/their party, and at least one person who blew through the scenario with no problems and wanted more of a challenge. That discrepancy is inevitable because dice (to say nothing of variable party makeup/optimization). So I discount those.
There are also folks who will give one star for any scenario involving a chase scene or some other mechanic they don't like, usually with no other comments other than that they hate chase scenes. Which tells me very little about the scenario and its quality, except that it has a chase scene. I generally discount these too.
Oftentimes there's also a positive individual(s) who give a five star review with a number complaints, though these are usually "workable" complaints.
On the other hand, there are plenty of reviews that say "I had a blast, but..." and give a whopping two stars.
To me, a "yes" answer to "did I have good fun?" nets a minimum of three stars. A "no" nets a maximum of two stars. Compression in either direction (towards zero or five) doesn't make much sense to me. I'm looking for a scenario that will provide a fun experience for me and my players. If a scenario does that, then whatever problems it has can be worked around.
(Of course, I have never actually submitted a review, but that's another issue...)
And only because it's related and discussed in the other thread:
I had good fun with this scenario. Was it the best scenario ever? No. It had plenty of problems (railroading, plot gaps, being needlessly complicated for the GM with the tracks, etc.). But all of these problems were manageable. In fact, without them, I probably never would have visited the GM Discussion on the topic, from which I garnered a wealth of awesome ideas. Yes, the railroading could have been reduced by just saying "these events should occur roughly in this order at a time that feels right for the GM/party." And yes, it could have provided all of the missing information, but I'm actually kinda glad it didn't because it offered GMs a chance to improvise a little and create a semi-unique experience, completely legally within PFS. So, all in all, folks had fun with a neat scenario that needed a few tweaks on the part of the GM. I'll take it.
|
For almost every scenario, I find that there's usually at least one person who died/suffered a TPK and decided to blame the scenario rather than themselves/their party, and at least one person who blew through the scenario with no problems and wanted more of a challenge. That discrepancy is inevitable because dice (to say nothing of variable party makeup/optimization). So I discount those.
I have absolutely no problem with the fact that scenarios with an excessive death toll get rated lower. The intent is to have fun and clearly there are more people who find death unfun than there are those who want extreme challenges.
But I don't think it is death that people REALLY dislike, I think it is what are perceived as unfair deaths.
|
|
Well, considering this is only the most reviewed ones, there might be another reality. Most downloaded ones would be interesting for example!
It's not really the "most" reviewed scenarios. The average scenario receives 12 reviews. If a scenario doesn't inspire at least the average number of reviews, how good or bad can it really be? Is a 4.5 rated scenario with 3 reviews after 2 years actually any good? It's certainly not better* than a 4.0 scenario with 20 reviews in its first year.
*better and good are relative to the statistics, not the actual quality of the scenario. Of course something with only a couple of reviews could in theory be a much better scenario, but it begs the question, if it's so good, then why haven't more people felt compelled to express their opinion?
|
|
| 1 person marked this as a favorite. |
(Of course, I have never actually submitted a review, but that's another issue...)
That *IS* a huge problem. Many people feel as though they need to wait to submit a review. Whether it's to run it multiple times, or both play and GM it, or whatever, I argue that it's worth more to write an initial review and explain that you're likely to edit it later.
A big part of why I did this (and have done similar in the past), is to inspire people to post reviews. If you see a scenario that you remember well, whether positively or negatively, write a review! Even just a few constructive sentences can make the difference in future scenario design.
|
|
I have absolutely no problem with the fact that scenarios with an excessive death toll get rated lower. The intent is to have fun and clearly there are more people who find death unfun than there are those who want extreme challenges.
But I don't think it is death that people REALLY dislike, I think it is what are perceived as unfair deaths.
I mostly agree, but I generally find that for scenarios that actually are overpowered against the PCs (rather than an unlucky roll, etc.) the topic of mass PC death comes up more than once or twice. Then I do pay attention. =)
Venture-Lieutenant, Indiana—Indianapolis
|
Correction: There are more people who find deadly scenarios "unfun" who post reviews than those who enjoy "extreme challenges" and post reviews.
This. I have been talking with other GMs and what not, and the conclusion that we reached was that, perhaps, people who don't like a scenario (for whatever reason) are my inclined/likely to write a review (which would be negative) than those who enjoyed the scenario (and thus, don't write the corresponding positive review.)
I think Kyle's point is on the money.
|
|
If I review an adventure, I prefer to have both played it and GMed so that I can see it from both sides of the screen. I absolutely won't review an adventure if I have not GMed it as I don't know whether problems I experienced as a player came from the author or the GM.
I do agree with this. Two of my least fun tables in PFS were because I was playing with a GM running cold who got half the scenario wrong. I won't necessary wait until I run it to review, but if I've had a bad experience playing a scenario, I do make a point of looking up the scenario online to see if my problem is a common complaint, or if my table was just bad because of the GM/players.
That said, I really should review scenarios more often. I almost never do. In fact, I just checked, and I've only written one review - 5 stars for Storming the Diamond Gate (so much for the theory that only people who had a bad time write reviews).
But Kyle's goal in starting these threads has worked on me - I'll try to make a point of writing some reviews. We'll have to go back in a month or two and revisit these stats after everyone who is motivated by these threads writes a bunch of new reviews.
|
|
If I review an adventure, I prefer to have both played it and GMed so that I can see it from both sides of the screen. I absolutely won't review an adventure if I have not GMed it as I don't know whether problems I experienced as a player came from the author or the GM.
I often read them after I've played them. Actual GM experience doesn't matter that much to me. I'm still getting my 1XP/2PP; I don't expect them to be written for GM experience.
|
|
| 1 person marked this as a favorite. |
In fact, I just checked, and I've only written one review - 5 stars for Storming the Diamond Gate (so much for the theory that only people who had a bad time write reviews).
To be fair, one instance does not disprove the theory. Also, the theory is that people are more likely to write a review for an very bad experience over a very good experience.
|
| 2 people marked this as a favorite. |
Looking at this data, it's clear to me that more people need to go submit reviews for Silver Tarn, Green Market and You Only Die Twice. :)
Honestly, I actually started writing reviews mostly for the best experiences I had. I think my 5-star review for Kalkamedes may have been my first.
I genuinely feel bad for the author when I submit low star reviews, and tend to hesitate submitting them until I get some corroborating evidence that a scenario really should be given a low rating (i.e. other local GMs).
|
|
Ok, I just tried to write a review and remembered the real reason why I never write them - I couldn't find the link. Is there some reason that "Write a review" isn't a prominently displayed link so people can actually find it? I did find it eventually, but it took some serious hunting around. This one's really not intuitive.
|
|
I will note that most of the high reviews I've left (and it seems others have left) almost always apply when there are "sandbox elements" to the scenario.
Off the top of my head, this included Disappeared, Blakros Matrimony, Green Market, Silver Tarn, etc.
In each of these scenarios, both the group I played with and later GM'd went off the rails for 1-2 hours with self-directed character roleplay.
For example, I don't know if many of the players at either table remember what really happened during Blakros Matrimony (except maybe the final combat). Most of them just identified that their character was at a wedding, and this allowed them a lot of free form RP time to explore what their character(s) would do at such a venue.
I was playing my nagaji "snake oil" salesman in Blakros Matrimony, and used it as a grand event to push free samples and product on all the guests. This same salesperson eventually set up shop, uninvited, at the center of the Green Market, which is actually what caused Zeeva to come rushing out, "No, no no! What are you doing?!".
So, my advice to authors if you want to snag a bunch of 5 star reviews from some of our local GMs and players: Include a sandbox element in your scenario.
(And then don't undo the opportunity by writing it likes Scars)
|
|
clearly there are more people who find death unfun than there are those who want extreme challenges.Correction: There are more people who find deadly scenarios "unfun" who post reviews than those who enjoy "extreme challenges" and post reviews.
Sure. But that is the data point that we have, the one that you specifically started this thread to discuss.
While its absolutely certain that the people who post reviews form a tiny minority and therefore quite possible that the data is skewed it is absolutely impossible to infer in what way it is skewed. Perhaps the reviewers like challenges MORE than most players and UNDERVALUE death.
Either we decide that the reviews are vaguely representative of overall opinion or we decide they have no value. What we can NOT do (at least, we can't if we want to be intellectually honest) is to accept the reviews when they support our opinions and reject them when they don't.
|
|
Correction: There are more people who find deadly scenarios "unfun" who post reviews than those who enjoy "extreme challenges" and post reviews.This. I have been talking with other GMs and what not, and the conclusion that we reached was that, perhaps, people who don't like a scenario (for whatever reason) are my inclined/likely to write a review (which would be negative) than those who enjoyed the scenario (and thus, don't write the corresponding positive review.)
I think Kyle's point is on the money.
That would tend to drive all review average numbers down. It doesn't speak to relative numbers between reviews
|
|
Well, considering this is only the most reviewed ones, there might be another reality. Most downloaded ones would be interesting for example!It's not really the "most" reviewed scenarios. The average scenario receives 12 reviews. If a scenario doesn't inspire at least the average number of reviews, how good or bad can it really be? Is a 4.5 rated scenario with 3 reviews after 2 years actually any good? It's certainly not better* than a 4.0 scenario with 20 reviews in its first year.
*better and good are relative to the statistics, not the actual quality of the scenario. Of course something with only a couple of reviews could in theory be a much better scenario, but it begs the question, if it's so good, then why haven't more people felt compelled to express their opinion?
I do guess, based on observations here on the board and in "wildlife", that there is some mouth to mouth propaganda about scenarios either being good and fun or having a good chronicle sheet. Of course a lot of that also depends on the GM who runs it and her abilities.
Anyway it´s interesting and informative. So thanks a lot.Writing reviews about scenarios is a fix point on my to do list though!
|
|
I've thought about writing reviews, but there's no obvious way from the scenarios page to submit a review, so I didn't bother spending time hunting for how to do it. If reviewing was more user friendly, I'd do it.
I know the feeling.
If you look on the product page you'll see ***s and under that
Write a review of Pathfinder Society Scenario #5–20: The Sealed Gate (PFRPG) PDF!
If you're just glancing at the page it looks like the first review.
|
|
| 2 people marked this as a favorite. |
Here are a few of my thoughts on the root cause behind the review dissonance in this most recent season. I don't think this is the only cause of the dissonance, but I think it's one of them.
What people have often said about Hitchcock (Alfred, not Tim), is that he had his movies so meticulously storyboarded and planned that anyone off the street could finish them if he fell over dead in the middle of a production. I think that's exactly the kind of mentality that Paizo has started taking, and you see that with the very stringent "run-as-written" paradigm as well.
They want anyone, regardless of skill, flexibility, or knowledge, to be able to pick up a scenario and run it satisfactorily. You start seeing situations where Destiny of the Sands 1 spends half a page telling you how to set DCs for PCs trying to improvise things, how to run skill challenges, and so forth. Good GMs should already know how to do that, and have been doing that without being told so, as they are comfortable with the “creative solutions” clause in the guide.
A part of it is also the players. If you have confident actors that are good at improvising, you can have a looser script. Robert Altman can get away with improvising half his movies because only because he has the right actors for it, and he's good enough at directing to know how to give them hooks to work with.
Here is where the dissonance comes in. Every scenario has tracks laid-out, if people want to find them. No matter how many stops you put up that say “come RP over here!” some people just won't get off the train. That's not even mentioning that a lot of GMs would never let them leave to begin with. Maybe the players lacks the confidence to push off on their own, or they just want to hurry up and get to the combat, or maybe they just want to see the scenario's story. They don't start pulling their hair when they play Scars of the Third Crusade and have no freedom because they didn't want it to begin with; they always just roll whatever the GM tells them to.
A lot of season 5 scenarios are railroads with very pretty scenery. For players that just want to ride the train, they are suddenly riding a passenger train through California wine country instead of the subway in New Jersey. Weapon in the Rift and The Traitor's Lodge can be very atmospheric and cinematic. The train doesn't stop as often, but when it does, you have to get off and look around. If you are on the railroad in Our Lady of Silver from season 0, you barely have to interact with any NPCs (especially without faction missions), and the ones you do have to talk to have very little in the way of hooks for the GM to work with. If you are on the railroad in The Hellknight's Feast from season 5, you HAVE to talk to people for your mission, and there's two pages of info on them. Suddenly, the inflexible GM HAS to let you “try something” in Destiny 1 because the script says so. Suddenly, he HAS to let you roll Knowledge (arcana) to influence the intellectual wizard lady. Players in The Stolen Heir might be getting forced to make an actual in-character decision for the first time. The kind of players that like the railroad would never have created these situations themselves without someone else laying out the tracks for them.
To the players that were happier providing their own RP experiences, this is all worse. The GM's that might have let you roll Knowledge (arcana) anyways if you had asked aren't running games any differently, and in fact, may feel more stifled. Many season 0 scenarios on the railroad can be over in 2-3 hours in a 4-5 hour slot, so the GM has plenty of room to let you wander if you want. Season 4 and 5 already tend to run over their allotted times, so a GM at a con checking his watch will feel like he has no room to maneuver.
I'm not saying that one way of playing is necessarily better than another. If one couple goes traveling and takes official tours of everything and another wants to go explore on their own, that's their choice. I do think that Paizo needs to have a better balance of catering to both styles, though.
RPG Superstar 2014 Top 32
|
| 1 person marked this as a favorite. |
Looking at these expanded numbers, I think that I can point out what ties them together compared to the bottom-rated ones. Each of the top-rated scenarios provides challenging but not impossible combat paired with a tight and streamlined story. These adventures tell one story and they tell it well without trying to go all over the map on tangents, and every combat advances that story.
Conversely, the bottom-rated scenarios have encounters placed for no reason other than having the requisite number of encounters and have plots that are all over the map. Combat is split here, with some of them being ridiculously easy and some being ridiculously hard. Let's take Rivalry's End - you have a dungeon hidden underneath a missed RP opportunity, followed by a plot twist that makes absolutely no sense and which the PCs have practically no ability to change. Let's take Wonders in the Weave II - you can choose to talk to or fight a bunch of people that have been duped by the Aspis Consortium, then you get to beat up the Aspis Consortium. Afterwards, you find a disco floor and fight a mummy which really has no reason to be in that room. First Steps III is just a mess, with a series of entirely unconnected encounters leading to an unfulfillable mission. Shadow's Last Stand I presents us with the bizarre concept that a group of hobgoblins managed to take over an entire lodge and hold a Venture Captain hostage, but a group of level 1s can easily save the day.
Two scenarios at the bottom are particularly notable - Scars of the Third Crusade and Trial by Machine. Scars is a bit of a tricky one. In the hands of an RP-minded GM, it can be great fun, but it does have problems. There are a lot of important details to the mystery that are just not explained, which causes a GM to have to improvise. It is largely, I feel, a victim of bad editing and word count. Trial by Machine has a decent (if slightly implausible) story, but is low-rated because the combat is so brutal.
The takeaway is simple: When writing a scenario, choose one thing to do. Do it well, and make sure that there's a reason for the PCs to be doing what they are doing.
Hrothdane: I really enjoyed your post, but I would argue that Season 5 scenarios are significantly less railroady than previous seasons. Hellknight's Feast, Scars of the Third Crusade and Library of the Lion are functionally railroads, but they do not feel that way to players. It is incumbent upon the GM to not make choices for the players in these scenarios, and to allow the players to stumble around until they hit upon the right solution. Also, a good PFS GM should remember that they are free to add social encounters to adventures, and that if the players get off the rails, the GM can be creative in returning them to the plot.
|
I find it really interesting that in season 4 and even more in season 5 the PI just went thorugh the roof.
Makes me wonder why newer scenarios polarize people so much more.
Makes me wonder if we have reviewers who intentionally vote high or low just to drive the overall rating for outside reasons (fake reviews). Unless you personally know all the reviewers its always a risk.
|
|
Its not just when combats are brutal its when combats are frustrating: absurd dr/hardness for the tier and the very over used swarms, guaranteed surprise rounds and incorporeals frustrate the player by leaving them sitting there for half an hour soaking up damage and drain when they can't do anything.
