|
I disagree 100%. If you haven't at least taken a look at the scenario yourself, and more preferably GMed it yourself, I'd prefer you not to review a scenario poorly because one GM ran it poorly for you.To be honest, Id much rather see player focused reviews than DMs. Reviews from DMs, due to the perspective that DMs get, knowing the backstory, seeing how the rules and suggestions are suppossed to be, is going to give a very different picture, likely boosting scenarios more than they probably should be.
There's always going to be off reviews. A DM can review something highly because they enjoyed running it but he table was not at all enjoying it. Or hate something, but have the table find the whole thing great from their side. And a "good DM" can make a poor scenario great, or a "poor DM" can drag a awesome scenario down.
Now, ideally, I would prefer if all reviews where from people that have both played and ran something. I just mean of the two, I'd rather see what the player's perspective of a given scenario is rather than the DM's. If there are multiple reviews, the ones that have a good or bad DM will show it.
|
I certainly agree that, as a player, I'm much more interested in reviews from players. Personally, I try to make it very clear what my role was (player vs. GM vs. both) when I review an adventure.To be honest, Id much rather see player focused reviews than DMs. Reviews from DMs, due to the perspective that DMs get, knowing the backstory, seeing how the rules and suggestions are suppossed to be, is going to give a very different picture, likely boosting scenarios more than they probably should be.
Honestly, when I'm looking at organizing events, which I've been doing more of lately, I'm interested in both player and GM reviews, but for different reasons. I want players to have fun and to make sure the scenario is fun, but I also try to be conscious of what the prep requirement is on the GM's part.
|
Kyle,
I've downloaded the spreadsheet and can add the info I mentioned in my email about the degree of confidence in the average ratings (based only upon number of reviews and making the huge presumption that these reflect a random sampling of participants).
Although top 20 are all likely excellent scenarios and bottom 20 are likely some of the worst, the relative standing of these and the ones in the middle (if we had responses from all who participated) could differ substantially. The "true" average rating could change by up to a star (or more) if you don't have a large sampling of reviews and the ones you have are not tightly grouped.
Another measure of a scenario's popularity would be number of play statistics. Granted, you'd have to look at replayable scenarios separately (yes, The Confirmation's popularity has a lot to do with that and you can't take all the credit). But if a player likes a scenario enough to want to GM it or a GM likes one enough to GM it again and again, that is meaningful feedback.
I don't have access to the Paizo stats on this, but we all can see the relative popularity of scheduled sessions in the Warhorn PFS Campaign Global Scenarios Listing. The top 25 include 2 replayables (Confirmation and We Be Goblins) and mostly season 5 stuff for the rest. Mists of Mwangi comes in at #25, and Trial by Machine is the only season 6 currently in the top 25. For an individual scenario, you can see number of past and upcoming plays as well as the total.
I expected the Destiny of the Sands trilogy would rate highly, but was surprised to see the Glass River Rescue, The Stolen Heir, Library of the Lion and the Wardstone Patrol right up there with those scenarios.
This is somewhat slanted as we didn't get the global scenario catalog up there until Jan/Feb 2014, and most folks had already played stuff from previous seasons using manually entered event descriptions (rather than picking from the global catalog). If you sort the information by season, you can see the relative (recent) popularity of scenarios within a specific season, or you can simply discount scenarios that were released after the global catalog went live as these will include a lot of the initial plays (there'll be a higher percentage of replays and lower totals for older games.) If you look at the top 50 you start seeing the most popular games outside of seasons 5 & 6. There are currently 271 offerings in the catalog, including scenarios, APs, modules and ACG Adventures.
|
Venture-Lieutenant, Minnesota—Minneapolis
|
Thanks Kyle for putting this together.
... I absolutely won't review an adventure if I have not GMed it as I don't know whether problems I experienced as a player came from the author or the GM.Agreed. At least three times my impression of a scenario as a player was significantly different after I had read it. A really great GM made a ho-hum scenario really memorable and twice I thought a scenario was a dog and it was really just the GM not knowing what he was doing.
I don't know if there is any way to gather this data, but if there were some way to run analysis on GM star rating vs what the players of the session rated it that might give a good idea where the expected skill of the GM was a major factor.
It has always been my experience that the skill of the GM and how well their style meshed with what I considered fun made more of a difference than anything else.
|
|
Kyle,
I just noticed that it is possible to give a zero star rating, and in some cases that occurred. Your formula for averaging excludes the count of the zero star ratings in the denominator, so the averages where there were any such reviews are artificially inflated.
I am nearly done with the calculation of confidence bands for the averages (estimated +/- error on the averages considering number of reviews) and am correcting the average columns for the zero star ratings now. Would you like me to post an update or send a copy to you first for review?
|
|
I just noticed that it is possible to give a zero star ratingThe review form does not provide 0 stars as a valid option, unless the form does not validate submissions where the star rating selector is left in the default position. It seems to me that 0 star ratings are indicative of bad data.
Kyle has about 1% of the reviews listed as zero stars - which I'd guess correspond to folks who wrote a text review but did not provide a star rating (the default choice, above "1 Star" is "Select Star Rating"). It could be that some of these were folks who neglected to provide a rating and some were those who though it didn't earn any stars. For 16 scenarios where this occurred, some had one "zero" and some had two. This ranged from 3% to 50% of the reviews for an individual scenario (avg. 7%).
I checked the ones that had 2 zeroes. In Rats of Round Mountain they were BS comments about Kyle's scenario killing players as well as PCs and destroying computers and setting fire to homes. In Temple of Empyreal Enlightenment they appeared to be omissions on positive reviews. In Sanos Abduction only one review appeared without stars (the total count also did not match the spreadsheet). That particular review was fairly negative, but it did not clearly show an intentional rating of zero stars.
Without checking the rest I would tend to agree that most of the reviews with zero stars were not intended to convey a zero rating. If you agree these are generally bad data, they should not count in any total of reviews that is used for statistical calculations. They don't have any impact on the "10 or more reviews" threshold as the scenarios in question either have under 10 reviews already or have more than 10 where a star rating was given.
|
Ah, interesting. I admittedly did not try to submit a review with the -Select Star Rating- item in the combobox because I was afraid if it going through and then not being able to fix it. I'm still somewhat hesitant to try it.
Kind of unfortunate, it sounds like some "valuable" reviews were lost because of what amounts to a validation error. Perhaps this constitutes a bug report against the website?
|
|
Ah, interesting. I admittedly did not try to submit a review with the -Select Star Rating- item in the combobox because I was afraid if it going through and then not being able to fix it. I'm still somewhat hesitant to try it.
Kind of unfortunate, it sounds like some "valuable" reviews were lost because of what amounts to a validation error. Perhaps this constitutes a bug report against the website?
I don't know. I presume they chose to code it that way on purpose in case someone wanted to comment but not provide a star rating. It appears that the average ratings on the website exclude those responses from the numerical averages, but the comments are still presented for others to peruse. It is a small percentage in most cases.
|
I've seen a number of scenarios come out with newer mechanics built in where review after review was of players complaining about something the mechanic did in direct opposition to what is actually part of the scenario...
I'd rather know the person has had a chance to actually see what is in the product before reviewing it based on another's interpretation.
I don't agree here - if a scenario is consistently generating bad reviews based on GMs handling it wrong, then that's evidence its writing isn't clear enough.
Also, I think relying overmuch on GM reviews causes too much bias toward rating what the scenario wants to do as opposed to what it actually does. For example, reviewing based on interesting backstory that never comes out in actual play. Sometimes GMs just get it wrong of course, but if there's review after review saying similar things went wrong in different runs then it points to a problem with the scenario.
|
|
I found some minor errors in the season averages (the last scenario was accidentally left out of the calculations for seasons 0, 1, 2 and 4).
I also added some double-check calculations and some further statistical stuff to the "All Scenarios" tab, such as response range, standard deviation, skewness, excess kurtosis and 95% confidence intervals for what the true average might be if we had more reviews but the same standard deviation for the distribution.
This last item suggests a slightly different ranking, taking into account that the average for a scenario with a smaller number of reviews is less likely to represent what we'd get if more people submitted reviews, but a tighter grouping of ratings suggests a higher degree of confidence even if there are few reviews. Many reviews combined with a tight grouping suggests the highest degree of confidence that the results are reliable or repeatable.
Unfortunately I'm still a newbie to Google Drive, and I didn't want to modify Kyle's original or overwrite a shared file while others may be accessing it. I've downloaded it and made changes offline and will send to Kyle to figure out if and how he may want to incorporate it.
There really is no absolute best ranking with the small amount of review ratings we have for some scenarios, but with filter and sort controls in the Excel version it is easy to generate a variety of top 10 or top 25 lists (most skewed, most split, most consistent, etc.)
In the season summary, season 3 had the most reviews, the highest average, the most skew towards a majority of high ratings, and the closest to a normal distribution of results. Distributions of ratings for other seasons were flatter, with averages closer to 3.
|
|
Two things to consider, is that he did specify scenarios with at least 10 reviews, and also that those numbers might have changed as people here have said "hey, I liked that scenario, but it's rated low, let me change that." or "wow, people didn't like that one, but that's my favorite, can't let that stand."
|
|
Two things to consider, is that he did specify scenarios with at least 10 reviews, and also that those numbers might have changed as people here have said "hey, I liked that scenario, but it's rated low, let me change that." or "wow, people didn't like that one, but that's my favorite, can't let that stand."
Understood. My point is that there is considerably higher confidence in the average for a scenario that has 30 reviews compared to one with only 10 reviews, and some meaning can be gleaned from sites with even fewer than 10 reviews, e.g. one that received 7 reviews and they were all 5 stars.
In terms of keeping up with the new data that comes in, that would be a daunting task, especially if it's done by scrolling through the reviews on the web pages to pick off the numbers for each one (both for changes and for new reviews). I'm not sure how often Kyle may try to update this, I'd guess not more than weekly.
It is also far from an unbiased system, as you point out. Everyone can look at all the other responses, and there is a tendency to not respond if you agree with the general consensus but submit a conflicting review if you disagree. That tends to flatten the distribution of responses (more near the extremes) and reduce the count of reviews for scenarios where there is close agreement on the rating.
There is still a lot of good data collected here, but folks should not be misled by averages reported to a precision of 0.01 stars. In many cases (even with 10 or more reviews) there isn't a lot of confidence that a more thorough polling would result in average ratings within half a star or even a whole star or more from the current values.
Of course, with more reviews, the confidence in the average ratings improves. But it is still just a one number rating system, so there is only so far you can go to improve the confidence without taking steps to eliminate bias and to separate the different aspects that go into creating a high or low rating on this scale.
|
|
Another way to look at it - the average isn't a great measure of typical value if there aren't a large percentage of responses near the average.
Many of the distributions of review ratings don't have a single clear mode (or two clear modes if responses are split). These distributions more closely approximate a uniform distribution than a unimodal, bimodal or normal distribution.
With more reviews, the most common ratings are likely to emerge, with counts at the peaks that exceed the counts for other rating values by more than just a couple or a handful.
10 reviews was a good cutoff to make sure the report could include a majority of the scenarios, but unless they are all tightly grouped you probably need more like 20 before the distribution of responses has a fairly clear shape. Unfortunately we currently only have a few scenarios that have received 20 or more reviews.
|
|
| 1 person marked this as a favorite. |
Would the flatter distributions be attributable to being more prone to influence by the individual GM than the scenario itself?
Possibly. The theory is that the table bias (due to GM, PC mix or player choices) should average out as more tables are included in the set of reviews. But the data sets are generally so small that I don't think you can say with much confidence that the scenario is prone to GM variation. More likely there were just a few people who had differing experiences (more due to the people than the scenario).
If you truly had two different ways the scenario might be run, which resulted in different levels of enjoyment, with enough reviews you might actually get a clearly bimodal distribution of ratings.
You actually can (and I did) calculate something called a bimodality coefficient for each distribution, and based on the result you can see if the distribution tends toward bimodal, unimodal, or uniform (equal values at all rating numbers). The flatter distributions most closely resemble a uniform distribution, which has no clear "typical" value(s) regardless of the result you'd get by calculating an average rating.
If there are a lot of responses and the distribution of ratings is still very uniform, then YMMV: your likely experience (and rating) with such a scenario might be nearly random. The scenarios that do have many reviews tend to NOT be like this; they tend to have a clear peak in the distribution of ratings and very few ratings far from that value.
Only a few scenarios have a highly bimodal distribution of reviews and more than a handful of reviews at the secondary peak: Library of the Lion, Murder on the Throaty Mermaid, The Elven Entanglement and The Waking Rune. It may be that these can go really well with the right PC mix and really poorly with the wrong mix. Other scenarios may be better balanced to work with a larger range of PC parties and capabilities.