| 1 person marked this as a favorite. |
Yes NN, you have, through pedantry, proven that anecdotes and technically everything is meaningless data. Props for that level of nihilism BTW. Though for pedantry’s sake I will say that the plural of “anecdote” is actually “anecdotes” ;)
The entire point of what I was saying and what I admittedly assume DMW and others were pointing out is a repeated tendency for BD to make sweeping statements about the gaming community, gaming industry, specific publishers and specific RPGs touting “anecdotes” as proof of his claims without providing any evidence whatsoever. No proper research or anything really. Not even quotations or references. I, and at least some others, find this tiring to say the least.
| 8 people marked this as a favorite. |
I think that the unstated word in the common idiom "the plural of anecdote is not [useful] data". Certainly all sorts of things could be perceived as data, per Hempel "these boots are blue" is evidence for "all ravens are black" since "all ravens are black" is logically equivalent to "all non-black things are non-ravens". But no one in their right mind would start looking at "all non-raven things" for some sort of survey on ravens. Nonetheless it is technically relevant data, it's just not data you want.
Any time it comes to self-selection (like "this person chose to share this anecdote in this community") you're introducing bias to your sample. It's like how on Amazon you get reviews clustered around 5 stars and 1 stars, even when it's something it's hard to see strong opinions forming around like "a hairbrush." People whose experience with a hairbrush is most likely going to be in the neighborhood of "It's fine, I guess."
So what do we see on the forums, on reddit, etc. when people are talking about the playtest? We see the people talking about the things, positive or negative, that they care most about. Whatever people are passionate about is important feedback since it identifies which things you should take a closer look at. But 8 people on the forums being angry about a thing is not necessarily evidence it needs to be fixed if 80,000 people filling out surveys thought it was just fine.
So "everyone I know" or "everyone I heard talking about it" is not really a good cross section for data collection purposes.
| 1 person marked this as a favorite. |
To devil's advocate my own position for a moment, though, it IS a good cross section of "people who care enough to comment", which is why it's a good thing that Paizo does take those 8 people into account even if they don't end up outweighing the 80,000.
|
| 4 people marked this as a favorite. |
This is incorrect, though the overall point is accurate. I've seen this stated twice now, and it needs to be corrected or it will become part of the forum zeitgeist.
This strikes me as an exceedingly pedantic semantic point. It's certainly true that, by the most technical definition, an anecdote is data inasmuch as anything is data if you use it as the basis for calculations.
But taken in isolation, anecdotes are pretty universally bad data. And as PossibleCabbage notes, the usual connotation of using data as a piece of terminology is 'useful data'. I'd strongly argue that anecdotes almost never fall under that umbrella on their own, which was where I was going with that...
| 1 person marked this as a favorite. |
It seems to me this discussion is kind of a case of talking past one another. I don't think you're really disagreeing substantively, it's merely a matter of terminology.
FWIW, "the plural of anecdote is not data" has always bugged me too - but I think Possible Cabbage's reading of the word [useful] into it will help me reconcile that somewhat.
My take is that anecdotal data is a different type of data that needs to be treated with very specific tools. It's useful to gather, but one shouldn't feel that when you present your experiences it deserves to be given equal weight to a more rigorously curated conclusion arising from a well-run survey of sufficient size. (The more scientific methods have their own limitations, too). It may provide more nuance or added insight, but it won't be as useful from a predictive/descriptive sense.
| 5 people marked this as a favorite. |
No, that's not what I said, and another example of twisting something to run out a straw army
He didn’t make a straw man argument, please stop using argumentative and scientific methodology terms that you don’t actually understand.
|
|
| 2 people marked this as a favorite. |
As I recall, you used this as counter in one of our discussions when I wasn't using an anecdote to claim scientific proof of anything. It came across as any anecdote doesn't count and a lot of them is multiplication by zero.
An isolated data point is pretty much always invalid if trying to generalize the results to any group at all.
Yes, large groups of anecdotes can provide information, but they don't provide good information on their own, which was sort of my point. Anecdotes inevitably ignore everyone who had an experience but didn't talk about it, which inevitably includes the majority of people who had experiences, and it's a very self selected group.
For example, Mark Seifter has noted several times that the survey results are generally much more positive than anecdotal forum responses are. This is because people are, in a vacuum, much more inclined to comment (ie: provide anecdotes) about things that they dislike than about things that they like.
That's the kind of sampling error that makes for extremely poor decisions if you base your choices on said sample, especially in isolation.
That's simply not accurate as a sweeping statement. When we're talking about information about someone's experiences, any data in isolation is proof nothing. A narrative can be the only way to capture information about subjective experiences, which is exactly one of the the things a playtest tries to do. Asking people to write down or talk about their experiences in freeform often gives you a far more complete picture of what is going on than survey questions which inherently limit the scope of the information you are collecting and themselves may bias the outcome.
For all the reasons you note, sometimes narratives are the only available way to provide information on a particular issue (ie: some thing aren't on the surveys), but that makes it the best of a bad lot in terms of providing info, not a good tool in and of itself.
Or to put it another way: Sometimes it's the only tool available, and any tool is better than none, but it's still an inherently flawed tool.
Somethings are simply not appropriate for narrative. I would not rely on anecdotes to determine humidity or how far someone traveled in an airplane. But that's not what we're discussing.
I would submit that Paizo absolutely values the data it received from the 1000's of anecdotes posted about the playtest. Again, I'm not saying you're wrong or right about Byron D, but your statement comes across as insisting that lots of anecdotes can't provide useful data.
As stated, it's a bad tool, but better than no tool at all. I suspect Paizo does indeed value the anecdotes about the playtest...but not as much as survey data, or actual mathematical analysis based on both. Nor should they.
My goal is to make sure your statements aren't taken out of context.
I feel like you're going about it the wrong way if that's the case. It's possible to offer a much shorter 'You're technically incorrect, but it's not a big deal' type answer to achieve this purpose much better than the extensive semantic discussion you started here.
| 6 people marked this as a favorite. |
In 1980 Roger G. Noll said, "The plural of anecdote is data," and attributed the statement to Berkley political scientist Raymond Wolfinger. The quote spread. Finally, statisticians took a stand and declared that the plural of anecdote is not data. An anecdote does not hold data in proper measured and unbiased form to be converted into information. In 1982 Kenneth Kernaghan and P. K. Kuruvilla set out the counter-meme, "The plural of anecdote is not data."
I am a statistician among my many mathematical fields, but I must move past the purely statistical view. I spent 5 years of my career finding ways to turn data into information so that experts could turn the information into knowledge. Anecdotes are not data; instead, they are something stronger than data. They are information. An anecdote does not have the same solid foundation as information extracted from reliable data, so it is shaky and could come tumbling down. Nevertheless, since an anecdote is already information, it can immediately by used in creating knowledge. That makes an anecdote valuable.
And with that, I think the anecdote debate has run its course, no? Can we get back to Pathfinder please?
|
|
Rather than go into this quote by quote, I'm going to try and wind this down.
Sure, I'll make this my last post on the subject.
Anecdotal "evidence" was a huge problem for the scientific community before the scientific method became widely accepted. The problem with anecdotes is not that the data is inherently unreliable, it's that the data was used inappropriately as "evidence" to prove something that could not be proven from a human accounting. e.g. People claiming to be cured, don't really know if they are cured of some disease after treatment. People don't know how fast that car was going. Nevertheless, there are countless scientific experiments that have produced incorrect data and led to incorrect conclusions. The difference is that based on the data being collected, we can examine the method used to identify things that skew the results. People can run the same experiment using the same method and see if they get the same results.
Anecdotal data isn't "scientific" because it isn't collected in a repeatable fashion. There's no method to test.
So far, I'm actually in complete agreement with you.
That doesn't make it inherently incorrect or "bad." One thing that it does really well in a playtest is provide insight on things that may be problematic and need more rigorous exploration, something that a survey is poorly suited to discover.
It's bad when taken in isolation. I agree entirely that it's useful in support of other, more scientific data. For example, I think it's very useful to get details of, say, what particular problems people have with some specific rules element that the surveys indicate is disliked.
Individual analysis, which is not quite the same thing as an anecdote, is also very useful and was very relevant in the playtest.
if you, or others want to write off anecdotes as invalid for the playtest, that's your prerogative. What matters to me, is that Paizo pay attention and give it the proper weight it deserves. Based on their statements and Mark et al presence in the forums, I think they have. Doesn't mean that I agree with all their decisions, but at least I feel like they've seen the anecdotal data and considered it.
I...literally just said I was sure they were paying attention to anecdotes and agreed they should (just less attention than, say, survey data). Your first sentence here is just flatly not what I said or what I believe.
Well, it appears you've doubled down on the I-can-discount-any-number-of-anecdotes mantra, so I think it's more than a semantic disagreement. In which case, we'll just have to agree to disagree.
To reiterate for one last time, my specific statement is that anecdotes are worth very little (not nothing, but I wouldn't call them reliable) in isolation. In support of other, more scientifically valid, data, they can be fairly useful.
| 7 people marked this as a favorite. |
I’m always amazed at how much of a garbage fire +1/lv threads turn into. But honestly, it might be time to call its end, just like it’s been for all the previous ones.
I kinda agree with Ediwir. Without much concrete to talk about within these forums, most of these topics are people arguing over semantics of this or that since there's nothing else to talk about.
I don't really even know why Paizo opened up these forums so early, lmao. Not much good conversation has happened.
| 2 people marked this as a favorite. |
I mean there has been *some* good discussion, but ever since the playtest started, +1/lv is the general thread (get it? Thread?) that just attracts garbage. We’d have them with or without a new forum.
But I’d rather just not have them.
-Ed
guy who used to ague against +1/lv, then read the rules, apologised and moved on
I mean there has been *some* good discussion, but ever since the playtest started, +1/lv is the general thread (get it? Thread?) that just attracts garbage.
I feel like it might be another thing entirely if we were talking about +1/lvl but we spent a day and a half going back and forth over why 5e should or should not be ignored, how that system is either good or bad and then spent another half day going on this pedantic anecdote sidebar.
It’s been at least 2 pages of threaded responses since this was on topic.
| 1 person marked this as a favorite. |
I mean there has been *some* good discussion, but ever since the playtest started, +1/lv is the general thread (get it? Thread?) that just attracts garbage. We’d have them with or without a new forum.
But I’d rather just not have them.-Ed
guy who used to ague against +1/lv, then read the rules, apologised and moved on
The new proficiency system with +1 to proficiency per level is the 3rd biggest change from Pathfinder 1st Edition to Pathfinder 2nd Edition.
The three-action system changed the pace of every single turn and forced rewriting every action. People liked it, so it did not gather argument.
The new class structure built around feats and no longer allowing multiclassing by level means that old characters cannot be readily adapted to PF2. People griped about that, but came to accept it.
The proficiency system made level the most important factor for the success of a d20 roll. People disliked that. Some people thought it flattened out character concepts, so that all wizards would pick up their daggers and join melee. Paizo increased the effect of proficiency rank and that calmed some people. Nevertheless, ClanPsi started this thread in the hope that Paizo had dropped the +1 per level entirely.
Disagreements attract comments, and long threads without a solution attract topic drift. The original question was answered within one hour, so this thread has no core topic.
I see strengths and weaknesses in all three major changes. Yet the time to voice opinions is past. Paizo made the changes and I will adapt to them because as a whole Pathfinder 2nd Edition is a much cleaner improvement.
I will still post a few pages of mathematical analysis over in the Playtest subforum, because I like mathematical analysis.
| 6 people marked this as a favorite. |
I just don't see that much of an issue with +level anymore. The biggest problems with +lvl were that it overshadowed any kind of measurable investment (proficiency).
Now with Proficiency getting bumped to +0/+2/+4/+6/+8 and with Skill Feats getting a scalar buff (allegedly anyways) there's not a lot of room to complain.
Now a level 1 Expert in Stealth still beats a level 5 character that's untrained (considerably due to the drop of level from Untrained). Even with Trained, Expert is enough to cover two levels of difference, which is pretty significant (especially since the person who is Trained did have to invest something at least).
It shows how experience and specialty are two sides of the same coin. Experience is just downright better for standard tasks, while better training means you can perform tasks that those that have the standard methods down can't perform or can do so with heavy penalties (Skill Feats).
And if both are the same experience, it comes down to specialty. Is that not how it should work? It seems pretty accurate in practice.
Everywhere else that +lvl is applied, it doesn't affect the narrative at all and it makes sense. They are all functions of combat (though proficiency still plays a part) where it is expected the longer you live and the more you train, the better at defending yourself and making lethal blows you will be.
While I'm sure there are other reasons to hate +lvl, they've adjusted the other parts of the game to make it work narrative wise, so I don't see the issue. A standard math scaling that's linear and easy to deal with means you can incorporate nebulous non-linear benefits via abilities (Skill Feats, Class Feats, etc.).
Maybe there's some glaring issue I'm just not seeing, but I'm not really hearing a lot of reasons to dislike it other than personally not caring for it (and as many have pointed out, it's not that hard to remove from the game entirely).
| 3 people marked this as a favorite. |
Well said, Midnight.
As an aside, my theory on why this thread hasn't been locked (or really moderated much) is that Paizo has realized that it's not going to die, and by allowing this thread to continue unabated they can at least contain this argument to one single thread instead of spread across the forum. :P
I'm honestly considering house-ruling untrained back to (level-4), minimum 0. I really liked the idea of general competence improving with level.
As near as I can tell, this shouldn't have a big impact on anything. If you like that flavor it seems perfectly reasonable as a house rule. One of the benefits of the system is how nicely house-rulable it is.
I'll certainly see and try the final system before I apply any house rules, but I considered that as well for sure; I was a strong proponent of level to untrained skills.
Sadly, the main reason to include level to untrained skills (it lets you use skills as defenses in combat) requires being built into the system, and it won't be, so it might not be worth the effort to house rule.
| 1 person marked this as a favorite. |
I'm honestly considering house-ruling untrained back to (level-4), minimum 0. I really liked the idea of general competence improving with level.
While I'm not crazy about the +0 no level either here are the items that I see negatively with the -4 + lvl:
- It decreases involvement at lower levels to the point of being unusable. -4 is a steep penalty. The +0 allows involvement in a skill with basic attributes, but means they don't progress near as fast.
- It heavily drops off at higher levels to the point where a person with no training in stealth at all can out stealth a "stealthy trained" person. I.E. Level 9 untrained Thievery Ranger is even with a level 1 Expert Thievery Rogue, in the context of "Thievery" it makes little to no sense for the Ranger to ever out do the Rogue (regardless of level).
Now there are obvious caveats to the +0 untrained, and I know they really wanted to avoid +1/2 level like the plague, but to me that's the only one that covers both the flaws of either implementation.
Overall, if I had to choose, I would choose +0 for the above reasons (if you really want to participate at higher levels, just invest in the skill for trained and you're good IMO) but I recognize not everyone feels that way.
My plan is to houserule a general feat (assuming something like this does not already exist) which fits in that feat 3 slot which allows you to make untrained checks at level -2.
I'm honestly considering house-ruling untrained back to (level-4), minimum 0. I really liked the idea of general competence improving with level.While I'm not crazy about the +0 no level either here are the items that I see negatively with the -4 + lvl:
- It decreases involvement at lower levels to the point of being unusable. -4 is a steep penalty. The +0 allows involvement in a skill with basic attributes, but means they don't progress near as fast.
- It heavily drops off at higher levels to the point where a person with no training in stealth at all can out stealth a "stealthy trained" person. I.E. Level 9 untrained Thievery Ranger is even with a level 1 Expert Thievery Rogue, in the context of "Thievery" it makes little to no sense for the Ranger to ever out do the Rogue (regardless of level).
Now there are obvious caveats to the +0 untrained, and I know they really wanted to avoid +1/2 level like the plague, but to me that's the only one that covers both the flaws of either implementation.
Overall, if I had to choose, I would choose +0 for the above reasons (if you really want to participate at higher levels, just invest in the skill for trained and you're good IMO) but I recognize not everyone feels that way.
The big question is how easy is it to get trained in most of the skills, armors, weapons, etc. If it is hard to get trained in things then it is a steep penalty. If however you are going to be trained in most things (especially saves) and it is your choice to not be trained in something. Then I don't see there being much of an issue.
Being legendary in one skill probably means being untrained in three more skills vs. just being trained. Skill feats will be competing with “+12 to an untrained skill” by mid-levels. Being a generalist is probably a bit more favored than before.
| 2 people marked this as a favorite. |
The big question is how easy is it to get trained in most of the skills, armors, weapons, etc. If it is hard to get trained in things then it is a steep penalty. If however you are going to be trained in most things (especially saves) and it is your choice to not be trained in something. Then I don't see there being much of an issue.
Based on what we have seen and dev comments, we know that no character will be untrained in any save, and that all classes are trained in unarmored (or expert, for monks). So as far as combat stats go, you'll never be untrained unless you intentionally pick up a weapon or armor you aren't proficient with.
I know that's a particular bugbear for another poster here, but it seems that Paizo hit that lovely definition of compromise where neither of us are happy - he wanted wizards to be untrained in unarmored, I wanted untrained skills to still get level, neither of us got what we wanted. :P
