Chief Technical Officer
|
I posted the following (or a very near approximation) during the playtest for the core rules, but it bears repeating here.
I think it's important that everyone have an understanding of what we mean when we talk about "playtest feedback."
It's actually pretty straightforward—"playtest" is a compound verb made up of two smaller verbs, and both of those verbs are equally important to us. We have some new ideas, and we want you to TEST them by PLAYING with them.
We're really not terribly interested in having people read the new material and then just imagine how things might play out. We already have talented folks on our payroll who are fully capable of doing that. What we need are people to put the rules into play in actual game situations, and tell us—good and bad—what happens.
I can promise you, when Jason's going through the playtest forums, reports of people actually playing sessions using the new material will have a far greater chance of making an impact on the finished product than the people who are merely theorizing about potential rules applications.
Now, don't get me wrong, it's perfectly fine to theorize—that's where the TEST part of playtesting comes in. If you think a particular rule is problematic, set up a scenario to exercise that rule—ideally, a somewhat "realistic" scenario that one might find in a published adventure—and try it out with some friends. But please do it by actually PLAYING—and then tell us about your playing and your testing in the playtest feedback forums.
Thanks!
I've done the math and I think your idea of what constitutes "playtesting" is really overpowered.
~laughs in RT~ Smartass!!!
Contributor, RPG Superstar 2008 Top 4, Legendary Games
|
I've done the math and I think your idea of what constitutes "playtesting" is really overpowered.
I strenuously and violently object to everything you just said! If you really think that, you must be blind and a n00b and have no idea how to play the game correctly! I've done the math and my math > your math! Your math sucks rocks, and not the good ones; those lumpy ones with the barnacles on the side... :)
I've done the math and I think your idea of what constitutes "playtesting" is really overpowered.I strenuously and violently object to everything you just said! If you really think that, you must be blind and a n00b and have no idea how to play the game correctly! I've done the math and my math > your math! Your math sucks rocks, and not the good ones; those lumpy ones with the barnacles on the side... :)
~laughs again in RT~ I reiterate! SMARTASS!!! ~grins~
To Hell with all of you! I want my THAC0 back, Damn It!!!
Methinks Smartass and Gamer are synonyms :) ROFL!
Methinks Smartass and Gamer are synonyms :) ROFL!
~grins~ Probably!
Did I win the playtest yet?
Be vewwy vewwy quiet. I'm hunting Stormwind fawwacies ..
AHAHAHAHAHAHAHAH!
I posted the following (or a very near approximation) during the playtest for the core rules, but it bears repeating here.
I think it's important that everyone have an understanding of what we mean when we talk about "playtest feedback."
It's actually pretty straightforward—"playtest" is a compound verb made up of two smaller verbs, and both of those verbs are equally important to us. We have some new ideas, and we want you to TEST them by PLAYING with them.
We're really not terribly interested in having people read the new material and then just imagine how things might play out. We already have talented folks on our payroll who are fully capable of doing that. What we need are people to put the rules into play in actual game situations, and tell us—good and bad—what happens.
I can promise you, when Jason's going through the playtest forums, reports of people actually playing sessions using the new material will have a far greater chance of making an impact on the finished product than the people who are merely theorizing about potential rules applications.
Now, don't get me wrong, it's perfectly fine to theorize—that's where the TEST part of playtesting comes in. If you think a particular rule is problematic, set up a scenario to exercise that rule—ideally, a somewhat "realistic" scenario that one might find in a published adventure—and try it out with some friends. But please do it by actually PLAYING—and then tell us about your playing and your testing in the playtest feedback forums.
Thanks!
I'm pretty excited. This weekend we will be integrating two new characters (Summoner and the Witch) into our usual Pathfinder game and giving them a real shakedown. I hope to have much to report come Monday.
Did I win the playtest yet?
No, it's a bit like thermonuclear war. The only way to win is to not play.
Sort of like Hello Kitty Island Adventure.
I've done the math and I think your idea of what constitutes "playtesting" is really overpowered.I strenuously and violently object to everything you just said! If you really think that, you must be blind and a n00b and have no idea how to play the game correctly! I've done the math and my math > your math! Your math sucks rocks, and not the good ones; those lumpy ones with the barnacles on the side... :)
You've both lost site of the purpose of the playtest! The rules don't matter so long as everyone is making out with eachother.
Ur all wrongz, ur rulz suxxors, u shuld yooz my house rulz and higher me to run pizo!
NOTE: The sad thing is I think I remember a letter to Dragon Magazine back when 3rd edition first came out wherein the letter writer stated the rules for 3rd edition were almost verbatim his house rules and that Wizards should just hire him as a designer. I'm pretty sure the guy was serious to boot.
Ur all wrongz, ur rulz suxxors, u shuld yooz my house rulz and higher me to run pizo!
NOTE: The sad thing is I think I remember a letter to Dragon Magazine back when 3rd edition first came out wherein the letter writer stated the rules for 3rd edition were almost verbatim his house rules and that Wizards should just hire him as a designer. I'm pretty sure the guy was serious to boot.
I can't understand people playtesting new classes using house rules, or changing this or that as it makes any test pointless.
Or ripping into a designer's post saying how they have got it all wrong without testing a rule change.
Or when you post playests after doing them by the guidelines and then being told you done it wrong because A, B or C was a bad tactic.
*Edit* Or "Sub-Optimal"
This attitude is very frustrating, Vic. If I can mathematically prove that fighters are underpowered compared to wizards, the Paizo staff apparently don't care very much. Instead, what they want to know is if the fighter players are having fun. For me, fun isn't dependent on balance. I can have fun playing a commoner.
The purpose of the playtest is to root out the flaws of the APG classes. If I can statistically prove that there is a problem with a class, it needs to be addressed. Again, fighters can be fun to play. That's not saying there isn't a balance problem with them.
|
This attitude is very frustrating, Vic. If I can mathematically prove that fighters are underpowered compared to wizards, the Paizo staff apparently don't care very much. Instead, what they want to know is if the fighter players are having fun. For me, fun isn't dependent on balance. I can have fun playing a commoner.
The purpose of the playtest is to root out the flaws of the APG classes. If I can statistically prove that there is a problem with a class, it needs to be addressed. Again, fighters can be fun to play. That's not saying there isn't a balance problem with them.
I disagree with this assessment of playtesting. It doesn't involve any play at all, and I am not now, nor have I ever been convinced that folks on the interwebs bearing "proof" of this or that have any idea what the hell they are talking about.
There is just so much more to actual play then crunching numbers, like for instance random variable from: situation/terrain, party/encounter composition, magic items available, flavor choices, different ability score brackets, equipment choices, feat choices, multiclassing, etc...
Just because you took a specific set of circumstances and tested raw damage output under those circumstances does not prove imbalance. All it means is that if we were not rolling dice (and living purely on set outputs for the rolls), and playing in a perfectly devoid microcosm that if two characters took those exact options that this "proof" would happen every time. But it does nothing to illustrate the classes overall balance in the thousands of other imaginable gameplay scenarios.
In that vein I often find that folks who have "proof", really don't have anything but an opinion they are trying to validate by creating a specific scenario that supports their opinion, while in most cases ignoring the rest of the variables of the game altogether.
love,
malkav
This attitude is very frustrating, Vic. If I can mathematically prove that fighters are underpowered compared to wizards, the Paizo staff apparently don't care very much. Instead, what they want to know is if the fighter players are having fun. For me, fun isn't dependent on balance. I can have fun playing a commoner.
The purpose of the playtest is to root out the flaws of the APG classes. If I can statistically prove that there is a problem with a class, it needs to be addressed. Again, fighters can be fun to play. That's not saying there isn't a balance problem with them.
I disagree with this assessment of playtesting. It doesn't involve any play at all, and I am not now, nor have I ever been convinced that folks on the interwebs bearing "proof" of this or that have any idea what the hell they are talking about.
There is just so much more to actual play then crunching numbers, like for instance random variable from: situation/terrain, party/encounter composition, magic items available, flavor choices, different ability score brackets, equipment choices, feat choices, multiclassing, etc...
Just because you took a specific set of circumstances and tested raw damage output under those circumstances does not prove imbalance. All it means is that if we were not rolling dice (and living purely on set outputs for the rolls), and playing in a perfectly devoid microcosm that if two characters took those exact options that this "proof" would happen every time. But it does nothing to illustrate the classes overall balance in the thousands of other imaginable gameplay scenarios.
In that vein I often find that folks who have "proof", really don't have anything but an opinion they are trying to validate by creating a specific scenario that supports their opinion, while in most cases ignoring the rest of the variables of the game altogether.
love,
malkav
I agree 100%.
+5 Holy Avenger.
I cleaned up some sniping. (And quotes thereof.) Be respectful.
Campaign playtest scenarios are just as arbitrary as theoretical/probability scenarios, only even less informative because of the following...
Blessings of the Polyhedral Gods: Die rolls will skew results from the expected; as an encounter where the PC rolls 16+ and the monster rolls 5 or less will make that PC look much more powerful than he is. The opposite can happen, where poor player rolls and good monster rolls make the PC look less powerful than he is.
Observation Bias: If you RP well and/or have lucky dice at a couple critical moments, memories of the rest of the session where you otherwise performed poorly will be glossed over and marginalized. Like the Blessings flaw, this can be mirrored by having critical junctures fail and poor RP makes everyone view the character as less than he actually is.
Voice in a Crowd: An entire party is near guaranteed success against an encounter where the CR matches the average party level, with minimal cost in resources. The expected party size being five instead of four in Pathfinder, which makes the resource loss even less. A single PC performing as little as half as much as any other character will not change the outcome of that fight. Heck, that single PC could be half again as powerful as he should be and two other characters perform noticeably subpar, and the final results would be the same.
Curse of the Time Sink: RL can make meeting times difficult, and standard adventure sessions take much longer than objective scenario analysis, which means you flat-out do less playtesting.
There are times where 'regular' playtests are appropriate. Finding system hiccups/abuse that you didn't think of or discover, mental investment comparisons (what's easy for you may be difficult for them), opinion polls on flavour, etc. But that's all for after you do the more rigorous and 'unfun' model of comparing a couple builds with a couple handfuls of highly varied scenarios; with an emphasis on comparing results with an established baseline.
I've done equivalent playtesting myself, but that was a long time ago, and I did need to compare it to one or two other builds at the same level. Technically, I actually did rolling rather than assign chances of survival, but that was because of this board's demand and reluctance to accept anything but; so I modified it to have each fight be done thrice. http://paizo.com/paizo/messageboards/paizoPublishing/olderProducts/pathfind erRPGBeta/feedback/alpha2/racesClasses/barbarianPlaytestPg811
Thank you, Virgil. You said it better than I could.
Campaign playtest scenarios are just as arbitrary as theoretical/probability scenarios, only even less informative because of the following...
Blessings of the Polyhedral Gods: Die rolls will skew results from the expected; as an encounter where the PC rolls 16+ and the monster rolls 5 or less will make that PC look much more powerful than he is. The opposite can happen, where poor player rolls and good monster rolls make the PC look less powerful than he is.
Observation Bias: If you RP well and/or have lucky dice at a couple critical moments, memories of the rest of the session where you otherwise performed poorly will be glossed over and marginalized. Like the Blessings flaw, this can be mirrored by having critical junctures fail and poor RP makes everyone view the character as less than he actually is.
Voice in a Crowd: An entire party is near guaranteed success against an encounter where the CR matches the average party level, with minimal cost in resources. The expected party size being five instead of four in Pathfinder, which makes the resource loss even less. A single PC performing as little as half as much as any other character will not change the outcome of that fight. Heck, that single PC could be half again as powerful as he should be and two other characters perform noticeably subpar, and the final results would be the same.
Curse of the Time Sink: RL can make meeting times difficult, and standard adventure sessions take much longer than objective scenario analysis, which means you flat-out do less playtesting.There are times where 'regular' playtests are appropriate. Finding system hiccups/abuse that you didn't think of or discover, mental investment comparisons (what's easy for you may be difficult for them), opinion polls on flavour, etc. But that's all for after you do the more rigorous and 'unfun' model of comparing a couple builds with a couple handfuls of highly varied scenarios; with an emphasis on comparing results with an established baseline.
...
Can't help but disagree. The intended AND expected use of all of this material is during standard gameplay, not a mathematical abstract. It's not a bad thing to do, it's just not valid to the end use of the material.
I think that it is apparent that Paizo does care about theoretical points. But as he said, they can do that, and it is not what they are asking for.
It is certainly helpful, but actually playing things reveals things that don't show up in theoretical/probability scenarios, making the actual playtesting the more informative option when they are doing theory already.
They have a preference for actual playtesting than other things because it is more useful to them right now. They aren't likely to ignore points just because they weren't actually playtested, but since that information is less useful toward contributing to what they are doing, it is understandable to prefer it.
Campaign playtest scenarios are just as arbitrary as theoretical/probability scenarios, only even less informative because of the following...
Blessings of the Polyhedral Gods: Die rolls will skew results from the expected; as an encounter where the PC rolls 16+ and the monster rolls 5 or less will make that PC look much more powerful than he is. The opposite can happen, where poor player rolls and good monster rolls make the PC look less powerful than he is.
Observation Bias: If you RP well and/or have lucky dice at a couple critical moments, memories of the rest of the session where you otherwise performed poorly will be glossed over and marginalized. Like the Blessings flaw, this can be mirrored by having critical junctures fail and poor RP makes everyone view the character as less than he actually is.
Voice in a Crowd: An entire party is near guaranteed success against an encounter where the CR matches the average party level, with minimal cost in resources. The expected party size being five instead of four in Pathfinder, which makes the resource loss even less. A single PC performing as little as half as much as any other character will not change the outcome of that fight. Heck, that single PC could be half again as powerful as he should be and two other characters perform noticeably subpar, and the final results would be the same.
Curse of the Time Sink: RL can make meeting times difficult, and standard adventure sessions take much longer than objective scenario analysis, which means you flat-out do less playtesting.There are times where 'regular' playtests are appropriate. Finding system hiccups/abuse that you didn't think of or discover, mental investment comparisons (what's easy for you may be difficult for them), opinion polls on flavour, etc. But that's all for after you do the more rigorous and 'unfun' model of comparing a couple builds with a couple handfuls of highly varied scenarios; with an emphasis on comparing results with an established baseline.
...
Math is hard for people, so expecting discussions based on it is unreasonable.
And people often make a habit of playing through 20 levels of characters in depth within the span of a couple of weeks by playing for 8-12 hours a day every day, so any weaknesses are bound to already show up.
[/snark]
Also, people need to keep in mind that internal math work is not a good idea (despite Vic's assertion that they can "do the math" on their own). 3e suffered massive problems because of this--just take harm for instance. In the 3.0 version, it was a touch attack that reduced the target to 1d4 HP--no save! But because WotC did internal playtesting (rather than letting math gurus at their system), they didn't catch it. Instead, the clerics played as healbots, so everything was fine and dandy.
Actually playing the game is not incompatible with theoretical analysis. For example, if one felt harm was too strong a spell and math backed one's assertion, one could actually make a cleric using harm to play and test.
If one is making them choose between playing and not, then they have given their answer, as actual playtesting is the goal of the playtest. But it is unclear, why forcing the choice is necessary at all. One is perfectly capable of doing both.
|
I think the idea behind playtest is very simple. Paizo has people that work for them whose job is to work on the theoretycal mechanics of the game, however there are things that cannot be seen with purely theoretical calculations, and so they do a playtest. As simple as that.
So theoretical analisis by people is redundant, and mostly (if thought about it) an excersice on self glorification. Pointing out details on the rule sis fine, but writing entyre essays ona class they have seen for 5 mins because of their ¨massive game experience and godlike math skills¨ is pretty silly.
Actually playing the game is not incompatible with theoretical analysis. For example, if one felt harm was too strong a spell and math backed one's assertion, one could actually make a cleric using harm to play and test.
I agree completely. However, sometimes there are issues so blatant--such as the aforementioned 3.0 harm--that they don't need to be playtested. The problem is that if I were to say, "Hey, guys, harm is clearly broken," I'd get naysayers like malkav telling me I'm full of it, that I don't know what I'm talking about, etc. However, even a glance at the spell from a person with a functioning brain leads to the conclusion that harm is really, really powerful, and no amount of playtesting is going to prove anything except that it's overpowered. (And then you'd get pretentious know-it-alls telling you how you're just trying to present your opinion as fact and that you're playing the game wrong.)
So theoretical analisis by people is redundant, and mostly (if thought about it) an excersice on self glorification. Pointing out details on the rule sis fine, but writing entyre essays ona class they have seen for 5 mins because of their ¨massive game experience and godlike math skills¨ is pretty silly.
Not really. For instance, in the early Beta playtest of the PRPG, Jason Bulmahn wrote up an ability that gave a bard a save-or-die with a DC equal to 10 + the bard's Perform skill modifier. That's going to be around a DC 40 Fort save. So, yes, the developers screw up and make math mistakes, and it's our duty as playtesters to say, "Hey, this isn't right."
I agree completely. However, sometimes there are issues so blatant--such as the aforementioned 3.0 harm--that they don't need to be playtested. The problem is that if I were to say, "Hey, guys, harm is clearly broken," I'd get naysayers like malkav telling me I'm full of it, that I don't know what I'm talking about, etc. However, even a glance at the spell from a person with a functioning brain leads to the conclusion that harm is really, really powerful, and no amount of playtesting is going to prove anything except that it's overpowered. (And then you'd get pretentious know-it-alls telling you how you're just trying to present your opinion as fact and that you're playing the game wrong.)
But your statements don't really have to persuade the entire forum. It is just necessary to get that information to the designer. Comments about your (or other posters) capabilities and understands should have little to no negative impact on your (or other posters) arguments when the designer reads them.
Just saying that the Bard's 20th level ability has an absurdly high DC might be enough. But, the argument would not be weakened if an actual bit of playtest data of the Bard being too successful implementing the ability, I would say that it would just strengthen the argument being made. Which I believe is a significant point of this thread. Actual playtesting is what they want and need most. Because of that, they show a definite preference for getting playtest data over other information (which they don't ignore, but doesn't take precedence as that is not the thing they want from the playtest.
Again, I agree completely. However, I honestly don't have the time to whip up mini-adventures for each and every level of play. If I can say, "The bard's ability requires a DC 40 Fort save or else the target dies, these CR 20 monsters all have less than a 15% chance of survival," then that should be good enough. What happens if my bard player doesn't use the ability very often in the playtest and the problem doesn't show up?
Y'all need to calm down.
|
|
Well if enought people palytest (which is kind of the point) statistically whatever bugs there are will pop up either here or there.
And I am pretty sure that in the case of that Bard ability they thought about it being overowered (even for a lvl20 capstone), but they left it there to give it a chance to see if it atually was so IN GAME. Do not think that the wheel was invented by yourself just because you see it and say ¨hey that thing is round, I think I´m onto something¨
And I am pretty sure that in the case of that Bard ability they thought about it being overowered (even for a lvl20 capstone), but they left it there to give it a chance to see if it atually was so IN GAME.
Actually, Frerezar, Jason admitted to making a mistake by putting it in the game. Anyone with a basic understanding of game mechanics can tell you that a DC 40 save-or-die is overpowered. It's not a matter of playtesting. It's a matter of basic competence. The argument that it needs to be playtested to see if it's broken is like saying, "Hey, let's serve sandwiches with ground up glass in them. I have a feeling that this is unhealthy, but we need to test them to see if anyone dies before making that conclusion."
More often than not, common sense and intelligence are more important than playtesting.
And I am pretty sure that in the case of that Bard ability they thought about it being overowered (even for a lvl20 capstone), but they left it there to give it a chance to see if it atually was so IN GAME.Actually, Frerezar, Jason admitted to making a mistake by putting it in the game. Anyone with a basic understanding of game mechanics can tell you that a DC 40 save-or-die is overpowered. It's not a matter of playtesting. It's a matter of basic competence. The argument that it needs to be playtested to see if it's broken is like saying, "Hey, let's serve sandwiches with ground up glass in them. I have a feeling that this is unhealthy, but we need to test them to see if anyone dies before making that conclusion."
More often than not, common sense and intelligence are more important than playtesting.
So they're (paizo) not competent or are somehow lacking in intelligence or common sense? Is that really what you meant to say? I'm not being sarcastic, I just want clarification.
What I meant is as follows:
Don't knock theoretical analysis. Math is important.
Well if enought people palytest (which is kind of the point) statistically whatever bugs there are will pop up either here or there.
This only works if you assume that you have a large enough sample, with people playing a wide enough range of levels. If you have 100,000 people testing, but 99.99% of them only test the levels between 6-10, and less than 10 people actually test the levels greater than 15, then it might not catch all of the problems. In fact, haphazard testing is often a very poor way to do it. Playtesting would be fine, but it should be assigned so that all levels and different options get equal testing.
What I meant is as follows:
Don't knock theoretical analysis. Math is important.
Nobody is knocking it. My main complaint is the large number of people who think they can do it better than anyone, and furthermore those who don't understand how vastly inappropriate it is in most situations when dealing with RPGs.
If there weren't droves of people trying to show off, I would think it a noble pursuit. Attitude counts for a lot in this. The people who use their analysis to try and passively help people (through guides and such) rather than simply barking at the Paizo staff and each other, well, they're more apt to hold my attention.
Pres man - a sample size of 32+ is usually considered pretty good, depending on the statistical tools you are using.
Pres man - a sample size of 32+ is usually considered pretty good, depending on the statistical tools you are using.
Yup, for each and every aspect. Can you guarantee that is happening? That there is in fact a uniform distribution and not, say a bell curve distribution, with most of the people testing a few aspects and very few people checking a wider range of aspects? How many level 19 play test groups are there out there? Are there enough to ensure that every single distribution of player choices, feats, equipment, class abilties, etc combos are taken into account?
Pres man - a sample size of 32+ is usually considered pretty good, depending on the statistical tools you are using.Yup, for each and every aspect. Can you guarantee that is happening? That there is in fact a uniform distribution and not, say a bell curve distribution, with most of the people testing a few aspects and very few people checking a wider range of aspects? How many level 19 play test groups are there out there? Are there enough to ensure that every single distribution of player choices, feats, equipment, class abilties, etc combos are taken into account?
It's a good question, and I support your assertion that a more structured playtest might be good — but it would come at a cost as well. We could segregate the volunteers into groups, and have quantified test scenarios, but letting people run wild tends to give better coverage of the way people are playing the game anyway.
It's a tough one.
While a certain amount of sound maths (which are hard) is a good idea (obviously there's a certain amount of arithmetic in the game), getting too crazy with it can cause issues. I used to love debating MMO (and RPG) systems and mechanics throughout the years but in the end found the games that were most satisfying were the ones that were not 100% balanced.
Equality in RPGs is boring (see: WoW). Everything is the same flavour of vanilla. However, the other extreme is also not good (OP or limp class causing frustration - see: DAoC or the 3.0 ranger).
I am also an avid board gamer and find the parallel to euro vs ameritrash board games apt here - pen and paper fantasy adventure games fall more in the latter category. In other words more visceral face-smash less abstracted systems = fun. Even if it's not 100% fair. Is a 3d6 mechanic a more mathematically accurate model of the predictability of the Sneak check of a well-trained thief? Yes. Is it as entertaining as the emotional range of the dreaded d20? No.
As usual, moderation and wisdom in all things will see you through. Now if we could just quantify wisdom... ;)
"Ameritrash"? That's a new one.
If there weren't droves of people trying to show off, I would think it a noble pursuit. Attitude counts for a lot in this. The people who use their analysis to try and passively help people (through guides and such) rather than simply barking at the Paizo staff and each other, well, they're more apt to hold my attention.
Whenever I do math analysis, I have no desire to "show off." I simply provide statistical analysis and support for why I believe something needs to be changed.
Myself, I don't use my theoretical analysis to support my argument, because that would assume I'm trying to prove a point ahead of time. I make my conclusions based on the results of the test.
Of course, I assume that the Same Game Test to be an extremely valid form of theoretical analysis.
Yes, pure number crunch on absolute maximum damage output (or AC, or Reflex save, or any other stat) without regard towards results before you can attain the build or the viability beyond that specific stat can result in inaccurate conclusions. Now, if you notice a new build is very close in various other traits with another, you can compare the differences; but that's similar to looking at the playtest results of the established build and superimposing as a shortcut. It's still a shortcut, but if done properly, can at least give you an impression of where theoretical analysis will lead.