I love that Paizo does open playtests. I think it is very much in the spirit of Open Gaming, and it likely results in a much more robust and polished product than closed-door design would. I think all those who participate in the playtests are really grateful for the opportunity to help build the game.
That said, I opted out of the APG playtest. Well, not true. I played the game and tested the material, I just wasn't active on the forums at all. At some point, I shut the APG forums and never looked back. I've actually felt sort of guilty, because I was playing and not giving feedback that might have helped the game!
Recent mention of a (possible) Epic level book and a (possible) psionic book which would also be playtested openly has got me thinking: What would it take for me to actually participate the next time around?
I believe it is possible for the "solo" poster without a group to contribute to the playtest, by looking at the rules and making judgements about what might happen. Unfortunately, there is a vocal minority of posters who simply seek to self-aggrandize and intellectually compete with other posters, and I find that to be counter-productive. That attitude is what drove me away, not the whole "playtest v. lookjudge" controversy.
I can't help but feel the signal-to-noise ratio for the designers is catastrophically low when so much of their feedback is merely intellectual jousting. My first thought, when applying the greater internet d**kwad theorem is to remove the "public" element of at least some of the feedback. If there were a "playtest report" form or somesuch thing built into the forum, that would solve two problems at once: remove the audience for the posters whose motives are not constructive, and allow posters to give feedback without going through the discussion forums.
I am not starting this thread to disparage other posters, but rather to gather ideas on how the playtest process might be improved to the advantage of both testers and designers. I'm especially keen to hear the ideas of others on this matter!
So, if you were in charge of the next open playtest for Pathfinder RPG, how would you manage it?
+(a lot)
1) I'd have a much more structured playtest in that I'd have online forms with specific questions I'd like testers to answer.
2) I'd have a forum thread where individuals would post their report or opinions, and only Paizo folks could respond.
3) I'd have very specific testing scenarios outlined, such as "low level testing", "mid level testing", and "high level testing"
4) I'd have a phase prior to release of the product where I ask X (25? 50?) number of fan volunteers to sign an NDA and then proofread the proposed final product. I'd give them a free signed copy of the printed product and maybe even include their name in a "playtest block" on the last page of the product. They would have 2 weeks to respond with any errors at which point the product goes to print or back for a round of corrections.
5) I'd make a point early in the playtest cycle to have a discussion amongst my fellow Paizo employees singling out specific fans who seem to have a very solid grasp of the rules and ability to post coherent thoughts, and then I'd contact them and ask them to sign an NDA then review specific subsets of the mechanics for errors I/my fellow Paizonites may have missed. I'd also give them a free signed copy of the printed product and maybe even include their name in a "playtest block" on the last page of the product.
Well what i would do is provide 2 sets of forums. One theory and one practical. I do believe that the 'intellectual' discussion of the rules by just looking at them is useful. The early realization that allowing eidolons to wear armor would lead to insane AC's was found by simple analysis of the rules. But it is also not final. What the goal here would be to have the discussion that is all theory, and then use the theories to set up situations for playtest.
So there would be 2 sections, general discussion and playtest reports/discussion. A sticking in the playtest reports/discussion would be a thread to allow poster to post ideas to be tested.
So for examples lets look back at the APG. Someone thinks all the small bonuses the Inquisitor gets would be too hard to track. There is a discussion about it in the first section of the playtest. People arent sure so someone (hopefully only 1 person) posts in the sticky thread 'hey we/i/the general concencus believes that an inquisitor of level x would be very hard to manage because of all the fiddly bits you have to track, someone try it out and get back to us'. Then in theory someone tries it, posts it and reports back both in the Playtest report section and the general discussion section what they found.
Basically I think we could benefit from taking play test and making it play TEST, with actual test conditions, and then experiementation to confirm or deny the premise of those test conditions.
Designers could also post up ideas they have. "Hey what if the eidolon could only wear magic items in slots the summoner isn't using". People try it out and report back.
I would still encourage people to use the material in 'natural' play, but I think encouraging direct testing of ideas and rules would be a very good addition to the playtest.
Well, I also think that the forum discussions work, I would just like to see the process refined and maybe more options added?
When the designers offer specific scenarios, that's good, but it also introduces possible blind-spots in the design that the scenarios didn't account for.
I guess what I'm saying is that I'm not in favor of removing or even significantly altering the Playtest forum model. I would like to see that stay, but add some additional channels for feedback...
Maybe a part of the site that was literally just playtest reports and threads which were only on-topic in response to those reports... sort of like the Blog and the blog forums. It would be interesting to see a section where all discussions were based on a playtest report — and maybe have a standard format for the report, not unlike the Obituaries threads. Informal enough to accommodate style-of-play variations, but more structured than "whoever is the most inflammatory gets the most responses".
So perhaps a "playtest reports" section of the forum. You start a new thread by filling out a form with slightly more detail (maybe even the statblocks of all players in the scene) and all discussion follows that topic.
The existing playtest/lookjudge forums can stay, but something a little more rigid would help me in filling out reports and keeping things on-topic.
Another interesting possibility would be more "Playtest by post",
where one poster actually assumes the role of a GM and lets people try to break his scene. I imagine being able to read actual encounters playing out would have a very high signal-to-noise ratio, and the playtesters could create a standardized summary at the end of each encounter.
Hmmmmmm... In fact, I think that's the way things should lean.
Another interesting possibility would be more "Playtest by post",
where one poster actually assumes the role of a GM and lets people try to break his scene. I imagine being able to read actual encounters playing out would have a very high signal-to-noise ratio, and the playtesters could create a standardized summary at the end of each encounter.Hmmmmmm... In fact, I think that's the way things should lean.
I actually did a fair amount of this with a couple people for the summoner. Its great in theory, but in the limited time scope of the playtests its unrealistic. Play by post takes alot of time.
I think both methods have validity.
During the APG play tests I saw a lot of "This is broken" types posts, but when you dig into the exchange of posts that followed you would find a host of player or DM errors that were involved. The exchange of posts often forces the person with the original complaint to provide more information that they did originally did.
Without that exchange I suspect half of the posts on summoners would have been like this.
"One of my players played a summoner, and he broken my game, nerf the summoner please"
Without more details on the whys and hows a post like that is pretty much useless.
It's the format, partly. The same problem exists with non-playtest forum discussions: The thread title and OP will always be more salient than any resolution achieved in the thread. You can discuss things till all involved feel they've got a solid grasp of the topic, and then a new poster will waltz in, read the OP, and post their (equally valid) opinion.
Anyway, I'm not saying we should abolish the old way, but suggest some additions perhaps? The current format is probably not perfect. Suggestions?
I think creating a standardized playtest report would be useful regardless. When I did playtest I wasnt always sure what information to record, what level a detail was useful, or I was just preoccupied with running my normal game if it wasnt exclusively a playtest. If I had a form handy that i could write notes for as we went, it would be much easier to give a more useful playtest report.
A standard format, and perhaps locking posts that don't use it would be nice.
A standard format, and perhaps locking posts that don't use it would be nice.
Even just a self-enforced standard format honestly. Something on the level of the Obit threads — you can break the format if you have to, but it just makes things easier for readers!
Some great ideas so far.
I would disagree with listing the names of the "chosen playtesters, proofreaders". As much as I would like to think I would be among the few, the proud, the band of... where was I... I think that EVERYONE who takes part in the playtests should get credit. And that is best done in the form of a brief paragraph, or single sentence in the book. I would much rather have a table, some art, or just more content, then see my own name in print, as gratifying as that would be.
If the play tests are going to be short, it would also be nice if we were provided with a few samples to work into our games. Or perhaps better yet, a section of community created NPC's.
I'm also a fan of very heavy moderation. I think Paizo does a great job of keeping the forums civil (and it shows!), but I would be totally happy with locking/ deleting threads, removing posts, etc. I know a few folks will complain, but it makes for a much better overall environment.
Some great ideas so far.
I would disagree with listing the names of the "chosen playtesters, proofreaders". As much as I would like to think I would be among the few, the proud, the band of... where was I... I think that EVERYONE who takes part in the playtests should get credit. And that is best done in the form of a brief paragraph, or single sentence in the book. I would much rather have a table, some art, or just more content, then see my own name in print, as gratifying as that would be.
If the play tests are going to be short, it would also be nice if we were provided with a few samples to work into our games. Or perhaps better yet, a section of community created NPC's.
I'm also a fan of very heavy moderation. I think Paizo does a great job of keeping the forums civil (and it shows!), but I would be totally happy with locking/ deleting threads, removing posts, etc. I know a few folks will complain, but it makes for a much better overall environment.
It wouldn't even need heavy moderation. If someone at least tries to use the proper format, let is slide. Even if they don't get the format 100% correct, the effort counts for something. The moderation is mainly a method to filter out the people who are venting and have not even tried to do so in the correct format.
It wouldn't even need heavy moderation. If someone at least tries to use the proper format, let is slide. Even if they don't get the format 100% correct, the effort counts for something. The moderation is mainly a method to filter out the people who are venting and have not even tried to do so in the correct format.
Or just build some fields into the submission form and let people use them. "Deciding Factors:", "Party Makeup:", "House rules employed:" etc etc.
That way people wouldn't have to educate themselves on the format before making a report, they could just work with the suggestions implicit in the fields. Sort of like how we all do our own thing with the profile fields on this board.
It wouldn't even need heavy moderation. If someone at least tries to use the proper format, let is slide. Even if they don't get the format 100% correct, the effort counts for something. The moderation is mainly a method to filter out the people who are venting and have not even tried to do so in the correct format.Or just build some fields into the submission form and let people use them. "Deciding Factors:", "Party Makeup:", "House rules employed:" etc etc.
That way people wouldn't have to educate themselves on the format before making a report, they could just work with the suggestions implicit in the fields. Sort of like how we all do our own thing with the profile fields on this board.
I was leaning the other way. Make educating yourself about the process part of the process. People who just want to rant are less likely to do this, and it creates a fairly quick indicator of who can be safely ignored.
I was leaning the other way. Make educating yourself about the process part of the process. People who just want to rant are less likely to do this, and it creates a fairly quick indicator of who can be safely ignored.
As much as I would love to correct the behavior of a few individuals, I don't think the status quo is all that bad if the designers are getting what they want out of it.
People are going to rant either way. I certainly believe the most hard core intellectual masturbators are going to learn the format. I'd be just as concerned about letting people contribute who don't want to really commit to learning a format (maybe because they spend a lot of time running their campaign).
Still, I think we're onto something here. Anyone have any thoughts on an actual report design?
For reporting PCs participating in a playtest encounter, what level of detail is necessary?
Ok, I'll take a stab at it.
Party composition: classes, levels, stat generation method. This is very important for getting a baseline. A group full of multiclass wonders is going to make any pure class in OP. It also gives you an idea of what level of play level you are dealing with.
Encounter 1: We fought a X in location Y. The fight lasted Z rounds. Give a general outline of how the encounter played out. The fighter ran up and oneshotted Monster A, charge B, and killed him with his full attack on round 3. A detailed blow by blow is not necessary, but the general performance of each class, buffs that were cast, etc.
Encounter 2: Same song, second verse.
Encounter X: The more examples, the better.
Conclusion: A description of the pluses and minuses encountered. Class destroyed encounters of type A, but floundered against flying opponents, etc.
As much as I would love to correct the behavior of a few individuals, I don't think the status quo is all that bad if the designers are getting what they want out of it.People are going to rant either way. I certainly believe the most hard core intellectual masturbators are going to learn the format. I'd be just as concerned about letting people contribute who don't want to really commit to learning a format (maybe because they spend a lot of time running their campaign).
Still, I think we're onto something here. Anyone have any thoughts on an actual report design?
For reporting PCs participating in a playtest encounter, what level of detail is necessary?
Yes, but everyone here already knows me and has me on ignore already....
Ok, I'll take a stab at it.
Party composition: classes, levels, stat generation method. This is very important for getting a baseline. A group full of multiclass wonders is going to make any pure class in OP. It also gives you an idea of what level of play level you are dealing with.
Encounter 1: We fought a X in location Y. The fight lasted Z rounds. Give a general outline of how the encounter played out. The fighter ran up and oneshotted Monster A, charge B, and killed him with his full attack on round 3. A detailed blow by blow is not necessary, but the general performance of each class, buffs that were cast, etc.
Encounter 2: Same song, second verse.
Encounter X: The more examples, the better.
Conclusion: A description of the pluses and minuses encountered. Class destroyed encounters of type A, but floundered against flying opponents, etc.
Information on any used house rules is also a must. As well as standards for character creation and distribution of gear. There is a difference between a level 8 fighter with a 10pt buy and 1k in gear and a level 8 fighter with a 30pt buy and 50k in gear.
Information on any used house rules is also a must.Ok, I'll take a stab at it.
Party composition: classes, levels, stat generation method. This is very important for getting a baseline. A group full of multiclass wonders is going to make any pure class in OP. It also gives you an idea of what level of play level you are dealing with.
Encounter 1: We fought a X in location Y. The fight lasted Z rounds. Give a general outline of how the encounter played out. The fighter ran up and oneshotted Monster A, charge B, and killed him with his full attack on round 3. A detailed blow by blow is not necessary, but the general performance of each class, buffs that were cast, etc.
Encounter 2: Same song, second verse.
Encounter X: The more examples, the better.
Conclusion: A description of the pluses and minuses encountered. Class destroyed encounters of type A, but floundered against flying opponents, etc.
I would limit it to applicable house rules, and put it the appropiate section. If a house rules didn't come into play, there is no point in listing it.
IE if the house rule pertains to character creation, then put it the party composition section. If the house rule pertains to an encounter, then put it in the encounter where it came into play.
My suggestion for heavy moderation applies to ALL of the message board threads, not just forms. From my experience, a few folks complain about "freedom of speech", but it makes for a much more cordial environment.
EL - "For reporting PCs participating in a playtest encounter, what level of detail is necessary"
I think some of the important things include: Point buy (or total ability bonuses), HP method , wealth, and anything not core races, classes, etc. I also think there should be a note about alternate rules, house rules, and difficulty. Although the later items are often almost unconscious habits.
I think also a mention of any crazy luck might help isolate extreme cases.
I was thinking if Paizo could provide a low level sample of the material, a mid level, and high level, that could get folks testing quickly, and perhaps provide some more even basis for comparisons. Of course, folks would still be able to come up with their own test cases.
EDIT: Damn I type slow! 10 minutes, and I get ninja'd left and right!
While I have very many ideas for improving the playtest process (the value of those opinions is certainly a matter for others to judge) I will refrain from expending the effort of posting those ideas until I have at least an inkling that Paizo might implement any of the ideas we post here.
|
I poked around the playtest forums some. It gets exhausting hearing the exact same complaints over and over over and over over and over over and over! Some feedback from the designers to alleviate the 600,000th post about the same thing would be nice.
All in all, due to a perceived lack of interest/involvement of the designers, I decided it just was not worth the effort.
Do I think the playtests could be improved with a bit more structure?
Yes.
Do I think that the playtests yielded useful results for both the Core Rules and the APG?
Yes.
Do I wish people at Paizo had more time to directly respond to some of the playtest data?
Yes.
Do I think Paizo is intentionally ignoring feedback and good advice due to lack of interest or some other reason?
No, not at all.
Do I think that it does any good to act snarky and tell everyone on the forums that you know better than Paizo and make dramatic statements about how much better you could have done it?
No, I think all that does is make it look like some individuals are fishing for attention, and want to be patted on the back for how brilliant and dedicated they are.
Can Paizo take what they have learned from playtests and refine the process?
Yes, and I think there were some improvements between the Core Rulebook forums and APG forums, but it still wasn't perfect. On the other hand, if people are just going to spend more time criticizing the process and Paizo because its not how "they" would do it, I wouldn't blame them for playtesting internally and avoiding the headache.
I'm a big fan of the notion of playtester-driven improvements. Standard Formatting, communal statblocks and encounters, and an emphasis on rigorous exploration are all things we can do on our own as playtesters.
If there is one thing I'd like to see from the designers that could improve the quality of the playtest, it would be clearer directives. For instance, knowing that a new class shouldn't be outshining another existing class in a specific category; or that a class should be able to hold its own in a given category; or that a deficiency has been included as a deliberate weakness. One full page of design objectives at the beginning of the playtest would allow the playtesters to really hack away and see if the goals were actually met or not.
+(a lot)
1) I'd have a much more structured playtest in that I'd have online forms with specific questions I'd like testers to answer.
2) I'd have a forum thread where individuals would post their report or opinions, and only Paizo folks could respond.
3) I'd have very specific testing scenarios outlined, such as "low level testing", "mid level testing", and "high level testing"
4) I'd have a phase prior to release of the product where I ask X (25? 50?) number of fan volunteers to sign an NDA and then proofread the proposed final product. I'd give them a free signed copy of the printed product and maybe even include their name in a "playtest block" on the last page of the product. They would have 2 weeks to respond with any errors at which point the product goes to print or back for a round of corrections.
5) I'd make a point early in the playtest cycle to have a discussion amongst my fellow Paizo employees singling out specific fans who seem to have a very solid grasp of the rules and ability to post coherent thoughts, and then I'd contact them and ask them to sign an NDA then review specific subsets of the mechanics for errors I/my fellow Paizonites may have missed. I'd also give them a free signed copy of the printed product and maybe even include their name in a "playtest block" on the last page of the product.
+ to all of this, especially #1
Okay, is there a way we can get all playtest-relevant PC info in one line?
Something like :
Easiest way to do a playtest of new character options:
Allow the Beta versions of those character options in PFS play for three or four months. Include a special playtest report form that players using the new character options must complete at the end of each scenario they play.
Limiting it to PFS play cuts out a huge portion of their player base, and leaves a very small subset of the potential situations to be tested. PFS is one of many ways to play the game, you want to playtest the new options in as many of those as possible
Ok So its not quite simple, but here is a start to what I think should go in a player driven playtest report.
Playtest Report:
General Information:
Level:
Character Creation Guidelines:
Adventure/Module (NA if homebrew):
Wealth/Treasure Guidelines:
Optimization in the group (Scale of 1(8 Int Wizard) – 10 (Codzilla))
Desired Difficulty of encounters (1(Chumps) – 10 (TPK))
Party Makeup:
Player 1
Class(es)
Intended Role:
Stats:
Important Magic Items:
Brief Description of the characters important mechanical Aspects:
[Repeat as Necessary]
Ecounter:
CR:
Brief Description of Setup and situation:
Enemies Involved:
Intended Result of encounter:
Actual Result of encounter:
Any relavent Houserules/homebrew elements that lended themselves to the Actual Result:
DM explanation for difference between Intended and actual result:
Player explanation for difference between intended and actual result:
[Repeat as necessary]
Impressions of Material Tested: