Thoughts on playtesting

[Catharz Godfoot]

I thought I'd quote a guy who's gotten a bit on (negative publicity) in the last few days. I'm sure the guys at Paizo (Jason et al) have already thought of this sort of thing, but I think it's a good reference for the rest of us interested in play testing.

FrankTrollman wrote:

...It is after all an unprecedented attempt - it's something which could theoretically achieve the kinds of results that the distributed network of D&D fans can already achieve - breaking the system within hours of publication. Regressed repeatedly, that could make a system that was resilient enough to withstand the vast majority of campaigns - maybe even all of them...

...set up a basic set of Bug Reports based on severity and Frequency, and then you'd have an entirely separate section where you discuss the equally important (but completely distinct) accessibility questions of Complexity, Cogency, and Coolness. So basic severities might look something like this:

A - An 'A' bug stops play or destroys the game world. An example of that would be The Shadow Over the Sun. Another example would be the Kokrachon from the [u]Book of Vile Darkness[/u]. It's CR 6 and it cast's blasphemy once per day at caster level 12. It will kill your entire party with no save, which stops play and is therefore of equal severity with any infinite power loop.

B - A 'B' bug severely impacts play or the campaign world. Gross balance disparities, economic inconsistencies, incorrect CRs on monsters, and the like are all B bugs because they produce a game which is not the game that is advertised. If people can fabricate or Chain Bind to get all the gold they can imagine and they can turn that into epic magic items, that's a B bug because the game claims that people of a certain level have certain magic items. If a Monk of whatever level is categorically incapable of contributing as much as other classes, that's a B bug because the game presents all of the classes as being viable in play.

C - A 'C' bug has a modest impact on play. Minor balance issues and weird numeric effects at the edges of the random number generator are C bugs because the game can shrug its shoulders and move on even when they come up. Things which can be played through and have the game mostly behave as advertised, without resorting to Rule Zero to make this happen are C bugs. Monsters which are too weak for their level are almost always C bugs because nothing bad happens to the game if the PCs get extra treasure and XP for smacking around an easy challenge. If one weapon is clearly superior to others it is a C bug, because the game doesn't stop being playable if everyone uses a lucerne hammer rather than a glaive (or whatever).

D - a 'D' bug is simply a matter of preference. If Rogues end up being classed as crossbow snipers and you want them to be dagger fighters, that's a D bug. There is nothing inherently wrong with characters fighting with swords or fists or golden teddy bears on sticks. So if a class is otherwise balanced, but is doing things you don't like, that's a D bug. It's a D bug because it is entirely possible that someone else does like the new stuffed animal fighting classes, and it is held up to a different standard than a mathematical anomaly would be.

[Leress]

This would really help in unifying how we are suppose to playtest this Alpha release and future releases.

[Cranus]

Good post. Something that everyone playtesting should keep in mind.

[Rhishisikk]

I would propose a second axis to determine how widespread the 'bug' is. How likely it is that this will enter the game.

[Catharz Godfoot]

Rhishisikk wrote:

I would propose a second axis to determine how widespread the 'bug' is. How likely it is that this will enter the game.

That would be nice too, although it can be hard to tell. Most things are conditional on whether they appear in a campaign. Classes, feats, spells, magic items, and monsters all have to be intentionally selected. Bugs in them will be very common in games they're in, and nonexistent without.

Bugs in things like economics, skill ranks, and basic actions available to all characters will probably be encountered in every campaign. For things like economics it can be difficult to find problems running one-shot or prewritten adventures.

Ideally one would want to fix any serious problem, regardless of how uncommon it seems to be. Normally that would be a little unrealistic, but with an open playtesting community like this it might be worth shooting for.

[Skeld]

Good in theory, but I don't think it will work well when the rubber meets the road. At least not in this kind of playtest environment.

-Skeld

[Psychic_Robot]

Skeld wrote:

Good in theory, but I don't think it will work well when the rubber meets the road. At least not in this kind of playtest environment.

-Skeld

How do you mean?

[SirUrza]

So he gets banned and now his friends are posting on the forums for him? Wtf.

[hogarth]

Catharz Godfoot wrote:

Bugs in things like economics, skill ranks, and basic actions available to all characters will probably be encountered in every campaign. For things like economics it can be difficult to find problems running one-shot or prewritten adventures.

Reporting bugs works well for obvious things like typos (e.g. in the SRD, a jann is listed as having the ability Plane Shift, but it doesn't say how many times per day it's usable, whether its only usable on itself, etc.).

Where bug reporting gets bogged down is where one person considers something "A" priority and someone else couldn't care less. For instance, person X says that being able to use Planar Binding on efreeti to get Wishes is campaign-shattering, but person Y argues "No way; if you tried to do that the whole City of Brass would come after you and try to kill you -- therefore it's not a problem." And then you get some lame-ass flame war over whether Rule Zero is a good solution or not. Ugh.

[Snorter]

SirUrza wrote:

So he gets banned and now his friends are posting on the forums for him? Wtf.

But at least this poster does so in a pleasant way.

And he has allowed us to state our opinions instead of telling us what our opinion should be.

[cappadocius]

SirUrza wrote:

So he gets banned and now his friends are posting on the forums for him? Wtf.

What? Are you new? This is the Internet. Of course this is how it goes.

At least this one's polite. :)

[Virgil]

The point is quite valid, so I don't see the point in trying to turn it hostile against the source regardless of its own merits.

While a typo can be a bug, not all typos are bugs. It's not a typo when you have the Shadow Over the Sun, but it remains no less a powerful A bug.

Rule Zero is used too often as a cop-out for when the rules prove to be flawed, and should not be of such importance when actually designing the RAW. It's like saying a tuna grinder is awesome as long you use ham instead of tuna.

[Skeld]

Psychic_Robot wrote:

How do you mean?

Hogarth hit on one point: what's "campaign world breaking" to one bug reporter, is someone else's "meh" problem.

With potentially thousands of people in a playtest, the sheer number of bug reports would probably be overwhelming for 1 person to process. Not to mention that each one would have to be dispositioned in some way, and many would likely be duplicates.

A non-trivial number of reports of "bugs" would be made by testers that had misread the rules or didn't understand how something peculiar to Pathfinder worked. Those would essentially be "waste of time" bug reports that would still have to be reviewed before someone could determine that it isn't really a problem.

Another non-trivial number of bug reports would be written from a standpoint of "user preference." A tester might write a report based on something that worked in a way that he just didn't like, like skill points or favored classes. Again, these would likely constitute "waste of time" reports because, technically, nothing would be broken.

IF the bug reports were in the form of message board posts, there would be no end to the amount of "this is broken!" vs. "no it isn't!" style arguements.

Those are just the ideas that quickly come to mind. I'm sure I can come up with more if you need.

I'm not saying that this would be an impossible system, I'm basing my comment on the facts that:
1)it isn't already in place ()and it would have to be in place before testing started),
2)Paizo has 1 full-time person dedicated to the task, and
3)has the potential to be very inefficient (based on the OP's proposal, barring any tweaks).

Bug reporting processes are complicated in the software/harware world. Port that to a RPG system that is highly subjective and completely reliant on Man-in-the-loop decisions and it could be a nightmare.

-Skeld

[James Griffin 877]

I hate to remind you guys, but WE care more about our input and opinions over this than THEY do. I mean...really. Whatever they produce, our geek subculture is going to buy it, or at least pirate it and use it. In my opinion, they've made something that has a good chance of standing up the more new and mysterious 4.0.

I mean our play-testing is really only to confirm or deny their own doubts about their mechanics changes and to get an idea of what they've produced will sell to us and we know it will.

So all this, "let's organize," "let's establish rules for x,y,z," are just pointless in the end. The alpha3 is just going to probably have the most glaring of errors changed from alpha2 with the addition of the ranger and monk i think. (i don't know about all that.)

Look at the boards. There may be new threads discussing material, but I haven't noticed a thread about any problem that someone didn't post last week in a similar fashion, and Paizo notices this too, I'm sure. So whatever changes we might have helped implement have more than likely already been make. They are on a production schedule you know.

Just keep giving them what they want- an unpaid test audience, and wait for the alpha3/beta.

Sorry for being pragmatic or pessimistic, but this is business. Just because it's for and by geeks/nerds doesn't mean the way business works is any different.

Happy posting!

[Praetor Gradivus]

SirUrza wrote:

So he gets banned and now his friends are posting on the forums for him? Wtf.

Nothing in the OP message s inappropiate so it does not matter where it originates from.

As to the suggestion... it is a fair idea except that people view certain areas of 3.5 in a different light so even if two people think the same occurence is a problem, one might think it an A Bug and the second a D Bug.

If we take sneak attack as an example: some people view Allpha2 sneak attack as no problem others as it is an A Bug. Some people view the Alpha 2 version a C Bug while the 3.5 version is not. And there are people that think the 3.5 version would be an A Bug so don't even want to consider the Alpha 2 version. Did i forget to mention the minority that think sneak attack isn't powerful enough because rogues have 3/4BAB instead of full BAB?

It's nice to try to categorize the problems as you see it, but many thing are rather subjective. I really don't believe that a DND game can be viewed purly in a mathematical way as it is a roleplaying game. It's not like a chess federation where the rules are applied the same regardless of where you play. DND by it's very nature is customized from group to group and so what unbalances a DND game in one group may not in another because each has their own rules.

But heck, try and list them with this system because it at leasts helps rate how big a problem the tester percieves it to be.

[James Griffin 877]

Is that you, Ogre, under a different alias? It sounds like you and has your picture. Lol.

[Catharz Godfoot]

Praetor Gradivus wrote:

If we take sneak attack as an example: some people view Allpha2 sneak attack as no problem others as it is an A Bug. Some people view the Alpha 2 version a C Bug while the 3.5 version is not. And there are people that think the 3.5 version would be an A Bug so don't even want to consider the Alpha 2 version. Did i forget to mention the minority that think sneak attack isn't powerful enough because rogues have 3/4BAB instead of full BAB?

That's a good point. I think it shows how important it is to base what you report on playtesting or running the numbers very carefully. Just eyeballing the rogue and saying 'that's no good' or 'it's fine' is no help to anyone.

Praetor Gradivus wrote:

It's nice to try to categorize the problems as you see it, but many thing are rather subjective. I really don't believe that a DND game can be viewed purly in a mathematical way as it is a roleplaying game. It's not like a chess federation where the rules are applied the same regardless of where you play. DND by it's very nature is customized from group to group and so what unbalances a DND game in one group may not in another because each has their own rules.

*My* hope is that when Pathfinder is released, it will not only will have all of the new improvements (skill ranks, combat feats, more class abilities for fighters and rogues, etc), but it will also be free of many of the balance and internal consistency problems of 3e. Enough at least that the rest can be taken care of with errata and not things like the polymorph fiasco.

Although many people will make changes in the rules that might throw it out of balance, at least then we can know it plays great before we mess it up. :)

[Cthulhu Dreams]

This sort of bug testing is only possible in an environment with

A) Clear objectives

B) Measurable test standards.

We are discussion a simulation that uses mathematical formula to take inputs and transform them into (flavorful) outputs.

Thus we can use maths and analysis, and that allows you to categories bugs by what they break in the test suites.

Say for example we have a 'unit test' for the 'Six Sigma Expert' class

That test should be defined, say as a series of encounters with proscribed tactics that he should win half of (or something else. I don't care, just as long as it is repeatable), and then various test results cause a certain level of bug reports.

For example, if the six sigma expert wins 100% of all encounters at levels 1, 3, 6, 9, 12, 15, 18 then we probably have a class A bug that needs to be identified.

If the six sigma expert just bugs down combat hugely and it takes 25 minutes on average to resolve each encounter instead of the 5 we've specified, then thats a probably class B

But if he wins 45% of encounters in 6 minutes on average then thats a class C or D

It shouldn't be a matter of debate, it should either be true and reproduceable, or not true and not reproduceable.