I feel like I'm in Bizzaro world.

I avidly followed RPGSS since 2010 when I found out about the contest a week too late to actually submit an item. I did submit last year and this year.

There were, in my mind, mostly fantastic items from year 1 through this year. As is everything in life, not everything is going to appeal to everyone, and to that truth, not every item this year (or any of the years for that matter) did I think good.

In the years past, the Judges had some wonderful commentary and feedback on the top 32 items.

This rant is not about how good or bad items that made it are, or why mine didn't.

This rant is taking the judges to task.

Why you ask?

I have read only the first column of items so far, and I think only two of the 16 have mostly favorable comments. The majority of the items were weak keeps by the strongest supporter, and outright rejects by at least two other judges.

In the past, the reveal of the top 32 round was a celebration of potential and even excellence in writing acumen. But this year feels different.

While I felt the actual items this year could stack up to the fantastic items from any prior year, the judge commentary seems to belie that statement.

Am I in Bizzaro world? Most of the commentary I'm reading from the top 32 actually feels negative enough that it seems I'm reading the "Critique my Item Thread".

I think the judges are being overly and hyper critical, perhaps even pedantic with their commentary. If I had to judge the items just by their commentary alone, then there wouldn't be a top 32, there would be a top 3 or 4. If these were the best of the best, the top 32, based solely on the judge commentary, the rest of the field (including my item) must have really, really sucked.

Is it just me, or is there a change in the level of standard required to get even a weak keep these days? If so, the level of standard didn't just raise a notch, it raised many several in the last year.

Ok, rant over...

I think what is concerning to you is the language we use and what we are discussing and when.

Remember, that is our discussion after the item has been put into the keep folder.

Essentially, we are sorting out the best of the best.

We've already decided that these items are in the top 3% of all items submitted. So seeing us discuss problems seems magnified. We are seeing what items are consensus keeps and we are very very picky. But remember, these items are items that we decided were keepers already. Maybe once an item is kept we should change our language to "lock" or "not a lock" or something like that.

Plus, I can attest that this year you are getting it a bit more raw and unedited than you ever got before. I did almost no editing of my comments (or very little). In the past we removed comments that might be seen as negative for fear of creating the very impression you are getting.

So please remember, these comments are us sorting through items we have already put in the keep folder--about 50 or 60 items out of hundreds upon hundreds upon hundreds upon hundreds of items. And the nature of the process is to find weaknesses, so that is what we are discussing. So it may seem negative but it isnt. Once you find 50 really good items, there is no other way to sort them.

We are not as negative about the items as we sound, but I understand what you mean.

In fact, I would say this might be one of our strongest fields yet.

I agree with the OP. Even prior to the final 32, the judges comments about the items they saw coming in seemed aggressively critical to me.

Remember, we are tough because we HAVE to be--this is RPG-freaking-SUPERSTAR!!! This is not "RPG Good enough to make a book of magic items-star".

We NEVER have 32 totally awesome, perfect items. It has never happened in the history of the contest. In fact, you only have a couple each year that are even total consensus items from the judges. Frankly, this year was impressive because we had 20 items there were 3 judge consensus items. That is impressive. I dont think you guys appreciate how hard it is to impress all 4 of us. Let alone 3 of us or even 2 of us enough to say "this person should be a Superstar contestant." That is hard, and it should be hard!

Why is that? Because if there were people that good out there they would likely already be freelancers.

We find good, new, raw talent and give them the chance to develop over the course of the contest. It's like American Idol. Look at the contestants when they do their first audition against when they do the top 8 or so. By then, those raw talents have been polished, have a new hairdo, new wardrobe, some extra vocal lessons, mentoring by industry insiders, extra work on their stage presence, etc. By the top 8 they are way better, more polished and developing nicely. Its the same thing here.

Andrew, Abbasax, I can only answer from my own perspective, not for the other top 32, but this is how I feel about the judge's feedback on my item.

1 - Good feedback is useful feedback, feedback that helps me improve
2 - Good feedback should mix praise with constructive criticism (on the keep doing, stop doing, start doing model)
3 - I already got all the praise I can handle. They picked my item. For RPG freaking Superstar. My item! Any more praise and I'd need a new hat.
4 - So the useful stuff is the criticism. How could this have been even better? What did I miss, flub, or not even consider? That's gold because I can take it into the next rounds and do better.

So it's not negative, and it doesn't feel negative, because the criticism is mixed with the best praise there is... "Keep".

Remember, we are tough because we HAVE to be--this is RPG-freaking-SUPERSTAR!!! This is not "RPG Good enough to make a book of magic items-star".

I try to chalk it up to the fact that there's no inflection when reading something on the internet- I can't read a person's face or judge intentions through body language- but read straight out, statements like that are incredibly condescending. It's an attitude I felt like I kept running into throughout the entire first round.

Of course we know it's RPG Superstar. We wouldn't have entered otherwise. Of course everyone who was serious about it tried to bring their A-game, and of course some A-games are better then others. There's nothing wrong with that.

I just feel it's completely dismissive of the those that didn't make the first cut. It heavily insinuates that they didn't make it, not because there were better entries, but because they suck.

I felt this way through out the entire process this year. This isn't the first time I've entered a my work into a contest, but this is the first time that I felt like the people in charge of it were actively waiting for me to fail.

It wasn't a good feeling.

I've never understood the use of negative feedback to inspire someone to do better (maybe because I've never watched America Idol).

I admit that I'm probably in the vast minority (if not the sole person) who feels this way about how it was handled. It's your guys' show, and you're welcome to run it anyway you want. If this is what it takes to allow you all to keep making a great product, then I support it.

Regardless of my sour taste on this, I want you to know that I really do appreciate Paizo for giving everyone the opportunity to grab the gold ring, and for all of the judges for taking the time and effort to look through all those crazy entries. Please don't let my mini-rant give the illusion that I have anything but respect for you all.

So it's not negative, and it doesn't feel negative, because the criticism is mixed with the best praise there is... "Keep".

Congratulations on that by the way!

I feel bad, because I co-oped the OP's thread for my own grindstone: That I felt like the feedback that was given while the first round entries were still coming in were sometimes too caustic. (Basically, I feel like a lot of them boiled down to: "We keep getting entries where [X] has been done wrong! Why can't you people be smarter?"). Which pretty much turned me off to the entire process.

As I said above though, it's probably just a me thing.

Looking over of the items, the great ones shine and the judges did a great job of recognizing that. Other borderline items, like mine, received honest opinions about what the judges thought of them. If only praise was given, how could a writer trying to improve know what to work upon? I would rather have a caustic and honest opinion, than a sugar coated "Almost there, try again next year!"

I tend to be brutally honest, which most people can't take. So when someone is harsh to me and I deserved it, I say "Roger" and move on. If not, I say "Roger" and ignore their [redacted].

I think what is concerning to you is the language we use and what we are discussing and when.

Remember, that is our discussion after the item has been put into the keep folder.

Essentially, we are sorting out the best of the best.

We've already decided that these items are in the top 3% of all items submitted. So seeing us discuss problems seems magnified. We are seeing what items are consensus keeps and we are very very picky. But remember, these items are items that we decided were keepers already. Maybe once an item is kept we should change our language to "lock" or "not a lock" or something like that.

Plus, I can attest that this year you are getting it a bit more raw and unedited than you ever got before. I did almost no editing of my comments (or very little). In the past we removed comments that might be seen as negative for fear of creating the very impression you are getting.

So please remember, these comments are us sorting through items we have already put in the keep folder--about 50 or 60 items out of hundreds upon hundreds upon hundreds upon hundreds of items. And the nature of the process is to find weaknesses, so that is what we are discussing. So it may seem negative but it isnt. Once you find 50 really good items, there is no other way to sort them.

We are not as negative about the items as we sound, but I understand what you mean.

In fact, I would say this might be one of our strongest fields yet.

Thanks for the response Clark. And I really do appreciate all the work you all go through to get this done.

I was just surprised at how negative the feedback felt for the top 32. But now that you've explained it, it makes sense.

Perhaps in future years, if you also included your initial responses that are, "ooh, look, shiny!" it would help mitigate the second review, which must of course be hypercritical.

Yeah. I had the same feeling.

But knowing that these are the comments after the initial keep puts things back into perspective.

That said I liked the item of last year generally more than this year and from two years ago.

Oh, I think the main issue I and others may have with the way you've presented the critique for the top 32, is that the comments don't feel any different than the comments given for those who didn't make it.

So my initial response on reading Jacob Kellog's item (I read his first, because he is a local PFS player and I see him on a regular basis... Yeah Jacob!) was, wow, this is pretty cool. Holy crap, the commentary is really slamming his item. With this commentary, how did he make the top 32?! If this is what they are putting in the top 32, with this type of commentary, how did some of the items that didn't make it, not make it?

Which is why I suggest actually putting the "Ooh! Shiney!" initial posts first, as well as the follow-up paring-down critiques.

Asking the general public to assume you all think these are great items because they made the top 32, while your words speak otherwise, I think is a big leap of logic and counterintuitive.

The presentation of the commentary could have been better.

I've never understood the use of negative feedback to inspire someone to do better (maybe because I've never watched America Idol).

*work hat* Negative feedback is part of life. Keep in mind you're asking to read the judges notes. When I critique something in the top 32, I try to be neutral to positive because I'm giving unsolicited advice. The judges comments aren't.

It's like when I work the help skill.

Rep: The member wants to know if this is covered.
Me: *after research* No, it's not.
Rep: What do I tell the member?
Me: Educated member 'sucks to be you.'
Rep: I can't say that!
Me: I can. Your job is to say it in a presentable way.

Same thing. The judges aren't discussing their feelings, or worrying about yours. They're shorthanding, "Guys, this sucks, here's why it sucks. Reject."

(Note, when I'm auditing, I'm as polite as can be. Sometimes 'can be' is not much)

It's what you do with the critism (good and bad) that is the end result. Why are the judges on reality TV shows so blunt? (besides it makes for good TV?) Because a) they see a lot more than we do on the tube. And B some of those people really do suck.*

I'm sure there are people that Randy, Simon, etc etc say "You're quite good, but you're not at the line we've set today. Keep practicing and I hope to see you next year." That's boring TV.

As to here? Read my round 2 2009 entry. It sucked. Hard. The judges were brutal. Deservedly so, I wasted everyone's time. But like Steve Austin, I rebuilt, faster, stronger, better.

So you can lament about the judges' brutality. You can argue to the heavens that they just don't see your true genius. Or you can suck it up, learn from it, and move on.

I suggest the later.

*

This is just a guess, but it seems like the judges have developed, over the life of the contest, a shorthand series of words, phrases, and statements -- macros, if you will -- for evaluating items. Doing so would save the judges considerable time, both in reading and writing.

This could lead comments to seem terse to people outside the reviewing process. The only analogy that I can offer comes from the world of physicians (I work in a hospital). When doctors communicate about patients with one another, to someone outside the process their comments can come across as cold, harsh, or uncaring. In reality, the doctors are doing what they can to be efficient, and they often rely on somewhat terse communication.

Just a thought.

This rant is taking the judges to task.

There's no way I can't respond to this, Andrew. I'll vex myself all day if I don't. So apologies in advance if I come off overly passionate here. First and foremost, I hear you. I understand your perspective. And I can understand how many others would share that perspective. But maybe let me add to what Clark's indicated by widening that perspective a little further...

...I think only two of the 16 have mostly favorable comments. The majority of the items were weak keeps by the strongest supporter, and outright rejects by at least two other judges.

We've got 4 judges now rather than just 3 like all the early years of the competition. Last year, we had 4, but three of us (me, Mark, and Ryan) were completely new to the judging responsibilities. I can't speak for Mark or Ryan, but I took my lead from Sean. He's the more experienced judge. And the more experienced developer. He routinely schools me on all kinds of gaming insights and I greatly admire and respect him. So, I think last year, I held back a little bit on asserting any especially strong critiques. As I found my bearing, I became more confident with it. And this year, even moreso, I'd say...

I bring this up, however, to kind of recast the comparison of prior years to this year. When you have four experienced judges (i.e., Sean and Clark have done this 4 times now, and this is the second go-around for Ryan and me), you get stronger opinions from each of them. In addition, you get four different perspectives and that's because each of us has a different perspective. Sean is a rules-guru. He can dissect things quickly and efficiently where it takes me four rambling paragraphs to sort it all out. Clark is a "big idea, see the designer behind the design" kind of guy. He champions mojo when he sees it and he recognizes people who have enough spark they can go on to do something really special and connect with the gaming community. Ryan has a really good eye for what's marketable. He can quickly determine if a designer is thinking in ways that'll help enhance the game and its position in the marketplace as viewed from the publisher's point of view. As for me? I'm just a guy who ran the gauntlet. I think I'm more of a jack-of-all-trades. I can do a little of most of those things. But, primarily, I think my strongest asset is that I can usually discern what led each designer to choose a particular design niche, what they're trying to do to stand out for the purposes of the contest, and...most importantly...how well-positioned I perceive them to be in taking on the rigors of becoming an actual freelancer.

As a result of our different perspectives, we all sort of add our own critiques along those lines. I don't know about the others, but I feel like that's the value we add to the discussion of each designer's work. Our goal, obviously, is to help them try and improve. Should we hold forth more carrot than stick in that regard? Not always, no. I often refer to RPG Superstar as a "crucible" or a "gauntlet." Both terms imply that you have to go through a series of hardships before you can emerge on the other side that much stronger for what you experienced. I think it's an appropriate way to look at things. In the old days, native American Indians would force their braves to pass through a line made up of the entire tribe. And everyone was expected to hit those coming through...all in an effort to both harden them and test their courage. So, as critical as I am (or can appear to be), I think most people realize it's with an eye towards making those who run the gauntlet that much better.

I'm sure, when you're on the outside looking in, it can seem especially harsh. In the formative years of RPG Superstar, I think it was necessarily more supportive and encouraging. Clark was always especially good at the "rah-rah" elements in his judging duties. I've tried to emulate that at times, too. But I'm also a very critical, strategic thinker. Always have been. I hold myself to a really high standard. And, sometimes unfairly, I hold others to that standard. But, a good way to view it is the role of a drill sergeant. While you're in training camp, that guy is your worst enemy. But that's because he's preparing you for what the enemy will be like when you're out on the battlefield. I think our comments at this stage of RPG Superstar are a lot like that. We're preparing folks for the battlefield. Those who absorb these lessons and apply them will become strong enough to come out on the other side much improved by the experience. And those who don't will likely wash out. That's the purpose of training camp in the military. And, in many ways, that's the purpose of the critical commentary from the judges in RPG Superstar.

At the same time, you're right in that we also have a role to play in being supportive and encouraging. The entire contest is in public view and we need to conduct ourselves in ways that make people want to get into RPG Superstar and the freelancing industry as a whole. And, we need to make sure that we're a good reflection on Paizo as we're essentially representing their interests in what we do here.

In the past, the reveal of the top 32 round was a celebration of potential and even excellence in writing acumen. But this year feels different.

I think this year feels different for a lot of reasons. Aside from the additional opinions from such a varied field of judges, this contest isn't in its infancy anymore. We've got 5-time submitters who still haven't cracked the Top 32. But, what they've become is pretty darn proficient nevertheless. We've also got 5-time submitters who misfired on the first four years and finally reached the promised land this go-around. So, those who have learned and applied the lessons of the prior competitions and all the advice/feedback given by the judges have raised the bar themselves. And, as a result, the judges have had to, as well.

Each year, it gets harder and harder to separate the Top 32 from the Keep pile. We hold up items that are pretty close to one another and we have to go on gut instinct or some pretty individual criticalities to distinguish who gets a shot at the competition. Sometimes, we base that on who we believe would benefit the most in going through the competition (i.e., who we perceive will grow the most from it, as based on our perception of their item design). Other times, we go with someone whose idea resonates in a way that we want to see what else they've got up their sleeve (i.e., did they just catch lightning in a bottle, or can they repeat this level of mojo despite some mechanical flaws).

Either way, we take especial care to point out all the areas where a designer fell short in being a unanimous pick. That's because we specifically want to see how they address them in future rounds. We're not expecting them to "fix" their wondrous item. Rather, we're hoping to see how they apply a universal lesson to their future work, both in an effort to solicit our approval as judges and to appeal to the voting public. If we call folks out on something, we want to see how they change our perception in future rounds. In addition, it's our duty as judges to point out all those things, not just to the designer, but also the voting public so they too can hold the designer accountable for demonstrating growth over the course of the competition.

Because each of the judges takes the time to highlight those opportunities for improvement in a designer's skillset, we're actually doing so from each of our areas of expertise. So, although it might seem like a litany of negativity, that's not its purpose. And, if a designer can't look past that and recognize the areas where we're pointing out they need to improve, it's unlikely they'll grow very much over the course of the competition, or in their actual work as a freelancer to hopefully craft things the consumer will enjoy. I think it just seems like we pile on a bit because we're all anxious to give them as many tools as possible to improve themselves, and the judges have a varied enough series of individual perspectives that we point out a lot more things than any single judge (or a collection of like-minded judges) would on their own.

While I felt the actual items this year could stack up to the fantastic items from any prior year, the judge commentary seems to belie that statement.

Nope. Much like Clark, I think we've got as strong a field as ever. Maybe even stronger. And I certainly hope so...

Am I in Bizzaro world? Most of the commentary I'm reading from the top 32 actually feels negative enough that it seems I'm reading the "Critique my Item Thread".

That might be because I put a special effort forward this year to elaborate as much as possible in each initial critique during the sorting. I knew we'd be cutting-and-pasting the judges' discussion as quickfire feedback to folks in the "Critique" thread and I wanted it to contain as much as I could include. If you've been monitoring the "Critique" thread, I'm still taking fire for not framing the commentary as a full-fledged critique or providing enough. So, it's a fine line to walk. People want to know everything where they might have made misstep. And yet, if I provide more of that than normal, we risk coming across as overly critical to those who make it through. It's really a two-edged blade... :-/

I think the judges are being overly and hyper critical, perhaps even pedantic with their commentary. If I had to judge the items just by their commentary alone, then there wouldn't be a top 32, there would be a top 3 or 4. If these were the best of the best, the top 32, based solely on the judge commentary, the rest of the field (including my item) must have really, really sucked.

I think you're reading way too much into the commentary. This is an internet forum. There's no voice inflection or facial expression for you to get a true representation of what we mean by the words written here in every critique. And I can assure you, if you met all these judges in person, we're about as far from that image as can be. There's no one here that doesn't want to see people succeed in the contest. If we're tough on anyone, it's because we think that's what'll help them succeed.

I'm heartened by the members of the current Top 32 who have come here indicating they appreciate this level of criticism. I think they recognize it's valuable and constructive to them. Whereas, to someone on the outside of the contest, I guess it seems more daunting and troublesome...sapping your energy to bounceback from missing the cut and braving it all again next year.

Is it just me, or is there a change in the level of standard required to get even a weak keep these days? If so, the level of standard didn't just raise a notch, it raised many several in the last year.

Actually, I think we just put the Weak Keep and Weak Reject variations into the analysis last year. It kind of gave us more wiggle room to acknowledge some Keeps were stronger than others. So, this year, the Weak Keep got trotted out more often...where, in the past, it would have just been labeled a Keep alongside everything else, even though the judges mentally viewed it as a little weaker than some others. In other words, I think the judges have been more transparent in how they categorize items, even though that categorization process was there all along. And you might be freaking out a bit at how "Weak" you assume a "Weak Keep" to be. Or, rather, how "Weak" you perceive the judges to view it. When, in fact, it's no different than how some of the "Keeps" were viewed in prior years.

Anyway, that's just my two cents,
--Neil

I also noticed the difference in the comments. While I'm not a potential competitor anymore, I actually consider it a good thing for those submitting next year. You can honestly see what the judges are saying about the Top 32 and what they want. They are giving you an inside look into the process they use to judge. That's got to be a good thing if you are someone looking to successfully move through that process.

Asking the general public to assume you all think these are great items because they made the top 32, while your words speak otherwise, I think is a big leap of logic and counterintuitive....The presentation of the commentary could have been better.

Normally, we go back and adjust our commentary a bit to clean up things in the Top 32. I don't know about the other judges, but I felt hard-pressed to do much of that this year. The flood of items coming through for review saps a lot out of you. And then, it's a rushed frenzy to establish the Top 32 plus alternates. We have to quickly turn that over so the guest judges can come in lay down their commentary. And, in turn, they're really rushed to do that so the tech team can everything ready for the big reveal. So, this is just a supposition on my part, but maybe we didn't have as much time/energy for enhancing the presentation of the commentary this go-around? I know Clark himself indicated he did very little modification of his comments, whereas in the past, because the number of submissions was lower (and didn't take as much out of you), maybe he felt freer to do so.

Either way, thanks for the feedback...
--Neil

Andrew Christian wrote:

Asking the general public to assume you all think these are great items because they made the top 32, while your words speak otherwise, I think is a big leap of logic and counterintuitive....The presentation of the commentary could have been better.

Normally, we go back and adjust our commentary a bit to clean up things in the Top 32. I don't know about the other judges, but I felt hard-pressed to do much of that this year. The flood of items coming through for review saps a lot out of you. And then, it's a rushed frenzy to establish the Top 32 plus alternates. We have to quickly turn that over so the guest judges can come in lay down their commentary. And, in turn, they're really rushed to do that so the tech team can everything ready for the big reveal. So, this is just a supposition on my part, but maybe we didn't have as much time/energy for enhancing the presentation of the commentary this go-around? I know Clark himself indicated he did very little modification of his comments, whereas in the past, because the number of submissions was lower (and didn't take as much out of you), maybe he felt freer to do so.

Either way, thanks for the feedback...
--Neil

I can appreciate everything you’ve said Neil, and I really do appreciate you taking the time to explain things as best you could.

Only thing I’d caution on, is when the feedback, when gauged on a scale of positive to negative, seems to hit the more negative marks on the sliding bar, you find it actually de-energizes those looking in from the outside. Especially if they submitted an item. When there isn’t much difference seen in the commentary between the Top 32 and the rest of the field, it actually is quite disheartening. It puts a huge damper on reading the top 32 items.

Seriously, my first reaction was, “What the F! They hated this item, how did it make it?!”

As a competitor, I want you to shred my item. It makes me better. Currently, I’m no longer a competitor, but an interested and somewhat invested observer. As an observer, I can tell you that the criticism only being harsh makes me not want to observe anymore. I want to see why you liked the item, not just what you didn’t like about it.

As an observer, I can tell you that the criticism only being harsh makes me not want to observe anymore. I want to see why you liked the item, not just what you didn’t like about it.

But the pendulum swings both ways, Andrew. If the judges frame their criticism just to tell you what they liked about the item (in order to help you as an observer), we run the risk of failing to focus on the very real areas where a design was weak and its designer needs to show improvement (in order to help them as a competitor). Certainly, there's a balance in there. We don't need to be all gloom-and-doom on the competitor, as they need positive reinforcement, too. As many have cited, making the Top 32 is some pretty darn amazing positive reinforcement. That's not just me speaking as a judge. Remember, I went through this thing same as they are. I didn't care how much more positive reinforcement the judges gave me. I wanted to know where I needed to improve. Because that's what would help me reach the next round. I wanted them to teach me more about game design. Not by focusing on what I did well, but what I didn't do well.

Additionally, whose needs are we really trying to address with our commentary here on the Top 32? Yours as an observer? Or theirs as competitors? Clearly, we want the contest to be a good experience for everyone. Entertaining to the audience. Educational to the contestants. But, if I've got to favor things in one direction more than the other, I'm going to frame my commentary so it's more helpful to the competitors. I have to. Or I'm not upholding my responsibility as an RPG Superstar judge.

Nevertheless, I definitely understand your point. Hopefully, you see mine as well. It's a worthwhile dialogue, regardless. And, as always, the Paizo community is head-and-shoulders above any other community when it comes to discussing the meta-aspects of this contest.

Sincerely,
--Neil

Thanks for the very well communicated reply Neil. For future reference, I felt last year was an incredible balance between both constructive criticism and positive feedback, both extremely helpful to not only the competitors who made the top 32, but other competitors as well, and entertaining as an observer.

I don’t think I need to keep harping on the subject, I’ve been heard, and that’s all I asked.

First off I want to say that I appreciate all the efforts of the judges and everyone who participates in the contest whether it be by submitting an item or just commenting on the threads constructively. This is only the second year I've participated, but just from one year to the next I can see the major improvement in the contest and the general field of submissions.

This is just a guess, but it seems like the judges have developed, over the life of the contest, a shorthand series of words, phrases, and statements -- macros, if you will -- for evaluating items. Doing so would save the judges considerable time, both in reading and writing.

This could lead comments to seem terse to people outside the reviewing process. The only analogy that I can offer comes from the world of physicians (I work in a hospital). When doctors communicate about patients with one another, to someone outside the process their comments can come across as cold, harsh, or uncaring. In reality, the doctors are doing what they can to be efficient, and they often rely on somewhat terse communication.

Just a thought.

I think this is probably one of the most insightful things I've seen today on this subject. When I read it all the judges comments suddenly clicked and made more sense if I assumed that like doctors their comments were short hand for much more complicated thought processes then they weren't necessarily making the snap judgments they short terse comments might have implied.

The judges said that their critiques were not intended for public consumption. I volunteered to see the raw critiques, and I'm perfectly happy with the result.

I will say that I'm really not as excited by this year's entries, but my own wasn't better so que sera.

If you're serious about being a content creator, you need to learn to take the best message from even vicious feedback. You should be most concerned with improving your craft, not winning praise.

The judges said that their critiques were not intended for public consumption. I volunteered to see the raw critiques, and I'm perfectly happy with the result.

I will say that I'm really not as excited by this year's entries, but my own wasn't better so que sera.

If you're serious about being a content creator, you need to learn to take the best message from even vicious feedback. You should be most concerned with improving your craft, not winning praise.

Oh, I’m not concerned for myself as far as feedback goes. I can take nasty feedback if it pertains to my item. The initial intent of the post was to indicate as an observer now (my item did not make it) I was not enjoying reading the feedback of the current top 32, which was certainly not the case in years past.

Ah I see. Sorry, I was knee-jerk posting from too much coffeecoffeecoffee.

I've never understood the use of negative feedback to inspire someone to do better (maybe because I've never watched America Idol).

First of all, we dont give ANY feedback unless a person asks for it. The only feedback you see is from the top 32 items, unless someone voluntarily asks for feedback in the critique my item thread.

Plus, you need to remember, we arent making those comments to help YOU grow, we are making them to help US judge and rank the items. Our comments are part of the process of winnowing out the keepers and then sorting among them for the top 32. We arent making comments that are intended to be for you, but due to public demand we have made our comments public on the top 32. So I think there is a misperception that our comments are intended in some way to help you, they aren't--at least not the comments we post in the initial thread--those were done for our benefit of sorting.

In fact, I would say this might be one of our strongest fields yet.

Actually, after I read the comments for the top 32 items I was put under the opposite impression. The often harsh public criticism made it seem like the winners' entries were simply the nicest of the damned. Until I started poking around the forum to see what was up, I wasn't even going to bother checking out the later rounds over fear of quality issues. >.>

Well, Clark - those comments may not be designed solely to be helpful to us, but as one of those whose entry advanced in spite of it's well-documented flaws, I'm glad those comments were included, as they do, indeed help me to rethink my own thought process as a wanna-be writer/designer.

Though I appreciate the sentiment of the post's original author (I, too noticed the commentary wasn't all hugs 'n bunnies this year, as opposed to previous years), I think the move towards more critical writing from the judges is a good thing. Like others mentioned, the biggest attaboy I can get is the "Kept." comment - that says a WORLD of good things - the rest I look at as how can I get better.

I found the "critique my item" commentary to be almost ironic - again, I know (I think) what the author was trying to say, about a more negative flavor to the posts this year (and I don't disagree) -- but isn't critiquing my item actually the point, top 32 or no? After all, if I wanted, "gee! Neato item! Can I play with it?" - I'd get feedback from my gaming friends (most of whom run players in my game and would have a vested interest in making me feel good, LOL...) What I'm looking for (even if it makes me wince a bit when I get it), is honest, critical feedback, that lets me know if I not only have a winning submission, but if I have the writing chops to successfully publish something commercially viable - something I expect is foremost on many of the judge's minds as well.

To Sean, Clark, Ryan and Neil, as well as Sam, Sean and Jerall - thanks so much for your input - and for all the hard work you do for this contest and the Paizo/Pathfinder community. It must seem sometimes that no good deed goes unpunished... but we really do appreciate your dedication!

In fact, I would say this might be one of our strongest fields yet.

I know this was said while entries were still coming in, but I'm coming from an opposite perspective than the OP: I haven't looked at the Top 32 but am just reading through the "Critique My Item" thread. It seems like in years past, there were more ultimately-rejected items which at least had a judge or two briefly championing them with a recommendation to Keep. This year, it's just a lot of quick "Reject, Reject, Rejected," even on items submitted by people with Top XX from previous years by their names. Maybe it's just the wrong subset of people asking for feedback, but it certainly seems like the judging was harsher this year.

It's not necessarily unreasonable that your standards would rise along with the quality of the submissions. It's possible that some of the winners from year one wouldn't even be considered in this year's contest. After several years of the contest and most of the judges having read thousands of wondrous item submissions, you probably get less tolerant of design that's not up to par.

Clark Peterson wrote:

In fact, I would say this might be one of our strongest fields yet.

Actually, after I read the comments for the top 32 items I was put under the opposite impression. The often harsh public criticism made it seem like the winners' entries were simply the nicest of the damned. Until I started poking around the forum to see what was up, I wasn't even going to bother checking out the later rounds over fear of quality issues. >.>

This was exactly why I started this thread and ranted in the first place.

The judges commentary on the top 32 is as much for the top 32, as it is for everyone else who eventually has to vote on the top 32/16/8/4/1 as the rounds commence.

If all we see is the raw harsh criticism and only reasons for why the item isn't good enough... its hard to separate that from why we think the item is cool.

I know I'm an adult, and I have a mind of my own. I am fully capable of reading for myself and forming my own opinion. But I trust and respect Clark, Ryan, Sean, and Neil. They are the professionals. They've all done this more than once, and they work in the gaming world as either producers, developers, CEOs, or freelancers. So, by default, I have to lend weight to their comments.

The way I see it, is that during the paring down process from Keeps to Top 32, the judges only threw out comments about why the item wasn't good enough. The praise is faint and far and few between, and often hidden within the other comments.

As such, I now basically have to read the judges comments as though they are an editor that is red-lining a draft to be rewritten, rather than as a comprehensive thought about how they liked/disliked the item. The later would be much more useful to the voting public, who ultimately will be choosing who writes the Pathfinder module and the PFS scenarios. The former is only useful if you can actually switch gears in your brain to read it that way.

Obviously, initially, I failed at switching that gear.

This is merely a comment for future commentary by the judges. Give us voters something else to work with other than just the negatives. I take my voting job very seriously, as I want to see the best writer win. But I'm finding it hard to objectively look at the items, when the commentary I see from the judges is not comprehensive. I have to formulate the whole picture on my own.

In the past, the Judges gave a more comprehensive view of each item in the top 32. I was able to use that commentary to help me make my decision.

This year, I feel I'm on my own. That is unfortunate, as I am not entirely qualified to make those design decisions.

Joana, I think that may also be due to the increasing numbers of entries. They have less time to comment or champion things that ultimately aren't going to make it, I assume.

Personally, any negative comments just inspired me (though honestly I thought the ones on my item were very positive). Except for Dancey's -- I've sworn blood feud against him. : )

Yeah Jacob!

Thanks Andy!

Though to be fair, I get the impression that I just barely made it in, so maybe reading mine first colored your impression of the group as a whole? I dunno, just a thought.

I think I made the mistake of making an item which, if printed, would've been in the plain old CRB instead of a sexy splat book. Oh well.

...It's not necessarily unreasonable that your standards would rise along with the quality of the submissions....After several years of the contest and most of the judges having read thousands of wondrous item submissions, you probably get less tolerant of design that's not up to par.

Both of these things are true.

...I think that may also be due to the increasing numbers of entries. They have less time to comment or champion things that ultimately aren't going to make it, I assume.

And so is this.

Personally, any negative comments just inspired me (though honestly I thought the ones on my item were very positive).

I felt the same way when I competed, Jacob. As Sean likes to say, "Prove us wrong. We want you to prove us wrong." Fuel for the fire. Let's see who comes roaring back and who just overheats with it...

Except for Dancey's -- I've sworn blood feud against him. : )

Yeah, that Ryan's a cagey guy. He may seem like he has a little "Simon" in him (from American Idol) now and then. But I can tell you right now, I think he makes some of the most astute observations sometimes about the relevancy and marketability of the ideas which designers hold forth. If you can get Ryan's buy-in on something, you've crossed a major threshold in your design. The same is true with regards to all of us, I think...and for all the various perspectives we bring. If you get something past Sean from a rules-fu perspective, good on ya. If you polish something down to make it look as professional as possible and win my approval, good on ya. And, if you can persuade Clark that your design has mojo and appeal to the gaming community at-large? Well, that's a major accomplishment. If you can do all those things across the board...you might just be the next Superstar.

--Neil

Joana, I think that may also be due to the increasing numbers of entries. They have less time to comment or champion things that ultimately aren't going to make it, I assume.

Personally, any negative comments just inspired me (though honestly I thought the ones on my item were very positive). Except for Dancey's -- I've sworn blood feud against him. : )

You'll have to get through me first! Ryan Dancey championed my item (at least that's what I gather from the comments), so I can't let you feud it up with him. I also hope I lived up to the promise he saw in my entry. :)

I've never read much RPG Superstar feedback in the past so I really don't have anything to compare it to but I do agree with the OP that the comments from the judges give an impression of "best of the worst" rather than "best of the best". I just thought this was deliberate behavior by the judges because you need to have thick skin to be an RPG designer and they need to expose you to that fact early in the process.

...and in the spirit of thick skin...I don't find Mr. Danceys criticisms to be "superstar judge" level. Maybe he's just too busy to put a great effort into the process but it seems like he is mailing it in compared to what the other judges provide. Running GW I'm sure is taking a ton of his time. I thought his mention of "beholders" in one of his criticisms was both funny and enlightening. Using auto-reject terminology (non-Paizo owned IP) in a criticism is a little odd.

Seriously, my first reaction was, “What the F! They hated this item, how did it make it?!”

That's a perfect example of what I am talking about. I can't change that. That's a YOU problem, not a ME problem.

You seriously thought we "hated" an item but put it in the top 32? Let alone in the keep folder? Think about that for a minute. Does that even seem reasonable to you?

I'm not going to put "oh my gosh, this is so shiny" at the start of a thread just to make sure you know I liked something that got put in the keep folder.

If your item was really good, it got put in the keep folder.

If your item was average, it might have been discussed, but rejected. Average doesnt cut it. That includes good execution of a "meh" idea as well as poor execution of a better idea.

if your item was a clear reject, that was a no brainer.

By definition, if you are in the keep folder, your item is really good and is more than just a "book of magic items" item.

If your item wasnt in the keep folder, you need to decide if you were a "clear reject" or a "good but not good enough". If you were good but not good enough, we probably said it. If you were a "this would have made a book of magic items," we probably said that.

I don't find Mr. Danceys criticisms to be "superstar judge" level.

You don't have any basis to have such an opinion. You aren't actually seeing everything Ryan writes. In fact, if an item is even remotely interesting, he had a checklist system he used for his evaluations that was great. You really need to be more careful with comments like this. Your comment seems more like sour grapes aimed at judges because you are mad. Given that Ryan saved D&D and oversaw the creation of 3E, made it open, which led to Pathfinder, I don't know that you could have a more appropriate judge. And his comments, while sometimes short, were always super-insightful and spot on, in my view. I loved his commentary, because he brings a view that is so different from mine. It enriched the contest, no doubt. To suggest otherwise is just not correct. Now, you may or may not see that. But I can assure you Ryan did not mail it in. Neil writes the most words in any given thread, Ryan probably the least but often most directly and with most impact, Sean and I in the middle with Sean's stuff being way more rules-fu.

[Edit] You know, I have to just pull back. It's silly that I even had to consider the idea of defending whether Ryan mailed it in. That's just crazy. Not sure why I even bothered to post this. It goes without saying.

Alright, even though I strongly disagree with some of the things in this thread, I am not one to disregard the heart felt complaints of members of the community.

So I decided to take a look back at some of the prior years. I'm not so sure that the OP's perception that "prior years were more positive" is really true.

This, for example, came from the thread from a top 32 contestant the second year:

=======

Clark Peterson:
golem in a can, literally. i'll admit, i kind of like it. it would make its way into a magic item book.

but not superstar.

reject

Wolfgang Baur:
I like it quite a bit, but probably not enough for Top 32.

Inclined to reject, willing to be talked out of it.

Clark Peterson

You know, I am coming around on this. It is so limited. It just lasts basically one round. And it only has 3 uses per day. Plus it is cool.

I am swinging to keep.

Sean K Reynolds:
It's limited and reactive enough that I like it, I was worried that it would be horrendously overpowered.

Keep.

==========

So here we are saying its a "monster in a can," its a "book of items but not superstar," the first two judges (me and Wolfgang, and it doesnt get any nicer as a judge than Wolfgang) vote to reject it initially, then I swing to keep (which this year I would call a weak keep), and Sean keeps it (again, a weak keep I am sure). And this item made the top 32.

Or here we are about the abyssal spurs:

========

Wolfgang Baur:
Every Chelaxian hellknight should own a pair of these.

Inclined to keep.

Sean K Reynolds:
6000gp seems too cheap to have a flying mount whenever you want, with the minimal "material component" cost of possibly killing your horse (or riding dog, let's make it really cheap).

The game doesn't really have a definition for "sentient" ... Int 3+? Int 1+?

Will save and Fort save should be capitalized.

Interesting idea, and a decent sense of the mechanics, but not quite there yet.

Reject.

Clark Peterson:
I like the idea. But the formatting problems are just not Superstar. Capitalization, I'm looking at you. And I'm with you on the "sentient" thing, Sean.

Clark Peterson:
Rejected.

=======

That means I actually rejected the item! This thing was in the reject pile! But I went back and pulled it out....

========

Clark Peterson:
You know what. I have rejection remorse. I wish I hadnt rejected these.

I think this is the kind of item that makes me want to see what the person could do in the contest itself. I know, that goes against things we've kicked other entries for. But there is some spark here.

I would reconsider rejecting this if you guys are on board. Heck, if one of you is on board.

=======

So I unilaterally put it back in the keep folder from the reject pile! And this item made top 32!

But all that aside, even though I dont think things are actually any different...I have to concede that the "tone" of our comments in past years may have seemed friendlier.

So I am going to take that observation to heart.

Andrew, even though I've given you a lot of grief in this thread at times, I thank you for raising the issue. And I am going to give it some more thought.

I see what you are saying.

So I unilaterally put it back in the keep folder from the reject pile! And this item made top 32!

And I thought I got lucky! Some of us need to start a Clark appreciation thread. :-)

Here's a different perspective.

My comments on the Top 32 are exactly the sort of comments I'd give to a new freelancer who I'd like to continue working with.

It's because I think they have potential.
They may need to improve their work, but I think they can do it. So I tell them what they need to fix, because if they don't fix it, I'm going to have to fix it for them... in everything they send to me, forever. By telling them "I want you to improve in this area," I'm telling them "it is worth my time to tell you how to improve, because I know you can do better."

If I don't think they have potential, I don't expect they'll be able to learn from that feedback, so I'll just fix it for them... and not send them any work ever again. Because it's not worth my time to tell them how to improve, because I don't think they can do better.

Just last week I sent feedback to one of my regular freelancers (who happens to be a Superstar veteran). I told him "this was well done, I don't have to cut any of it, now work on improving these five issues." 75 words of praise, 300 words of critique. His response was "thank you for the feedback"... because he knows it took me a half an hour to type out that criticism, he knows I value my time, and by spending that time on him I'm saying his work is good and I want to use him in future projects.

Look, I've been doing this for a long time. I've had verbal arguments with Gary Gygax. I've shook hands with Dave Arneson. I've had scary meetings in Lorraine Williams's office. I grew up reading Margaret Weiss and Tracy Hickman and Ed Greenwood and Bob Salvatore, and now they're my peers. I played D&D when most people thought it was Satanic. I've been gaming for longer than some of you have been alive. I've spent hours and hours and days and days writing. For AD&D. And D&D. And Alternity. And Marvel. And DC. And The Last Airbender. And Pirates of the Caribbean. And Warcraft. And Pathfinder. I've left behind friends in all corners of this country as I've moved to follow jobs in the game industry. I've had people say they love my work. I've had people say they hate my work. I've had people say horrible things about me and my family because of my history in the gaming industry. I've been lucky to have a salary as a game designer and unlucky enough to be working contract to contract with $10 in my bank account. I ask my peers what I can do to improve my work because I take writing absolutely seriously. Because I know that I am incredibly, incredibly lucky to be able to get paid for doing something that has been a happy part of my life since I was eight years old. Because professional writing isn't just the current project, or the next project, it's about the next next project. And that means always trying to do better, because there are thousands of other people who'd love to take your place.

As Clark said, I see your point: it does look like we're being hard on the Top 32. So I'll make an effort to be more positive at the start of my comments. But I'm not going to coddle anyone. I'll tell you what I like about your submission... and what I don't like.

To the Top 32: Congratulations. You made the first cut. Now you're in boot camp, and for the next nine weeks I'm going to ride your ass and push you to work hard so you can do your best work. Because you want to get paid to write games and I can make that happen, but only if you're willing to make the effort. You're good... now listen to how you can be even better.

...You wrote for The Last Airbender? I loved that show!

Another way to look at it... my item got very short shrift, gave one judge a headache, sent another home early, my biggest pan entry ever.

I didn't cry foul, I took it to the chin.

I came back, carried on working on my feedback to the winners, carried on with working on my template explained thread, in short learnt.

I made two fatal mistakes this year and got royally reamed for it.

1) I forgot about how they had so many coins and bags in the first year, blinkered by my inside out move in my item.
2) Because I forgot, I thought I was being different and so didn't check, sheesh, the judges reactions put me straight.

Did I get item feedback, in a way you could say no, but in a very strong way, you could say the judges said in no uncertain terms (come on Ant, that's pathetic, wake your ideas up, you are better than this)

Message received - and whilst here - many, many apologies for the migraines.

So what am I doing - yeah, I'm looking at those winners feedback and comparing what I had done - yes there were technical issues, some markedly so - that I can beat with my eyes shut now. So it's all about the mojo now. mojo and practice.

I started a practice thread, setting very direct challenges to myself and anyone wanting to come along for the ride. I now wish I has started this practice earlier.

I think my Passage of the Lotus Petal Stream wouldnt have made Neil whinge about my item name and the other judges may even have said something nice.

Yes, I could have said after 5 years, thats enough. But I didn't. I got up, pushed aside my embarrassment and decided to scare next years top 32.

So instead of say things are disappointing, look at the comments, do they apply to your item, can you learn from them.

I can't answer for sure, but I'm going to have a dang good try.

If anyone wants to join me in that thread, you are very welcome. I wont promise we will get Top 32 next year or the year after, but I can promise we will get in touching distance of the keep folder at minimum.

Once again, apologies to the judges, I made a complete ass of myself with that item - I wont do it again.

And in case anyone is wondering, yeah, Im still dreadfully embarrased by my entry and you know something, I ruddy well should be.

More importantly, what did you do for Marvel, Sean?

...You wrote for The Last Airbender? I loved that show!

I'm re-watching season one right now, as I type. The Last Airbender is in my top five fantasy works of the last decade, in any media. It's wonderfully written and directed and works on so many levels.

Anyway, back to the anxious waiting for me and Jacob...

So I'll make an effort to be more positive at the start of my comments. But I'm not going to coddle anyone. I'll tell you what I like about your submission... and what I don't like.

That's exactly what I'm asking for. Thanks for hearing what I had to say.

Ha, I wish! No, I wrote flavor text for The Last Airbender TCG, the Pirates of the Caribbean TCG, and the Vs. TCG (which features the Marvel and DC characters), all of them published under license by Upper Deck. I didn't mean to mislead anyone with the above statement... they all were licensed and thus had to be approved by the owners of said properties, but it's not like I was writing dialogue for the Airbender cartoon or the Pirates movies...

Ha, I wish! No, I wrote flavor text for The Last Airbender TCG, the Pirates of the Caribbean TCG, and the Vs. TCG (which features the Marvel and DC characters), all of them published under license by Upper Deck. I didn't mean to mislead anyone with the above statement... they all were licensed and thus had to be approved by the owners of said properties, but it's not like I was writing dialogue for the Airbender cartoon or the Pirates movies...

That's probably for the best - had I successfully connected you to an episode of TLA, my head might have exploded, thus removing me from the contest.

But all that aside, even though I dont think things are actually any different...I have to concede that the "tone" of our comments in past years may have seemed friendlier.

Clark, what I think we need is more Dokken comments. That was awesome (referencing your comments from the critique thread)! My brother is a huge, huge, Lynch fan- he's got all the CDs, instructional DVDS, the skeleton guitar, and just got a Kamikaze one. I saw the original line-up in 97 in a small club and got Lynch to sign my excuse note from school for the next day. Dokken makes everything better :)

To the Top 32: Congratulations. You made the first cut. Now you're in boot camp, and for the next nine weeks I'm going to ride your ass and push you to work hard so you can do your best work. Because you want to get paid to write games and I can make that happen, but only if you're willing to make the effort. You're good... now listen to how you can be even better.

I never expected anything less.

Here's a different perspective.

*Inspirational perspective*

Now that is why I want to be part of this. That's a seriously motivational speech. I'm both terrified and thrilled that you and the other judges will be tearing my next entry apart to find both the good and bad in it (let's hope you find more of the former).

Putting my work out there for all the gaming world to see and analyze is a real buzz. I reckon this competition is as much about character as anything else. Can you take all that criticism and turn it into a whole can of awesome the next time 'round? I think we're going to see some pretty special entries as the competition goes on, judges initial comments notwithstanding.