| 5 people marked this as a favorite. |
Was that a dig at me, Clark? Was it? ;)
No :)
I think the "sufficiently awesome" bit also trips people up. Voters and judges agree that SIAC items are bad, BUT it seems like at least one gets in every year. I saw a few items that maybe tread close to that line or otherwise come off as unoriginal/overdone in their very core concept, but they were presented in a neat way that made me like them anyway. Is there any sort of litmus test for this on the judging end, or is it just gut feelings (or, more probably, honed intuition)?
Not really. That is one of the reasons I push so hard to make this decision process so open and why I was so happy to permit those first "critique my item" thread back for the first RPGS. We have long acknowledged that there is a subjective element to the judging. The best way to help people understand that subjectivity is to let people see the process.
I have distilled a few things, though.
1. Inspired design. A fresh approach to an old problem, creativity, showing an understanding of the design space you are in and how to work with the rules, originality.
2. Cinematic. I actually just figured that one out this year, thanks to help from some of the guest judges. I found that I was really drawn to items that had a strong cinematic component--not just the description, but that I could actually see the item in play and how it would work. I noticed the guest judges kept calling the various really good items "cinematic" and I don't know why I never put that together before but I agree with them. I have used that word in later rounds before but I guess I never consciously tied my preference to this component. Go read the top 32 items and see what I mean.
3. Just darn good writing. In the end, freelancers and designers are going to be writers. I am certainly drawn to evocative, well written items. And I don't mean backstory or silly descriptions. I mean good, tight, concise yet still evocative writing.
Remember, this process is a learning experience for all of us, too. OK, well, for me. Sean and Wolfgang's design-fu is way better than mine. They could win this contest. I couldn't. :)
Those are the things that I think take an item from "just a really, really good item" to "superstar". And look at that. I've been a judge for all but one year of Superstar and it has taken ME that long to crystallize my thoughts on this issue. So don't punish yourselves overmuch if you don't figure it out right away.
Thanks for taking the time to give me a well thought out response. :)
I definitely know I have a lot to learn, but like you said, I'm trying to understand and grapple with these sorts of issues as much as I can. After looking over everything, it's much clearer to me why the Top 32 were chosen, though it's still hard for me to put my finger on the exact reasons or fully explain them.
I don't know about anyone else, but I'm definitely just on cloud nine over the fact that people liked my item, even if it ultimately wasn't Superstar. Even a little. As a first time submitter, just getting my item far enough for serious consideration feels like a mammoth achievement.
Well I can say one thing for sure. This year I'm gonna get active in the 9 Blazing Months thread or its successor. As I've only heard of the contest in the first decade of December I've never had any time to prepare or participate in the communal writing. But now I see how things are supposed to be done around here, and how much more work people put in their design than me. So, even though I didn't make even the top 89, I've learned a lot this time.
RPG Superstar 2012 Top 16
,
Marathon Voter Season 6, Marathon Voter Season 7, Marathon Voter Season 8, Star Voter Season 9
aka Locke1520
|
| 2 people marked this as a favorite. |
Thanks Clark for suggesting this to Paizo and a special thank you to Lisa, Vic and everyone else for letting Clark post the top 89 I think it will help people wrap their heads around how to design for two audiences at once.
The above list really demonstrates what it will take in future contests to appeal to two separate audiences although the two audiences do not seem to be that far apart. To those who made the top 36 & 32 Congratulations again looking forward to seeing all of your efferts in round two and beyond. To the top 89 of vote-getters congratulations for making that first hurdle, you were in good company.
And to the rest of us congratulations for entering I hope you are all still as proud of your effort now as you were when you entered. This contest isn't about beating the competition. Its about finding new talent, in this community and in ourselves.
Contributor; Publisher, Kobold Press; RPG Superstar Judge
|
| 2 people marked this as a favorite. |
A lot of what makes an item Superstar rather than just a good item boils down to "Don't be boring" and "Inspire people to want to play with this." It's what the SF writers refer to as sensawunda.
All the items in the Top 89 were imaginative and interesting in different ways, enough to catch the attention of voters and get the nod. The ones that stood out to the judges were cinematic, inspiring, imaginative, as well as generally well-designed on mechanical grounds.
Congrats on doing well your first year out, agirlnamedbob!
Thanks for sharing your thoughts, Wolfgang. Like so many things, it really seems to boil down to "simple, but not easy." And thanks for the kudos!
Andrew - Sorry you didn't make the list. I really hope to see you in the critique thread, though. I'm curious what your item was! (Sorry if you already mentioned it somewhere and I missed it.)
Orcus - I'll probably be seeing you in the 9 Blazing Months thread, or whatever similar thing happens this year. We can help each other.
|
RPG Superstar 2012 Top 16
,
Marathon Voter Season 6, Marathon Voter Season 7, Marathon Voter Season 8, Star Voter Season 9
aka Locke1520
|
Andrew - Sorry you didn't make the list. I really hope to see you in the critique thread, though. I'm curious what your item was! (Sorry if you already mentioned it somewhere and I missed it.)
Thanks Bob. I'll PM you my item if you'd like and I'll definitely post to the Critique thread but I hadn't yet mentioned it. I wasn't even going to really bring it up until Friday but with the top 89 now visible I wanted to say something to the others in the 600+ entrants who might be a little disappointed by this announcement.
What's really interesting to me is that about 90% or so of the total items (going by the lists others have provided) ended up with negative vote totals. In other words, 90% of the items were voted against more times than they were voted for.
I'm sure the 25% cull shifted the math in that direction some, but that's still a striking idea. It means that the top 89 were easy, clear winners when they came up against any of the rest. That's a real, meaningful division in what the public liked and did not like. It's worth devoting some serious attention to.
| 4 people marked this as a favorite. |
I think one has to understand that there is no "magic bullet" for making the top 32. You can do everything right. You can create an item that is evocative and cinematic. That fills an untapped design space and bends the rules in a new direction. That does something super fresh and cool. That appeals to the general voter and to the judges. And you might still get cut from the top 32. It might be because Clark stubbed his toe that morning and decides he doesn't like open toe sandal items today. Or Wolfgang was cut off in traffic by a red car and now has a subconscious bias against items that are red. The point is, a lot of items must be cut, and there are many, miniscule reasons the judges might not advance your item. It is largely subjective. And on a different day, the judges may see something in your item to advance it that they wouldn't have seen the day before.
That is what I learned from my first few failures in this contest, and it made rejection much easier to swallow for the next.
And then I learned to do the things I mentioned above to increase my chances. I had one more failure mostly due to chance. My submission last year was the Artumian Branding Iron and the judges thought I was trying to pander to judge Niel Spicer who wrote the Artume section of the Guide to the River Kingdoms. I actually had no idea Niel was the author or that the Guide even existed. I had just searched the Golarion wiki for places that had to do with herding and came up with Artume. Maybe my item was just not good enough. But it may be that the judges were just subconsciously or consciously turned off by the thought that I might be pandering to a judge to see the qualities of my item. I'll never really know and they might not be able to unbiasedly say that wasn't a big part of it.
Anyway, you have to do the things I mentioned above to have a shot, but even that will only get you most of the way there. Then you are up to fate and the judges' fancies for that day. Good luck next year to all the hopefuls. Unless I go all the way and make myself ineligible, I'll be submitting again next year along with you.
|
Marathon Voter Season 6
|
| 1 person marked this as a favorite. |
, it also says a lot about how Paizo feels about the quality of this community that you can receive such information and not have it cause hurt feelings and other negative reactions.
I have both of those honestly, it is more a matter of handling them like a mature adult, and transmuting them into determination.
| 2 people marked this as a favorite. |
Here's another bit of data:
The top-voted item was voted on over 500 times, and was upvoted more than 400 times more than it was downvoted... but that still means it was downvoted more than 100 times (i.e., 100+ times, someone saw it paired with another item and picked the other item instead of it).
So if there's an item you really liked but isn't in the 89, or there are items you definitely didn't like that somehow made it into the 89, remember that the highest-voted item still only had a 75% approval rating. Everyone's tastes are a little different, but the collective voting did push the better items to the top.
I will be starting the Blazing 9 tradition sometime after the contests ends. :) I am toying with the idea of running a 'what would you have submitted' for each of the rounds over the summer (instead of when the contestants are running the gauntlet when we normally do. :)
I agree with Meepo and St.C in that it feels like a lot of this practice was for naught. It feels in many ways a complete waste of time. I'm obviously not getting any better. Waste of time. My dad was right. My mate used to say the same thing. But then we hosted game night and she saw a group of friends having fun. She does not think it a waste of time anymore, it is FUN way to spend time.
Could I put my daily writing into something more productive? Yes. Would it be more fun? Sometimes. Would I regret not playing. Definately.
Besides I wouldn't want Clark to eat my soul. :)
I will be starting the Blazing 9 tradition sometime after the contests ends. :) I am toying with the idea of running a 'what would you have submitted' for each of the rounds over the summer (instead of when the contestants are running the gauntlet when we normally do. :)
I agree with Meepo and St.C in that it feels like a lot of this practice was for naught. It feels in many ways a complete waste of time. I'm obviously not getting any better. Waste of time. My dad was right. My mate used to say the same thing. But then we hosted game night and she saw a group of friends having fun. She does not think it a waste of time anymore, it is FUN way to spend time.
Could I put my daily writing into something more productive? Yes. Would it be more fun? Sometimes. Would I regret not playing. Definately.
Besides I wouldn't want Clark to eat my soul. :)
Glad to hear you'll be sticking around to start the Blazin' Nine. To be blunt if you didn't do it this year someone would have I think there's allot of us that were RPGSS virgins this year and I for 1 intend to practice!
...My submission last year was the Artumian Branding Iron and the judges thought I was trying to pander to judge Niel Spicer who wrote the Artume section of the Guide to the River Kingdoms. I actually had no idea Niel was the author or that the Guide even existed. I had just searched the Golarion wiki for places that had to do with herding and came up with Artume. Maybe my item was just not good enough. But it may be that the judges were just subconsciously or consciously turned off by the thought that I might be pandering to a judge to see the qualities of my item.
For the record, it had no bearing on how your item was judged, Nickolas. The connection to Artume simply caught my eye because I wrote that one-page write-up for the Guide to the River Kingdoms and I was actually kind of delighted to see someone make a reference to it. And, while it crossed my mind to wonder if someone had noted the author by-line on that page (i.e., it's one of the few products where Paizo listed which author contributed each section of the book), I obviously set that aside as any sort of evaluation criteria to remain unbiased. And, obviously, it had no bearing on the other judges anyway. Only Sean, as the book's developer, would probably have made the connection. And he's the least likely person to ever become biased by something like that...positively or negatively.
Instead, you should probably know that that item fell through on a number of other concerns...not the least of which was that it involved holding creatures down and branding their flesh to magically function. For the purposes of standing out in RPG Superstar, it's usually not a good idea to go that route. Building in elements to how your wondrous item functions which rely on maiming, deforming, or even being fueled by body parts, injurying yourself, etc. doesn't usually draw approval for the Top 32. Doesn't mean you'll never see an item like that get through (though it's probably very unlikely). I'm just saying you're creating a much harder hurdle for yourself if you pursue an inspiration like that.
What's really interesting to me is that about 90% or so of the total items (going by the lists others have provided) ended up with negative vote totals. In other words, 90% of the items were voted against more times than they were voted for.
"Voted against" sounds negative; it isn't the same as "not voted for." There were a very few items I voted against, as I thought they were really awful. But in most cases it was much like choosing a candidate for hiring or promotion; they were both good but one was a better fit.
My item was basically utilitarian, and likely ended up way far down in the list (though it somehow survived the cull), but I don't expect people actively hated it or thought it was stupid; it was just easily outstripped in the voting by [probably hundreds of] better items. In retrospect I realize that, despite avoiding most of the pitfalls referenced in Rules 1 thru 26, I wasn't writing for Superstar, I was writing for a paragraph of filler space in Ultimate Equipment 2: Eclectic Boogaloo. :)
I like that idea, Curaigh.
I mentioned last go-around that we should practice other rounds (I feel like I'd have done much better on monsters this time, but was still very nervous about it) and was suggested trying the homebrew forum.
But as Standback pointed out in the thread asking for a critique sub-forum, I feel like the goals are a little different. There I feel like I'm more asking about whether something's usable for a game; here I'm more practicing design work for mass appeal, at least in my mind.
Here's another bit of data:
The top-voted item was voted on over 500 times, and was upvoted more than 400 times more than it was downvoted... but that still means it was downvoted more than 100 times (i.e., 100+ times, someone saw it paired with another item and picked the other item instead of it).
I'm curious. Do we know if all the items were indeed voted on more or less the same number of times (within 10 percent, at least)? I certainly felt like I saw certain items time after time while others I never saw at all.
For the record, it had no bearing on how your item was judged.
I didn't mean to imply that you had conscious bias because of the Artume tie in. You said you didn't and I believe you. The other judges may not have either. Certainly it had other flaws. I missed the "no plot device items" rule last year and that is really what it was. I didn't really picture it being used in the hands of a PC, but as a villainous item used to start an adventure. I was just using it as an example of how little random things could potentially influenced a person's thoughts on a item, whether consciously or subconsciously. I meant to offense.
Oh, and now I have a PDF of the Guide to the River Kingdoms and am having a hard time deciding what kingdom to set my archetype in. I can't say more.
Go Artume. Make 'em it be an Artumian Brander (um, figher? maybe?). : )
| 1 person marked this as a favorite. |
I'm curious. Do we know if all the items were indeed voted on more or less the same number of times (within 10 percent, at least)? I certainly felt like I saw certain items time after time while others I never saw at all.
The difference in the number of votes on #1 and #89 is less than 20.
The difference in the number of votes on #1 and on #200 is less than 20.The difference in the number of votes on #200 and #400 is less than 20.
The difference in the number of votes on #1 and #400 is less than 20.
So yeah, more or less the same number of times for each item (within 10%).
it feels like a lot of this practice was for naught. It feels in many ways a complete waste of time. I'm obviously not getting any better. Waste of time. My dad was right. My mate used to say the same thing.
This connects well with Clark's "Olympic" example. If anyone has ever played competitive sports you'll know that you train and train and train and train and then all that work comes down to one (or a few) key moments. And when the other team rushes onto the field/court/whatever after winning, or the other competitor has gold placed around his or her neck, it's really hard not to think, "well, all that hard work was for nothing."
It's not until after a few days, or months, or even years that you start to gain some perspective and realize that you really did get a lot out of all that training. If you make this realization soon enough it can be the fire that pushes you to keep going in hopes that the next time you get the opportunity, if the bounces happen to go your way, you you'll be ready to run with them.
| 1 person marked this as a favorite. |
So...math not being my strong suit, how is it if every item was paired with another, that only 89 out of over 800 got positive vote records, esp if #1 had onoy a 75% positive ratio?
One possibility is that items in the lower 90% nearly tied with each other. Sometimes when items in that group were against each other, they got votes, sometimes they lost votes, on a more or less even basis. So if the pool was just those items, they be more or less vote neutral. Call it the "mediocre effect".
But when items in the lower 90% went against an item in the upper 10%, they lost more times than they won. That would be enough to shift the lower 90% to negative votes and the upper 10% to positive.
It's complicated somewhat by the bottom 25% getting cut mid contest, but I think that would have enhanced the mediocre effect since the "disliked effect" items would have been swept out, leaving those items that were mixed reaction and positive reaction.
I think.
Oversimplifying the voting a bit...
Say there's 900 items total, which seems to be the ballpark.
90ish items, or 10%, ended up positive.
The most positive item was roughly +100 votes.
Say all 90 positive items averaged +50 votes, that would be +4500 votes for the top 10%, which means the bottom 810 would be -4500 collectively. That's more than enough leeway to make the math work out.
Spread those 4500 votes out over 800ish items and it seems quite possible.
Thanks, Sean. As always, I'm amazed at just how open and responsive you and the rest of the Paizo staff are. I think it's one of the greatest strengths of the company.
(Actually, amazed isn't the right word since it's mostly expected by now; as always, I'm so pleased.)
Orcus - I'll probably be seeing you in the 9 Blazing Months thread, or whatever similar thing happens this year. We can help each other.
Awesome! I'll give my best to both learn and help out.
| 1 person marked this as a favorite. |
Thank you very much for posting the top 89!
I have followed this contest for years, this is the first time I entered. Voting definitely shed a lot of light on what I can do better for item design. My item was the mage's battlecloak, and I happily admit it needs some polishing and a few errors fixed.
Kenton, for what it's worth, the mage's battlecloak was the first item I saw that went into my Keep pile. It was very simple and definitely had "why haven't I already got one of these?" going on. Well done.
So...math not being my strong suit, how is it if every item was paired with another, that only 89 out of over 800 got positive vote records, esp if #1 had onoy a 75% positive ratio?
Anyway, that bit of confusion aside, let's say a factor of two got loose in there somewhere and we had 800 items in the cull [1]. Since the number 1 item was upvoted more than 400 more times than downvoted and had over 500 votes, the item had around an 80% approval rating. Sean says 75% later, so we'll go with that [2].
Now, we'll take as given that we know only 89 items had a positive net total [3]. The best-case scenario for this to be true would be if every single item in the Top 89 had around a 75% approval rating, just like the top item (the lower the approval of the 89, the harder it is to make [3] be true). Since the culled items probably pushed up every other item's approval rating, excluding the cull makes [3] more likely to be true, so we'll ignore the cull for now. Also, because it makes [3] more likely to work, we'll assume that the top 89 items generated all their losses against other top 89 items, never giving any victories to the bottom items.
Let us consider the 90th best item, which by [3] must have a losing record. The worst the 90th item could have done was to completely tie with all the items below 89th in its record (otherwise it will gain an edge from beating the other low-ranked items and be more likely to have a positive record). Since we assumed the top 89 items never lost to the bottom items, that means if there is a complete tie amongst the lower items, that the 90th best item is guaranteed to have a losing record.
TLDR--It's definitely possible, though it says a lot about the top 89 items rarely losing to the bottom items and the bottom items mostly having fairly homogeneous slightly losing records.
@RE
Bottom of the top 89 had close to 50/50 ration according to post...
@RE
Bottom of the top 89 had close to 50/50 ration according to post...
Breaking that assumption won't break the math, as long as they got many of those losses to the other Top 89.
@RE
Bottom of the top 89 had close to 50/50 ration according to post...
Well, to avoid confusion, we had two mathematical models used on the data. The "top89" is from a subset of the full data, and doesn't have quite the same parameters.
In other words, in the interest of providing more information, I've confused some people.
I'm not 100% on the math they used for each method, but what I think happened is we did a (faster but less intensive) mathematical ranking of all the items post-cull, and then a second (slower but more intensive) ranking of the top 200 items from that post-cull list (which is where Clark's "top 89" list comes from). And I think the math-intensive second list is only comparing cross-votes in that top 200 list, which is why #89 in that list is "tied" (same number of upvotes as downvotes). But if I look at the fast mathematical ranking of all items, #89 isn't "tied" (it has nearly 100 more upvotes than downvotes).
So my comments about "The difference in the number of votes on #1 and #400 is less than 20" and such is referring to votes in the full data list, not the shorter list drawn from the top 200 of that full data list.
Why didn't we do the slower but more intensive ranking of all post-cull items instead of just the top 200? Because it would take several days to compile that data, which means we'd get a voter-ranked list right around the time we're scheduled to reveal the Top 32, which means the judges would not have time to select the Top 32 or make comments on the items. So we went with a faster method for a preliminary (top 200) subset list, and used that as a general reference while waiting for the slower math on that subset. As it turns out, the lists were very, very similar overall... but because the first (full{ set was using a larger vote set, each item in that set got more votes.
TLDR: Just because #89 was "tied" in counting votes in a subset doesn't contradict that the number of votes for #1 and #89 were within 20 of each other in the full set. Likwise, #1 in the subset was way ahead in subset upvotes compared to subset downvotes compared to subset "tied" #89, and there's no contradiction with the "within 20 of each other" vote total in the full set.
And I think the math-intensive second list is only comparing cross-votes in that top 200 list, which is why #89 in that list is "tied" (same number of upvotes as downvotes). But if I look at the fast mathematical ranking of all items, #89 isn't "tied" (it has nearly 100 more upvotes than downvotes).
Aha, it's only within the best-of-the-best Top 200 that #90 has a negative record. Well that makes a lot of sense, and it's pretty much what you would expect (that the equivalence point is around the 50% percentile of the items). As a Failed-to-Get-Into-Top-89 guy myself, that also makes me feel a lot better. You rock, Sean!
I'm hopeful the judges will also throw in on the critique my ride... I mean, critique my item thread. Will that be at all possible?
By the way, I appreciate the feedback on my prise de fer plastron from a couple years back. I even listened to the no french part. I also think I listened to other stuff as well, but that doesn't mean I got the whole point. Slow learner here.
Much appreciate, everyone.
I agree with Meepo and St.C in that it feels like a lot of this practice was for naught.
I thought I would feel that way, but I don’t. I had a good time designing, it was fun.
In the past few weeks, I’ve been planning the framework for each and every round of the contest. I created 10 wondrous items, 6 archetypes (the twist eliminated almost all of them and I had to start again), 3 creatures, 3 encounters, 1 module proposal.
I’m the happiest with the module proposal because when I entered RPG Superstar, I honestly didn’t think I could design an entire module. After creating my proposal, I know I can do it. And it was going to be very cool, innovative, and fun. It was fun brainstorming about it on my lunch hours. So this entire RPG Superstar process was worth my time, despite the mixed results.
However, I think I'll do my practicing in private, with my home group, and saving my best ideas for the next contest or perhaps another publication.
Anyway, thanks for posting the top 90, but I think one of the judges said it best already, if you're not in the top 32, does it really matter?
I'm more confused about the numbers than I was before. Heh.
I would expect a sort of power law at work in the + voting numbers, but maybe not.
Moving onward!
Chief Technical Officer
|
...And I think the math-intensive second list is only comparing cross-votes in that top 200 list, which is why #89 in that list is "tied" (same number of upvotes as downvotes). But if I look at the fast mathematical ranking of all items, #89 isn't "tied" (it has nearly 100 more upvotes than downvotes).
That's correct: the ratio Clark mentioned actually represents how well-liked items did against *other* well-liked items. In the overall voting, as one might expect, a little less than half the items had "winning records."
I am astounded by the number of items in the top 89 I never even saw. After the cull, it got a bit easier but the initial grind of the same 50 or so items (by my count) really wore me out (and my wife's patience, too!..."Just one more vote, Hun!"). I thought by making Dedicated Voter I would see most of the items but now I doubt that I got anywhere close.
The m dolls of planar cadre and swarmbane gauntlets were in my top 5. The one missing is the seppuku blossoms. Whoever wrote those, I hope they put it up for critique.
Thanks Clark, Sean & Vic for providing the top 89 list and explanations of voting data.
As someone who didn't make the 89, it tells me I have a lot to improve on for next year.
Great contest. I've enjoyed the whole process.
RPG Superstar 2013 Top 32
,
Dedicated Voter Season 6, Dedicated Voter Season 7, Dedicated Voter Season 8, Star Voter Season 9
aka Steven T. Helt
|
And the process isn't over. We'll all continue to learn a lot by seeing the contestants go at it, reading critiques and seeing how well they received. Especially as regards mojo and cinema, learning to write a better monster or tactical encounter may also help one write a better wondrous item!
Especially as regards mojo and cinema, learning to write a better monster or tactical encounter may also help one write a better wondrous item!
Not to mention give us plenty of inspiration for our own home brew campaigns!
I was quite pleased to see I hadn't been in the cull, making the top 89 has already got me thinking about next year - hmm, how about some chalk that.... :P
Without having yet read the 97 posts in this forum, I have to say that seeing that my item was not in this list cheers me up a bit, strange as that sounds.
While I didn't make the top 89. I sincerely appreciate that the judges made this list available. I agree that more transparency can only make the end results better.
If the RPG Superstar process can help discover and mold future gaming talent, we are all the better for it.
Thank you.