I'm really kind of annoyed with this voting system, I wish it had a little more... variation in it. Like rating each item on a 1-10 scale, that way each item gets an over-all rating instead of which item is comparatively better. Being able to rate items on a 1-10 scale would allow judges to really see just where the voters think an item is. It would reveal a lot of data about items in the game. For instance, an item that is literally hit-or-miss could have people voting 10's and 1's almost exclusively, which would let the Judges see that this item is either really good, for some people, or really bad for others, and let them just carefully go over such items. If everyone is voting 8's and 9's on an item, the judges could review in, knowing that such an item is really popular. It would help the judges, and Pazio, gauge just where the audience is at in the voting.
Even splitting the "Both items are equally good/bad" would help. Some items are just plain awful, and I've seen them paired with an equally awful item, and I want an option to make sure my vote counts them as awful items, not possibly good, or possibly bad, but tied with the other item.Just having an "Equally Good" button and then an "Equally Bad" button would make the voting better, at least in my opinion.
You understand that splitting the button to "Equally Good" and "Equally Bad" doesn't make a difference with the current voting system?
You understand that splitting the button to "Equally Good" and "Equally Bad" doesn't make a difference with the current voting system?
There does seem to be a lot of confusion about the voting system.
Voting for an item over another item is not equivalent to giving that item one vote. It is a statement that x item is better than y item. If you are looking at two bland items, you are really just deciding if which is going to be ranked #678 and which will be ranked #679 (numbers are random examples and subject to change!). This is why you shouldn't give yourself an aneurism over deciding between two bad items, but two good items deserve a lot of consideration.
At least that is how I understand it.
I think it's clear that the voting system is not as clear and as intuitive as could be hoped.
I know a little bit about the math, so to me the system seems more natural. The trick is that we're basically taking one big task - "Find the top 100 or so best entries" - but, we're trying to distribute it across a crowd.
Most "intuitive" breakdowns would be pretty problematic. Offering voters a "keep/reject" button, or grading individual entries on a scale, would be tricky - (A) each voter has his own subjective yardstick, so votes are very tough to bring together into a final ranking, and (B) it's ripe for abuse - a participant is incentivized to vote down items that aren't his. You can analyse any particular system, but most "simple" voting systems have difficulties, which the Condorcet system is resilient against.
But it is harder to understand. And it does require a LOT of extra work that won't be significantly represented in the final results. I sincerely hope that the organizers can take this year's experience, and improve on it for next time around. But please don't think that just because the voting system is a little nonintuitive, or because the number of exciting items is low, that the system isn't working. It's working just fine. It's not unadulterated fun for the voters, sure. But, y'know, judging never has been. And the slog-voting is purely voluntary. And a lot of the voters seem to be enjoying it well enough to feel involved, to participate on the boards, and to keep plugging at it :)
A ranked pair voting system is actually the best voting system ever devised.
There is a specific variation known as the Shulze Method that would make an excellent political voting system.
It is a little odd compared to what we're used to in voting systems, but it really is the best way to ensure the best items get to the top.
It allows a hierarchy to form naturally over time and does basically rank items without the need for arbitrary numbers.
Item 1 and 2 come up, you vote 1.
Item 3 and 4 come up, you vote 4.
Item 2 and 3 come up, you vote 3.
Item 1 and 4 come up, you vote 4.
You just ranked those items:
4 is best, 1 is second, 3 is third, 2 is last.
So you did pretty much score them, you just did it without actually using a scoring system.
A ranked pair voting system is actually the best voting system ever devised.
There is a specific variation known as the Shulze Method that would make an excellent political voting system.
It is a little odd compared to what we're used to in voting systems, but it really is the best way to ensure the best items get to the top.
It allows a hierarchy to form naturally over time and does basically rank items without the need for arbitrary numbers.
Item 1 and 2 come up, you vote 1.
Item 3 and 4 come up, you vote 4.
Item 2 and 3 come up, you vote 3.
Item 1 and 4 come up, you vote 4.
You just ranked those items:
4 is best, 1 is second, 3 is third, 2 is last.
So you did pretty much score them, you just did it without actually using a scoring system.
Indeed. This is the "Swiss-system" of tournament pairings, which will be familiar to anyone who has entered a large tournament such as a CCG tournament or badminton or bridge. Swiss-system is a Conddorcet method. Paizo haven't revealed the exact maths they are using (at least I don't think so), but it will probably be something similar to this.
What is also interesting is that, after a while, the matches should be between roughly equivalent candidates. This does seem to be the case with the voting - by now you should be offered two mediocre items or two good items etc to try and rank, you should rarely see a really good item vs a really bad item. My experience of voting is that the matches are now becoming closer. (This could be just me ofc).
If this is correct, this means that when you spot your own item (I haven't yet) look at the item it is paired against. If it is really good, your item is probably doing well as this is a "top-ranked" match. If the other item is really bad, well, you are probably lower down the ladder at this point.
| 2 people marked this as a favorite. |
I've just had the same pairing appear to me three times in a row.
A ranked pair voting system is actually the best voting system ever devised.A ranked pair voting system is the best voting system ever designed for ranking a few dozen candidates. Unfiltered ranked pair voting is thousands of man-hours of wasted time in a field with hundreds of candidates.
But since the manpower is currently free and voluntary, it works out fine.
Remember, the voting isn't determining who wins, it is only determining the best items so the real judges can look through them and choose the top 32.
This is just the masses separating chaff from wheat for them.
PostMonster General
|
| 1 person marked this as a favorite. |
Because I'm paranoid that the voting system might not work properly—nobody's ever done anything quite like this, to my knowledge—over the weekend I've been doing quick "what if" counts on the rankings. It looks to me like the good items are being shuffled to the top of the list, and the bad items are being sent to the bottom. And the rankings are getting adjusted over time, with some items bubbling up to the top of the list that weren't there before. So while we may want to adjust some things for next year if we do public voting on round 1 again, I think it'll turn out pretty well this year. Better than I think any of us expected, actually.
I get the feeling that I will not see many of the top 32 items this year. I would love to see all the items.
I propose a simple system where I could give a grade from 1 to 100 for each item. Also give me the ability to go back and change my vote on items I already saw. (Optionally, when I put in a score of 80 on an item, give me another item with a close score so I can adjust one or both items.)
When I've seen all the items, I could sort by how I've ranked the items and start to adjust my own grades relative to other items.
The order in which I first see items in should be based on items which have received the fewest grades. I like the idea that all items can be seen by someone taking the time to vote.
Keep a timer for new items, but maybe make it 20 seconds? I can start reading some items and know they will not be top 100.
After voting closes, the judges can see how my grades compare to everyone else for all items, a specific item, top 32 items, etc. If my voting is really off, all the bad items are given 100 and the great items are given 1, my weighted score would be adjusted down. (If I am not following the guidelines, my weighted score could be dropped to 0.5, 0.1, 0.01, etc.)
Every item will then get an overall average score, highest average gets looked at first by the judges.
I'd be more than happy to help with the math and algorithm, although I'm sure you don't need my help with specifics.
The problem I have now is that I'm not seeing any new items and rarely seeing items which make me go wow.
If they're going to keep this voting system for next year, I actually want to see a single item repeatedly compared against the next 5 items I see, and the timer reduced to 30 seconds. This way I don't have to re-read items I've already reviewed as often when giving comparisons (some of them are very similarily themed). It would save me time and would make the people who desire shorter wait periods happier.
I can say with utmost confidence that I've seen some items 10-15 times now.
Because I'm paranoid that the voting system might not work properly—nobody's ever done anything quite like this, to my knowledge—over the weekend I've been doing quick "what if" counts on the rankings. It looks to me like the good items are being shuffled to the top of the list, and the bad items are being sent to the bottom. And the rankings are getting adjusted over time, with some items bubbling up to the top of the list that weren't there before. So while we may want to adjust some things for next year if we do public voting on round 1 again, I think it'll turn out pretty well this year. Better than I think any of us expected, actually.
I think people are dissatisfied because they see so many bad items reappear over and over again.
After seeing so many bad items repeats and no superstar quality items, I find the voting system not very "rewarding", even if I understand how the voting works and know it is efficiant.
Right now I'm having a good deal of bad items reappear for me, with some new ones appearing once in a while. Its getting weird.
Dedicated Voter Season 6, Star Voter Season 7
|
The weird thing for me is I keep on seeing eerily-matched pairs.
Intellectually, I know that this is coincidence, but I have repeatedly been seeing pairs of, lets say, two similar crafting items or two SIAC-type items, or two very similarly-priced items.
Even though I am tired of seeing some of the same stinkers again and again, I have faith that the math (which other people seem to understand) will work itself out the way that its supposed to.
The process has been interesting so far and very educational for me. For reference, I have voted on probably about 400 pairs so far.
Marathon Voter Season 6
|
If we get to rate on 1-10 you would have people giving their item a 10 and every other a 1.
You have to keep in mind we are here to separate out the bad. Once an item gets up to the top the judges decide on it, they dont need our input up there. They need us to make sure good items that got creamed early by better items still have a chance to get to the middle rankings.
The weird thing for me is I keep on seeing eerily-matched pairs.
Intellectually, I know that this is coincidence, but I have repeatedly been seeing pairs of, lets say, two similar crafting items or two SIAC-type items, or two very similarly-priced items.
I'm getting a lot of that, too. I am often choosing between two goggles or cloaks or saddles or quills. Especially quills. They are the Starbucks of my little voting corner.
Might be word recognition. I've gotten pairings of items that have the same word in the title, sometimes the descriptor rather than the actual item itself.
| 2 people marked this as a favorite. |
There is no word recognition, or attempts to pair similar items. It's just chance, and the human tendency to see patterns everywhere.
| 1 person marked this as a favorite. |
Or Ross inadvertently created the beginnings of Skynet...
| 1 person marked this as a favorite. |
(raises thread from grave by the light of a full moon)
From a mathematical POV, it's plausible that once an item has been declared Worse often enough that there's only, say, a 0.1% chance of that happening randomly, there's no point in testing it any more: it's not going to get into the top 32. Or 100. Might save people a lot of effort in trawling through the dross.
0.1% is about 3.8 standard deviations. This corresponds to, for example, 19/20 or 39/50 votes for Worse. Pick your own level of significance.
The obvious disadvantage to this is that there won't be a complete final ranking, but I don't expect there will be anyway. Paizo don't normally reveal the number of entries.