I should mention that I only spend about 10 to 15 minutes voting and then I move on to something else. Since my last post, I have had two items come up that I've seen before. Once I start seeing them more often, I can see that it will be faster to judge them.
| 2 people marked this as a favorite. |
The timer would be an almost non-existent issue if we weren't seeing the same items so often.
If anything, that's the place to focus, it would make all the voters much happier.
I understand it places a load on the server, but there are highly-efficient methods for solving the no-repeat problem that should be fine for the (relatively) small number of items involved here.
If anything, that's the place to focus, it would make all the voters much happier.
+1 for fresh choices
After 15 hours of voting, very few seem new anymore...
I just saw a new one item, after a dozen matches of items I've seen many times. I'm not a professional programmer, but I do know how I could program in BASIC to give me at least one new item each vote until I've seen the all. I don't see how it would slow everything down too much if only one item was new each time, but it would make voting more interesting.
About to call it a day and see if I manage some more voting tomorrow....
After 15 hours of voting, very few seem new anymore...I just saw a new one item, after a dozen matches of items I've seen many times. I'm not a professional programmer, but I do know how I could program in BASIC to give me at least one new item each vote until I've seen the all. I don't see how it would slow everything down too much if only one item was new each time, but it would make voting more interesting.
About to call it a day and see if I manage some more voting tomorrow....
Imagine trying to track number of views for each item, or worse, each possible pairing, against the number of voters. That's a lot of computational time needed. A dataset that large will also guarantee worst case search performance a few times or more.
Also imagine needing to keep track of those views across all users multiple times for when X users are requesting a new pairing at a time.
As a professional programmer, my hats off to the Paizo team.
| 2 people marked this as a favorite. |
It's 60 seconds. Have we really become so impatient that we can't wait 60 seconds? C'mon folks, pick real problems to complain about.
We have, Jerry, we have. O tempora! O mores! ;)
RPG Superstar 2012 Top 16
,
Star Voter Season 6, Dedicated Voter Season 7, Dedicated Voter Season 8
aka FaxCelestis
|
After 15 hours of voting, very few seem new anymore...I just saw a new one item, after a dozen matches of items I've seen many times. I'm not a professional programmer, but I do know how I could program in BASIC to give me at least one new item each vote until I've seen the all. I don't see how it would slow everything down too much if only one item was new each time, but it would make voting more interesting.
About to call it a day and see if I manage some more voting tomorrow....
Imagine trying to track number of views for each item, or worse, each possible pairing, against the number of voters. That's a lot of computational time needed. A dataset that large will also guarantee worst case search performance a few times or more.
Also imagine needing to keep track of those views across all users multiple times for when X users are requesting a new pairing at a time.
As a professional programmer, my hats off to the Paizo team.
Credit where credit is due, obviously, but I don't see what's so computationally intense about having a lookup table comparing voters to items, with the table having a yes/no flag, and requiring one of the items be previously unseen. Counting total votes is unnecessary for the purposes of matchmaking: as long as one item is new for the voter, you can guarantee it is a match they have not seen before. Once they get through the entire pack, then you can start pairing seen items and not care about the table (which can be bypassed via a user-based flag: "Has this person seen all the items? If yes, skip the table; if no, find one he hasn't seen and throw it against a random item.").
@James Raine
This will give you a background on analyzing algorithms.
Growth and computation times get messy.
The biggest problem, though, is going to be searching through a database. You have to look at worst case scenarios and how long that will take. This voting system is probably something like NlogN but Gary would have to comment on that.
To be honest, I think the process as it exists is a good thing. It subtly discourages a few motivated people from having a grossly exaggerated effect on the outcome, and in addition lets late entrants to the process experience the same voting conditions as those who started madly reloading their browser at second 0 of the voting.
I've come to think the process is as it should be :)
|
RPG Superstar 2012 Top 16
,
Star Voter Season 6, Dedicated Voter Season 7, Dedicated Voter Season 8
aka FaxCelestis
|
@James Raine
This will give you a background on analyzing algorithms.
Growth and computation times get messy.
The biggest problem, though, is going to be searching through a database. You have to look at worst case scenarios and how long that will take. This voting system is probably something like NlogN but Gary would have to comment on that.
Yeah, I get that. However, what I'm proposing (not even really a proposal, so I guess I'm just musing) is something like this fauxcode:
if($SeenAll == 1)
{ get(RandomLeft,RandomRight) }
else
{ get(RandomLeft,RunLookup() }
RunLookup{
get(
where($ItemSeen == 0)
{RandomItem()}
)
}
Wouldn't that mean that each item would gain a data point (seen=yes/no) for every person who votes? That multiplies the amount of data you're referencing.
Wouldn't that mean that each item would gain a data point (seen=yes/no) for every person who votes? That multiplies the amount of data you're referencing.
When I log on to the message boards, it knows how many new messages have been posted since I last looked at each thread. Does every thread have it's own information, or is this data specific to me? My guess is the latter.
Couldn't something similar be used for the Round 1 voting? If I have a look up table assigned just to me, it does not matter how many items are in the contest or who else is voting. It couldn't be much more difficult than keeping track of the message boards.
|
RPG Superstar 2012 Top 16
,
Star Voter Season 6, Dedicated Voter Season 7, Dedicated Voter Season 8
aka FaxCelestis
|
Wouldn't that mean that each item would gain a data point (seen=yes/no) for every person who votes? That multiplies the amount of data you're referencing.
Sure, but it's not a quantity value ("seen 30 times"), just a per-item yes/no flag, so it's a linear scale rather than a quadratic one (+X data points per voter, where X = number of entries). As far as lookup time that's a negligible difference, but for the math involved in picking which item to show you it is a bigger distinction: you don't need to care whether or not the item in question has been viewed forty times, as long as it hasn't been viewed by Voter X.
You still get new data to populate the voting pool with, and it has the added advantage of ensuring a broader view of the items submitted. The current system doesn't promote breadth of view, but it does have some advantages for depth of view: that is, per user, you have a lot of data points for fewer items, rather than a lot of data points distributed evenly across a bunch of items.
In the long run, it frankly doesn't matter from a statistical perspective. However, from an end-user perspective, ensuring a new item every pull would at the very least keep voters engaged.
The thing that Sean is hitting on, and what I've alluded to, is the fact that this has to happen X times at any given second.
If a bunch of people are trying to read/write to a database with a large amount of data at the same time then searching through the database for your particular records becomes a longer process. Adding another column to check across all these datapoints and access/seek times just adds to that.
Making sure you get a "new" item every round either has to be tracked by the pairings (because getting items A:B then B:A is NOT a new item, but it looks new to the algorithm) which makes the dataset grow even larger.
There are myriad other considerations, but I'll wrap it up by saying that all this has to happen when a lot of people are accessing the same data at the same time and response rates need to be kept up, not to mention making sure there is no data collision if the same record is trying to be accessed by 2 or more people.
Right ... but the point was that it was a tradeoff between server load and ideal algorithms (for particular definitions of "ideal").
Furthermore, while a method that ensures all items are seen as quickly as possible might make a voter happier, I don't believe it would actually result in better data, since it's a lot more likely a voter would walk away after gettting duplicate items, thus resulting in a smaller data and potentially lower quality data set.
My guess is the current process works better for the purpose intended (getting a large quantity of useful data), even if it's not ideal from the viewpoint of someone who just wants to see all the items. After all, many, many quizzes and tests ask the same question multiple times in order to ensure an answer given is the actual answer. This is no different.
|
RPG Superstar 2012 Top 16
,
Star Voter Season 6, Dedicated Voter Season 7, Dedicated Voter Season 8
aka FaxCelestis
|
I guess part of my issue is that I'm running into a plethora of items I've seen before. If there was a way that the cooldown could be, say, halved if both items are ones you've already voted on, that would probably solve my issue as well. Not that I'm not voting now, just that I'd like to improve the process if I could.
Assuming you could do this, this may help without needing to search for unseen items... it doesn't force extra searchs for items not seen, it tackles it by reducing the 60 second "gah seen these" wait times.
Start by doing what you do now to select 2 items from the pool
... no change/cost on time/cpu ...
Set wait time to 60 seconds...
Get unique key of account and unique key of item - combine them in some way (two parts of a unique key to a binary switch table)
Try to access table for that combined key value...
If not found, store a "true" value into the table for that key.
If found, halve the current wait time (i.e becomes 30 seconds).
Cost - one fetch and possibly one store, rest is maths.
Now repeat for the second item and the account key.
This means you would with a couple of look ups and a decreasing volume of record creations be able to offer 2 items presented with an appropriate wait time that doesn't cause so much voting friction when items are repeated.
60 seconds wait time if neither item seen before
30 seconds if one of them has been seen before
15 seconds if both have been seen before.
For at worst case, 2 fetches and 2 stores when an account starts voting to 2 fetches and the occasional store after lots of voting.
However, from an end-user perspective, ensuring a new item every pull would at the very least keep voters engaged.
Definately. I stopped voting because the repeats of bad items was getting ridiculus.
It does mean I can spend more time on my archetype, but I would have liked to have seen 1 superstar quality item.
Worst case scenario in this sense means how long it takes to find the record you are looking for.
Not how many fetch operations...although that will be very important as well.
For example, if record A exists somewhere just past the 3/4 mark of the entire dataset, then it will be about the most time it can take to actually find that record. 10 milliseconds? 1 second? Not sure what it is in this particular case, but fetch times will increase as more people try to access the dataset.
| 2 people marked this as a favorite. |
However, from an end-user perspective, ensuring a new item every pull would at the very least keep voters engaged.Definitely. I stopped voting because the repeats of bad items was getting ridiculous.
It does mean I can spend more time on my archetype, but I would have liked to have seen 1 superstar quality item.
I have also stopped voting because of repeat items.
I like my item, but I doubt I'll make the top 32 as a first year entrant. Thus, I'm basically done until next year. Unless they find a way to let me see all items or top 100 items or something more interesting than deciding between two items I've already seen a dozen times, I do have better things to do with my limited time.
If Paizo wants to decrease interest in voting over time, they have the right system. If they want to keep me engaged before mid-January, something needs to change.
Edit: I see they have changed things a little, dropped the bottom items and given tags. I will give it another go and see if it is better....
I like my item, but I doubt I'll make the top 32 as a first year entrant.
I thought the exact same thing last year. Heck, one person made the Top 32 without ever having played Pathfinder.
|
RPG Superstar 2012 Top 16
,
Star Voter Season 6, Dedicated Voter Season 7, Dedicated Voter Season 8
aka FaxCelestis
|
| 1 person marked this as a favorite. |
I like my item, but I doubt I'll make the top 32 as a first year entrant.I thought the exact same thing last year. Heck, one person made the Top 32 without ever having played Pathfinder.
Wasn't that Garrett? The guy with the monkey avatar? I think that's a monkey, anyway. I liked that item.
Credit where credit is due, obviously, but I don't see what's so computationally intense about having a lookup table comparing voters to items, with the table having a yes/no flag, and requiring one of the items be previously unseen.
I don't think this would be easy at all.
- It would require referring to "current user" in the selection of the pair, which AFAIK is not currently done. So that'd be inserting an extra dependence into the pair-selector.
- What's much, much more important than "a single user shouldn't see the same item two often" is "each item should get roughly the same number of votes". Right now, weighing pairs to undervoted items is easy - you just always pull out two items with the fewest votes. But if you need to tailor each pair to individual voters' past history, that's much more difficult. A voter who has seen lots of "undervoted" items will need to see "overvoted" items, making them more overvoted. Or else you need to play some game with constraints and weighting, which adds a ton to computation time.
| 1 person marked this as a favorite. |
It's a Vanara, which is basically a sentient monkey. So close enough!
| 1 person marked this as a favorite. |
Wasn't that Garrett? The guy with the monkey avatar? I think that's a monkey, anyway. I liked that item.I hope that's a monkey. If that's not a monkey, I don't want to know what it is.
Given the image is named avatar/PZO9040-Monkey.jpg, I feel safe saying it's a monkey :)
| 1 person marked this as a favorite. |
You won't have seen those two items paired together, though. That's the important part, judging each item against the other, not each item individually.
Not so. I've seen the same two items paired together three times in a row.
I considered it my punishment for voting them equal.
The combination of one minute delay and endlessly repeating items is making the voting process truly torturous. It's like working overtime as a line worker in a boredom factory.
In years gone by, I spent a summer working in an electric lamp production line on quality control - does the lamp light, yes, pass, does the lamp light, no, bin, does the lamp light, yes, pass...
I was SO glad to get back to Uni.
You won't have seen those two items paired together, though. That's the important part, judging each item against the other, not each item individually.Not so. I've seen the same two items paired together three times in a row.
I considered it my punishment for voting them equal.
You are correct, sir. Voting two items equal forfeits the "won't see this pair again" guarantee. As long as the pair goes un-bejudged, it can show up again - and very often does.
| 1 person marked this as a favorite. |
The combination of one minute delay and endlessly repeating items is making the voting process truly torturous. It's like working overtime as a line worker in a boredom factory.
You could always, y'know, stop voting. :P
I'm spending most of my days now coddling my newborn and trying to get him (and his mom...) to sleep. So I'm already reduced to a mechanical lulling device. Voting gives me marvellous little nuggets that don't require more attention than I can allot 'em.
The combination of one minute delay and endlessly repeating items is making the voting process truly torturous. It's like working overtime as a line worker in a boredom factory.
But it starts off fun, and gets decreasingly fun as you continue. I kinda like this, as it prevents one voter from skewing the results too much. A broad swath of participation is probably better than a few die-hard voters.
That being said, it is also designed such that every voter walks away with a bad taste in their mouth, having reached their individual pain threshold. This is less good.
| 1 person marked this as a favorite. |
Take child to car seat.
Drive car in warm dressing gown.
Baby asleep in under 5 minutes
Do not try to take back into house, snooze in the car.
:)
Take child to car seat.
Drive car in warm dressing gown.
Baby asleep in under 5 minutes
Do not try to take back into house, snooze in the car.
:)
Bolded part is probably the MOST IMPORTANT!!! :)
Right ... but the point was that it was a tradeoff between server load and ideal algorithms (for particular definitions of "ideal").
I dunno, you can probably streamline this client-side.
Give each user a cookie that is basically numbers 1-N (N being the number of entries) in a pseudo-random order. This would be the queue for the order they see entries. Encode it to discourage cheating, naturally.
When they load the page, they pull the next two numbers, and send a request to the server to get the textual details.
By pseudo-random, I mean two things:
* the server can shuffle them so that each user gets a unique order
* if you submitted an item, it's always last in your queue :P
Right ... but the point was that it was a tradeoff between server load and ideal algorithms (for particular definitions of "ideal").I dunno, you can probably streamline this client-side.
Give each user a cookie that is basically numbers 1-N (N being the number of entries) in a pseudo-random order. This would be the queue for the order they see entries. Encode it to discourage cheating, naturally.
When they load the page, they pull the next two numbers, and send a request to the server to get the textual details.
By pseudo-random, I mean two things:
* the server can shuffle them so that each user gets a unique order
* if you submitted an item, it's always last in your queue :P
+1, please.
| 1 person marked this as a favorite. |
I dunno, you can probably streamline this client-side.
No you can't. Because it's much much much more important to balance the relative number of votes each submission receives, than it is to balance the variety of submissions each voter views.
The method you propose destroys any control the server has over which entries get voted on next. If the 1,000 users supposed to vote on the tennis racket of splendiforousness somewhere in their first 1,000 votes happen to be the voters who get bored after 10 pairs, then you have no idea where that racket is supposed to be ranked.
The result would be that "which items get voted on enough for a reliable ranking" would be randomly determined by "which pre-calculated sequence of items was assigned to dedicated voters." You really don't want that.
I dunno, you can probably streamline this client-side.No you can't. Because it's much much much more important to balance the relative number of votes each submission receives, than it is to balance the variety of submissions each voter views.
The method you propose destroys any control the server has over which entries get voted on next. If the 1,000 users supposed to vote on the tennis racket of splendiforousness somewhere in their first 1,000 votes happen to be the voters who get bored after 10 pairs, then you have no idea where that racket is supposed to be ranked.
The result would be that "which items get voted on enough for a reliable ranking" would be randomly determined by "which pre-calculated sequence of items was assigned to dedicated voters." You really don't want that.
My method does assume that:
a) having your item at the end of the queue encourages you to press on.b) we've got a large enough voting pool that any one item getting significantly fewer votes is statistically unlikely.
c) the items end up being picked by an electoral college anyway, so our votes really just serve to sift out the junk a little, and give us something to do to make us feel useful while waiting to see who makes round 2.
d) it is more important to Paizo that this experience is optimized for our (their customer's) enjoyment rather than for absolute fairness. It just has to be fair enough.
The server side can always send back a response to re-order your list if it needs more or less votes on a particular item. The point is, it doesn't have to get bogged down tracking which user has seen which item. It can be done.
I agree with Adam above. It can be done, and would be much more enjoyable. I had given up, but the cull got me back to voting again. Since the cull, I've only seen a dozen new items and the last one was many votes ago.
I'm about to give up again.
If we could see every item at least once, then rank our top 32 only after we voted, I'm sure there is an algorithm which could both be fair and more interesting for the voters.
| 1 person marked this as a favorite. |
Adam, I suspect the organizers didn't design the system with Marathon Voters in mind :) Seriously, on the first update there were 505 people who voted 100 times or more - and at least 1000 entries. The dedicated voters are obviously the most vocal on the forum boards, but they're hardly the entire picture.
I'm confident that if this year's experiment is a success (and I'm sure it will be), then next year they'll be able to tweak the system, or use a new system that better addresses voter burnout. That being said, modifying the existing system in the way you're suggesting is not an easy fix.
Working with big numbers means dealing with a lot of randomness - consider that at the time of the cull, if there were 2,000 original submissions in the pool, then each one of them was only voted on a mere 200 times. If you let that 200 roam randomly, then most items will be fine, but some items would get drastically undervoted - and there would be nothing that could be done about it. And we did want the cull, didn't we?
It'd be really nice if there were an easy fix for this. There might be one! But this isn't it. For a fix to be viable, it would have to include some sort of built-in balance with the guarantee of roughly equal votes per entry. The two are inherently in conflict, so any fix that doesn't address the balance is going to cause more trouble than it's worth.
| 1 person marked this as a favorite. |
Working with big numbers means dealing with a lot of randomness - consider that at the time of the cull, if there were 2,000 original submissions in the pool, then each one of them was only voted on a mere 200 times. If you let that 200 roam randomly, then most items will be fine, but some items would get drastically undervoted - and there would be nothing that could be done about it. And we did want the cull, didn't we?
It'd be really nice if there were an easy fix for this. There might be one! But this isn't it. For a fix to be viable, it would have to include some sort of built-in balance with the guarantee of roughly equal votes per entry. The two are inherently in conflict, so any fix that doesn't address the balance is going to cause more trouble than it's worth.
It's actually the opposite way. Statistically with more voters there is less randomness than with a small sample size. With the number of voters and the number of times people are voting, I believe they can get rid of the part of the algorithm that ensures items are seen an equal number of times. That was important with a small sample size, but not when there are 100,000+ votes being cast.
I believe that Standback was referring to high randomness in the number of votes per item due to the large number of *items*, which is correct. Except that it's even worse than he said, since with 203,909 votes cast as of Dec. 28, that would only be about 102 votes each per 2000 items. That's really not much at all, and the variation could be huge without a thumb on the scale for less-balloted items to appear.
PostMonster General
|
| 2 people marked this as a favorite. |
Keep in mind the most important rule of web development: Never trust anything the client sends you. (Those who have attempted to bypass the countdown via Greasemonkey scripts or whatever will have noticed the extent to which we take that dictum to heart.) So we won't be encoding item selection or voting or whatever on the client side in cookies or what have you.
Personally I figured there would be a handful of voters who would get to "dedicated" status, and maybe one or two would make it to "marathon". I knew that wouldn't be how it actually shook out in practice but didn't know where my estimates would be wrong. So we erred on the side of trying to make everything as mathematically sound as possible.
If we do this again next year we'll be making changes to the process based on our experiences this year. Who knows, next year we might just do an advancing bracketed "hot or not"-type number-of-wins comparison or something. We won't know until all the votes are in and we've had a chance to do a complete post-mortem on the process. But so far we're super pleased with how it's going.
I believe that Standback was referring to high randomness in the number of votes per item due to the large number of *items*, which is correct. Except that it's even worse than he said, since with 203,909 votes cast as of Dec. 28, that would only be about 102 votes each per 2000 items.
I also thought it was 100 votes per item at first glance, but remember every vote is between two items. Every vote means 2 items seen; item distribution is roughly equal; so each item should be seen twice as many times as the total-votes/#items average.
But so far we're super pleased with how it's going.
Great to hear! Looking forward to the post-mortem. :)
Keep in mind the most important rule of web development: Never trust anything the client sends you.
You can always heap on encoding and CRC checks, but I hear what you're saying.
By no means am I saying you did it wrong, or suggesting you change mid-stream. You did your homework and went with a solid algorithm and an interface that was least likely to fail under strain.
I think it safe to say that it was generally well received by the fan-base. I liked it, certainly, and it drove me to greater enthusiasm and participation that did last year's event.
What I am saying is that I hope you do it again (but here's an itch I'd like you to scratch that might make it even better next time).
By the by, do you have a bug report thing for this? I don't want to just post it, as it might be exploitable. And I don't want to look silly, as I'm 99% sure you already thought of and mitigated this.