Most of you will know that as part of dpreview's 10th anniversary celebrations we recently launched the new Challenges feature. The reception has been fantastic and the 5 initial challenges launched (with many more to come) saw incredible participation (most filling reaching the 500 entry initial limit inside 48 hours). Old news you say? Well now that voting is underway I thought I'd talk a little about how that's progressing. In a word ... fantastic.
The really juicy numbers are locked away in our private (for now) stats pages (manna from heaven for visualization and stats geeks) but even our publicly available numbers are interesting.
- The least popular challenge already has 3969 votes with the most popular receiving several times that (with several days voting to go).
- The challenges feature is seeing about 5% of overall site traffic
- A vote is placed every 6 seconds (average)
- 32687 votes total (as of writing)
- 4674 entries across all challenges (~10% disqualified)
Even taking into account the traffic profile for new features (launch spike, trough, slow build) these are impressive numbers (we're certainly happy). We're really excited to see what happens when we throw open the doors and enable user-created challenges.
It's written in the stars
We've currently implemented only a single voting scheme: stars (½ to 5 stars in ½ star increments) but have plans to introduce others eventualy. The success of the challenges system hinges on votes reflecting image quality, but with the subjective nature of image evaluation and the diversity of voters we weren't quite sure what the distribution of votes would be. Some staffers expected a normal distribution (most votes clustering around the mean). Others expected more of a 'hot or not' style distribution (i.e. mostly 'extreme' votes). Of course the reality ended up being more complicated.
| Indoor portrait ... | Backyard Safari | Shadows | Wrapped Up | Compact/Abstract |
| Average: 2.77 | Average: 2.63 | Average: 2.62 | Average: 2.04 | Average: 2.41 |
|
|
|
|
|
|
There's a lot to be gleaned from the above. Not least:
- Significant variation between challenge averages (2.04 on the low end, 2.77 on the other).
- Distribution resembles classic normal distribution, but not in all cases (see Wrapped Up).
- Voters are generally less likely to award ½ star ratings than full star ratings.
Just to satisfy the curious, here's the overall distribution of votes (across all 5 open challenges) as of writing.
Leave it to Bayes
There's been a little concern in the forums that determining winners will be impossible due to our use of a star-rating system combined with the effects of outliers and variable votes per image. Fret not, our algorithm does takes into account both score and 'confidence' using some fairly established Bayesian techniques. There will be winners announced in a few days and the voting/ranking combo is definitely sorting the wheat from the chaff (it's fun to watch winners emerge). If there's enough interest I might explore this issue in a further post. Rest assured that we're on top of it.
Vote early and often
Whilst you can only vote for a single entry once, nothing prevents (or obliges) you voting for every image (in fact someone's already tried). Voting makes the whole thing work so make your vote(s) count now.
You've got to be in it to win it
Several challenges have begun since the initial wave with new challenges opening virtually every day. Make sure to get your photos in the mix and good luck.
You need to put a link somewhere for access to the challenges. (somewhere on the main page..) I would love to vote, but I hate trying to find the old news article to get the link to the challenges...
Posted by: CrashE3 | Jan 8, 2009 11:50:34 AM
The lesson of the previous comment is that Labs>Challenges (beta) is not an intuitive place on the dpreview menu for finding the challenges. I think the challenges deserve a top level menu slot and their own discussion forum.
Posted by: morton_goldberg | Jan 8, 2009 3:23:56 PM
I find 500 shots rather too many to wade through. I give shots at the beginning rather more time and consideration than the ones near the end, by which time I've become a bit bored. I wonder if the scores tend to be a bit higher near the beginning, indeed if a higher proportion of shots are scored near the start than near the end. If so, that would tend to put a weighting in favour of earlier shots.
Posted by: HarryLally | Jan 8, 2009 3:38:56 PM
That assumes that it's always that same shots at 'the start.' And I don't think we're expecting many people to vote for every image in a challenge.
Posted by: Richard Butler | Jan 8, 2009 4:21:25 PM
@CrashE3 and @morton_goldberg
This has been a recurring theme in the feedback we've been getting (including yours). It was always our intention to publicisize challenges heaviliy (but of course we didn't want to get egg on our face in the event that the beta flopped). However, things are going well so we've put a 'challenges' link on the main menu
@HarryLally
Furthering on from richard's comment, each user receives a uniquely 'shuffled' view of challenge entries so even if each user 'gives up' after the first page, no entry is at any appreciable disadvantage (as all entries receive approximately equal attention)
Posted by: Jaysen Marais | Jan 8, 2009 5:21:57 PM
Personally I'd love a better system to be put in place for high volume voting. At this point I'm basically going to a page of 16, middle-clicking every one to open them in new tabs, then wading through the tabs. Click (kind of small and hard to hit) star, close tab, click star, close tab, over and over. It's ok but I think it could be done much better. How about a page that displays, say, 9 random photos at once? I'd probably vote on a lot more images if that were the case.
Also, I know you guys are probably trying to save bandwidth, but both the thumbnails and especially the full sized images are way too small.
Great effort though in general, I just want a higher-volume voting solution.
Posted by: Andy | Jan 8, 2009 5:46:58 PM
I take that back about the large size, I guess I just realized you could click them. Though I still would like the non-clicked version a bit bigger. But the volume voting thing is definitely still an issue!
Posted by: Andy | Jan 8, 2009 5:48:50 PM
I visit every day at lunch. Your addition of challenges makes peanut butter gormet! I am hoping for a DSLR in my near future but have been confused by all of the len's and the many options. This feature has put a face on the lens' and has given me valuable feed back (ie what is being used for portraiture vs landscapes). I am disappointed that some fine entries are missing that information. To all posters, this is very helpful.
God Bless, have a great lunch.
Posted by: Gary Leland | Jan 13, 2009 5:35:01 PM
Hello
I saw the thread bemoaning that the 2nd place in "Shadows" had more votes, and more 5'stars than the 1st place picture.
And after some thought, was promted to make the following comments:
- You are recording more than 2 metrics.
1 - could be "popularity", does this picture get any vote of any stars.
2 - could be "excellence" how many stars does this picture get.
I think you should divide all pictures into say 10 groups,
the top 10% most popular, the 10-20% group , the 20-30% group etc.
Then within each "Popularity group"
Compare the pictures for the highest average of stars of "excellence".
The 1st place getter would be the most popular and with the most stars.
You could then also see interesting comparisons like most popular, but with only a few stars.
Or not very popular, but rated a 5 by all the people that rated it...
You also actually (probably) have a third metric,
3 - most viewed (irrespective of voting).
For particular chalenges you could add futher metrics like "Original/colourful/technically chalenging..." etc
So you would end up with a richer way to vote for/assess the pictures.
(where one star is better than none...
and one photo with five votes of five stars will not be seen as better than one with 100 votes of 4 stars)
I'd like to add some other points.
a) You should be able to page surf. (I am interested to go to what I see as random pages)
b) There should be a discussion thread available on any photo, so people can say "Fred how did you do that?"
Or people may want to make some other comments
(another metric - comments - tease out the controversion/interesting photos...)
(I have only read about Bayesim techniques for filtering spam...)
Excellent work.
Posted by: Evan | Jan 14, 2009 11:41:20 AM
I think that half-stars are unnecessary and just add to the voting complication ("hmm... 4.5 or 5 stars?...")
i agree about the multiple metrics - glad its not my problem to sort out!
great stuff!!
Posted by: Simon Neville | Jan 15, 2009 12:29:56 AM
The results notification following voting is brilliant! (even if my result wasn't!) well done
Posted by: Simon Neville | Jan 15, 2009 12:34:05 AM
What I understood from the various exchanges in the threads devoted to this subject is the following :
The winner is the photo which bears, according to your formula :
The highest probability to have received the highest simple average score, had each voter actually voted for every single photo in the challenge.
Is that correct ?
Posted by: jplh | Jan 17, 2009 1:04:23 PM
Could you elaborate on the use of Bayesian stats to take cre of outliers? I saw cases of distributions with outliers (classical analysis) and it seems they didn't get removed.
Posted by: rhlpetrus | Feb 16, 2009 9:24:45 PM