It's long been obvious that the existing dpr review format - even in its seriously truncated 'concise' form - is unsuitable for the majority of what can be broadly termed 'point and shoot' compact cameras, and I've been wanting to develop a group test format for some time now. There are simply too many compact cameras released each year for us to keep up with (even the most basic camera takes a minimum of a week to put through our battery of tests and to produce a gallery), and the truth is that image quality amongst competing models doesn't - for the most part - vary that much. But there are always going to be exceptions, cameras with image quality that falls well below or rises way above the mean, and unless we test a far higher proportion of the models on the market there's a risk we'll miss these occasional stars and lemons. The group test format may not give you as much detail about the individual cameras, but it does give you a much better idea of the relative quality of a far greater number of competing models.
The recently published group tests were designed specifically to offer a snapshot of the market in the run up to the peak buying season, and to help the huge number of dpreview visitors who arrive looking for help choosing a camera - those that aren't regulars or forum activists. The format was chosen to reflect the way digital cameras are typically used, and to compare how the various models performed in everyday situations. Unusually for dpreview the image quality was assessed first and foremost in terms of typical viewing magnifications, rather than at a pixel level, with the emphasis on accuracy and reliability of the autofocus, metering and white balance systems and on issues that impact on the quality of a typical print (color, contrast, dynamic range and so on). This doesn't reflect a shift in the dpreview ethos or any intention to change the way our full reviews are conducted; it simply recognizes that the typical buyer of a budget digital camera - or one designed to slip into a shirt pocket - is less likely to be concerned with pixel peeping than the typical dpreview regular.
We've grown to expect anything and everything we publish to receive far more brickbats than bouquets on the forums - a thick skin is an essential attribute for anyone writing for dpreview - but our impression is that the idea of a group test has been fairly well received (and they've certainly had a lot of traffic), even if there are disagreements on the cameras chosen and the conclusions drawn. We're almost certainly going to be doing more of them in the future, and to be honest I can't see many simple point and shoot cameras getting full reviews given the number of DLSR and 'prosumer' compacts launched each year.
So now we're working on refining the group test format to offer a little more than those just published without getting to the point where the articles become unwieldy. I've actually delayed publication of the last group test (the super zooms) so we can add a couple more tests, as befits cameras that in many cases are almost as sophisticated as the typical entry-level DSLR.
I'm sure everyone has their own idea of what's important and of what should be included in the group tests, and we have been following the feedback on the current crop closely. For the most basic cameras I think the format used for the articles just published is probably sufficient, but for more advanced cameras we probably need to add at least some of the following to what we included this time around:
- Raw comparison
- Noise analysis
- Basic performance measurements and timings
- Optical aberrations (CA etc)
- Distortion
- Dynamic range
- Macro performance
- Resolution
- Movie mode
This is what we're currently debating, and hopefully you'll see the group test format developing into the genuinely useful comparison tool we envisage. We also see the group test format as being something we could possibly add to our full SLR reviews, allowing us to, for example, gather together all the entry-level digital SLRs we've already reviewed to produce a more concise comparison article once or twice a year.
Obviously high-end compacts such as the Canon G10 and Panasonic LX3 will still warrant full reviews, as will many of the superzoom cameras, and the more advanced premium models, but for the scores of other compacts launched each year the group test format will allow us to broaden our coverage to a significantly bigger proportion of the market without compromising our 'get under the skin' approach to reviewing.
I'm part of the reason you get so many more brickbats than bouquets in the forums; I suspect I'm part of a silent majority who reads and likes pretty much everything on the site but rarely feels the need to comment. So, consider this a general high compliment to your entire publication.
I really agree with your reasoning for creating the group test format, and would like to see it expanded in the way you outlined. For simpler compacts, being part of a group test is all the information that any prospective buyer will need. And for high end compacts or low end DSLRs, being part of a group test in addition to having its own full review will really help prospective buyers evaluate their choices in relation to each other.
And, as always, don't listen to the flak you catch for your conclusions. The important work you do is creating the review data and writing about it. Then, anyone who wants to draw a different conclusion can do so themselves, since you've kindly made available such a comprehensive set of test data. Remember, they're only able to disagree because you've done the hard work of producing and making available the information.
Posted by: Alex | 28 Dec 2008 01:23:31
I think the group format is a great idea. SLR users are sold to a system for many years and for most of us an in-depth revew of a specific camera is far more important than a comparison between different ones. For point and shoot the opposite is true and I've been eagerly waiting for the group reviews - again, as probably the most of the people here. I find the format very good. I would like to see how (and if) the camera can balance ambient and flash exposure compensation - something I use extensively for indoor shooting and many cameras don't allow to do, if I understand it right. In your prosumer P&S review I was bitterly disappointed you did not include one of the FujiFilm F's. From what I could see in your own pictures, image quality of Fuji F series competes directly with Panasonic LX3 and I would like to hear your opinion to decide which one to purchase. Otherwise, thank you for your hard work, I visit the site almost daily looking for the most competent and comprehensive reviews available for the general public now. Happy New Year!
Posted by: Maxim | 28 Dec 2008 03:37:35
I think that way would be great.
For Maxim: DSRL entry level are for people who usually don't have anything and are trying to figure out what to buy as their first DSRL camera. Yes... maybe there are some who are willing just for a good review to upgrade their old entry level DSRL, but still they should take into account what other systems are offering, as they are probably people who doesn't have thousands in a camera system, as serious amateurs and pros do.
Posted by: alanschamber | 28 Dec 2008 04:31:06
I really like the new format - thanks!!
I'd love to see coverage of movie modes,
Posted by: Avi Flax | 28 Dec 2008 06:43:22
I agree with Alex comments.
Like the new formet. Very usefull.
Additional info?
Movie mode:
Some info over movie modes would be great:general quality, IS and zoom still working in video mode?, minimal in-camera editing or not ...
Timing: AF delay and S1-S2 delay. A simple "best-good-average-poor" mention would be sufficient (note: S1-S2 delay may increse noticealy using the flash even without red eye reduction due to pre-flash meteting time. If it is the case, it desserve a mention.
Aberration: your general comment on quality and specific image issue seems enough for me for point and shoot cam.
Posted by: Francesco | 28 Dec 2008 08:17:16
The group tests have been great, very useful and exactly as informative as they should be. Certainly looking forward to reading more of them in the future. :-)
Posted by: TheHacker | 28 Dec 2008 11:00:52
I think it's a great idea, these group tests that you're doing right now. And I think that for a lot of the new users & for people looking to upgrade to a dSLR system the notion of group tests as informed & thorough as yours would be invaluable. Not all of us have decades worth of legacy glass worth thousands of dollars. And I feel as if that is an aspect that gets overlooked all too often.
And as a side note, about your brickbats vs bouquets issue, I feel that Alex has got it spot on in that people who don't already have the camera in question & are looking for validation by you, who don't feel like their fav camera got undervalued will not scream about how unfair you are.
If someone likes what he sees in your reviews, they're likely to bookmark your page & refer it to a friend or look over it again (if that even)... If someone dislikes it however, they're more than likely to go to the forums & rant about how you've all sold your souls to your corporate masters...
Posted by: Gaurav S. | 29 Dec 2008 13:36:44
As a dslr owner i would actually like to see a concise group test of mid-range and pro/enthusiast dslr cameras. yes they will still warrant full reviews, but even in these categories it can be very difficult weighing up which one to buy.
At the moment that means visiting each forum, just to get a totally biased opinion of that manufacturers offering...
Posted by: p tomlin | 29 Dec 2008 16:21:00
I also like the group test format for the P&S cameras. A few different friends asked for recommendations on budget cameras for Christmas gifts this year, and I had to basically make guesses based on specs and experience with past models. As it turns out my recommendation (Canon A590IS) performed reasonably well in your tests; and it is nice to see that Panasonic seems to finally have gotten image quality up to match their build and feature sets.
For future tests though, it would seem that if you add all the options you suggested, you are almost all the way back to the concise tests. Movie mode considerations are the only thing I see on that list that would probably be relevant to most P&S cameras. Some of the others (resolution, noise) are adequately addressed through the test images, and others (timings, distortion/CA, dynamic range) were typically addressed in the text of the group test. Perhaps to save time after the current group testing is done, pick the top few for additional testing. For example, check the quality of the P6000 in RAW to see if its' problems are all processing related, but if RAW is disappointing as well there isn't much point to going further, as it does not appear to stack up to the LX3 or G10.
Posted by: Steve | 29 Dec 2008 16:55:49
There are only few DSLR on each level and buying one is a major decision, not only because of the cost of the initial investment, but also because it is a long-term commitment to the system. For these, full review is the way to go, comparison is done to a great extent in the "compared to" pages and even if not, we can do some work ourselves. Open several windows, compare the data yourself side-by-side (DPR, thank you for using the same set-up over the years!). Reviews are expensive undertaking after all.
For P&S, on the other hand, group review is a must - bless you heart for doing them! The cameras come by dozens, specs are confusing, you don't have tome to review them all, we don't have time and enough incentive to read and compare them. After all, buying one is just this - getting a relatively inexpensive tool for two-three years without any further obligation.
Yes, please present the RAW data and please comment about flash and ambient light exposure complensation - I wonder, how do people shoot indoor without using it when they don't have time to experiment with manual setting?
Otherwise, please thrive and prosper the next year and after. You are the best!
Posted by: Maxim | 29 Dec 2008 19:38:52
Group test format sounds like a great idea. However Simon, I cant help thinking that some of these changes are influenced by Amazon to encourage funnelled marketing and purchasing of their goods?
Posted by: Zoe Hellar | 30 Dec 2008 03:35:22
Zoe: Amazon has no influence over the editorial content of this site, nor has it shown any interest in having any. Did it not occur to you that maybe, just maybe, we could have ideas of our own? Or that maybe, just maybe, we're trying to do something that's right for our visitors?
Posted by: Simon Joinson | 30 Dec 2008 21:39:39
As an almost daily viewer I find your site unquestionably the best. Others are good..... I feel yours is better. You provide knowledge and incite to those of us looking for it in an honest and straightforward manner. And it's free. What more can people want. Well, strike that....people always want more and will critize no matter what. I'm happy that your starting the new P&S tests also. After all, one can only buy so many Dslr's and I have my share.....or do I? Ron
Posted by: Ronald Swadley | 31 Dec 2008 03:45:54
Group tests are a great tool, as they provide buyers of compacts with COMPARATIVE information, which is exactly the kind of information they need when making a purchase. Explicit comparison allows inexperienced buyers to answer the question, "which are the best cameras in my price range?" without the need to develop a mental catalog of camera reviews against which to measure a particular model.
Now, what would be really cool is a widget that would allow site users to create their own comparisons (even if they lacked editorial content) of, for example, dynamic range, or the standard studio noise shot.
Posted by: Tim | 31 Dec 2008 09:56:26
Simon, Thank you for responding to my comment and clarifying your position with regards to editorial content. I am pleased to read that the artistic and content flair of this site is not compromised. please do not take any offence regarding my previous comment. It was not intended to offend, merely to state a presumption based on activities from previous acquistiions.
Posted by: Zoe Hellar | 31 Dec 2008 10:19:52
I have been looking forward to your reviews of super-zooms. I finally purchased one on the basis of user comments and other reviews. On reading of the plans you now have for expanding the content of reviews for superzooms, I am happy to wait some more. None of the extant reviews touch on all the matters you will now include.
Posted by: Alan E. Mayers | 31 Dec 2008 18:20:11
Simon, I completely agree with the principles outlined in your opening post. Those in the market for a basic camera will be overwhelmed by/disinterested in a great deal of information. Also, as you stated, past a point, there isn't a huge amount of performance difference. Two subjective attributes that I believe would be helpful in the group test format would be ratings for ergonomic design and intuitive operation.
Happy New Year and happy tenth year!
Posted by: Steve Bryant | 31 Dec 2008 20:06:02
For more log onto www.uploaded.tv it's a coolone
Posted by: shantanu sarkar | 31 Dec 2008 23:35:58
Regardless of the methodology you choose for the test, I would like to suggest a way to choose the compact cameras to review: review only cameras that are worth a review because they are "interesting". If a camera does not offer anything special, simply do non review it. Do not waste your time. Choose cameras that can be interesting to dpreview readers. Try to understand what readers are looking for from dpreview: they see (IMO) a compact camera that looks promising, because it has an interesting zoom range, or interesting features, cameras that in any way promise something different from the standard compacts designed for the mass market, and they ask themselves: ok, this camera looks nice, it has a wide angle/manual modes/whatever, but what about image quality, performance, user interface, hidden problems that can be discovered only by accurate testing? We don't need reviews of standard cameras with 3x zooms and no particular functions. We already know how they perform, and your budget cameras test just confirms this. They are more or less all the same, what's more they are probably not so interesting for dpreview readers. So the idea is: you cannot review everything, so you choose and review only compact cameras that are interesting for the "advanced amateur" that is looking for features and image quality instead of megapixels and 150x zoom range. Your job is not only to review cameras, but also to choose, with your experience as editor, the cameras that are worth a review. That's my opinion.
Posted by: Tommy | 2 Jan 2009 11:40:16
I agree for almost everything with what Simon states. Generally I have read the posted 'group reviews' with much interest and growing enthousiasm, as the reviewers were able to show the differences in performance of exactly the camera's the normal buyer would like to take into consideration. One remark however. Especially in the last group test a number of camera's were compared who had already had a 'full' review recently. IMO it would be preferable to avoid this as much as possible and include only camera's not reviewed earlier.
Happy New Year to all and the very best for DPReview on its 10th year.
Posted by: Marius | 2 Jan 2009 16:59:50
I suggest you put the conclusions first. I skip the detail and go to the conclusions. If I want more detail then I go back and look at it.
Posted by: Byron Prinzmetal | 4 Jan 2009 00:27:27
I personally really enjoy the group reviews, as it gives me a better chance to see your(valued) comparisons between similar cameras. Keep up the great work!
Posted by: Supr X | 4 Jan 2009 04:42:52
I love the new group tests, as it shortens the time for amateurs to consider things and also help them from getting lost in sea of megapixels, wide angles, face detections, bla bla bla....
Perhaps in the future, there will be group test of SLR lenses :D ?
Posted by: Zenstrive | 4 Jan 2009 04:51:23
Maybe the group test is a good starting point. From this test the best/more interesting cameras can be selected for a more in-depth test.
Posted by: MarcV | 5 Jan 2009 11:45:17
The group tests, especially for compact "phd" style cameras make sense. I would be curious about how a given years crop, i.e. fall 2008 might compare to models from one or two years earlier. It seems that a sweet spot for small sensor cameras was between 6 and 8 megapixels and the newest cameras are not necessarily better. Also, can the RAW performance of an Olympus SP-320/310 be compared the RAW performance of a Canon G10 or Panasonic LX-3? So, a different sort of comparison might be interesting...
Posted by: mravitz | 6 Jan 2009 14:01:03