Step One - Product shots

The first step when a lens arrives is to take all of the product shots which are used on the front page and the 'Body and Design' page. This is best done when the lens is still in pristine cosmetic condition, as it can sometimes get marked during the course of testing. While there's only about 10-15 different views used per review, each is shot multiple times. Total number of shots for this review was 45 (obviously of the lens, not with it).

Step Two - Studio Tests

Producing the profile data that gets displayed in the lens review widget is a time-consuming process. Test charts have to be very precisely aligned and focused, and the data shot at least twice to assure reproducibility.

Resolution and CA - 7 marked focal lengths, at least 7 aperture settings at all focal lengths (and 8 at 50mm). Two complete data sets shot = 100 shots precisely
Distortion - This requires the most critical alignment to make the data as useful as possible. 7 marked focal lengths, but an average of 14 shots at each to achieve optimum alignment. As it happens, also 100 shots total.
Falloff - 7 marked focal lengths, third-stop increments from wide open to F11, data obtained from averaging three exposures. This required a total of 171 shots.
Macro - the 18-270mm, with its rather extreme focus shift on stopping down, was a tricky lens for this deceptively simple test. Eventually, to work out precisely what was going on, we resorted to shooting the macro chart using a focusing rail with the lens set to its minimum focus distance. Shots were taken at each full stop in the aperture range, at 1mm distance increments. This procedure generated 77 shots to define the lens's behavior accurately. A further set of 14 shots of a 3-dimensional test subject were also taken to illustrate the focus shift visually. Total = 91 shots.
Image stabilization - 10 shots at each of five shutter speeds with IS on and off, repeated for three focal lengths. 10x5x2x3 = 300 shots

Together, this gives a grand total number of shots used in the studio tests of 762.

Step Three - Real World Shooting

One of the more mystifying criticisms we hear of our reviews is that they aren't from 'real users'. Strangely enough each of the reviewers here is (probably) a 'real' person, and we tend to carry the camera/lens we're working on around with us pretty well all of the time. It's currently late autumn (fall for our transatlantic friends)/early winter here in London, which means the weather is mainly gray and damp, with the occasional bright but cold day thrown in. Good shooting conditions are therefore rare, and we have to make use of them whenever they turn up.

Even so, I managed to fit in several good sessions with the Tamron 18-270mm. I actually prefer to shoot with the lens fairly extensively before processing all of the studio data, to get an initial impression of how it performs which isn't colored by those results. Once I've digested all of the test data I'll then go out looking to illustrate any specific weaknesses identified in the tests (and if I don’t find them, repeat the tests). Generally the studio data is remarkably predictive of how the lens will perform in actual use, but this ‘shoot, test, shoot, test’ approach mitigates well against drawing incorrect conclusions based on just the studio data alone. It’s also important to realize that a full lens review simply can’t be done purely in a studio environment: the technical data is hugely valuable but it says nothing about autofocus speed under various lighting conditions, or flare, or bokeh; for these we need to go out and take 'real' pictures.

In the case of the Tamron 18-270mm, I shot a total of about 500 'real-world' exposures, 200 on the D90 and 300 on the D300. Normally I would also aim to shoot on a lower end body too (i.e. the D60), but didn't get a chance in the timescale of this review.

So in summary, that's the work that goes into delivering you a lens review. In this example it adds up to about 1250 shots with the lens under test, in the course of slightly less than three weeks. I suspect this is rather more intensive use and analysis than most 'real users' ever manage, and it’s the sum total of all of this experience that goes into writing the final review.