Unless you have specialized equipment that can measure the performance of just the lens, the only way to test a lens is to take pictures with it, and analyze the pictures.

That is a *huge* problem.

The reason is that you are now measuring not just the lens, but the camera, the circumstances of the picture, the photographer's skill and judgment, and their PP skills.

Allow me to list some *critical* factors in assessing the lens. Was the picture accurately focused? How much of the image is within the DOF? Was a tripod and MLU used? If not, what support was used and how fast was the shutter? What is the sensor size and MP of the camera that was used? Was it shot in RAW or jpg? If it was shot in RAW, which converter was used, and on what settings? If jpg, what were the in-camera settings? Was any PP aside from the basic RAW conversion applied? If so, precisely what was done? When saved as a jpg for the web, what quality was used? All of the these factors, *all* of them, *critically* affect the image.

That all said, whatever results a person obtains from a lens test shows the *lower limit* of the quality of the lens. That is, it is quite possible that the lens is better than the test result indicates, but the flaws in any particular test do not allow that potential to show.

However, we then have the issue of copy variation. That is, any particular person's review of a lens, at best, only describes the capability of particular lens that was used in the test, not all copies of that lens. I honestly do not know how much variation exists within lenses, nor do I know the nature of the variation.

That is, perhaps it's possible that my 50 / 1.4 has better center performance than someone else's, but theirs has better edge performance. Maybe mine has more internal flare, but theirs has more PF. Or maybe mine is sharper at close distances, theirs sharper at infinity, etc., etc. I simply do not know enough about lenses to say if these situations are possible, and, if so, how different, or how likely, such situations are.

It's even worse for zooms. When I owned a Tamron 28-75 / 2.8, it was soft at 28mm, f / 2.8, focused at infinity, but sharp elsewhere. When I called for a service ticket, the tech was surprised, saying that most people complained about the long end, not the short.

So, what to do? I mean, you simply cannot expect all these variables to be equal from different people. But you most certainly can expect them to be equal from the same person.

That is, if you are comparing two lenses that you currently own, then it is a slam dunk that you should use the same camera with the same settings (i.e., f-ratio, and exposure, and ISO) and process the pics identically. One should also take at least three shots of each scene and choose the best pic, to help eliminate focus errors. Of course, it's also best to try to take pics of the same scene in the same circumstances, or as close as possible.  Additionally, higher ISOs rob an image of detail, so care should be taken to use the lowest possible ISO.  Sometimes, it is not relevant, however, when comparing, say, lens performance at different apertures, so long as all images have the same ISO settings.  For example, if you were comparing the 50 / 1.4 to *itself* at f / 1.4 and f / 2.8, then, as long as the ISO were the same for both settings, all is fair.

However, it's rare that someone does a shoot off between two lenses. Usually, people merely post pics of the one lens they have and put them out there so people can view and make their own decisions. Obviously, the more who do this, the more data you have to make an educated decision.

Many decry the posting of sharpened pics. I have the opposite opinion. That is, we want to present the lens in the best possible light; thus, you should sharpen the pic to make it look its best. As already discussed, all the other variables, camera model, variation between copies, different scenes, etc., already make an equivalent comparison impossible. Some say that the less processing you do, the more even the playing field. I completely disagree. I feel that processing the images levels the playing field. In any event, simply the brand of RAW converter you use already *completely* makes direct comparisons null and void, unless the viewer processes their images with the same converter and same settings.  On the other hand, if you use in-camera jpgs, then, again, unless the viewer also uses in-camera jpgs with the same settings, then their results will not be the same.

It is important to note, however, that if you choose to process the image, you should process the *entire* image with the *same* parameters.  For example, I hope it's obvious that if you were to selectively sharpen different areas of the images by different amounts, that completely skews the perception of the lens performance across the frame.  In the example I give below, the image was corrected for distortion, but that is an automatic function of the RAW converter.  However, for a proper lens test, I should really turn that feature off for that purpose.

So, to that end, here's what I propose: when posting images as a function of a lens test, edit the image as you see fit, but give all relevant information: camera model, exif, software used, and processing done. For example, the following image:

http://www.pbase.com/joemama/image/69193869/original

was shot with a Canon 5D + 100 / 2 @ f / 2, 1/1600, ISO 100, unsupported handheld. It was processed using BreezeBrowser Pro, combined, aRGB, +0 saturation (middle setting), -4 contrast (minimum setting), +3 sharpness (on a scale of 0-7), standard picture style, HQ style sharpening, PT lens distortion correction (incorporated into the RAW converter), saved as a 16 bit tiff, then resaved as a jpg using IrfanView with a 90% quality. Ideally, of course, one would post the original RAW (the best option, but the most time and memory consuming), or original in-camera jpg, as well.

If one is posting 100% crops instead of fullsize images, then, in addition to the above information, it is good to say where the crop came from (center, lower left, etc.). However, posting resized images for a lens test is not nearly as useful. But you can get an idea of sharpness and contrast, and a good idea of color and flare control, even from a resized image. It is mainly in the sharpness and contrast that resized images lose a lot of information. Since those are the two primary points of interest (usually), it's best to avoid using these images for lens tests unless they are accompanied with 100% crops and/or full sized images.

Lastly, I would like to note that there is a difference between a "good" test and a "complete" test.  For example, if you test a lens only within a certain aperture range, then you are only measuring the quality of the lens in that range.  For example, testing a 100 / 2 at f / 5.6 does not tell us about its performance wide open.  Also important to consider is the subject-camera distance.  A lens may perform substantially differently at its minimum AF distance as opposed to infinity.

One should take care when extrapolating conclusions to situations not covered in a particular test.  That is, if a lens is tested at f / 2.8, it's probably safe to conclude that  the lens performance of the higher apertures (to the diffraction limit)  are going to be at least as good, but it's quite another thing to extrapolate at wider apertures.  Similarly, testing a lens with the subject at the minimum AF distance may give substantially different results than when the lens is focused at infinity.

In other words, a lens review can never be "complete", but, if care is taken, it can be "useful" within the range of the images taken.

So, to recap:

1) There's a big difference between comparing two lenses and posting test images of your particular lens.

2) Taking multiple pics and choosing the best of each scene helps weed out camera/operator error. In addition, take care to pay close attention to the camera's settings -- a low shutter speed off tripod tests your hands more than the lens, and the 1 / FL rule does not apply for a lens test -- 1 / 4FL is a much safer bet.  Higher ISOs will also rob detail from an image.

3) Post fullsize images and/or original files. If you're going to resize the image, then at least accompany it with 100% crops from various areas of interest.

4) Do not hesitate to post-process, but apply the *same* processing to the *entire* image (e.g., no selective sharpening).  All images are processed. It's just a matter of whether the camera or computer software is doing it. There is no reason to assume that the camera's defaults, or the software editor's defaults, are any more valid than other options. However, it is useful to note what software was used with which settings, and/or any additional processing.

5) A lens review only measures the one copy of the camera-lens combination being used in the test, and only for the circumstances of the images in that test.  We can extrapolate its performance outside the images posted in the test, but should take care in the validity of such conclusions.