Test setup details


The test chart used in all reviews is the ISO 12233-alike lens test chart by Stephen H Westin. Test patterns on the chart are marked with numbers from 1 to 20, indicating the resolution in lines per picture height (lph) divided by 100. The highest depicted resolution of the patterns is 2000 lph near the center of the image. The chart is printed in ISO 216 A1 format on regular matte 80g/m² paper and hung on a flat metal sheet screwed to a wall in a darkened room. The chart is illuminated by two 35 W halogen light sources which are positioned about 1 m above and 3 m away from the chart. The camera is positioned on a tripod with the sensor being positioned on a straight line from the center of the chart and normal to the chart and wall.

The cameras utilized are a Sony NEX-5N or NEX-5T with full resolution and an aspect ratio of 3:2. The cameras sensors are practically identical 16 MP APS-C CMOS units and should give indistinguishable results. The lenses are mounted with a simple, purely mechanical adapter with correct infinity focus. The cameras are set to ISO 100 with center-weighed auto exposure. The focus on all but the Sony and Sigma lenses is – off course – done manually with assist of focus peaking and lots of patience. Shutter release is automatic with either a two or a ten second delay, depending on the shutter speed.

The three largest apertures (smallest f-stops) on each lens are shot roughly ten times with refocusing between shots. Larger f-stops only get two to four shots after focusing. Starting at f/11 or f/16, diffraction usually limits the focus peaking ability of the camera. Those and higher apertures are therefore captured with the lens focused at f/8 and stopped down afterward.

All shots are converted to lossless PNG using Adobe CameraRaw with untouched standard settings only correcting the white balance, black level and white level.

 

What it is good for

By looking at the charts and comparing different apertures and lenses, you should get a pretty good idea of the main lens characteristics: Distortion, chromatic aberrations, center sharpness, an estimate of corner sharpness and apparent strong vignetting. You will also get an impression of hazing – strong loss of contrast and bleeding of lighter areas into darker ones at wide open aperture. A ‘feature’ often found on older lenses that is mainly attributable to uncorrected longitudinal chromatic aberrations and inadequate coatings.

Further on, I compute the T-stops of all lenses at their widest aperture. This is feasable under the assumption of constant lighting conditions and identical framing and metering. Based upon the DxOMark review of Sony’s SELP1650, I was able to determine the exposure value (EV) of my test setup. With a constant EV and the individual exposure times, the T-stops directly follow. Because of the necessary assumptions, this method has some margin of error which I would estimate to ±0.15 EV based on all results to date.

When comparing T-stop and f-stop of the lenses, keep in mind that they usually differ more, the wider the largest aperture and the higher the sensor resolution of your camera. This is a feature of digital sensors which was documented by DxOMark in 2010. If you read the DxOMark investigation, keep in mind that the pixel pitch of the NEX-5T and 5N is 4.76 µm and the α7R is nearly identical with 4.86 µm.

Last but not least, it would actually be possible to compute MTF-values from the slanted edges of the black squares and trapezoids. However, I don’t have the necessary software for this, yet. You can drop me an e-mail, if you’re willing (and qualified) to implement this in MATLAB. If I did have the software, the values could still not be compared to professional lab tests, as my test setup is very different. Also, the corrected white and black levels result in increased contrast, which probably pushes MTF numbers for lower frequencies. But comparisons between the lenses presented on this site would be feasible.

 

What it is not good for / Limitations

  • The resolution of the printer used to print the chart is finite and the chart is converted from vector- to pixel-format for printing by the printer driver. Therefore, slight steps are visible on angled edges and the central resolution test patterns are slightly uneven above 1600 lph (number 16).
  • Since the test chart is not glued to the sheet and the wall behind is not perfectly flat either, the chart is slightly wavy (+5/-0 mm). As the smallest tested depth of field tested to date is 18 mm (@ f/1.2), this should have no visible effect on the images.
  • Positioning of the tripod and camera is done without any measuring equipment and therefore not repeatable and not perfect.
  • The camera might not be oriented perfectly perpendicular to the chart when shooting and sharpness might therefore be slightly different in each corner.
  • The highest distinguishable resolution is affected – amongst others – by the correct framing of the chart. If the black borders on top and bottom are visible, the actual resolution of the lens is slightly higher than it seems. The largest framing error I have seen in my charts so far is just below 3% of picture height.
  • The light sources are setup from scratch each time I perform a test and lighting tends to be slightly brighter in the center of the chart than in the corners. Therefore, very light local shadowing may be visible and lens vignetting might appear more pronounced than it actually is.
  • The change of the focal point with apertures, also called focus shift, can not be evaluated from the charts since each shot is refocused (with the exceptions being f/11 or f/16 and higher).
  • Lens contrast cannot be judged by the shots, since the black and white levels are corrected in the RAW conversion process.
  • Since all tests to-date were performed on a APS-C cameras, the performance of these lenses on full frame cameras like the Sony α7 series can hardly be evaluated. Especially corner-sharpness and vignetting will differ significantly.