Smartphones have long become ubiquitous, but mobile technology is still advancing faster than many other sectors of consumer technology. That is especially true for the camera modules, where manufacturers surprise us with new features, designs, and spec improvements with every new-generation device.
At DXOMARK we are keeping a close eye on these advancements so we can adjust and modify our testing protocols and methodologies accordingly and make sure that DXOMARK remains the most relevant smartphone camera testing out there. We launched smartphone camera tests in 2012 and released the first update to our protocol in 2017, adding dedicated tests for simulated bokeh modes and zoom quality, as well as for low-light testing down to 1 lux and motion-based test scenes. In September 2019, we introduced the Wide and Night scores to take into account the increased availability of ultra-wide cameras on smartphones as well as improved low-light and night shooting capabilities.
One year later and it’s time for another update to the DXOMARK smartphone main camera protocol. In version 4 we are adding tests for image quality in the camera preview and what we call trustability. Trustability is designed to make our Camera test protocol more exhaustive and challenging. It brings more use cases and an updated scoring system for increased user relevance. We have also improved the testing methodology for autofocus, with new objective lab tests in low light, for handheld shooting, and under HDR conditions.
Trustability for Photo
Even the most modern and high-end smartphone cameras can deliver excellent results in one test scene, only to fail in a similar but different scene. In other words, just because a camera does well in some situations does not mean you can trust it to do well all the time. At the same time, as smartphone camera quality has improved over the years, users are now using their phones in any kind of photographic situation, even under difficult conditions that would previously have required a DSLR or mirrorless camera, such as shooting low-light scenes or fast-moving subjects. It’s therefore absolutely critical for smartphone cameras to provide good image quality on a consistent basis across all scenes.
This is why we have introduced trustability. Trustability measures the camera’s ability to deliver consistent still image and video quality across all shooting scenarios, not just a finite number of selected test scenes. It makes the DXOMARK Camera protocol both more exhaustive and more challenging.
More still image sample scenes
To implement trustability, we had to expand our current set of sample scenes and images. We have introduced a wider range of new photo use cases that cover portrait, multi-plane, close-up, and moving scenes across all light conditions. There are also entirely new sample sets for portrait and low-light shooting. Below you can see a selection of our new test scenes.
People scenes like the one below are designed to let us check if a camera is capable of delivering accurate face exposure, natural rendering of various skin tones, and good rendering of fine detail on facial features. All these attributes are important in any portrait shot, including family snapshots, pictures of children, and typical vacation photos.
In this comparison we can see that portrait rendering can vary a lot among cameras. This is especially true for the tradeoff between texture preservation and noise reduction, but also in terms of exposure. For example, the iPhone captures much brighter skin tones here than the Xiaomi. However, the latter has strong highlight clipping in the sky in the background.
Randomized shooting plan
In addition, we have now added a random element to what we call the shooting plan (which is all the sample images we shoot for a review). These random scenes reward those devices that can produce consistently good results in any condition, and they help diversify testing even more.
In addition to all our precisely-defined sample scenes that we capture for every test, we now also shoot a series of samples that are defined only in terms of composition, scene content, and lighting conditions, but not in terms of location, subject, or framing. For example, such a scene could be a backlit portrait or a low-light cityscape. Our testers then capture an image at any location and with any subject that fulfills the defined requirements.
Let’s have a look at a couple of examples:
The “people on sofa” random scene has to meet the following conditions: two motionless subjects, low-contrast indoor lighting conditions, and a wide range of colors. Within those parameters, the DXOMARK testers are free to capture whatever scene they want. All three images below comply the requirements and could therefore be used in the test to evaluate color, exposure, and artifacts, among other attributes.
More low-light and HDR scenes, more challenging motion for Video
Trustability has of course also brought improvements to our video testing, offering a much wider range of tests overall. While all of our objective lab testing has largely remained unchanged, we have added more use cases, low-light and HDR scenes, and portrait and closeup videos, as well as more challenging motion (of both photographer and subjects) to our perceptual video testing. This makes our video test protocol both more complete and more challenging. Like for stills, smartphone cameras also have to deliver consistently good image quality across a wide range of conditions and scenes in video mode in order to achieve high scores.
To integrate trustability into our video testing, we have also modified the video scoring system for all attributes (exposure, color, texture, noise, artifacts, autofocus, and stabilization) as well as the computation of the overall score.
For example, this backlit portrait scene tests the HDR capabilities of a camera as well as its exposure strategy. As you can see, results differ vastly between devices. The Sony Xperia 1 prioritizes detail in the bright background, strongly underexposing the darker foreground. As a result, it renders the subject of the portrait pretty much invisible. The Samsung Galaxy Note 10+ 5G goes the other way and exposes for the subject. This is a better strategy than the Sony’s, but results in a strongly overexposed background. The Google Pixel 4 strikes the best balance in this instance, but still isn’t quite as good as we would like, with strong underexposure on the subject.
Up to now our smartphone camera reviews have focused on image recording. However, the performance of a camera app’s preview mode gives you a very first impression of image quality and can have a big impact on image results as well. For example, it’s impossible to get the framing of a shot right if preview framing does not align with the framing of the captured image. Similarly, a preview image that is too bright or dark might make you necessarily apply exposure compensation. In the worst-case scenario, a photographer might simply choose not capture an image at all if the preview looks like it’s not worth pressing the shutter button, even though the recorded image would be absolutely fine.
Among the current generation of devices, preview performance varies quite significantly. In the sample scene below, the Samsung Galaxy S10+ shows the biggest difference between the preview and the final image. Looking at the preview image, you might not even bother pressing the shutter button because of the extensive highlight clipping in the background. However, the camera applies effective HDR processing to the final actual image, which in total contrast to the preview, shows decent color and detail in the background.
Much of the background in the Huawei P30 Pro preview image is clipped as well, but the P30 Pro does not apply any HDR processing to the output image. While the output isn’t great, at least the preview is fairly accurate. The iPhone 11 Pro arguably does the best job in this comparison. It uses HDR processing on the actual output image and gives us a fairly accurate preview of the end result on its display.
All preview analysis is undertaken perceptually while looking at the preview and output images side-by-side. For HDR testing, we use a specific setup that includes a mannequin and an Analyzer HDR slide. Our testers look at target exposure on the mannequin’s face and entropy on the slide for EV+4 and EV+7. We undertake HDR testing for preview at three light levels, 1000 lux D65, 100 lux TL84, and 20Lux A.
Trustability and Preview are the core elements of version 4 of the DXOMARK Camera test protocol, but there is a range of additional updates, mainly to our objective testing methodologies in a controlled lab environment, but also to some of our perceptual testing methods in real-life scenes.
New objective autofocus measurements
Previously, the lowest light level we used for objective autofocus testing in the lab was 20 lux. We now also test autofocus at the extremely low light level of 5 lux. There are also anew objective autofocus tests under high dynamic range conditions and with simulated handheld motion (with the device mounted on a motion platform) to replicate even more use cases and shooting situations in the lab.
New HDR measurements
Previously our objective testing in the lab was undertaken under homogeneous light conditions, with high dynamic range scenes covered by our real-life shooting and perceptual evaluation. Now we have expanded our objective tests to include HDR recording as well.
Expanded zoom testing
Finally, with new devices offering ever-expanding zoom ranges and quality, we have modified our zoom tests accordingly. We previously undertook zoom testing to cover zoom factors of approximately 2x to 8x. We have now expanded zoom tests to deliver results across a wider zoom range from approximately 1.5x to 10x and beyond.
We have also improved our resolution measurements for zoom; further, our testing setup is now fully automated as well as more difficult for the autofocus. Our experts apply a strong defocus between shots and then the shutter is automatically triggered one second later, which is a real challenge for any smartphone camera when zooming. In addition, we updated our shooting plan with more portrait and HDR scenes as well as other complex uses cases for zoom testing, and we introduced a new AI measurement on our perceptual chart for texture evaluation.