Case Study - Arm - DXOMARK - Electronic Products Quality Evaluation Solutions

Challenges

Benchmarking ISP performance against automotive reference camera and consumer -grade cameras
to evaluate their technology’s competitiveness.
Developing a testing protocol for front-facing automotive cameras, focusing on key use case
challenges and the relative importance of relevant sub-attributes.
Iterative ISP tuning by capturing and analyzing RAW data in both lab and natural environments.

Solutions

Delivered a tailored testing service with a custom protocol combining objective lab evaluations and real-world perceptual tests during driving sessions.

Deliverable

Delivered a detailed report comparing Arm ISP performance with an industry-standard automotive camera, the same module with Arm’s ISP, and a consumer camera
Captured RAW data in both controlled lab settings and real-world scenarios, empowering the customer to fine-tune their ISP iteratively.

Arm is a leading provider of semiconductor technology and design solutions for the
automotive industry. Their offerings include advanced processors, ISP (Image Signal Processing) solutions, and system architectures tailored for applications such as autonomous driving, ADAS, and in-vehicle systems. Arm’s technology focuses on enabling efficient, high-performance, and scalable solutions for automotive manufacturers and Tier 1 suppliers. Widely adopted in the industry, Arm’s solutions form the backbone of many automotive platforms, supporting the growing need for computing power and image processing in modern vehicles.

Excerpt from the debrief session presentation showcasing key issues observed in a typical automotive corner case scenario: a pedestrian in black clothing, invisible to the camera under certain night-time tuning configurations. The camera details have been anonymized for confidentiality.

Camera 1

Camera 2

Camera 3

Testing protocol

The testing protocol was specifically tailored for Arm, demonstrating our ability to design custom solutions that meet unique customer requirements. It combined perceptual real-world testing, including a driving session with a carefully curated list of scenarios to address common use cases and challenging corner cases, alongside laboratory objective testing. Quantified results were delivered using automotive-standard metrics powered by our Analyzer software, ensuring precise and reliable evaluations.

Laboratory Objective Tests

The objective of our laboratory testing was to provide quantified results for each KPI under evaluation. This was achieved using our Analyzer suite, which incorporates most of the metrics prescribed by IEEE P2020, including Flare, Noise, and LED Flicker Mitigation. In addition to IEEE P2020 metrics, we utilized a wide range of standard metrics such as SFR, Tone Mapping, Color Shift, Distortion, and more. Our capabilities also extend to testing dynamic range up to 170 dB and CTA. Furthermore, we offer the flexibility to develop custom testing protocols tailored to specific client needs.

Real-world tests

To complement and illustrate the objectives evaluation, driving sessions were conduct to ensure automotive cameras perform reliably in real-world conditions. Our scenarios include night driving, glare, tunnels, and fast motion. Those scene are use to perceptually evaluate the same KPIs like dynamic range, low-light performance, sharpness, and so on, for optimal safety and driver experience.

Itinerary of the driving session for real-world perceptual analysis and sample of typical cases included in the driving session.