Validation of a candidate instrument to assess image quality in digital mammography using ROC analysis

Joana Boita,Ruben E Van Engen,Alistair Mackenzie,Anders Tingberg,Hilde Bosmans,Anetta Bolejko,Sophia Zackrisson,Matthew G Wallis,Debra M Ikeda,Chantal Van Ongeval,Ruud Pijnappel,Mireille Broeders,Ioannis Sechopoulos

doi:10.1016/j.ejrad.2021.109686

Abstract

PurposeTo validate a candidate instrument, to be used by different professionals to assess image quality in digital mammography (DM), against detection performance results. MethodsA receiver operating characteristics (ROC) study was conducted to assess the detection performance in DM images with four different image quality levels due to different quality issues. Fourteen expert breast radiologists from five countries assessed a set of 80 DM cases, containing 60 lesions (40 cancers, 20 benign findings) and 20 normal cases. A visual grading analysis (VGA) study using a previously-described candidate instrument was conducted to evaluate a subset of 25 of the images used in the ROC study. Eight radiologists that had participated in the ROC study, and seven expert breast-imaging physicists, evaluated this subset. The VGA score (VGAS) and the ROC and visual grading characteristics (VGC) areas under the curve (AUCROC and AUCVGC) were compared. ResultsNo large differences in image quality among the four levels were detected by either ROC or VGA studies. However, the ranking of the four levels was consistent: level 1 (partial AUCROC: 0.070, VGAS: 6.77) performed better than levels 2 (0.066, 6.15), 3 (0.061, 5.82), and 4 (0.062, 5.37). Similarity between radiologists’ and physicists’ assessments was found (average VGAS difference of 10 %). ConclusionsThe results from the candidate instrument were found to correlate with those from ROC analysis, when used by either observer group. Therefore, it may be used by different professionals, such as radiologists, radiographers, and physicists, to assess clinically-relevant image quality variations in DM.

Highlights

Image quality plays an important role in the early detection of breast cancer during mammographic screening
The quality of a digital mammogram can be affected by the parameters with which the image was acquired and by the image processing used to change its Abbreviations: AUC, Area Under the Curve; confidence in terval (CI), Confidence Interval; digital mammography (DM), Digital Mammography; multi-reader multi-case (MRMC), Multi-Reader Multi-Case; pAUC, partial Area Under the Curve; receiver operating characteristics (ROC), Receiver Operating Characteristics; visual grading analysis (VGA), Visual Grading Analysis; VGA score (VGAS), Visual Grading Analysis Score; visual grading characteristics (VGC), Visual Grading Characteristics
The outcomes of this ROC study were compared to the results obtained with eight expert breast screening radiologists performing a VGA study with the candidate instrument to assess the quality of a subset of 25 of the images used in the ROC study degraded to the same levels of quality

Summary

Introduction

Image quality plays an important role in the early detection of breast cancer during mammographic screening. Guidelines for evaluation of DM systems have been developed to assess, optimise, and approve these devices before they are introduced into clinical practice and after major changes are made An example of such evaluation procedures of DM systems is type testing, which is included in the Supplement to the European Guidelines for Quality Assurance in Breast Cancer Screening and Diagnosis [11]. In type testing, both physical and clinical assessments are performed, to identify issues in the DM systems that may negatively affect lesion detection and interpretation

Objectives

Methods

Results

Discussion

Conclusion