Comparative Performance of Artificial Intelligence Algorithms for Screening Mammography.

Abstract

HomeRadiology: Imaging CancerVol. 2, No. 6 PreviousNext Research HighlightsFree AccessComparative Performance of Artificial Intelligence Algorithms for Screening MammographyMichio TayaMichio TayaMichio TayaPublished Online:Nov 27 2020https://doi.org/10.1148/rycan.2020209034MoreSectionsPDF ToolsImage ViewerAdd to favoritesCiteTrack CitationsPermissionsReprints ShareShare onFacebookTwitterLinked In Take-Away Points■ Major Focus: To evaluate the performance of three commercially available artificial intelligence algorithms in screening mammography assessment in an external population both independently and in combination with radiologists.■ Key Result: One of the three artificial intelligence algorithms had significantly higher accuracy for breast cancer detection (area under receiver operating characteristic curve of 0.96 vs 0.92 vs 0.92; P < .001). Combining this best algorithm with first-reader radiologists achieved the highest accuracy at 88.6% sensitivity and 93.0% specificity.■ Impact: A commercially available artificial intelligence algorithm can assess screening mammograms with comparable diagnostic accuracy as radiologists. Whether such an algorithm can act as independent readers prospectively in a screening population remains unanswered.Screening mammography is currently the clinical standard for early detection of breast cancer. However, performance of individual radiologists in screening mammography programs varies widely. Efforts to develop artificial intelligence (AI) algorithms for screening mammography have yielded mixed outcomes owing to variability in quality of training data and study methodology. External validation studies comparing the performance of different AI algorithms in a screening population are lacking.This external validation study compared the diagnostic performance of three commercially available AI algorithms both independently and in combination with radiologists in a retrospective population-based screening cohort. Women with implants or prior breast cancer were excluded. The study population included 739 women with pathology-verified breast cancer diagnosed at screening or within 12 months and 8066 randomly sampled healthy controls with negative findings by 2-year cancer-free follow-up. The AI algorithm determined suspicion of cancer on a continuous scale from 0 to 1.0 with a decision threshold for normal and abnormal set at the mean specificity of the first-reader radiologist (96.6%).Algorithm 1 performed significantly better than the other two algorithms with area under receiver operating characteristic curve of 0.96 versus 0.92 versus 0.92 (P < .001). No significant difference in sensitivity existed compared with radiologist consensus at the same specificity (81.9% vs 85.0%, P = .11), despite the algorithm having no inputs from prior imaging or clinical history. When choosing an operating point corresponding to the U.S. Breast Cancer Surveillance Consortium benchmark of 88.9% specificity, algorithm 1’s sensitivity of 88.6% was comparable to the benchmark sensitivity of 86.9%.In combination with a radiologist where abnormal was defined by either the algorithm or radiologist, the cancer detection rate increased 8% from 5.32 per 1000 for algorithm 1 alone to 5.76 per 1000 in combination with the first reader with a corresponding 77% increase in recall rate from 39.1 to 69.1 per 1000.The most accurate algorithm was trained on the largest training set and utilized pixel-level annotations and higher capacity network architecture, despite training on a patient population and vendor system different from the external validation data set. In conclusion, only one of the three algorithms performed at an independent level comparable to radiologists, setting the groundwork for prospective trial evaluation of AI systems in screening mammography.Highlighted ArticleSalim M, Wåhlin E, Dembrower K, et al. External evaluation of 3 commercial artificial intelligence algorithms for independent assessment of screening mammograms. JAMA Oncol 2020;6(10):1581–1588. doi: 10.1001/jamaoncol.2020.3321Highlighted ArticleSalim M, Wåhlin E, Dembrower K, et al. External evaluation of 3 commercial artificial intelligence algorithms for independent assessment of screening mammograms. JAMA Oncol 2020;6(10):1581–1588. doi: 10.1001/jamaoncol.2020.3321 Crossref, Medline, Google ScholarArticle HistoryPublished online: Nov 27 2020 FiguresReferencesRelatedDetailsRecommended Articles MRI for Evaluation of Complications of Breast AugmentationRadioGraphics2022Volume: 42Issue: 4pp. 929-946Does Reader Performance with Digital Breast Tomosynthesis Vary according to Experience with Two-dimensional Mammography?Radiology2017Volume: 283Issue: 2pp. 371-380Invited Commentary: High-Quality MRI after Breast AugmentationRadioGraphics2022Volume: 42Issue: 4pp. E113-E114Artificial Intelligence for Reducing Workload in Breast Cancer Screening with Digital Breast TomosynthesisRadiology2022Volume: 303Issue: 1pp. 69-77Breast Imaging in Transgender Patients: What the Radiologist Should KnowRadioGraphics2019Volume: 40Issue: 1pp. 13-27See More RSNA Education Exhibits What the Breast Radiologist Needs to Know About Imaging Transgender PatientsDigital Posters2018Evaluation of Breast Implants by Traditional Methods, Mammography, and Ultrasonography: When Suspicious of RuptureDigital Posters2019Imaging the Post Mastectomy and Post Implant Breast: Challenges and PitfallsDigital Posters2019 RSNA Case Collection Intracapsular Implant RuptureRSNA Case Collection2021Silicone lymph node on 18F-FDG PET/CTRSNA Case Collection2020Breast implant-associated anaplastic large cell lymphoma RSNA Case Collection2021 Vol. 2, No. 6 Metrics Altmetric Score PDF download