R2 ImageChecker Research Articles

Improved Cancer Detection Using Artificial Intelligence: a Retrospective Evaluation of Missed Cancers on Mammography

To determine whether cmAssist™, an artificial intelligence-based computer-aided detection (AI-CAD) algorithm, can be used to improve radiologists’ sensitivity in breast cancer screening and detection. A blinded retrospective study was performed with a panel of seven radiologists using a cancer-enriched data set from 122 patients that included 90 false-negative mammograms obtained up to 5.8 years prior to diagnosis and 32 BIRADS 1 and 2 patients with a 2-year follow-up of negative diagnosis. The mammograms were performed between February 7, 2008 (earliest) and January 8, 2016 (latest), and were all originally interpreted as negative in conjunction with R2 ImageChecker CAD, version 10.0. In this study, the readers analyzed the 122 studies before and after review of cmAssist™, an AI-CAD software for mammography. The statistical significance of our findings was evaluated using Student’s t test and bootstrap statistical analysis. There was a substantial and significant improvement in radiologist accuracy with use of cmAssist, as demonstrated in the 7.2% increase in the area-under-the-curve (AUC) of the receiver operating characteristic (ROC) curve with two-sided p value < 0.01 for the reader group. All radiologists showed a significant improvement in their cancer detection rate (CDR) with the use of cmAssist (two-sided p value = 0.030, confidence interval = 95%). The readers detected between 25 and 71% (mean 51%) of the early cancers without assistance. With cmAssist, the overall reader CDR was 41 to 76% (mean 62%). The percentage increase in CDR for the reader panel was significant, ranging from 6 to 64% (mean 27%) with the use of cmAssist. There was less than 1% increase in the readers’ false-positive recalls with use of cmAssist. With the use of cmAssist TM, there was a substantial and statistically significant improvement in radiologists’ accuracy and sensitivity for detection of cancers that were originally missed. The percentage increase in CDR for the radiologists in the reader panel ranged from 6 to 64% (mean 27%) with the use of cmAssist, with negligible increase in false-positive recalls.

Impact of computer-aided detection systems on radiologist accuracy with digital mammography.

The purpose of this study was to assess the impact of computer-aided detection (CAD) systems on the performance of radiologists with digital mammograms acquired during the Digital Mammographic Imaging Screening Trial (DMIST). Only those DMIST cases with proven cancer status by biopsy or 1-year follow-up that had available digital images were included in this multireader, multicase ROC study. Two commercially available CAD systems for digital mammography were used: iCAD SecondLook, version 1.4; and R2 ImageChecker Cenova, version 1.0. Fourteen radiologists interpreted, without and with CAD, a set of 300 cases (150 cancer, 150 benign or normal) on the iCAD SecondLook system, and 15 radiologists interpreted a different set of 300 cases (150 cancer, 150 benign or normal) on the R2 ImageChecker Cenova system. The average AUC was 0.71 (95% CI, 0.66-0.76) without and 0.72 (95% CI, 0.67-0.77) with the iCAD system (p = 0.07). Similarly, the average AUC was 0.71 (95% CI, 0.66-0.76) without and 0.72 (95% CI 0.67-0.77) with the R2 system (p = 0.08). Sensitivity and specificity differences without and with CAD for both systems also were not significant. Radiologists in our studies rarely changed their diagnostic decisions after the addition of CAD. The application of CAD had no statistically significant effect on radiologist AUC, sensitivity, or specificity performance with digital mammograms from DMIST.

Assessing the Stand-Alone Sensitivity of Computer-Aided Detection With Cancer Cases From the Digital Mammographic Imaging Screening Trial

The purpose of this study was to assess the sensitivities and false-detection rates of two computer-aided detection (CADe) systems when applied to digital or film-screen mammograms in detecting the known breast cancer cases from the Digital Mammographic Imaging Screening Trial (DMIST) breast cancer screening population. Available film-screen and digital mammograms of 161 breast cancer cases from DMIST were analyzed by two CADe systems, iCAD Second-Look and R2 ImageChecker. Three experienced breast-imaging radiologists reviewed the CADe marks generated for each available cancer case, recording the number and locations of CADe marks and whether each CADe mark location corresponded with the known location of the cancer. For the 161 cancer cases included in this study, the sensitivities of the DMIST reader without CAD were 0.43 (69/161, 95% CI 0.35-0.51) for digital and 0.41 (66/161, 0.33-0.49) for film-screen mammography. The sensitivities of iCAD were 0.74 (119/161, 0.66-0.81) for digital and 0.69 (111/161, 0.61-0.76) for film-screen mammography, both significantly higher than the DMIST study sensitivities (p < 0.0001 for both). The average number of false CADe marks per case of iCAD was 2.57 (SD, 1.92) for digital and 3.06(1.72) for film-screen mammography. The sensitivity of R2 was 0.74 (119/161, 0.66-0.81) for digital, and 0.60 (97/161, 0.52-0.68) for film-screen mammography, both significantly higher than the DMIST study sensitivities (p < 0.0001 for both). The average number of false CADe marks per case of R2 was 2.07 (1.57) for digital and 1.52 (1.45) for film-screen mammography. Our results suggest the use of CADe in interpretation of digital and film-screen mammograms could lead to improvements in cancer detection.

Comparison of Two Commercial CAD Systems for Digital Mammography

I n late 2006, an outpatient imaging center began its conversion to digital mammography (DM). Part of this transition was a plan to purchase software for computer-aided detection (CAD) to assist in analysis of the digital mammography images. The preparation to purchase one of two systems included a comparison of several specifications, including DICOM compatibility and the ability of the systems to fit well into the digital mammography program of the imaging center. A small study was designed to determine whether one of the two different commercially available systems being considered [R2 ImageChecker (version 8.3.17) and iCAD Second Look (version 7.2-H)] was superior to the other for assisting in interpretation of digital mammography images from screening mammograms obtained with the newly installed General Electric Senographe DS unit. Patient images included in this study were given retrospective approval from the Institutional Review Board with waiver of informed consent for publication of the study results.

Evaluation of Computer-aided Detection Systems in the Detection of Small Invasive Breast Carcinoma

To retrospectively compare two CAD systems for detecting invasive breast cancers manifesting as noncalcified masses smaller than 16 mm. Waiver of informed consent was granted by the Institutional Review Board that approved this HIPAA-compliant study. Mammograms obtained from two institutions providing consecutive invasive carcinomas manifesting as noncalcified masses smaller than 16 mm were evaluated by using two commercially available CAD systems (R2 ImageChecker M1000, version 5.0A and iCAD Second Look, version 6.0 mid operating point). To provide statistical power to test for a possible 10% difference in the sensitivity performance between the systems, 192 consecutive mammographic studies (182 unifocal, six multifocal, and four bilateral cancers) were collected. Masses were characterized using the Breast Imaging Reporting and Data System (BI-RADS). Per study specificity and mass false marker rate were determined by using 51 normal four-view studies, while scoring only the mass false-positive marks for noncalcified masses. Associations between mass characteristics and supplying institution were compared by using chi2 tests. A P value of .05 was considered to indicate a significant difference. The respective per study sensitivity, per image (ie, per view) sensitivity, per study specificity, and mass false-positive marker rates were 81.8%, 64.7%, 39.2%, and 1.08 for the R2 ImageChecker M1000 system, and 60.9%, 42.6%, 31.4%, and 1.41 for the iCAD Second Look system. The overall per study and per image sensitivities were significantly better for R2 than for iCAD (McNemar test, all P<.001), with a nonsignificant higher per study specificity and lower mass false marker rate on normal studies. CAD results demonstrated at least a 20% variation between BI-RADS categories 4a and 5 for per study and per image sensitivity. A statistically significant difference was observed in per study and per image sensitivity in our mammography data set with small (<16 mm), noncalcified invasive breast malignancies between two CAD systems. Differences in per study specificity and mass false marker rate were noted but were not statistically significant.

Impact of computer-aided detection prompts on the sensitivity and specificity of screening mammography

To determine the value of computer-aided detection (CAD) for breast cancer screening. Two sets of mammograms with known outcomes were used in two studies. Participants in both studies read the films with and without the benefit of a computer aid. In both studies, the order of reading sessions was randomised separately for each reader. The first set of 180 films, used in study 1, included 20 false-negative interval cancers and 40 screen-detected cancers. The second set of 120 films, used in study 2, was designed to be favourable to CAD: all 44 cancer cases had previously been missed by a film reader and cancers prompted by CAD were preferentially included. The studies were conducted at five UK screening centres between January 2001 and April 2003. Thirty radiologists, five breast clinicians and 15 radiographers participated. All cases in the trial were digitised and analysed using the R2 ImageChecker version 2.2. Participants all received training on the use of CAD. In the intervention condition, participants interpreted cases with a prompt sheet on which regions of potential abnormality were indicated. The sensitivity and specificity of participants were measured in both intervention and control conditions. No significant difference was found for readers' sensitivity or specificity between the prompted and unprompted conditions in study 1 [95% confidence index (CI) for sensitivity with and without CAD is 0.76 to 0.80, for specificity it is 0.81 to 0.86 without CAD and 0.81 to 0.87 with CAD]. No statistically significant difference was found between the sensitivity and specificity of different groups of film reader (95% CI for unprompted sensitivity of radiologists was 0.75 to 0.81, for radiographers it was 0.71 to 0.81, prompted sensitivity was 0.76 to 0.81 for radiologists and 0.69 to 0.79 for radiographers). Thirty-five readers participated in study 2. Sensitivity was improved in the prompted condition (0.81 from 0.78) but the difference was slightly below the threshold for statistical significance (95% CI for the difference -0.003 to 0.064). Specificity also improved (0.87 from 0.86); again, the difference was not significant at 0.05 (95% CI -0.003 to 0.034). A cost-effectiveness analysis showed that computer prompting increases cost. No significant improvement in film readers' sensitivity or specificity or gain in cost-effectiveness was established in either study. This may be due to the system's low specificity, its relatively poor sensitivity for subtle cancers or the fact the prompts cannot serve as aids to decision-making. Readers may have been better able to make use of the prompts after becoming more accustomed to working with them. Prompts may have an impact in routine use that is not detectable in an experimental setting. Although the case for CAD as an element of the NHS Breast Screening Programme is not made here, further research is required. Evaluations of new CAD tools in routine use are underway and their results should be given careful attention.

Quality assurance of routinely collected data in the Computer Aided Detection Evaluation Trial

A study may only be feasible when routinely collected data (RCD) are used to identify participants. RCD from one centre in a retrospective study of computer-aided detection (CAD) in breast screening are considered. Double-read mammograms were re-read by a different reader using CAD (R2 ImageChecker®) and the cancer detection and recall rates were compared. Subjects attended routine screening during 1996 and were aged 50 years or over. Among RCD subjects, 11,947 (92.5%) were eligible. From the study sample (n = 5037), a subsample of 650 (13%) subjects was examined. Previous attendance was incorrect in all four subjects with earlier screening at a different centre. The reader was correctly recorded in all but one subject where it was missing on the paper copy. Whether a subject was recalled for further examination was correct for all subjects; however, single-reader recall, when recall was requested by only one radiologist, was incorrect in 22 (3%) cases; each was among the 53 (8%) recalls showing 42% were incorrect (95% confidence interval, 28–56%). Single-reader recall was unreliable and previous attendance did not have sufficient detail to report the whether a screen was prevalent. Both were checked throughout the study. Issues raised when using RCD are discussed.

Reproducibility of Prompts in Computer-aided Detection (CAD) of Breast Cancer

AIM: We evaluated the reproducibility of prompts using the R2 ImageChecker M2000 computer-aided detection (CAD) system. MATERIALS AND METHODS: Forty selected two-view mammograms of women with breast cancer were digitized and analysed using the ImageChecker on 10 separate occasions. The mammograms were chosen to provide both straightforward and subtle signs of malignancy. Data analysed included mammographic abnormality, pathology, and whether the cancer was prompted or given an emphasized prompt. RESULTS: Correct prompts were generated in 86 out of 100 occasions for screen-detected cancers. Reproducibility was less in the other categories of more subtle cancers: 21% for cancers previously missed by CAD, a group that contained more grade 1 and small (<10 mm) tumours. Prompts for calcifications were more reproducible than those for masses (76% versus 53%) and these cancers were more likely to have an emphasized prompt. CONCLUSIONS: Probably the most important cause of variability of prompts is shifts in film position between sequential digitizations. Consequently subtle lesions that are only just above the threshold for display may not be prompted on repeat scanning. However, users of CAD should be aware that even emphasized prompts are not consistently reproducible.

Does incorrect computer prompting affect human decision making? A case study in mammography

The goal of the data collection and analyses described in this paper was to investigate the effects of incorrect output from a CAD tool on the reliability of the decisions of its human users. Our work follows on a clinical trial that evaluated the impact of introducing a computerised prompting tool (R2 ImageChecker) as part of the breast screening programme in the UK. Our goal was to use data obtained in this trial to feed into probabilistic models (similar to those used in “reliability engineering”) which would allow us to find and assess possible ways of improving the interaction between the automated tool and its human user. A crucial requirement for this modelling approach is estimating the probability that a human will fail in his/her task when the output of the automated tool is incorrect. The data obtained from the clinical trial was not sufficient to inform this aspect of our probabilistic model. Therefore, we conducted a follow-up study to elucidate further the effects of computer failure on human performance. Preliminary analyses of the data resulting from the follow-up study are reported and discussed.

An evaluation of CAD for mammography in the context of a multi-professional screening service

We have evaluated the impact of the R2 Imagechecker, a CAD system for mammography in two studies using both radiologists and appropriately trained radiographers. We first compared the performance of 30 radiologists and 15 film-reading radiographers reading 120 normals and 60 cancer cases. The readers read all cases, with and without CAD prompts. The cancer cases included 40 screening-detected cancers and 20 interval cancers. We looked at sensitivity and specificity and found no significant difference between radiologists and film-reading radiographers with or without CAD. Our second study was designed to maximise the potential of CAD and included an additional 40 subtle cancers, previously missed by radiologists, but which R2 was able to prompt. We have carried out an evaluation of CAD involving a large number of film-readers and two test sets of cancers drawn from the UK screening programme. Our results suggest that the impact of prompts on readers' recall decisions is substantially lower than previous authors had anticipated.

A Pilot Study in the Analysis of Digitized Mammograms

This article explores the development and utility of computer-assisted interpretation of mammograms with use of an academically developed mass enhancement system and the R2 ImageChecker M1000 System, a commercial mass detector system. This exploratory pilot study examines five mammogram views, three with abnormalities and two without, that were extracted from the public database known as the Digital Database for Screening Mammography. Digitized images were analyzed by an academic system (producing 25 different enhancement images) and original films were analyzed by the commercial R2 system (producing a set of five output views with region-of-interest prompts). Quantitative assessment was derived from radiologist-marked database ground truth information, whereas qualitative evaluation relied on a 210-question survey presented to two radiologists active in a breast cancer screening program. Academically developed segmentation-based enhancements appeared useful in aiding interpretation for the mass-prompting system by enhancing medically relevant features in prompt areas. Dual feature–based enhancements were preferred to single feature enhancements. Edge-based features appeared useful for facilitating clinical interpretation of possible spicules and for differentiating abnormal and normal tissues. This provisional study does not allow direct comparisons of the R2 system to other commercial or academic systems. However, this research does help to provide a framework for presenting medically relevant issues that can impact the development of digital mass-prompting systems and suggest alternative visualization techniques.