Multireader Multicase Research Articles

Deep learning improves detection of aortic stenosis in b-mode ultrasound: results of a pivotal multicenter validation study using artificial intelligence

Abstract Background Aortic stenosis (AS) is a highly prevalent and morbid condition, with moderate or severe AS present in 9% of patients over 75 years. Despite improved diagnostic and therapeutic methods, AS remains underdiagnosed, particularly in Black patients and underserved communities. Echocardiography (echo) is effective for detecting AS but requires expertise to quantify and interpret valve gradients and valve area. We hypothesized that a deep learning (DL) algorithm could be trained to detect AS severity using B-mode (non-Doppler) echo alone, the modality most likely to be available in point-of-care ultrasound (POCUS) studies. Here, we present the results of a multicenter pivotal trial to test whether artificial intelligence (AI) can help increase the detection of AS from B-mode echoes. Methods A convolutional neural network (CNN) was trained to classify the severity of aortic stenosis from 3 B-mode echo views of the aortic valve (parasternal long- and short-axis and apical 5-chamber). The training dataset had over 500,000 cine loops from 29,527 echoes in 26,981 patients. A fully crossed multireader multicase study was conducted using 220 echoes never seen during algorithm development. 5 level II-trained cardiologists (referred to as POCUS readers as they only reviewed B-mode clips) were presented with these echoes twice at least a month apart, once with and again without AI interpretation of AS severity. Readers identified 2 dichotomous severity levels (either "suggestive of moderate to severe AS" or not), with ground truth established by 3 level III-trained cardiology echo experts. While experts had access to full studies (including Doppler) to adjudicate severity, POCUS readers had no Doppler or measurements of any kind available. Sensitivity and specificity were compared for the 5 POCUS readers when aided by the AI algorithm vs unaided, with similar tests of inter-reader agreement. Results There was a statistically significant improvement in sensitivity (+5.5%, 95% CI [1.5%, 9.5%], p=0.007) with stable specificity (-0.3%, 95% CI [-3.6%, 3.0%], p=NS) when the 5 POCUS readers received the assistance of the AI software. Fig 1 shows 2,500 bootstrapping assessments to better show consistency of these findings. In addition, the POCUS inter-reader agreement (overall percent agreement) improved by approximately 7% (p<0.001), becoming comparable to that of expert readers (Fig 2). Conclusions In a comprehensive multireader pivotal trial, an AI algorithm improved the performance of POCUS cardiologist readers to identify significant AS from limited B-mode echoes as might be obtained in screening or outreach settings. Sensitivity for detection of moderate-severe AS increased significantly by 5.5% with stable specificity. Inter-reader agreement improved to resemble that of expert cardiologists given access to full study information, including Doppler. AI may have a role in assisting detection of significant AS from limited B-mode echoes.Fig 1Fig 2

Evaluating Biparametric Versus Multiparametric Magnetic Resonance Imaging for Diagnosing Clinically Significant Prostate Cancer: An International, Paired, Noninferiority, Confirmatory Observer Study

Background and objectiveBiparametric magnetic resonance imaging (bpMRI), excluding dynamic contrast-enhanced (DCE) magnetic resonance imaging (MRI), is a potential replacement for multiparametric MRI (mpMRI) in diagnosing clinically significant prostate cancer (csPCa). An extensive international multireader multicase observer study was conducted to assess the noninferiority of bpMRI to mpMRI in csPCa diagnosis. MethodsAn observer study was conducted with 400 mpMRI examinations from four European centers, excluding examinations with prior prostate treatment or csPCa (Gleason grade [GG] ≥2) findings. Readers assessed bpMRI and mpMRI sequentially, assigning lesion-specific Prostate Imaging Reporting and Data System (PI-RADS) scores (3–5) and a patient-level suspicion score (0–100). The noninferiority of patient-level bpMRI versus mpMRI csPCa diagnosis was evaluated using the area under the receiver operating curve (AUROC) alongside the sensitivity and specificity at PI-RADS ≥3 with a 5% margin. The secondary outcomes included insignificant prostate cancer (GG1) diagnosis, diagnostic evaluations at alternative risk thresholds, decision curve analyses (DCAs), and subgroup analyses considering reader expertise. Histopathology and ≥3 yr of follow-up were used for the reference standard. Key findings and limitationsSixty-two readers (45 centers and 20 countries) participated. The prevalence of csPCa was 33% (133/400); bpMRI and mpMRI showed similar AUROC values of 0.853 (95% confidence interval [CI], 0.819–0.887) and 0.859 (95% CI, 0.826–0.893), respectively, with a noninferior difference of –0.6% (95% CI, –1.2% to 0.1%, p < 0.001). At PI-RADS ≥3, bpMRI and mpMRI had sensitivities of 88.6% (95% CI, 84.8–92.3%) and 89.4% (95% CI, 85.8–93.1%), respectively, with a noninferior difference of –0.9% (95% CI, –1.7% to 0.0%, p < 0.001), and specificities of 58.6% (95% CI, 52.3–63.1%) and 57.7% (95% CI, 52.3–63.1%), respectively, with a noninferior difference of 0.9% (95% CI, 0.0–1.8%, p < 0.001). At alternative risk thresholds, mpMRI increased sensitivity at the expense of reduced specificity. DCA demonstrated the highest net benefit for an mpMRI pathway in cancer-averse scenarios, whereas a bpMRI pathway showed greater benefit for biopsy-averse scenarios. A subgroup analysis indicated limited additional benefit of DCE MRI for nonexperts. Limitations included that biopsies were conducted based on mpMRI imaging, and reading was performed in a sequential order. Conclusions and clinical implicationsIt has been found that bpMRI is noninferior to mpMRI in csPCa diagnosis at AUROC, along with the sensitivity and specificity at PI-RADS ≥3, showing its value in individuals without prior csPCa findings and prostate treatment. Additional randomized prospective studies are required to investigate the generalizability of outcomes.

Enhancing Radiologist Reading Performance by Ordering Screening Mammograms Based on Characteristics That Promote Visual Adaptation.

Background Mammographic background characteristics may stimulate human visual adaptation, allowing radiologists to detect abnormalities more effectively. However, it is unclear whether density, or another image characteristic, drives visual adaptation. Purpose To investigate whether screening performance improves when screening mammography examinations are ordered for batch reading according to mammographic characteristics that may promote visual adaptation. Materials and Methods This retrospective multireader multicase study was performed with mammograms obtained between September 2016 and May 2019. The screening examinations, each consisting of four mammograms, were interpreted by 13 radiologists in three distinct orders: randomly, by increasing volumetric breast density (VBD), and based on a self-supervised learning (SSL) encoding (examinations automatically grouped as "looking similar"). An eye tracker recorded radiologists' eye movements during interpretation. The area under the receiver operating characteristic curve (AUC), sensitivity, and specificity of random-ordered readings were compared with those of VBD- and SSL-ordered readings using mixed-model analysis of variance. Reading time, fixation metrics, and perceived density were compared using Wilcoxon signed-rank tests. Results Mammography examinations (75 with breast cancer, 75 without breast cancer) from 150 women (median age, 55 years [IQR, 50-63]) were read. The examinations ordered by increasing VBD versus randomly had an increased AUC (0.93 [95% CI: 0.91, 0.96] vs 0.92 [95% CI: 0.89, 0.95]; P = .009), without evidence of a difference in specificity (89% [871 of 975] vs 86% [837 of 975], P = .04) and sensitivity (both 81% [794 of 975 vs 788 of 975], P = .78), and a reduced reading time (24.3 vs 27.9 seconds, P < .001), fixation count (47 vs 52, P < .001), and fixation time in malignant regions (3.7 vs 4.6 seconds, P < .001). For SSL-ordered readings, there was no evidence of differences in AUC (0.92 [95% CI: 0.89, 0.95]; P = .70), specificity (84% [820 of 975], P = .37), sensitivity (80% [784 of 975], P = .79), fixation count (54, P = .05), or fixation time in malignant regions (4.6 seconds, P > .99) compared with random-ordered readings. Reading times were significantly higher for SSL-ordered readings compared with random-ordered readings (28.4 seconds, P = .02). Conclusion Screening mammography examinations ordered from low to high VBD improved screening performance while reducing reading and fixation times. © RSNA, 2024 Supplemental material is available for this article. See also the editorial by Grimm in this issue.

Implementing multiple imputations for addressing missing data in multireader multicase design studies

BackgroundIn computer-aided diagnosis (CAD) studies utilizing multireader multicase (MRMC) designs, missing data might occur when there are instances of misinterpretation or oversight by the reader or problems with measurement techniques. Improper handling of these missing data can lead to bias. However, little research has been conducted on addressing the missing data issue within the MRMC framework.MethodsWe introduced a novel approach that integrates multiple imputation with MRMC analysis (MI-MRMC). An elaborate simulation study was conducted to compare the efficacy of our proposed approach with that of the traditional complete case analysis strategy within the MRMC design. Furthermore, we applied these approaches to a real MRMC design CAD study on aneurysm detection via head and neck CT angiograms to further validate their practicality.ResultsCompared with traditional complete case analysis, the simulation study demonstrated the MI-MRMC approach provides an almost unbiased estimate of diagnostic capability, alongside satisfactory performance in terms of statistical power and the type I error rate within the MRMC framework, even in small sample scenarios. In the real CAD study, the proposed MI-MRMC method further demonstrated strong performance in terms of both point estimates and confidence intervals compared with traditional complete case analysis.ConclusionWithin MRMC design settings, the adoption of an MI-MRMC approach in the face of missing data can facilitate the attainment of unbiased and robust estimates of diagnostic capability.

Comparison of Mammography and Mammography with Supplemental Whole-Breast US Tomography for Cancer Detection in Patients with Dense Breasts.

Women with dense breasts benefit from supplemental cancer screening with US, but US has low specificity. To evaluate the performance of breast US tomography (UST) combined with full-field digital mammography (FFDM) compared with FFDM alone for breast cancer screening in women with dense breasts. This retrospective multireader multicase study included women with dense breasts who underwent FFDM and UST at 10 centers between August 2017 and October 2019 as part of a prospective case collection registry. All patients in the registry with cancer were included; patients with benign biopsy or negative follow-up imaging findings were randomly selected for inclusion. Thirty-two Mammography Quality Standards Act-qualified radiologists independently evaluated FFDM followed immediately by FFDM plus UST for suspicious findings and assigned a Breast Imaging Reporting and Data System (BI-RADS) category. The superiority of FFDM plus UST versus FFDM alone for cancer detection (assessed with area under the receiver operating characteristic curve [AUC]), BI-RADS 4 sensitivity, and BI-RADS 3 sensitivity and specificity were evaluated using the two-sided significance level of α = .05. Noninferiority of BI-RADS 4 specificity was evaluated at the one-sided significance level of α = .025 with a -10% margin. Among 140 women (mean age, 56 years ±10 [SD]; 36 with cancer, 104 without), FFDM plus UST achieved superior performance compared with FFDM alone (AUC, 0.60 [95% CI: 0.51, 0.69] vs 0.54 [95% CI: 0.45, 0.64]; P = .03). For FFDM plus UST versus FFDM alone, BI-RADS 4 mean sensitivity was superior (37% [428 of 1152] vs 30% [343 of 1152]; P = .03) and BI-RADS 4 mean specificity was noninferior (82% [2741 of 3328] vs 88% [2916 of 3328]; P = .004). For FFDM plus UST versus FFDM, no difference in BI-RADS 3 mean sensitivity was observed (40% [461 of 1152] vs 33% [385 of 1152]; P = .08), but BI-RADS 3 mean specificity was superior (75% [2491 of 3328] vs 69% [2299 of 3328]; P = .04). In women with dense breasts, FFDM plus UST improved cancer detection by radiologists versus FFDM alone. Clinical trial registration nos. NCT03257839 and NCT04260620 Published under a CC BY 4.0 license. Supplemental material is available for this article. See also the editorial by Mann in this issue.

Assistive AI in Lung Cancer Screening: A Retrospective Multinational Study in the United States and Japan.

Purpose To evaluate the impact of an artificial intelligence (AI) assistant for lung cancer screening on multinational clinical workflows. Materials and Methods An AI assistant for lung cancer screening was evaluated on two retrospective randomized multireader multicase studies where 627 (141 cancer-positive cases) low-dose chest CT cases were each read twice (with and without AI assistance) by experienced thoracic radiologists (six U.S.-based or six Japan-based radiologists), resulting in a total of 7524 interpretations. Positive cases were defined as those within 2 years before a pathology-confirmed lung cancer diagnosis. Negative cases were defined as those without any subsequent cancer diagnosis for at least 2 years and were enriched for a spectrum of diverse nodules. The studies measured the readers' level of suspicion (on a 0-100 scale), country-specific screening system scoring categories, and management recommendations. Evaluation metrics included the area under the receiver operating characteristic curve (AUC) for level of suspicion and sensitivity and specificity of recall recommendations. Results With AI assistance, the radiologists' AUC increased by 0.023 (0.70 to 0.72; P = .02) for the U.S. study and by 0.023 (0.93 to 0.96; P = .18) for the Japan study. Scoring system specificity for actionable findings increased 5.5% (57% to 63%; P < .001) for the U.S. study and 6.7% (23% to 30%; P < .001) for the Japan study. There was no evidence of a difference in corresponding sensitivity between unassisted and AI-assisted reads for the U.S. (67.3% to 67.5%; P = .88) and Japan (98% to 100%; P > .99) studies. Corresponding stand-alone AI AUC system performance was 0.75 (95% CI: 0.70, 0.81) and 0.88 (95% CI: 0.78, 0.97) for the U.S.- and Japan-based datasets, respectively. Conclusion The concurrent AI interface improved lung cancer screening specificity in both U.S.- and Japan-based reader studies, meriting further study in additional international screening environments. Keywords: Assistive Artificial Intelligence, Lung Cancer Screening, CT Supplemental material is available for this article. Published under a CC BY 4.0 license.

Impact of a Categorical AI System for Digital Breast Tomosynthesis on Breast Cancer Interpretation by Both General Radiologists and Breast Imaging Specialists.

Purpose To evaluate performance improvements of general radiologists and breast imaging specialists when interpreting a set of diverse digital breast tomosynthesis (DBT) examinations with the aid of a custom-built categorical artificial intelligence (AI) system. Materials and Methods A fully balanced multireader, multicase reader study was conducted to compare the performance of 18 radiologists (nine general radiologists and nine breast imaging specialists) reading 240 retrospectively collected screening DBT mammograms (mean patient age, 59.8 years ± 11.3 [SD]; 100% women), acquired between August 2016 and March 2019, with and without the aid of a custom-built categorical AI system. The area under the receiver operating characteristic curve (AUC), sensitivity, and specificity across general radiologists and breast imaging specialists reading with versus without AI were assessed. Reader performance was also analyzed as a function of breast cancer characteristics and patient subgroups. Results Every radiologist demonstrated improved interpretation performance when reading with versus without AI, with an average AUC of 0.93 versus 0.87, demonstrating a difference in AUC of 0.06 (95% CI: 0.04, 0.08; P < .001). Improvement in AUC was observed for both general radiologists (difference of 0.08; P < .001) and breast imaging specialists (difference of 0.04; P < .001) and across all cancer characteristics (lesion type, lesion size, and pathology) and patient subgroups (race and ethnicity, age, and breast density) examined. Conclusion A categorical AI system helped improve overall radiologist interpretation performance of DBT screening mammograms for both general radiologists and breast imaging specialists and across various patient subgroups and breast cancer characteristics. Keywords: Computer-aided Diagnosis, Screening Mammography, Digital Breast Tomosynthesis, Breast Cancer, Screening, Convolutional Neural Network (CNN), Artificial Intelligence Supplemental material is available for this article. © RSNA, 2024.

AI assisted reader evaluation in acute CT head interpretation (AI-REACT): protocol for a multireader multicase study

IntroductionA non-contrast CT head scan (NCCTH) is the most common cross-sectional imaging investigation requested in the emergency department. Advances in computer vision have led to development of several artificial intelligence...

Development and Evaluation of a Surveillance System for Follow-Up After Colorectal Polypectomy

The adherence of physicians and patients to published colorectal postpolypectomy surveillance guidelines varies greatly, and patient follow-up is critical but time consuming. To evaluate the accuracy of an automatic surveillance (AS) system in identifying patients after polypectomy, assigning surveillance intervals for different risks of patients, and proactively following up with patients on time. In this diagnostic/prognostic study, endoscopic and pathological reports of 47 544 patients undergoing colonoscopy at 3 hospitals between January 1, 2017, and June 30, 2022, were collected to develop an AS system based on natural language processing. The performance of the AS system was fully evaluated in internal and external tests according to 5 guidelines worldwide and compared with that of physicians. A multireader, multicase (MRMC) trial was conducted to evaluate use of the AS system and physician guideline adherence, and prospective data were collected to evaluate the success rate in contacting patients and the association with reduced human workload. Data analysis was conducted from July to September 2022. Assistance of the AS system. The accuracy of the system in identifying patients after polypectomy, stratifying patient risk levels, and assigning surveillance intervals in internal (Renmin Hospital of Wuhan University), external 1 (Wenzhou Central Hospital), and external 2 (The First People's Hospital of Yichang) test sets; the accuracy of physicians and their time burden with and without system assistance; and the rate of successfully informed patients of the system were evaluated. Test sets for 16 106 patients undergoing colonoscopy (mean [SD] age, 51.90 [13.40] years; 7690 females [47.75%]) were evaluated. In internal, external 1, and external 2 test sets, the system had an overall accuracy of 99.91% (95% CI, 99.83%-99.95%), 99.54% (95% CI, 99.30%-99.70%), and 99.77% (95% CI, 99.41%-99.91%), respectively, for identifying types of patients and achieved an overall accuracy of at least 99.30% (95% CI, 98.67%-99.63%) in the internal test set, 98.89% (95% CI, 98.33%-99.27%) in external test set 1, and 98.56% (95% CI, 95.86%-99.51%) in external test set 2 for stratifying patient risk levels and assigning surveillance intervals according to 5 guidelines. The system was associated with increased mean (SD) accuracy among physicians vs no AS system in 105 patients (98.67% [1.28%] vs 78.10% [18.01%]; P = .04) in the MRMC trial. In a prospective trial, the AS system successfully informed 82 of 88 patients (93.18%) and was associated with reduced burden of follow-up time vs no AS system (0 vs 2.86 h). This study found that an AS system was associated with improved adherence to guidelines among physicians and reduced workload among physicians and nurses.

Effect of a deep learning-based automatic upper gastrointestinal endoscopic reporting system: a randomized crossover study.

Esophagogastroduodenoscopy (EGD) is essential for gastrointestinal disorders, and reports are pivotal to facilitating post-procedure diagnosis and treatment. Manual report generation lacks sufficient quality and is labor intensive. We first reported and validated an artificial intelligence-based endoscopy automatic reporting system (AI-EARS). The AI-EARS was designed for automatic report generation, including real-time image capturing, diagnosis and textual description. It was developed using multicenter datasets from eight hospitals in China, including 252111 images for training, 62706 images and 950 videos for testing.Twelve endoscopists and 44 endoscopy procedures were consecutively enrolled to evaluate the effect of AI-EARS in a multi-reader multi-case (MRMC) crossover study. The precision and completeness of the reports were compared between endoscopists using AI-EARS and conventional reporting systems. In video validation, AI-EARS achieved completeness of 98.59% and 99.69% for esophageal and gastric abnormality records, accuracies of 87.99% and 88.85% for esophageal and gastric lesion location records, and 73.14% and 85.24% for diagnosis.Compared to the conventional reporting systems, AI-EARS achieved greater completeness (79.03% vs. 51.86%, p<0.001), accuracy (64.47% vs. 42.81%, p<0.001) of the textual description, and completeness of the photo-documents of landmarks (92.23% vs. 73.69%, p<0.001). The mean reporting time for an individual lesion was significantly reduced (80.13±16.12 vs. 46.47±11.68 seconds, P<0.001) after AI-EARS assistance. AI-EARS showed its efficacy in improving the accuracy and completeness of the EGD reports. It might facilitate the generation of complete endoscopy reports and post-endoscopy patient management. (ClinicalTrails.gov, number NCT05479253).

Roe and Metz identical-test simulation model for validating multi-reader methods of analysis for comparing different radiologic imaging modalities.

The most frequently used model for simulating multi-reader multi-case (MRMC) data that emulate confidence-of-disease ratings from diagnostic imaging studies has been the Roe and Metz model, proposed by Roe and Metz in 1997 and later generalized by Hillis (2012), Abbey etal. (2013), and Gallas and Hillis (2014). These models have been used for evaluating MRMC analysis and sample size methods. The models suggested in these papers for assessing type I error have been null models, where the expected area under the receiver-operating-characteristic curve across readers is the same for each test. However, for these null models, there are other differences that would not exist if the two tests were identical. None of the papers mentioned above discuss how to formulate a null model that is also an identical-test model, where the two tests are identical in all respects. The purpose of this paper is to show how to formulate a Roe and Metz identical-test model and to show its usefulness for validating the error covariance constraints employed by the Obuchowski-Rockette (1995) method. For a given Roe-and-Metz model, the corresponding Roe-and-Metz identical-test model is derived by modifying the Roe and Metz null model under the assumption that the two tests are identical. The importance of the Obuchowski-Rockette model constraints for avoiding negative variance estimates is established using data simulated from the Roe and Metz identical-test model. It is also shown that negative variance estimates can occur at nontrivial rates when the two tests are not identical but somewhat "close" to being identical. The findings of this paper are important because it has recently been shown (Hillis, 2022) that the commonly used MRMC method proposed by Gallas (2006) and Gallas etal. (2009) uses the same test statistic as the unconstrained Obuchowski-Rockette method.

Low frequency 3D transmission ultrasound tomography: technical details and clinical implications

A novel 3D ultrasound tomographic (3D UT) method (called volography) that creates a speed of sound (SOS) map and a reflection modality that is co-registered are reviewed and shown to be artifact free even in the presence of high contrast and thus shown to be applicable for breast, orthopedic and pediatric clinical use cases. The 3D UT images are almost isotropic with mm resolution and the reflection image is compounded over 360 degrees to create sub-mm resolution in plane. MethodsThe physics of ultrasound scattering requires 3D modeling and the concomitant high computational cost is ameliorated with a bespoke algorithm (paraxial approximation – discussed here) and Nvidia GPUs. The resulting reconstruction times are tabulated for clinical relevance. The resulting SOS map is used to create a refraction corrected reflection image at ∼3.6 MHz center frequency. The transmission data are highly redundant, collected over 360 degrees and at 2 mm levels by true matrix receiver arrays yielding 3D data.The high resolution SOS and attenuation maps and reflection images are used in a segmentation algorithm that optimally utilizes this information to segment out glandular, ductal, connective tissue, fat and skin. These volumes are used to estimate breast density, an important correlate to cancer. ResultsMultiple SOS images of breast, knee and segmentations of breast glandular and ductal tissue are shown. Spearman rho is calculated between our volumetric breast density estimates and Volpara™ from mammograms, as 0.9332. Multiple timing results are shown and indicate the variability of the reconstruction times with breast size and type but are ∼30 minutes for average size breast. The timing results with the 3D algorithm indicate ∼60 minute reconstruction times for pediatrics with two Nvidia GPUs. Characteristic variations of the glandular and ductal volumes over time are shown. The SOS from QT images are compared with literature values.The results of a multi-reader multi-case (MRMC) study are shown that compares the 3D UT with full field digital mammography and resulted in an average increase in ROC AUC of 10%. Orthopedic (knee) 3D UT images compared with MRI indicate regions of zero signal in the MRI are clearly displayed in the QT image.Explicit representation of the acoustic field is shown, indicating its 3D nature. An image of in vivo breast with the chest muscle is shown and speed of sound agreement with literature values are tabulated. Reference is made to a recently published paper validating pediatric imaging. ConclusionsThe high Spearman rho indicates a monotonic (not necessarily linear) relation between our method and industry gold standard Volpara™ density. The acoustic field verifies the need for 3D modeling. The MRMC study, the orthopedic images, breast density study, and references, all indicate the clinical utility of the SOS and reflection images. The QT image of the knee shows its ability to monitor tissue the MRI cannot. The included references and images herein indicate the proof of concept for 3D UT as a viable and valuable clinical adjunct in pediatric and orthopedic situations in addition to the breast imaging.

Obuchowski-Rockette analysis for multi-reader multi-case (MRMC) readers-nested-in-test study design with unequal numbers of readers.

The Obuchowski-Rockette method has been an important tool for analyzing multi-reader multi-case (MRMC) radiologic imaging data. Although the typical study design for such studies has been the factorial design, where each reader reads each case using each test (modality), sometimes a reader-nested-in-test design is more appropriate. We consider such an example in this talk, where 53 Australian and 15 Singaporean breast radiologists interpreted the same test in their respective locations. In this paper we show how the Obuchowski-Rockette method can be used for analysis of such data, without assuming that the number of readers is the same for each test.

Assessment of deep learning assistance for the pathological diagnosis of gastric cancer

AbstractPrevious studies on deep learning (DL) applications in pathology have focused on pathologist-versus-algorithm comparisons. However, DL will not replace the breadth and contextual knowledge of pathologists; rather, only through their combination may the benefits of DL be achieved. A fully crossed multireader multicase study was conducted to evaluate DL assistance with pathologists' diagnosis of gastric cancer. A total of 110 whole-slide images (WSI) (50 malignant and 60 benign) were interpreted by 16 board-certified pathologists with or without DL assistance, with a washout period between sessions. DL-assisted pathologists achieved a higher area under receiver operating characteristic curve (ROC-AUC) (0.911 vs. 0.863, P = 0.003) than unassisted in interpreting the 110 WSIs. Pathologists with DL assistance demonstrated higher sensitivity in detection of gastric cancer than without (90.63% vs. 82.75%, P = 0.010). No significant difference was observed in specificity with or without deep learning assistance (78.23% vs. 79.90%, P = 0.468). The average review time per WSI was shortened with DL assistance than without (22.68 vs. 26.37 second, P = 0.033). Our results demonstrated that DL assistance indeed improved pathologists' accuracy and efficiency in gastric cancer diagnosis and further boosted the acceptance of this new technique.

Determining Roe and Metz model parameters for simulating multireader multicase confidence-of-disease rating data based on real-data or conjectured Obuchowski-Rockette parameter estimates.

Purpose: The most frequently used model for simulating multireader multicase (MRMC) data that emulates confidence-of-disease ratings from diagnostic imaging studies has been the Roe and Metz (RM) model, proposed by Roe and Metz in 1997 and later generalized by Hillis (2012), Abbey etal. (2013), and Gallas and Hillis (2014). A problem with these models is that it has been difficult to set model parameters such that the simulated data are similar to MRMC data encountered in practice. To remedy this situation, Hillis (2018) mapped parameters from the RM model to Obuchowski-Rockette (OR) model parameters that describe the distribution of the empirical AUC outcomes computed from the RM model simulated data. We continue that work by providing the reverse mapping, i.e., by deriving an algorithm that expresses RM parameters as functions of the OR empirical AUC distribution parameters. Approach: We solve for the corresponding RM parameters in terms of the OR parameters using numerical methods. Results: An algorithm is developed that results in, at most, one solution of RM parameter values that correspond to inputted OR parameter values. The algorithm can be implemented using an R software function. Examples are provided that illustrate the use of the algorithm. A simulation study validates the algorithm. Conclusions: The resulting algorithm makes it possible to easily determine RM model parameter values such that simulated data emulate a specific real-data study. Thus, MRMC analysis methods can be empirically tested using simulated data similar to that encountered in practice.

Hypothesis testing methods for multi-reader multi-case studies

ABSTRACT Rationale and Objectives: Multi-reader multi-case (MRMC) studies are widely used to compare the diagnostic accuracies of different imaging modalities. Although Dorfman–Berbaum–Metz (DBM) method is the most popular one among the MRMC methods, the adaptions of ANOVA statistic for linear mixed model (LMM) are not based on solid theory and the assumption of ANOVA that all groups have the same number of samples might not be met in some situations. The purpose of the article is to investigate whether the statistics for testing fixed effect in linear mixed model can yield a closer type I error rate to nominal level. Materials and Methods: We proposed to use the statistics such as likelihood ratio test (LRT) and Wald statistic to test the hypothesis of equivalence in several imaging modalities. Extensive simulations were conducted and the application to a clinical example dataset was illustrated. Results: The simulation results showed the type I error rates of Wald statistic were closer to the nominal level under many simulated situations, especially when the simulated data was ordinal and the number of diseased and non-diseased were 100. Conclusion: The Wald statistic whose degrees of freedom ware approximated by Satterthwaite's method showed competitive performance, indicating the potential of the statistic applied in DBM model for MRMC analysis.

Relationship between Obuchowski–Rockette–Hillis and Gallas methods for analyzing multi-reader diagnostic imaging data with empirical AUC as the reader performance measure

For analyzing multireader multicase (MRMC) diagnostic imaging data when the reader performance measure of interest is the area under the receiver-operating-characteristic curve (AUC), two popular methods of analysis that allow conclusions to generalize to both the reader and case populations are the method developed by Obuchowski, Rockette and Hillis (ORH) and the method primarily developed by Gallas (Gallas). While the ORH method is a general method that is applicable to most reader performance metrics, the Gallas method is limited to those metrics for which an unbiased variance estimate exists. Previously it was not known if the ORH method could be adapted so as to produce the same variance estimate as the Gallas method. In this paper, I show that a recently proposed version of the OR method produces the same unconstrained variance statistic as the Gallas method. However, the two methods differ in their approaches to constraining the variance estimate to be nonnegative and in their degrees-of-freedom estimates. These two differences are discussed and recommendations given. In addition, several contributions to the development of the ORH method are made, including determining sufficient conditions for unbiased variance estimates and providing justification for the ORH variance constraints and covariance estimation method.

Artificial Intelligence Tool for Assessment of Indeterminate Pulmonary Nodules Detected with CT.

Background Limited data are available regarding whether computer-aided diagnosis (CAD) improves assessment of malignancy risk in indeterminate pulmonary nodules (IPNs). Purpose To evaluate the effect of an artificial intelligence-based CAD tool on clinician IPN diagnostic performance and agreement for both malignancy risk categories and management recommendations. Materials and Methods This was a retrospective multireader multicase study performed in June and July 2020 on chest CT studies of IPNs. Readers used only CT imaging data and provided an estimate of malignancy risk and a management recommendation for each case without and with CAD. The effect of CAD on average reader diagnostic performance was assessed using the Obuchowski-Rockette and Dorfman-Berbaum-Metz method to calculate estimates of area under the receiver operating characteristic curve (AUC), sensitivity, and specificity. Multirater Fleiss κ statistics were used to measure interobserver agreement for malignancy risk and management recommendations. Results A total of 300 chest CT scans of IPNs with maximal diameters of 5-30 mm (50.0% malignant) were reviewed by 12 readers (six radiologists, six pulmonologists) (patient median age, 65 years; IQR, 59-71 years; 164 [55%] men). Readers' average AUC improved from 0.82 to 0.89 with CAD (P < .001). At malignancy risk thresholds of 5% and 65%, use of CAD improved average sensitivity from 94.1% to 97.9% (P = .01) and from 52.6% to 63.1% (P < .001), respectively. Average reader specificity improved from 37.4% to 42.3% (P = .03) and from 87.3% to 89.9% (P = .05), respectively. Reader interobserver agreement improved with CAD for both the less than 5% (Fleiss κ, 0.50 vs 0.71; P < .001) and more than 65% (Fleiss κ, 0.54 vs 0.71; P < .001) malignancy risk categories. Overall reader interobserver agreement for management recommendation categories (no action, CT surveillance, diagnostic procedure) also improved with CAD (Fleiss κ, 0.44 vs 0.52; P = .001). Conclusion Use of computer-aided diagnosis improved estimation of indeterminate pulmonary nodule malignancy risk on chest CT scans and improved interobserver agreement for both risk stratification and management recommendations. © RSNA, 2022 Online supplemental material is available for this article. See also the editorial by Yanagawa in this issue.

Convolutional neural network assistance significantly improves dermatologists’ diagnosis of cutaneous tumours using clinical images

BackgroundConvolutional neural networks (CNNs) have demonstrated expert-level performance in cutaneous tumour classification using clinical images, but most previous studies have focused on dermatologist-versus-CNN comparisons rather than their combination. The objective of our study was to evaluate the potential impact of CNN assistance on dermatologists for clinical image interpretation. MethodsA multi-class CNN was trained and validated using a dataset of 25,773 clinical images comprising 10 categories of cutaneous tumours. The CNN's performance was tested on an independent dataset of 2107 images. A total of 400 images (40 per category) were randomly selected from the test dataset. A fully crossed, self-control, multi-reader multi-case (MRMC) study was conducted to compare the performance of 18 board-certified dermatologists (experience: 13/18 ≤ 10 years; 5/18＞10 years) in interpreting the 400 clinical images with or without CNN assistance. ResultsThe CNN achieved an overall accuracy of 78.45% and kappa of 0.73 in the classification of 10 types of cutaneous tumours on 2107 images. CNN-assisted dermatologists achieved a higher accuracy (76.60% vs. 62.78%, P < 0.001) and kappa (0.74 vs. 0.59, P < 0.001) than unassisted dermatologists in interpreting the 400 clinical images. Dermatologists with less experience benefited more from CNN assistance. At the binary classification level (malignant or benign), the sensitivity (89.56% vs. 83.21%, P < 0.001) and specificity (87.90% vs. 80.92%, P < 0.001) of dermatologists with CNN assistance were also significantly improved than those without. ConclusionsCNN assistance improved dermatologist accuracy in interpreting cutaneous tumours and could further boost the acceptance of this new technique.

Three-Way Mixed Effect ANOVA to Estimate MRMC Limits of Agreement

When evaluating the clinical performance of a medical imaging device, a multi-reader multi-case (MRMC) analysis is usually applied to account for both case and reader variability. For a clinical task that equates to a quantitative measurement, an agreement analysis such as a limits of agreement (LOA) method can be used to compare different measurement methods. In this work, we introduce four types of comparisons; these types differ depending on whether the measurements are within or between readers and within or between modalities. A three-way mixed effect ANOVA model is applied to estimate the variances of individual differences, which is an essential step for estimating LOA. To verify the estimates of LOA, we propose a hierarchical model to simulate quantitative MRMC data. Two simulation studies were conducted to validate both the simulation and the LOA variance estimates. From the simulation results, we can conclude that our estimate of variance is unbiased, and the uncertainty of the estimation drops as the number of readers and cases increases and rises as the value of true variance increases.