Multi-reader Multi-case Study Research Articles

Comparison of Mammography and Mammography with Supplemental Whole-Breast US Tomography for Cancer Detection in Patients with Dense Breasts.

Women with dense breasts benefit from supplemental cancer screening with US, but US has low specificity. To evaluate the performance of breast US tomography (UST) combined with full-field digital mammography (FFDM) compared with FFDM alone for breast cancer screening in women with dense breasts. This retrospective multireader multicase study included women with dense breasts who underwent FFDM and UST at 10 centers between August 2017 and October 2019 as part of a prospective case collection registry. All patients in the registry with cancer were included; patients with benign biopsy or negative follow-up imaging findings were randomly selected for inclusion. Thirty-two Mammography Quality Standards Act-qualified radiologists independently evaluated FFDM followed immediately by FFDM plus UST for suspicious findings and assigned a Breast Imaging Reporting and Data System (BI-RADS) category. The superiority of FFDM plus UST versus FFDM alone for cancer detection (assessed with area under the receiver operating characteristic curve [AUC]), BI-RADS 4 sensitivity, and BI-RADS 3 sensitivity and specificity were evaluated using the two-sided significance level of α = .05. Noninferiority of BI-RADS 4 specificity was evaluated at the one-sided significance level of α = .025 with a -10% margin. Among 140 women (mean age, 56 years ±10 [SD]; 36 with cancer, 104 without), FFDM plus UST achieved superior performance compared with FFDM alone (AUC, 0.60 [95% CI: 0.51, 0.69] vs 0.54 [95% CI: 0.45, 0.64]; P = .03). For FFDM plus UST versus FFDM alone, BI-RADS 4 mean sensitivity was superior (37% [428 of 1152] vs 30% [343 of 1152]; P = .03) and BI-RADS 4 mean specificity was noninferior (82% [2741 of 3328] vs 88% [2916 of 3328]; P = .004). For FFDM plus UST versus FFDM, no difference in BI-RADS 3 mean sensitivity was observed (40% [461 of 1152] vs 33% [385 of 1152]; P = .08), but BI-RADS 3 mean specificity was superior (75% [2491 of 3328] vs 69% [2299 of 3328]; P = .04). In women with dense breasts, FFDM plus UST improved cancer detection by radiologists versus FFDM alone. Clinical trial registration nos. NCT03257839 and NCT04260620 Published under a CC BY 4.0 license. Supplemental material is available for this article. See also the editorial by Mann in this issue.

A deep-learning model for intracranial aneurysm detection on CT angiography images in China: a stepwise, multicentre, early-stage clinical validation study

Artificial intelligence (AI) models in real-world implementation are scarce. Our study aimed to develop a CT angiography (CTA)-based AI model for intracranial aneurysm detection, assess how it helps clinicians improve diagnostic performance, and validate its application in real-world clinical implementation. We developed a deep-learning model using 16 546 head and neck CTA examination images from 14 517 patients at eight Chinese hospitals. Using an adapted, stepwise implementation and evaluation, 120 certified clinicians from 15 geographically different hospitals were recruited. Initially, the AI model was externally validated with images of 900 digital subtraction angiography-verified CTA cases (examinations) and compared with the performance of 24 clinicians who each viewed 300 of these cases (stage 1). Next, as a further external validation a multi-reader multi-case study enrolled 48 clinicians to individually review 298 digital subtraction angiography-verified CTA cases (stage 2). The clinicians reviewed each CTA examination twice (ie, with and without the AI model), separated by a 4-week washout period. Then, a randomised open-label comparison study enrolled 48 clinicians to assess the acceptance and performance of this AI model (stage 3). Finally, the model was prospectively deployed and validated in 1562 real-world clinical CTA cases. The AI model in the internal dataset achieved a patient-level diagnostic sensitivity of 0·957 (95% CI 0·939-0·971) and a higher patient-level diagnostic sensitivity than clinicians (0·943 [0·921-0·961] vs 0·658 [0·644-0·672]; p<0·0001) in the external dataset. In the multi-reader multi-case study, the AI-assisted strategy improved clinicians' diagnostic performance both on a per-patient basis (the area under the receiver operating characteristic curves [AUCs]; 0·795 [0·761-0·830] without AI vs 0·878 [0·850-0·906] with AI; p<0·0001) and a per-aneurysm basis (the area under the weighted alternative free-response receiver operating characteristic curves; 0·765 [0·732-0·799] vs 0·865 [0·839-0·891]; p<0·0001). Reading time decreased with the aid of the AI model (87·5 s vs 82·7 s, p<0·0001). In the randomised open-label comparison study, clinicians in the AI-assisted group had a high acceptance of the AI model (92·6% adoption rate), and a higher AUC when compared with the control group (0·858 [95% CI 0·850-0·866] vs 0·789 [0·780-0·799]; p<0·0001). In the prospective study, the AI model had a 0·51% (8/1570) error rate due to poor-quality CTA images and recognition failure. The model had a high negative predictive value of 0·998 (0·994-1·000) and significantly improved the diagnostic performance of clinicians; AUC improved from 0·787 (95% CI 0·766-0·808) to 0·909 (0·894-0·923; p<0·0001) and patient-level sensitivity improved from 0·590 (0·511-0·666) to 0·825 (0·759-0·880; p<0·0001). This AI model demonstrated strong clinical potential for intracranial aneurysm detection with improved clinician diagnostic performance, high acceptance, and practical implementation in real-world clinical cases. National Natural Science Foundation of China. For the Chinese translation of the abstract see Supplementary Materials section.

AI assisted reader evaluation in acute CT head interpretation (AI-REACT): protocol for a multireader multicase study

IntroductionA non-contrast CT head scan (NCCTH) is the most common cross-sectional imaging investigation requested in the emergency department. Advances in computer vision have led to development of several artificial intelligence...

Artificial intelligence-assisted diagnosis of congenital heart disease and associated pulmonary arterial hypertension from chest radiographs: A multi-reader multi-case study

ObjectivesTo explore the possibility of automatic diagnosis of congenital heart disease (CHD) and pulmonary arterial hypertension associated with CHD (PAH-CHD) from chest radiographs using artificial intelligence (AI) technology and to evaluate whether AI assistance could improve clinical diagnostic accuracy. Materials and MethodsA total of 3255 frontal preoperative chest radiographs (1174 CHD of any type and 2081 non-CHD) were retrospectively obtained. In this study, we adopted ResNet18 pretrained with the ImageNet database to establish diagnostic models. Radiologists diagnosed CHD/PAH-CHD from 330/165 chest radiographs twice: the first time, 50% of the images were accompanied by AI-based classification; after a month, the remaining 50% were accompanied by AI-based classification. Diagnostic results were compared between the radiologists and AI models, and between radiologists with and without AI assistance. ResultsThe AI model achieved an average area under the receiver operating characteristic curve (AUC) of 0.948 (sensitivity: 0.970, specificity: 0.982) for CHD diagnoses and an AUC of 0.778 (sensitivity: 0.632, specificity: 0.925) for identifying PAH-CHD. In the 330 balanced (165 CHD and 165 non-CHD) testing set, AI achieved higher AUCs than all 5 radiologists in the identification of CHD (0.670–0.858) and PAH-CHD (0.610–0.688). With AI assistance, the mean ± standard error AUC of radiologists was significantly improved for CHD (ΔAUC + 0.096, 95 % CI: 0.001–0.190; P = 0.048) and PAH-CHD (ΔAUC + 0.066, 95 % CI: 0.010–0.122; P = 0.031) diagnosis. ConclusionChest radiograph-based AI models can detect CHD and PAH-CHD automatically. AI assistance improved radiologists' diagnostic accuracy, which may facilitate a timely initial diagnosis of CHD and PAH-CHD.

Mammographic Breast Density Model Using Semi-Supervised Learning Reduces Inter-/Intra-Reader Variability.

Breast density is an important risk factor for breast cancer development; however, imager inconsistency in density reporting can lead to patient and clinician confusion. A deep learning (DL) model for mammographic density grading was examined in a retrospective multi-reader multi-case study consisting of 928 image pairs and assessed for impact on inter- and intra-reader variability and reading time. Seven readers assigned density categories to the images, then re-read the test set aided by the model after a 4-week washout. To measure intra-reader agreement, 100 image pairs were blindly double read in both sessions. Linear Cohen Kappa (κ) and Student's t-test were used to assess the model and reader performance. The model achieved a κ of 0.87 (95% CI: 0.84, 0.89) for four-class density assessment and a κ of 0.91 (95% CI: 0.88, 0.93) for binary non-dense/dense assessment. Superiority tests showed significant reduction in inter-reader variability (κ improved from 0.70 to 0.88, p ≤ 0.001) and intra-reader variability (κ improved from 0.83 to 0.95, p ≤ 0.01) for four-class density, and significant reduction in inter-reader variability (κ improved from 0.77 to 0.96, p ≤ 0.001) and intra-reader variability (κ improved from 0.89 to 0.97, p ≤ 0.01) for binary non-dense/dense assessment when aided by DL. The average reader mean reading time per image pair also decreased by 30%, 0.86 s (95% CI: 0.01, 1.71), with six of seven readers having reading time reductions.

Low frequency 3D transmission ultrasound tomography: technical details and clinical implications

A novel 3D ultrasound tomographic (3D UT) method (called volography) that creates a speed of sound (SOS) map and a reflection modality that is co-registered are reviewed and shown to be artifact free even in the presence of high contrast and thus shown to be applicable for breast, orthopedic and pediatric clinical use cases. The 3D UT images are almost isotropic with mm resolution and the reflection image is compounded over 360 degrees to create sub-mm resolution in plane. MethodsThe physics of ultrasound scattering requires 3D modeling and the concomitant high computational cost is ameliorated with a bespoke algorithm (paraxial approximation – discussed here) and Nvidia GPUs. The resulting reconstruction times are tabulated for clinical relevance. The resulting SOS map is used to create a refraction corrected reflection image at ∼3.6 MHz center frequency. The transmission data are highly redundant, collected over 360 degrees and at 2 mm levels by true matrix receiver arrays yielding 3D data.The high resolution SOS and attenuation maps and reflection images are used in a segmentation algorithm that optimally utilizes this information to segment out glandular, ductal, connective tissue, fat and skin. These volumes are used to estimate breast density, an important correlate to cancer. ResultsMultiple SOS images of breast, knee and segmentations of breast glandular and ductal tissue are shown. Spearman rho is calculated between our volumetric breast density estimates and Volpara™ from mammograms, as 0.9332. Multiple timing results are shown and indicate the variability of the reconstruction times with breast size and type but are ∼30 minutes for average size breast. The timing results with the 3D algorithm indicate ∼60 minute reconstruction times for pediatrics with two Nvidia GPUs. Characteristic variations of the glandular and ductal volumes over time are shown. The SOS from QT images are compared with literature values.The results of a multi-reader multi-case (MRMC) study are shown that compares the 3D UT with full field digital mammography and resulted in an average increase in ROC AUC of 10%. Orthopedic (knee) 3D UT images compared with MRI indicate regions of zero signal in the MRI are clearly displayed in the QT image.Explicit representation of the acoustic field is shown, indicating its 3D nature. An image of in vivo breast with the chest muscle is shown and speed of sound agreement with literature values are tabulated. Reference is made to a recently published paper validating pediatric imaging. ConclusionsThe high Spearman rho indicates a monotonic (not necessarily linear) relation between our method and industry gold standard Volpara™ density. The acoustic field verifies the need for 3D modeling. The MRMC study, the orthopedic images, breast density study, and references, all indicate the clinical utility of the SOS and reflection images. The QT image of the knee shows its ability to monitor tissue the MRI cannot. The included references and images herein indicate the proof of concept for 3D UT as a viable and valuable clinical adjunct in pediatric and orthopedic situations in addition to the breast imaging.

Diagnostic performance of deep learning-based vessel extraction and stenosis detection on coronary computed tomography angiography for coronary arterydisease: a multi-reader multi-case study.

Post-processing and interpretation of coronary CT angiography (CCTA) imaging are time-consuming and dependent on the reader's experience. An automated deep learning (DL)-based imaging reconstruction and diagnosis system was developed to improve diagnostic accuracy and efficiency. Our study including 374 cases from five sites, inviting 12 radiologists, assessed the DL-based system in diagnosing obstructive coronary disease with regard to diagnostic performance, imaging post-processing and reporting time of radiologists, with invasive coronary angiography as a standard reference. The diagnostic performance of DL system and DL-assisted human readers was compared with the traditional method of human readers without DL system. Comparing the diagnostic performance of human readers without DL system versus with DL system, the AUC was improved from 0.81 to 0.82 (p < 0.05) at patient level and from 0.79 to 0.81 (p < 0.05) at vessel level. An increase in AUC was observed in inexperienced radiologists (p < 0.05), but was absent in experienced radiologists. Regarding diagnostic efficiency, comparing the DL system versus human reader, the average post-processing and reporting time was decreased from 798.60s to 189.12s (p < 0.05). The sensitivity and specificity of using DL system alone were 93.55% and 59.57% at patient level and 83.23% and 79.97% at vessel level, respectively. With the DL system serving as a concurrent reader, the overall post-processing and reading time was substantially reduced. The diagnostic accuracy of human readers, especially for inexperienced readers, was improved. DL-assisted human reader had the potential of being the reading mode of choice in clinical routine.

Assessment of deep learning assistance for the pathological diagnosis of gastric cancer

AbstractPrevious studies on deep learning (DL) applications in pathology have focused on pathologist-versus-algorithm comparisons. However, DL will not replace the breadth and contextual knowledge of pathologists; rather, only through their combination may the benefits of DL be achieved. A fully crossed multireader multicase study was conducted to evaluate DL assistance with pathologists' diagnosis of gastric cancer. A total of 110 whole-slide images (WSI) (50 malignant and 60 benign) were interpreted by 16 board-certified pathologists with or without DL assistance, with a washout period between sessions. DL-assisted pathologists achieved a higher area under receiver operating characteristic curve (ROC-AUC) (0.911 vs. 0.863, P = 0.003) than unassisted in interpreting the 110 WSIs. Pathologists with DL assistance demonstrated higher sensitivity in detection of gastric cancer than without (90.63% vs. 82.75%, P = 0.010). No significant difference was observed in specificity with or without deep learning assistance (78.23% vs. 79.90%, P = 0.468). The average review time per WSI was shortened with DL assistance than without (22.68 vs. 26.37 second, P = 0.033). Our results demonstrated that DL assistance indeed improved pathologists' accuracy and efficiency in gastric cancer diagnosis and further boosted the acceptance of this new technique.

A concurrent, deep learning-based computer-aided detection system for prostate multiparametric MRI: a performance study involving experienced and less-experienced radiologists.

To evaluate the effect of a deep learning-based computer-aided diagnosis (DL-CAD) system on experienced and less-experienced radiologists in reading prostate mpMRI. In this retrospective, multi-reader multi-case study, a consecutive set of 184 patients examined between 01/2018 and 08/2019 were enrolled. Ground truth was combined targeted and 12-core systematic transrectal ultrasound-guided biopsy. Four radiologists, two experienced and two less-experienced, evaluated each case twice, once without (DL-CAD-) and once assisted by DL-CAD (DL-CAD+). ROC analysis, sensitivities, specificities, PPV and NPV were calculated to compare the diagnostic accuracy for the diagnosis of prostate cancer (PCa) between the two groups (DL-CAD- vs. DL-CAD+). Spearman's correlation coefficients were evaluated to assess the relationship between PI-RADS category and Gleason score (GS). Also, the median reading times were compared for the two reading groups. In total, 172 patients were included in the final analysis. With DL-CAD assistance, the overall AUC of the less-experienced radiologists increased significantly from 0.66 to 0.80 (p = 0.001; cutoff ISUP GG≥1) and from 0.68 to 0.80 (p = 0.002; cutoff ISUP GG ≥ 2). Experienced radiologists showed an AUC increase from 0.81 to 0.86 (p = 0.146; cutoff ISUP GG ≥ 1) and from 0.81 to 0.84 (p = 0.433; cutoff ISUP GG ≥ 2). Furthermore, the correlation between PI-RADS category and GS improved significantly in the DL-CAD + group (0.45 vs. 0.57; p = 0.03), while the median reading time was reduced from 157 to 150 s (p = 0.023). DL-CAD assistance increased the mean detection performance, with the most significant benefit for the less-experienced radiologist; with the help of DL-CAD less-experienced radiologists reached performances comparable to that of experienced radiologists. • DL-CAD used as a concurrent reading aid helps radiologists to distinguish between benign and cancerous lesions in prostate MRI. • With the help of DL-CAD, less-experienced radiologists may achieve detection performances comparable to that of experienced radiologists. • DL-CAD assistance increases the correlation between PI-RADS category and cancer grade.

Artificial Intelligence Tool for Assessment of Indeterminate Pulmonary Nodules Detected with CT.

Background Limited data are available regarding whether computer-aided diagnosis (CAD) improves assessment of malignancy risk in indeterminate pulmonary nodules (IPNs). Purpose To evaluate the effect of an artificial intelligence-based CAD tool on clinician IPN diagnostic performance and agreement for both malignancy risk categories and management recommendations. Materials and Methods This was a retrospective multireader multicase study performed in June and July 2020 on chest CT studies of IPNs. Readers used only CT imaging data and provided an estimate of malignancy risk and a management recommendation for each case without and with CAD. The effect of CAD on average reader diagnostic performance was assessed using the Obuchowski-Rockette and Dorfman-Berbaum-Metz method to calculate estimates of area under the receiver operating characteristic curve (AUC), sensitivity, and specificity. Multirater Fleiss κ statistics were used to measure interobserver agreement for malignancy risk and management recommendations. Results A total of 300 chest CT scans of IPNs with maximal diameters of 5-30 mm (50.0% malignant) were reviewed by 12 readers (six radiologists, six pulmonologists) (patient median age, 65 years; IQR, 59-71 years; 164 [55%] men). Readers' average AUC improved from 0.82 to 0.89 with CAD (P < .001). At malignancy risk thresholds of 5% and 65%, use of CAD improved average sensitivity from 94.1% to 97.9% (P = .01) and from 52.6% to 63.1% (P < .001), respectively. Average reader specificity improved from 37.4% to 42.3% (P = .03) and from 87.3% to 89.9% (P = .05), respectively. Reader interobserver agreement improved with CAD for both the less than 5% (Fleiss κ, 0.50 vs 0.71; P < .001) and more than 65% (Fleiss κ, 0.54 vs 0.71; P < .001) malignancy risk categories. Overall reader interobserver agreement for management recommendation categories (no action, CT surveillance, diagnostic procedure) also improved with CAD (Fleiss κ, 0.44 vs 0.52; P = .001). Conclusion Use of computer-aided diagnosis improved estimation of indeterminate pulmonary nodule malignancy risk on chest CT scans and improved interobserver agreement for both risk stratification and management recommendations. © RSNA, 2022 Online supplemental material is available for this article. See also the editorial by Yanagawa in this issue.

Multireader Diagnostic Accuracy Imaging Studies: Fundamentals of Design and Analysis.

The design and analysis of multireader multicase (MRMC) studies are quite challenging. These studies differ from most medical studies because they need a reference standard and sampling from two populations (ie, reader and patient populations). They are quite expensive to conduct, requiring a good deal of readers' time for image interpretation. One common problem is the use of imperfect reference standards, often correlated with the test or tests being evaluated. Another common issue is oversimplification of the multidimensional MRMC data. In this study, the fundamentals of MRMC study design and analysis are reviewed. The goal is to provide investigators with a guide to the fundamentals of MRMC design and analysis, with references to more detailed discussions. In addition, readers are updated on newer areas of research, including correction for studies with multiple diagnostic accuracy end points and adjustment for location bias.

Can a Computer-Aided Mass Diagnosis Model Based on Perceptive Features Learned From Quantitative Mammography Radiology Reports Improve Junior Radiologists' Diagnosis Performance? An Observer Study.

Radiologists’ diagnostic capabilities for breast mass lesions depend on their experience. Junior radiologists may underestimate or overestimate Breast Imaging Reporting and Data System (BI-RADS) categories of mass lesions owing to a lack of diagnostic experience. The computer-aided diagnosis (CAD) method assists in improving diagnostic performance by providing a breast mass classification reference to radiologists. This study aims to evaluate the impact of a CAD method based on perceptive features learned from quantitative BI-RADS descriptions on breast mass diagnosis performance. We conducted a retrospective multi-reader multi-case (MRMC) study to assess the perceptive feature-based CAD method. A total of 416 digital mammograms of patients with breast masses were obtained from 2014 through 2017, including 231 benign and 185 malignant masses, from which we randomly selected 214 cases (109 benign, 105 malignant) to train the CAD model for perceptive feature extraction and classification. The remaining 202 cases were enrolled as the test set for evaluation, of which 51 patients (29 benign and 22 malignant) participated in the MRMC study. In the MRMC study, we categorized six radiologists into three groups: junior, middle-senior, and senior. They diagnosed 51 patients with and without support from the CAD model. The BI-RADS category, benign or malignant diagnosis, malignancy probability, and diagnosis time during the two evaluation sessions were recorded. In the MRMC evaluation, the average area under the curve (AUC) of the six radiologists with CAD support was slightly higher than that without support (0.896 vs. 0.850, p = 0.0209). Both average sensitivity and specificity increased (p = 0.0253). Under CAD assistance, junior and middle-senior radiologists adjusted the assessment categories of more BI-RADS 4 cases. The diagnosis time with and without CAD support was comparable for five radiologists. The CAD model improved the radiologists’ diagnostic performance for breast masses without prolonging the diagnosis time and assisted in a better BI-RADS assessment, especially for junior radiologists.

Automated tumor proportion score analysis for PD-L1 (22C3) expression in lung squamous cell carcinoma

Programmed cell death ligend-1 (PD-L1) expression by immunohistochemistry (IHC) assays is a predictive marker of anti-PD-1/PD-L1 therapy response. With the popularity of anti-PD-1/PD-L1 inhibitor drugs, quantitative assessment of PD-L1 expression becomes a new labor for pathologists. Manually counting the PD-L1 positive stained tumor cells is an obviously subjective and time-consuming process. In this paper, we developed a new computer aided Automated Tumor Proportion Scoring System (ATPSS) to determine the comparability of image analysis with pathologist scores. A three-stage process was performed using both image processing and deep learning techniques to mimic the actual diagnostic flow of the pathologists. We conducted a multi-reader multi-case study to evaluate the agreement between pathologists and ATPSS. Fifty-one surgically resected lung squamous cell carcinoma were prepared and stained using the Dako PD-L1 (22C3) assay, and six pathologists with different experience levels were involved in this study. The TPS predicted by the proposed model had high and statistically significant correlation with sub-specialty pathologists’ scores with Mean Absolute Error (MAE) of 8.65 (95% confidence interval (CI): 6.42–10.90) and Pearson Correlation Coefficient (PCC) of 0.9436 (p < 0.001), and the performance on PD-L1 positive cases achieved by our method surpassed that of non-subspecialty and trainee pathologists. Those experimental results indicate that the proposed automated system can be a powerful tool to improve the PD-L1 TPS assessment of pathologists.

Effect of a comprehensive deep-learning model on the accuracy of chest x-ray interpretation by radiologists: a retrospective, multireader multicase study

Chest x-rays are widely used in clinical practice; however, interpretation can be hindered by human error and a lack of experienced thoracic radiologists. Deep learning has the potential to improve the accuracy of chest x-ray interpretation. We therefore aimed to assess the accuracy of radiologists with and without the assistance of a deep-learning model. In this retrospective study, a deep-learning model was trained on 821 681 images (284 649 patients) from five data sets from Australia, Europe, and the USA. 2568 enriched chest x-ray cases from adult patients (≥16 years) who had at least one frontal chest x-ray were included in the test dataset; cases were representative of inpatient, outpatient, and emergency settings. 20 radiologists reviewed cases with and without the assistance of the deep-learning model with a 3-month washout period. We assessed the change in accuracy of chest x-ray interpretation across 127 clinical findings when the deep-learning model was used as a decision support by calculating area under the receiver operating characteristic curve (AUC) for each radiologist with and without the deep-learning model. We also compared AUCs for the model alone with those of unassisted radiologists. If the lower bound of the adjusted 95% CI of the difference in AUC between the model and the unassisted radiologists was more than -0·05, the model was considered to be non-inferior for that finding. If the lower bound exceeded 0, the model was considered to be superior. Unassisted radiologists had a macroaveraged AUC of 0·713 (95% CI 0·645-0·785) across the 127 clinical findings, compared with 0·808 (0·763-0·839) when assisted by the model. The deep-learning model statistically significantly improved the classification accuracy of radiologists for 102 (80%) of 127 clinical findings, was statistically non-inferior for 19 (15%) findings, and no findings showed a decrease in accuracy when radiologists used the deep-learning model. Unassisted radiologists had a macroaveraged mean AUC of 0·713 (0·645-0·785) across all findings, compared with 0·957 (0·954-0·959) for the model alone. Model classification alone was significantly more accurate than unassisted radiologists for 117 (94%) of 124 clinical findings predicted by the model and was non-inferior to unassisted radiologists for all other clinical findings. This study shows the potential of a comprehensive deep-learning model to improve chest x-ray interpretation across a large breadth of clinical practice. Annalise.ai.

Impact of artificial intelligence support on accuracy and reading time in breast tomosynthesis image interpretation: a multi-reader multi-case study

ObjectivesDigital breast tomosynthesis (DBT) increases sensitivity of mammography and is increasingly implemented in breast cancer screening. However, the large volume of images increases the risk of reading errors and reading time. This study aims to investigate whether the accuracy of breast radiologists reading wide-angle DBT increases with the aid of an artificial intelligence (AI) support system. Also, the impact on reading time was assessed and the stand-alone performance of the AI system in the detection of malignancies was compared to the average radiologist.MethodsA multi-reader multi-case study was performed with 240 bilateral DBT exams (71 breasts with cancer lesions, 70 breasts with benign findings, 339 normal breasts). Exams were interpreted by 18 radiologists, with and without AI support, providing cancer suspicion scores per breast. Using AI support, radiologists were shown examination-based and region-based cancer likelihood scores. Area under the receiver operating characteristic curve (AUC) and reading time per exam were compared between reading conditions using mixed-models analysis of variance.ResultsOn average, the AUC was higher using AI support (0.863 vs 0.833; p = 0.0025). Using AI support, reading time per DBT exam was reduced (p < 0.001) from 41 (95% CI = 39–42 s) to 36 s (95% CI = 35– 37 s). The AUC of the stand-alone AI system was non-inferior to the AUC of the average radiologist (+0.007, p = 0.8115).ConclusionsRadiologists improved their cancer detection and reduced reading time when evaluating DBT examinations using an AI reading support system.Key Points• Radiologists improved their cancer detection accuracy in digital breast tomosynthesis (DBT) when using an AI system for support, while simultaneously reducing reading time.• The stand-alone breast cancer detection performance of an AI system is non-inferior to the average performance of radiologists for reading digital breast tomosynthesis exams.• The use of an AI support system could make advanced and more reliable imaging techniques more accessible and could allow for more cost-effective breast screening programs with DBT.

An Exploratory Multi-reader, Multi-case Study Comparing Transmission Ultrasound to Mammography on Recall Rates and Detection Rates for Breast Cancer Lesions

Three-dimensional Quantitative Transmission (QT) ultrasound imaging is an emerging modality for improving the detection and diagnosis of breast cancer. QT ultrasound has high resolution and high contrast to noise ratio, making it effective in evaluating breast tissue. This study compares radiologists' performance of noncancer recall rates and lesion detection rates using QT Ultrasound versus full-field digital mammography (FFDM) in a cross section of female subjects. In this multi-reader multi-case (MRMC) study, we examined retrospective data from two clinical trials conducted at five sites. All subjects received FFDM and QT scans within 90 days. Data were analyzed in a reader study with full factorial design involving 22 radiologists and 108 breast cases (42 normal, 39 pathology-confirmed benign, and 27 pathology-confirmed cancer cases). The main results used a random-reader random-case analysis adjusted for location bias performed after a primary predefined random-reader fixed-case analysis. The readers' mean rate of detecting lesions of any type was 4% higher (p-value > 0.05) with QT imaging. The mean non-cancer recall rate improved significantly, showing a decrease of 16% with QT (p-value = 0.03), at the expense of a 2% decrease in the mean cancer recall rate (p-value >0.05) in comparison to FFDM. Combining performance on cancer and noncancer recall rates, the mean area under the receiver operator curve of confidence scores improved significantly by 10% with QT (p-value = 0.01). This MRMC study indicates that QT improves non-cancer recall rates without substantially affecting cancer recall rates. The main limitation is the small number of cases from retrospective data. A larger prospective MRMC study is warranted for further assessment.

Multi-Reader Multi-Case Study for Performance Evaluation of High-Risk Thyroid Ultrasound with Computer-Aided Detection

Physicians use sonographic characteristics as a reference for the possible diagnosis of thyroid cancers. The purpose of this study was to investigate whether physicians were more effective in their tentative diagnosis based on the information provided by a computer-aided detection (CAD) system. A computer compared software-defined and physician-adjusted tumor loci. A multicenter, multireader, and multicase (MRMC) study was designed to compare clinician performance without and with the use of CAD. Interobserver variability was also analyzed. Excellent, satisfactory, and poor segmentations were observed in 25.3%, 58.9%, and 15.8% of nodules, respectively. There were 200 patients with 265 nodules in the study set. Nineteen physicians scored the malignancy potential of the nodules. The average area under the curve (AUC) of all readers was 0.728 without CAD and significantly increased to 0.792 with CAD. The average standard deviation of the malignant potential score significantly decreased from 18.97 to 16.29. The mean malignant potential score significantly decreased from 35.01 to 31.24 for benign cases. With the CAD system, an additional 7.6% of malignant nodules would be suggested for further evaluation, and biopsy would not be recommended for an additional 10.8% of benign nodules. The results demonstrated that applying a CAD system would improve clinicians’ interpretations and lessen the variability in diagnosis. However, more studies are needed to explore the use of the CAD system in an actual ultrasound diagnostic situation where much more benign thyroid nodules would be seen.

Evaluation of Pseudoreader Study Designs to Estimate Observer Performance Results as an Alternative to Fully Crossed, Multireader, Multicase Studies

To examine the ability of a pseudoreader study design to estimate the observer performance obtained using a traditional fully crossed, multireader, multicase (MRMC) study. A 10-reader MRMC study with 20 computed tomography datasets was designed to measure observer performance on four novel noise reduction methods. This study served as the foundation for the empirical evaluation of three different pseudoreader designs, each of which used a similar bootstrap approach for generating 2000 realizations from the fully crossed study. Our three approaches to generating a pseudoreader varied in the degree to which reader performance was matched and integrated into the pseudoreader design. One randomly selected simulation was selected as a "mock study" to represent a hypothetical, prospective implementation of the design. Using the traditional fully crossed design, figures of merit) (95% CIs) for the four noise reductions methods were 68.2 (55.5-81.0), 69.6 (58.4-80.8), 70.8 (60.2-81.4), and 70.9 (60.4-81.3), respectively. When radiologists' performances on the fourth noise reduction method were used to pair readers during the mock study, there was strong agreement in the estimated figures of merits with estimates using the pseudoreader design being within ±3% of the fully crossed design. Fully crossed MRMC studies require significant investment in resources and time, often resulting in delayed implementation or minimal human testing before dissemination. The pseudoreader approach accelerates study conduct by combining readers judiciously and was found to provide comparable results to the traditional fully crossed design by making strong assumptions about exchangeability of the readers.

Can Digital Breast Tomosynthesis Replace Full-Field Digital Mammography? A Multireader, Multicase Study of Wide-Angle Tomosynthesis.

OBJECTIVE. The purpose of this study was to test the hypothesis whether two-view wide-angle digital breast tomosynthesis (DBT) can replace full-field digital mammography (FFDM) for breast cancer detection. SUBJECTS AND METHODS. In a multireader multicase study, bilateral two-view FFDM and bilateral two-view wide-angle DBT images were independently viewed for breast cancer detection in two reading sessions separated by more than 1 month. From a pool of 764 patients undergoing screening and diagnostic mammography, 330 patient-cases were selected. The endpoints were the mean ROC AUC for the reader per breast (breast level), ROC AUC per patient (subject level), noncancer recall rates, sensitivity, and specificity. RESULTS. Twenty-nine of 31 readers performed better with DBT than FFDM regardless of breast density. There was a statistically significant improvement in readers' mean diagnostic accuracy with DBT. The subject-level AUC increased from 0.765 (standard error [SE], 0.027) for FFDM to 0.835 (SE, 0.027) for DBT (p = 0.002). Breast-level AUC increased from 0.818 (SE, 0.019) for FFDM to 0.861 (SE, 0.019) for DBT (p = 0.011). The noncancer recall rate per patient was reduced by 19% with DBT (p < 0.001). Masses and architectural distortions were detected more with DBT (p < 0.001); calcifications trended lower (p = 0.136). Accuracy for detection of invasive cancers was significantly greater with DBT (p < 0.001). CONCLUSION. Reader performance in breast cancer detection is significantly higher with wide-angle two-view DBT independent of FFDM, verifying the robustness of DBT as a sole view. However, results of perception studies in the vision sciences support the inclusion of an overview image.

Concurrent Computer-Aided Detection Improves Reading Time of Digital Breast Tomosynthesis and Maintains Interpretation Performance in a Multireader Multicase Study.

Digital breast tomosynthesis (DBT) is more accurate than full-field digital mammography alone but requires a longer reading time. A radiologist reader study evaluated the use of concurrent computer-aided detection (CAD) to shorten the reading time while maintaining interpretation performance. A CAD system was developed to detect suspicious soft-tissue densities in DBT planes. Abnormalities are extracted from the plane in which they are detected and blended into the corresponding synthetic image. The study used an enriched sample of 240 DBT cases with 68 malignancies in 61 patients. Twenty radiologists retrospectively reviewed all 240 cases in a multireader multicase crossover design to compare reading time and performance with and without CAD. The performance of CAD alone was also evaluated. Reading time improved by 29.2% with CAD (95% CI, 21.1-36.5%; p < 0.01). Reader performance, measured by ROC AUC, was noninferior with CAD (p < 0.01). The mean AUC increased from 0.841 without to 0.850 with CAD (95% CI, -0.012 to 0.030). Mean sensitivity increased from 0.847 without to 0.871 with CAD (difference 95% CI, -0.005 to 0.055), showing a 0.033 increase in sensitivity for cases with soft-tissue densities (95% CI, -0.002 to 0.068). Mean specificity decreased from 0.527 without to 0.509 with CAD (difference 95% CI, -0.041 to 0.005), and mean recall rate for noncancers slightly increased from 0.474 without to 0.492 with CAD (difference 95% CI, -0.006 to 0.041). Concurrent use of CAD with DBT resulted in 29.2% faster reading time, while maintaining reader interpretation performance.