Multiple-reader, Multiple-case Research Articles

Assessing Detection Accuracy of Computerized Sonographic Features and Computer-Assisted Reading Performance in Differentiating Thyroid Cancers.

For ultrasound imaging of thyroid nodules, medical guidelines are all based on findings of sonographic features to provide clinicians management recommendations. Due to the recent development of artificial intelligence and machine learning (AI/ML) technologies, there have been computer-assisted detection (CAD) software devices available for clinical use to detect and quantify the sonographic features of thyroid nodules. This study is to validate the accuracy of the computerized sonographic features (CSF) by a CAD software device, namely, AmCAD-UT, and then to assess how the reading performance of clinicians (readers) can be improved providing the computerized features. The feature detection accuracy is tested against the ground truth established by a panel of thyroid specialists and a multiple-reader multiple-case (MRMC) study is performed to assess the sequential reading performance with the assistance of the CSF. Five computerized features, including anechoic area, hyperechoic foci, hypoechoic pattern, heterogeneous texture, and indistinct margin, were tested, with AUCs ranging from 0.888~0.946, 0.825~0.913, 0.812~0.847, 0.627~0.77, and 0.676~0.766, respectively. With the five CSFs, the sequential reading performance of 18 clinicians is found significantly improved, with the AUC increasing from 0.720 without CSF to 0.776 with CSF. Our studies show that the computerized features are consistent with the clinicians’ findings and provide additional value in assisting sonographic diagnosis.

Diagnostic value of the stand-alone synthetic image in digital breast tomosynthesis examinations.

To demonstrate the non-inferiority of synthetic image (SI) mammography versus full-field digital mammography (FFDM) in breast tomosynthesis (DBT) examinations. An observational, retrospective, single-centre, multireader blinded study was performed, using 2384 images to directly compare SI and FFDM based on Breast Imaging Reporting and Data System (BIRADS) categorisation and visibility of radiological findings. Readers had no access to digital breast tomosynthesis slices. Multiple reader, multiple case (MRMC) receiver operating characteristic (ROC) methodology was used to compare the diagnostic performance of SI and FFDM images. The kappa statistic was used to estimate the inter-reader and intra-reader reliability. The area under the ROC curves (AUC) reveals the non-inferiority of SI versus FFDM based on BIRADS categorisation [difference between AUC (ΔAUC), -0.014] and lesion visibility (ΔAUC, -0.001) but the differences were not statistically significant (p=0.282 for BIRADS; p=0.961 for lesion visibility). On average, 77.4% of malignant lesions were detected with SI versus 76.5% with FFDM. Sensitivity and specificity of SI are superior to FFDM for malignant lesions scored as BIRADS 5 and breasts categorised as BIRADS 1. SI is not inferior to FFDM when DBT slices are not available during image reading. SI can replace FFDM, reducing the dose by 45%. • Stand-alone SI demonstrated performance not inferior for lesion visibility as compared to FFDM. • Stand-alone SI demonstrated performance not inferior for lesion BIRADS categorisation as compared to FFDM. • Synthetic images provide important dose savings in breast tomosynthesis examinations.

Lymph Node Metastases from Gastric Cancer: Gadofluorine M and Gadopentetate Dimeglumine MR Imaging in a Rabbit Model

To compare the diagnostic performance of gadofluorine M with that of gadopentetate dimeglumine in the diagnosis of lymph node metastases with magnetic resonance (MR) imaging in a rabbit model of gastric cancer. The study protocol was approved by the institutional animal care committee. VX2 carcinomas were inoculated into the wall of the stomach in 20 rabbits. Gadopentetate dimeglumine-enhanced MR imaging was performed 4-6 weeks after inoculation, and gadofluorine M-enhanced MR imaging was performed approximately 24 hours later. Both MR imaging sets were analyzed separately and independently by four radiologists with respect to confidence level regarding the presence of metastases in lymph nodes and lymph node conspicuity. Statistical analysis was performed by using multiple-reader multiple-case (MRMC) receiver operating characteristic curve analysis and the Wilcoxon signed rank test. Metastases were confirmed at pathologic analysis in 32 of 104 lymph nodes from 16 rabbits. The area under the receiver operating characteristic curve (AUC) for confidence regarding the presence of metastases in lymph nodes was significantly greater for gadofluorine M than for gadopentetate dimeglumine (AUC, 0.947 vs 0.894; P = .009). However, most (81%, 25 of 32) metastatic nodes were necrotic, and no significant difference was obtained in nonnecrotic nodes. For lymph node conspicuity, the gadofluorine M MR imaging set was assigned a significantly higher score than was the gadopentetate dimeglumine MR imaging set by all readers (P < .001). Gadofluorine M showed significantly higher accuracy and better conspicuity than gadopentetate dimeglumine in the diagnosis of metastatic nodes, most of which were necrotic, in this animal model of gastric cancer.

Prediction Accuracy of a Sample-size Estimation Method for ROC Studies

Sample-size estimation is an important consideration when planning a receiver operating characteristic (ROC) study. The aim of this work was to assess the prediction accuracy of a sample-size estimation method using the Monte Carlo simulation method. Two ROC ratings simulators characterized by low reader and high case variabilities (LH) and high reader and low case variabilities (HL) were used to generate pilot data sets in two modalities. Dorfman-Berbaum-Metz multiple-reader multiple-case (DBM-MRMC) analysis of the ratings yielded estimates of the modality-reader, modality-case, and error variances. These were input to the Hillis-Berbaum (HB) sample-size estimation method, which predicted the number of cases needed to achieve 80% power for 10 readers and an effect size of 0.06 in the pivotal study. Predictions that generalized to readers and cases (random-all), to cases only (random-cases), and to readers only (random-readers) were generated. A prediction-accuracy index defined as the probability that any single prediction yields true power in the 75%-90% range was used to assess the HB method. For random-case generalization, the HB-method prediction-accuracy was reasonable, approximately 50% for five readers and 100 cases in the pilot study. Prediction-accuracy was generally higher under LH conditions than under HL conditions. Under ideal conditions (many readers in the pilot study) the DBM-MRMC-based HB method overestimated the number of cases. The overestimates could be explained by the larger modality-reader variance estimates when reader variability was large (HL). The largest benefit of increasing the number of readers in the pilot study was realized for LH, where 15 readers were enough to yield prediction accuracy >50% under all generalization conditions, but the benefit was lesser for HL where prediction accuracy was approximately 36% for 15 readers under random-all and random-reader conditions. The HB method tends to overestimate the number of cases. Random-case generalization had reasonable prediction accuracy. Provided about 15 readers were used in the pilot study the method performed reasonably under all conditions for LH. When reader variability was large, the prediction-accuracy for random-all and random-reader generalizations was compromised. Study designers may wish to compare the HB predictions to those of other methods and to sample-sizes used in previous similar studies.

Evaluating the decision accuracy and speed of clinical data visualizations

Clinicians face an increasing volume of biomedical data. Assessing the efficacy of systems that enable accurate and timely clinical decision making merits corresponding attention. This paper discusses the multiple-reader multiple-case (MRMC) experimental design and linear mixed models as means of assessing and comparing decision accuracy and latency (time) for decision tasks in which clinician readers must interpret visual displays of data. These tools can assess and compare decision accuracy and latency (time). These experimental and statistical techniques, used extensively in radiology imaging studies, offer a number of practical and analytic advantages over more traditional quantitative methods such as percent-correct measurements and ANOVAs, and are recommended for their statistical efficiency and generalizability. An example analysis using readily available, free, and commercial statistical software is provided as an appendix. While these techniques are not appropriate for all evaluation questions, they can provide a valuable addition to the evaluative toolkit of medical informatics research.

Digital breast tomosynthesis versus digital mammography: a clinical performance study

To compare the clinical performance of digital breast tomosynthesis (DBT) with that of full-field digital mammography (FFDM) in a diagnostic population. The study enrolled 200 consenting women who had at least one breast lesion discovered by mammography and/or ultrasound classified as doubtful or suspicious or probably malignant. They underwent tomosynthesis in one view [mediolateral oblique (MLO)] of both breasts at a dose comparable to that of standard screen-film mammography in two views [craniocaudal (CC) and MLO]. Images were rated by six breast radiologists using the BIRADS score. Ratings were compared with the truth established according to the standard of care and a multiple-reader multiple-case (MRMC) receiver-operating characteristic (ROC) analysis was performed. Clinical performance of DBT compared with that of FFDM was evaluated in terms of the difference between areas under ROC curves (AUCs) for BIRADS scores. Overall clinical performance with DBT and FFDM for malignant versus all other cases was not significantly different (AUCs 0.851 vs 0.836, p = 0.645). The lower limit of the 95% CI or the difference between DBT and FFDM AUCs was -4.9%. Clinical performance of tomosynthesis in one view at the same total dose as standard screen-film mammography is not inferior to digital mammography in two views.

A Probabilistic Model for the MRMC Method, Part 1: Theoretical Development

Current approaches to receiver operating characteristic (ROC) analysis use the MRMC (multiple-reader, multiple-case) paradigm in which several readers read each case and their ratings (or scores) are used to construct an estimate of the area under the ROC curve or some other ROC-related parameter. Standard practice is to decompose the parameter of interest according to a linear model into terms that depend in various ways on the readers, cases, and modalities. Though the methodologic aspects of MRMC analysis have been studied in detail, the literature on the probabilistic basis of the individual terms is sparse. In particular, few articles state what probability law applies to each term and what underlying assumptions are needed for the assumed independence. When probability distributions are specified for these terms, these distributions are assumed to be Gaussians. This article approaches the MRMC problem from a mechanistic perspective. For a single modality, three sources of randomness are included: the images, the reader skill, and the reader uncertainty. The probability law on the reader scores is written in terms of three nested conditional probabilities, and random variables associated with this probability are referred to as triply stochastic. In this article, we present the probabilistic MRMC model and apply this model to the Wilcoxon statistic. The result is a seven-term expansion for the variance of the figure of merit. We relate the terms in this expansion to those in the standard, linear MRMC model. Finally, we use the probabilistic model to derive constraints on the coefficients in the seven-term expansion.

Effect of soft-copy display supported by CAD on mammography screening performance

Diagnostic performance and reading speed for conventional mammography film reading is compared to reading digitized mammograms on a dedicated workstation. A series of mammograms judged negative at screening and corresponding priors were collected. Half were diagnosed as cancer at the next screening, or earlier for interval cancers. The others were normal. Original films were read by fifteen experienced screening radiologists. The readers annotated potential abnormalities and estimated their likelihood of malignancy. More than 1 year later, five radiologists reread a subset of 271 cases (88 cancer cases having visible signs in retrospect and 183 normals) on a mammography workstation after film digitization. Markers from a computer-aided detection (CAD) system for microcalcifications were available to the readers. Performance was evaluated by comparison of A(z)-scores based on ROC and multiple-Reader multiple-case (MRMC) analysis, and localized receiver operating characteristic (LROC) analysis for the 271 cases. Reading speed was also determined. No significant difference in diagnostic performance was observed between conventional and soft-copy reading. Average A(z)-scores were 0.83 and 0.84 respectively. Soft-copy reading was only slightly slower than conventional reading. Using a mammography workstation including CAD for detection of microcalcifications, soft-copy reading is possible without loss of quality or efficiency.

Solitary pulmonary nodule diagnosis on CT: Results of an observer study 1

To investigate the performance of observers with different levels of experience in distinguishing between benign and malignant solitary pulmonary nodules (SPN) on CT, and to determine the effects on interpretation of three different conditions: image data alone, the addition of clinical data, and the addition of output from a computer-aided diagnosis (CAD) system. 28 thin-section CT datasets of SPNs with proven diagnoses (15 malignant and 13 benign) were used to measure observer performance. Readers were categorized according to their experience and read the cases in random order. For each case readers were asked to assign a level of confidence on a scale from 0.0-1.0 (0.0 benign, 1.0 malignant) for the diagnosis of the nodule. Each reader scored the cases based on review of image data alone (phase 1), then with limited clinical data (phase 2), and finally with CAD output (phase 3). To assess performance, multiple reader multiple case (MRMC) receiver operating characteristic (ROC) analysis was used. 2 thoracic radiologists, 1 thoracic radiology fellow, 2 nonthoracic radiologists, and 3 radiology residents read the cases. The average area under the ROC curve for all readers (A(z)) at each stage was 0.68, 0.75, and 0.81, for image data alone, with clinical data, and with CAD output respectively. The difference in performance between phases (2 and 3) and (1 and 3) was significantly different (P = 0.018 and P = 0.020). However, the difference between phases (1 and 2) was not significantly different (P = 0.155). Diagnostic performance increased significantly with the addition of CAD output. With further validation CAD output may play a significant role in SPN management.

Reader Variability in Mammography and Its Implications for Expected Utility over the Population of Readers and Cases

The multiple-reader, multiple-case (MRMC) approach to receiver operating characteristic (ROC) analysis is becoming the dominant assessment paradigm in medical imaging. Its most common version involves having many readers read every patient case in the study, a critical feature since differences among competing imaging modalities are often dominated by differences in reader performance. The present authors have carried out MRMC ROC analysis on a uniquely large data set for mammography. The analysis quantifies the great range of observed reader skill in that data set. It also demonstrates that the sample sizes are sufficiently large that the conclusions generalize to the populations sampled here with little uncertainty from the finite sample size. A schematic approach to bracketing the utility matrix is then used to study trends in the resulting expected utility functions that correspond to the range of observed ROC curves. This is done for both the screening and the diagnostic context. The results raise 2 hypotheses for further investigation. First, it is possible that the present ambiguity surrounding the effectiveness of mammography is due in part to the observed range of reader skills and corresponding expected utility functions. Second, it is possible that computer-assisted modalities for mammography may lead to improvements in the expected utility function not only for screening but also in the diagnostic context, especially for the lower performing readers.

Assessment of Medical Imaging and Computer-Assist Systems: Lessons from Recent Experience

In the last 2 decades major advances have been made in the field of assessment methods for medical imaging and computer–assist systems through the use of the paradigm of the receiver operating characteristic (ROC) curve. In the most recent decade this methodology was extended to embrace the complication of reader variability through a dvances in the multiple–reader, multiple–case (MRMC) ROC measurement and analysis paradigm. Although this approach has been widely adopted by the imaging research community, some investigators appear averse to it, possibly from concern that it could place a greater burden on the scarce resources of patient cases and readers compared to t he requirements of alternative methods. The present communication argues, however, that the MRMC ROC approach to assessment in the context of reader variability may be the most resource–efficient approach available. Moreover, alternative approaches may also be statistically uninterpretable with regard to estimated summary measures of performa nce and their uncertainties. The authors propose that the MRMC ROC approach be considered even more widely by the larger community with responsibilities for the introduction and dissemination of medical imaging technologies to society. General principles of study design are reviewed, and important contemporary clinical trials are used as examples.