Multi-reader Multi-case Analysis Research Articles

Roe and Metz identical-test simulation model for validating multi-reader methods of analysis for comparing different radiologic imaging modalities.

The most frequently used model for simulating multi-reader multi-case (MRMC) data that emulate confidence-of-disease ratings from diagnostic imaging studies has been the Roe and Metz model, proposed by Roe and Metz in 1997 and later generalized by Hillis (2012), Abbey etal. (2013), and Gallas and Hillis (2014). These models have been used for evaluating MRMC analysis and sample size methods. The models suggested in these papers for assessing type I error have been null models, where the expected area under the receiver-operating-characteristic curve across readers is the same for each test. However, for these null models, there are other differences that would not exist if the two tests were identical. None of the papers mentioned above discuss how to formulate a null model that is also an identical-test model, where the two tests are identical in all respects. The purpose of this paper is to show how to formulate a Roe and Metz identical-test model and to show its usefulness for validating the error covariance constraints employed by the Obuchowski-Rockette (1995) method. For a given Roe-and-Metz model, the corresponding Roe-and-Metz identical-test model is derived by modifying the Roe and Metz null model under the assumption that the two tests are identical. The importance of the Obuchowski-Rockette model constraints for avoiding negative variance estimates is established using data simulated from the Roe and Metz identical-test model. It is also shown that negative variance estimates can occur at nontrivial rates when the two tests are not identical but somewhat "close" to being identical. The findings of this paper are important because it has recently been shown (Hillis, 2022) that the commonly used MRMC method proposed by Gallas (2006) and Gallas etal. (2009) uses the same test statistic as the unconstrained Obuchowski-Rockette method.

Osteomyelitis of the Lower Limb: Diagnostic Accuracy of Dual-Energy CT versus MRI.

MRI is the preferred imaging technique for the identification of osteomyelitis. The key element for diagnosis is the presence of bone marrow edema (BME). Dual-energy CT (DECT) is an alternative tool which is able to identify BME in the lower limb. To compare the diagnostic performance of DECT and MRI for osteomyelitis, using clinical, microbiological, and imaging data as reference standards. This prospective single-center study enrolled consecutive patients with suspected bone infections undergoing DECT and MRI imaging from December 2020 to June 2022. Four blinded radiologists with various experience levels (range of 3-21 years) evaluated the imaging findings. Osteomyelitis was diagnosed in the presence of BMEs, abscesses, sinus tracts, bone reabsorption, or gaseous elements. The sensitivity, specificity, and AUC values of each method were determined and compared using a multi-reader multi-case analysis. A p value < 0.05 was considered significant. In total, 44 study participants (mean age 62.5 years ± 16.5 [SD], 32 men) were evaluated. Osteomyelitis was diagnosed in 32 participants. For the MRI, the mean sensitivity and specificity were 89.1% and 87.5%, while for the DECT they were 89.0% and 72.9%, respectively. The DECT demonstrated a good diagnostic performance (AUC = 0.88), compared with the MRI (AUC = 0.92) (p = 0.12). When considering each imaging finding alone, the best accuracy was achieved by considering BME (AUC for DECT 0.85 versus AUC of MRI of 0.93, with p = 0.07), followed by the presence of bone erosions (AUC 0.77 for DECT and 0.53 for MRI, with p = 0.02). The inter-reader agreement of the DECT (k = 88) was similar to that of the MRI (k = 90). Dual-energy CT demonstrated a good diagnostic performance in detecting osteomyelitis.

Determining Roe and Metz model parameters for simulating multireader multicase confidence-of-disease rating data based on real-data or conjectured Obuchowski-Rockette parameter estimates.

Purpose: The most frequently used model for simulating multireader multicase (MRMC) data that emulates confidence-of-disease ratings from diagnostic imaging studies has been the Roe and Metz (RM) model, proposed by Roe and Metz in 1997 and later generalized by Hillis (2012), Abbey etal. (2013), and Gallas and Hillis (2014). A problem with these models is that it has been difficult to set model parameters such that the simulated data are similar to MRMC data encountered in practice. To remedy this situation, Hillis (2018) mapped parameters from the RM model to Obuchowski-Rockette (OR) model parameters that describe the distribution of the empirical AUC outcomes computed from the RM model simulated data. We continue that work by providing the reverse mapping, i.e., by deriving an algorithm that expresses RM parameters as functions of the OR empirical AUC distribution parameters. Approach: We solve for the corresponding RM parameters in terms of the OR parameters using numerical methods. Results: An algorithm is developed that results in, at most, one solution of RM parameter values that correspond to inputted OR parameter values. The algorithm can be implemented using an R software function. Examples are provided that illustrate the use of the algorithm. A simulation study validates the algorithm. Conclusions: The resulting algorithm makes it possible to easily determine RM model parameter values such that simulated data emulate a specific real-data study. Thus, MRMC analysis methods can be empirically tested using simulated data similar to that encountered in practice.

Hypothesis testing methods for multi-reader multi-case studies

ABSTRACT Rationale and Objectives: Multi-reader multi-case (MRMC) studies are widely used to compare the diagnostic accuracies of different imaging modalities. Although Dorfman–Berbaum–Metz (DBM) method is the most popular one among the MRMC methods, the adaptions of ANOVA statistic for linear mixed model (LMM) are not based on solid theory and the assumption of ANOVA that all groups have the same number of samples might not be met in some situations. The purpose of the article is to investigate whether the statistics for testing fixed effect in linear mixed model can yield a closer type I error rate to nominal level. Materials and Methods: We proposed to use the statistics such as likelihood ratio test (LRT) and Wald statistic to test the hypothesis of equivalence in several imaging modalities. Extensive simulations were conducted and the application to a clinical example dataset was illustrated. Results: The simulation results showed the type I error rates of Wald statistic were closer to the nominal level under many simulated situations, especially when the simulated data was ordinal and the number of diseased and non-diseased were 100. Conclusion: The Wald statistic whose degrees of freedom ware approximated by Satterthwaite's method showed competitive performance, indicating the potential of the statistic applied in DBM model for MRMC analysis.

Identical-test Roe and Metz simulation model for validating multi-reader methods of analysis for comparing different radiologic imaging modalities.

The most frequently used model for simulating multireader multicase (MRMC) data that emulate confidence-of-disease ratings from diagnostic imaging studies has been the Roe and Metz model, proposed by Roe and Metz in 1997 and later generalized by Hillis (2012), Abbey et al (2013) and Gallas and Hillis (2014). All of these models generate continuous confidence-of-disease ratings based on an underlying binormal model for each reader, with the separation between the normal and abnormal rating distributions varying across readers. Numerous studies have used these models for evaluating MRMC analysis and sample size methods. The models suggested in these papers for assessing type I error have been "null" models, where the expected AUC across readers is the same for each test. However, for the null models that have been suggested, there are other differences that would not exist if the two tests were identical. None of the papers cited above discuss how to formulate a null model that is also an "identical-test" model, where the two tests are identical in all respects. The purpose of this paper is to show how to formulate an identical-test model and to discuss the importance of this model. Using the identical-test model, I show through simulations the importance of the Obuchowski-Rockette model constraints to avoid a negative variance estimate, a result which had not previously been empirically demonstrated.

Multi-reader multi-case analysis of variance software for diagnostic performance comparison of imaging modalities.

A common study design for comparing the diagnostic performance of imaging modalities is to obtain modality-specific ratings from multiple readers of multiple cases whose true statuses are known. Typically, there is overlap between the modalities, readers, and/or cases for which special analytical methods are needed to perform statistical comparisons. We describe our new R software package MRMCaov, which is designed for multi-reader multi-case comparisons of two or more imaging modalities. The software allows for the comparison of reader performance metrics, such as area under the receiver operating characteristic curve (ROC AUC), with analysis of variance methods originally proposed by Obuchowski and Rockette (1995) and later unified and improved by Hillis and colleagues (2005, 2007, 2008, 2018). MRMCaov is an open-source package with an integrated command-line interface for performing multi-reader multi-case statistical analysis, plotting, and presenting results. Features of the package include (1) ROC curves estimated parametrically or non-parametrically; (2) reader-specific ROC curves and performance metrics; (3) user-definable performance metrics; (4) modality-specific estimates of mean performance along with confidence intervals and p-values for statistical comparisons; (5) support for factorial, nested, or partially paired study designs; (6) inference for random readers and cases, random readers and fixed cases, or fixed readers and random cases; (7) DeLong, jackknife, or unbiased covariance estimation; and (8) compatibility with Microsoft Windows, Mac OS, and Linux.

Verification of modified receiver-operating characteristic software using simulated rating data.

ROCKIT, which is a receiver-operating characteristic (ROC) curve-fitting software package, was developed by Metz et al. In the early 1990s, it is a very frequently used ROC software throughout the world. In addition to ROCKIT, DBM-MRMC software was developed for multi-reader multi-case analysis of the difference in average area under ROC curves (AUCs). Because this old software cannot run on a PC with Windows 7 or a more recent operating system, we developed new software that employs the same basic algorithms with minor modifications. In this study, we verified our modified software and tested the differences between the index of diagnostic accuracies using simulated rating data. In our simulation model, all data were generated using target AUCs and a binormal parameter b. In ROC curve fitting with simulated rating data, we varied four factors: the total number of case samples, the ratio of positive-to-negative cases, a binormal parameter b, and the preset AUC. To investigate the differences between the statistical test results obtained from our software and the existing software, we generated simulated rating data sets with three levels of case difficulty and three degrees of difference in AUCs obtained from two modalities. As a result of the simulation, the AUCs estimated by the new and existing software were highly correlated (R > 0.98), and there were high agreements (85% or more) in the statistical test results. In conclusion, we believe that our modified software is as capable as the existing software.

Comparison of breast cancer detection and depiction between planar and rotating synthetic mammography generated from breast tomosynthesis

PurposeTo compare breast cancer detection and depiction between planar synthetic mammography (SM) and rotating synthetic mammography (RM) generated from digital breast tomosynthesis (DBT). Materials and methodsIn a fully-crossed multi-reader multi-case (MRMC) study, three radiologists retrospectively reviewed 190 cases (27 malignant, 31 benign, 132 normal), once with SM alone and once with RM alone, the DBT stack of slices was not reviewed. Lesions were scored using BI-RADS® and level of suspiciousness (1–10). Sensitivity, specificity, and area under the receiver operating characteristic curve (AUC) were computed using MRMC Analysis of Variance using the open-access software iMRMC. Additionally, readers were asked to make a visual grading analysis (VGA) on visibility of calcifications and soft tissue lesions (1–5 scale with 5 = Excellent visualization). The VGA scores were analyzed using the visual grading characteristics (VGC) method. ResultsOn average, the AUC was similar between SM and RM (0.66 versus 0.67, P = 0.818). The sensitivity was equivalent (0.62 versus 0.60, P = 0.794), while specificity was significantly lower in SM than in RM (0.66 versus 0.72, P = 0.028). Radiologists significantly (P < 0.05) preferred the display of all types of lesions in RM over SM. The average reading time per case was higher for RM than for SM (30 s versus 23 s, P < 0.05). ConclusionRadiologists achieve similar cancer detection with RM as with SM. They prefer the 3D-like rotating representation of soft tissue lesions and calcifications in comparison to the 2D visualization, which might improve their specificity, but at the expense of longer reading time.

MRMC analysis of agreement studies.

The purpose of this work is to present and evaluate methods based on U-statistics to compare intra- or inter-reader agreement across different imaging modalities. We apply these methods to multi-reader multi-case (MRMC) studies. We measure reader-averaged agreement and estimate its variance accounting for the variability from readers and cases (an MRMC analysis). In our application, pathologists (readers) evaluate patient tissue mounted on glass slides (cases) in two ways. They evaluate the slides on a microscope (reference modality) and they evaluate digital scans of the slides on a computer display (new modality). In the current work, we consider concordance as the agreement measure, but many of the concepts outlined here apply to other agreement measures. Concordance is the probability that two readers rank two cases in the same order. Concordance can be estimated with a U-statistic and thus it has some nice properties: it is unbiased, asymptotically normal, and its variance is given by an explicit formula. Another property of a U-statistic is that it is symmetric in its inputs; it doesn't matter which reader is listed first or which case is listed first, the result is the same. Using this property and a few tricks while building the U-statistic kernel for concordance, we get a mathematically tractable problem and efficient software. Simulations show that our variance and covariance estimates are unbiased.

Comparative Statistical Properties of Expected Utility and Area Under the ROC Curve for Laboratory Studies of Observer Performance in Screening Mammography

Our objective is to determine whether expected utility (EU) and the area under the receiver operator characteristic (AUC) are consistent with one another as endpoints of observer performance studies in mammography. These two measures characterize receiver operator characteristic performance somewhat differently. We compare these two study endpoints at the level of individual reader effects, statistical inference, and components of variance across readers and cases. We reanalyze three previously published laboratory observer performance studies that investigate various x-ray breast imaging modalities using EU and AUC. The EU measure is based on recent estimates of relative utility for screening mammography. The AUC and EU measures are correlated across readers for individual modalities (r = 0.93) and differences in modalities (r = 0.94 to 0.98). Statistical inference for modality effects based on multi-reader multi-case analysis is very similar, with significant results (P < .05) in exactly the same conditions. Power analyses show mixed results across studies, with a small increase in power on average for EU that corresponds to approximately a 7% reduction in the number of readers. Despite a large number of crossing receiver operator characteristic curves (59% of readers), modality effects only rarely have opposite signs for EU and AUC (6%). We do not find any evidence of systematic differences between EU and AUC in screening mammography observer studies. Thus, when utility approaches are viable (i.e., an appropriate value of relative utility exists), practical effects such as statistical efficiency may be used to choose study endpoints.

New methods for using computer-aided detection information for the detection of lung nodules on chest radiographs

To investigate two new methods of using computer-aided detection (CAD) system information for the detection of lung nodules on chest radiographs. We evaluated an interactive CAD application and an independent combination of radiologists and CAD scores. 300 posteroanterior and lateral digital chest radiographs were selected, including 111 with a solitary pulmonary nodule (average diameter, 16 mm). Both nodule and control cases were verified by CT. Six radiologists and six residents reviewed the chest radiographs without CAD and with CAD (ClearRead +Detect™ 5.2; Riverain Technologies, Miamisburg, OH) in two reading sessions. The CAD system was used in an interactive manner; CAD marks, accompanied by a score of suspicion, remained hidden unless the location was queried by the radiologist. Jackknife alternative free response receiver operating characteristics multireader multicase analysis was used to measure detection performance. Area under the curve (AUC) and partial AUC (pAUC) between a specificity of 80% and 100% served as the measure for detection performance. We also evaluated the results of a weighted combination of CAD scores and reader scores, at the location of reader findings. AUC for the observers without CAD was 0.824. No significant improvement was seen with interactive use of CAD (AUC = 0.834; p = 0.15). Independent combination significantly improved detection performance (AUC = 0.834; p = 0.006). pAUCs without and with interactive CAD were similar (0.128), but improved with independent combination (0.137). Interactive CAD did not improve reader performance for the detection of lung nodules on chest radiographs. Independent combination of reader and CAD scores improved the detection performance of lung nodules. (1) Interactive use of currently available CAD software did not improve the radiologists' detection performance of lung nodules on chest radiographs. (2) Independently combining the interpretations of the radiologist and the CAD system improved detection of lung nodules on chest radiographs.

An Analytic Expression for the Binormal Partial Area under the ROC Curve

The partial area under the receiver operating characteristic (ROC) curve (pAUC) is a useful summary measure for diagnostic studies. Unlike most summary measures that are functions of the ROC curve, researchers have not been aware of an analytic expression available for computing the pAUC for an ROC curve based on a latent binormal model. Instead, the pAUC has been computed using numerical integration or a rational polynomial approximation. Our purpose is to provide analytic expressions for the two forms of pAUC. We discuss the two fundamentally different types of pAUC. We present analytic expressions for both types, provide corresponding proofs, and illustrate their application with an example comparing the performances of spin echo and cine magnetic resonance imaging for detecting thoracic aortic dissection. We provide an example of using the pAUC as the outcome in a multireader multicase analysis. We find that using the pAUC results in a more significant finding. We have provided analytic expressions for both types of pAUC, making it easier to compute the pAUCs corresponding to binormal ROC curves.

Comparison of conventional and automated breast volume ultrasound in the description and characterization of solid breast masses based on BI-RADS features

To compare the performance of radiologists in the use of conventional ultrasound (US) and automated breast volume ultrasound (ABVU) for the characterization of benign and malignant solid breast masses based on breast imaging and reporting data system (BI-RADS) criteria. Conventional US and ABVU images were obtained in 87 patients with 106 solid breast masses (52 cancers, 54 benign lesions). Three experienced radiologists who were blinded to all examination results independently characterized the lesions and reported a BI-RADS assessment category and a level of suspicion of malignancy. The results were analyzed by calculation of Cohen's κ coefficient and by receiver operating characteristic (ROC) analysis. Assessment of the agreement of conventional US and ABVU indicated that the posterior echo feature was the most discordant feature of seven features (κ = 0.371 ± 0.225) and that orientation had the greatest agreement (κ = 0.608 ± 0.210). The final assessment showed substantial agreement (κ = 0.773 ± 0.104). The areas under the ROC curves (Az) for conventional US and ABVU were not statistically significant for each reader, but the mean Az values of conventional US and ABVU by multi-reader multi-case analysis were significantly different (conventional US 0.991, ABVU 0.963; 95 % CI -0.0471 to -0.0097). The means for sensitivity, specificity, positive predictive value, and negative predictive value of conventional US and ABVU did not differ significantly. There was substantial inter-observer agreement in the final assessment of solid breast masses by conventional US and ABVU. ROC analysis comparing the performance of conventional US and ABVU indicated a marginally significant difference in mean Az, but not in mean sensitivity, specificity, positive predictive value, or negative predictive value.

Breast MRI lesion classification: Improved performance of human readers with a backpropagation neural network computer‐aided diagnosis (CAD) system

To develop and test a computer-aided diagnosis (CAD) system to improve the performance of radiologists in classifying lesions on breast MRI (BMRI). A CAD system was developed that uses a semiautomated segmentation method. After segmentation, 42 features based on lesion shape, texture, and enhancement kinetics were computed, and the 13 best features were selected and used as inputs to a backpropagation neural network (BNN). The BNN was trained and tested using the leave-one-out method on 80 BMRI lesions (37 benign, 43 malignant). Lesion histopathology was used as the reference standard. Five human readers classified the 80 lesions first without and then with CAD assistance. The performance of the computer classifier and the human readers was assessed using receiver operating characteristic curves; the performance of the human readers was also evaluated using multireader multicase (MRMC) analysis. The performance of the human readers significantly improved when aided by the CAD system (P < 0.05). MRMC analysis showed that human reader performance with and without CAD system assistance can be generalized to the population of cases (P < 0.001). A CAD system based on lesion morphology and enhancement kinetics can improve the performance of human readers in classifying lesions on breast MRI.