Receiver Operating Characteristic Studies Research Articles

Establishment of an endoplasmic reticulum stress-associated lncRNAs model to predict prognosis and immunological characteristics in hepatocellular carcinoma.

The endoplasmic reticulum stress (ERS) and unfolded protein response (UPR) pathways play an essential role in the pathophysiology of hepatocellular carcinoma (HCC), and activation of the UPR pathway is strongly associated with tumor growth. However, the function of ERS-associated long non-coding RNAs (lncRNAs) in HCC is less recognized. We have used TCGA (The Cancer Genome Atlas) to obtain clinical and transcriptome data for HCC patients and the GSEA (Gene Set Enrichment Analysis) molecular signature database to get the ERS gene. ERS-associated prognostic lncRNA was determined using univariate Cox regression study. Then, least absolute shrinkage and selection operator and multivariate Cox regression study were used to construct ERS-associated lncRNAs risk model. Next, we use Kaplan-Meier (KM) survival study, time-dependent receiver operating characteristic (ROC) curve, univariate and multivariate Cox regression study to validate and evaluate the risk model. GSEA reveals the underlying molecular mechanism of the risk model. In addition, differences in Immune cell Infiltration Study, half-maximal inhibitory concentration (IC50) and immune checkpoints blockade (ICB) treatment between high and low risk groups were analyzed. We constructed a risk model consisting of 6 ERS-associated lncRNAS (containingMKLN1-AS, LINC01224, AL590705.3, AC008622.2, AC145207.5, and AC026412.3). The KM survival study showed that the prognosis of HCC patients in low-risk group was better than that in high-risk group. ROC study, univariate and multivariate Cox regression study showed that the risk model had good predictive power for HCC patients. Our verification sample verified the aforesaid findings. GSEA suggests that several tumor- and metabolism-related signaling pathways are associated with risk groups. Simultaneously, we discovered that the risk models may help in the treatment of ICB and the selection of chemotherapeutic drugs. In this article, we created an ERS-associated lncRNAs risk model to help prognostic diagnosis and personalized therapy in HCC.

Human Observer Net: A Platform Tool for Human Observer Studies of Image Data.

Background Current software applications for human observer studies of images lack flexibility in study design, platform independence, multicenter use, and assessment methods and are not open source, limiting accessibility and expandability. Purpose To develop a user-friendly software platform that enables efficient human observer studies in medical imaging with flexibility of study design. Materials and Methods Software for human observer imaging studies was designed as an open-source web application to facilitate access, platform-independent usability, and multicenter studies. Different interfaces for study creation, participation, and management of results were implemented. The software was evaluated in human observer experiments between May 2019 and March 2021, in which duration of observer responses was tracked. Fourteen radiologists evaluated and graded software usability using the 100-point system usability scale. The application was tested in Chrome, Firefox, Safari, and Edge browsers. Results Software function was designed to allow visual grading analysis (VGA), multiple-alternative forced-choice (m-AFC), receiver operating characteristic (ROC), localization ROC, free-response ROC, and customized designs. The mean duration of reader responses per image or per image set was 6.2 seconds ± 4.8 (standard deviation), 5.8 seconds ± 4.7, 8.7 seconds ± 5.7, and 6.0 seconds ± 4.5 in four-AFC with 160 image quartets per reader, four-AFC with 640 image quartets per reader, localization ROC, and experimental studies, respectively. The mean system usability scale score was 83 ± 11 (out of 100). The documented code and a demonstration of the application are available online (https://github.com/genskeu/HON, https://hondemo.pythonanywhere.com/). Conclusion A user-friendly and efficient open-source application was developed for human reader experiments that enables study design versatility, as well as platform-independent and multicenter usability. © RSNA, 2022 Online supplemental material is available for this article. See also the editorial by Thompson in this issue.

Phase and dark-field imaging with mesh-based structured illumination and polycapillary optics.

X-ray phase and dark-field (DF) imaging have been shown to improve the diagnostic capabilities of X-ray systems. However, these methods have found limited clinical use due to the need for multiple precision gratings with limited field of view or requirements on X-ray coherence that may not be easily translated to clinical practice. This work aims to develop a practicable X-ray phase and DF imaging system that could be translated and practiced in the clinic. This work employs a conventional source to create structured illumination with a simple wire mesh. A mesh-shifting algorithm is used to allow wider Fourier windowing to enhance resolution. Deconvolution of the source spot width and camera resolution improves accuracy. Polycapillary optics are employed to enhance coherence. The effects of incorporating optics with two different focal lengths are compared. Information apparent in enhanced absorption images, phase images, and DF images of fat embedded phantoms were compared and subjected to a limited receiver operator characteristic (ROC) study. The DF images of the moist and dry porous object (sponges) were compared. The mesh-based phase and DF imaging system constructs images with three different information types: scatter-free absorption images, differential phase images, and scatter magnitude/DF images, simultaneously from the same original image. The polycapillary optic enhances the coherence of the beam. The deblurring technique corrects the phase signal error due to geometrical blur and the limitation of the camera modulation transfer function (MTF) and removes image artifacts to improve the resolution in a single shot. The mesh-shifting method allows the use of a wider Fourier processing window, which gives even higher resolution, at the expense of an increased dose. The limited ROC study confirms the efficacy of the system over the conventional system. DF images of moist and dry porous object show the significance of the system in the imaging of lung infections. The mesh-based X-ray phase and DF imaging system is an inexpensive and easy setup in terms of alignment and data acquisition and can produce phase and DF images in a single shot with wide field of view. The system shows significant potential for use in diagnostic imaging in a clinical setting.

Validation of a candidate instrument to assess image quality in digital mammography using ROC analysis

PurposeTo validate a candidate instrument, to be used by different professionals to assess image quality in digital mammography (DM), against detection performance results. MethodsA receiver operating characteristics (ROC) study was conducted to assess the detection performance in DM images with four different image quality levels due to different quality issues. Fourteen expert breast radiologists from five countries assessed a set of 80 DM cases, containing 60 lesions (40 cancers, 20 benign findings) and 20 normal cases. A visual grading analysis (VGA) study using a previously-described candidate instrument was conducted to evaluate a subset of 25 of the images used in the ROC study. Eight radiologists that had participated in the ROC study, and seven expert breast-imaging physicists, evaluated this subset. The VGA score (VGAS) and the ROC and visual grading characteristics (VGC) areas under the curve (AUCROC and AUCVGC) were compared. ResultsNo large differences in image quality among the four levels were detected by either ROC or VGA studies. However, the ranking of the four levels was consistent: level 1 (partial AUCROC: 0.070, VGAS: 6.77) performed better than levels 2 (0.066, 6.15), 3 (0.061, 5.82), and 4 (0.062, 5.37). Similarity between radiologists’ and physicists’ assessments was found (average VGAS difference of 10 %). ConclusionsThe results from the candidate instrument were found to correlate with those from ROC analysis, when used by either observer group. Therefore, it may be used by different professionals, such as radiologists, radiographers, and physicists, to assess clinically-relevant image quality variations in DM.

The effect of breast density on the missed lesion rate in screening digital mammography determined using an adjustable-density breast phantom tailored to Japanese women.

ObjectiveDespite the high risk of missing lesions in mammography, the missed lesion rate is yet to be clinically established. Further, no breast phantoms with adjustable breast density currently exist. We developed a novel, adjustable-density breast phantom with a composition identical to that of actual breasts, and determined the quantitative relationship between breast density and the missed lesion rate in mammography.MethodsAn original breast phantom consisting of adipose- and fibroglandular-equivalent materials was developed, and a receiver operating characteristic (ROC) study was performed. Breast density, which is the fraction by weight of fibroglandular to total tissue, was adjusted to 25%, 50%, and 75% by arbitrarily mixing the two materials. Microcalcification, mass lesions, and spiculated lesions, each with unique characteristics, were inserted into the phantom. For the above-mentioned fibroglandular densities, 50 positive and 50 negative images for each lesion type were used as case samples for the ROC study. Five certified radiological technologists participated in lesion detection.ResultsThe mass-lesion detection rate, according to the area under the curve, decreased by 18.0% (p = 0.0001, 95% Confidence intervals [CI] = 0.1258 to 0.1822) and 37.8% (p = 0.0003, 95% CI = 0.2453 to 0.4031) for breast densities of 50% and 75%, respectively, compared to that for a 25% breast density. A similar tendency was observed with microcalcification; however, spiculated lesions did not follow this tendency.ConclusionsWe quantified the missed lesion rate in different densities of breast tissue using a novel breast phantom, which is imperative for advancing individualized screening mammography.

57 Study of the camera’s limit of detectability with Iodine-131

Introduction In differentiated thyroid cancer, scintigraphic images are performed after therapeutic administration of iodine-131. Several courses are proposed in case of thyroid markers arise due to thyroid’s cell regrowth or metastatic disease appearance. As disease evolves, in some cases, thyroid cells no longer absorb iodine-131 and thus, remain unseen on scintigraphic images. This situation leads to the end of iodine-131 treatments for the patient. Nevertheless, lack of detection does not equally correspond to absence of thyroid tissues or metastasis. In this context, we have evaluated the limit of detectability of our gamma-camera. Methods Gate (v7) Monte-Carlo code was used to model our gamma-camera ( Discovery NM / CT 690, GEHC). Acquired and modelled spatial resolution, sensitivity and contrast (SR, S, C) were compared. Collimator Angular response function (ARF) was used to speed simulations up and get images of a cylindrical object in different geometrical conditions close to clinical settings (radioactive concentration, tumor’s volume and depth) used to conduct a Receiver Operating Characteristic (ROC) study with several physicians. We have also studied different models of automatic observers in order to get closest results as possible to human observer ROC study. Results Simulated and acquired images have identical SR, S and C characteristics. Simulations were 20 times faster with ARF table than without. ROC/LROC human study shows an important variability in detection performances. Some automatic observers such as “2–Alternative Forced Choice” method present close performances to human observers whereas others like CHO outperformed them. Those different results allow us to derive an empirical law, relating the minimal radioactive concentration to detect a lesion as function of its depth and size. Regardless of lesion depth, a radioactive concentration of 25 kBq/mL is necessary to detect all lesions superior to 15 mm. Conclusions We performed a ROC/LROC study on simulated images to evaluate our camera detectability limit and we confirmed that lesions might stay unseen on scintigraphic images as long as radioactive concentration is below some threshold.

RECEIVER OPERATING CHARACTERISTIC (ROC) ANALYSIS

Visual expertise covers a broad range of types of studies and methodologies. Many studies incorporate some measure(s) of observer performance or how well participants perform on a given task. Receiver Operating Characteristic (ROC) analysis is a method commonly used in signal detection tasks (i.e., those in which the observer must decide whether or not a target is present or absent; or must classify a given target as belonging to one category or another), especially those in the medical imaging literature. This frontline paper will review some of the core theoretical underpinnings of ROC analysis, provide an overview of how to conduct an ROC study, and discuss some of the key variants of ROC analysis and their applications.

ROC解析を利用した一般撮影とポータブル撮影の患者識別能の比較

To evaluate the patient identification ability of radiographers, previous and current chest radiographs were assessed with observer study utilizing a receiver operating characteristics (ROCs) analysis. This study included portable and conventional chest radiographs from 43 same and 43 different patients. The dataset used in this study was divided into the three following groups: (1) a pair of portable radiographs, (2) a pair of conventional radiographs, and (3) a combination of each type of radiograph. Seven observers participated in this ROC study, which aimed to identify same or different patients, using these datasets. ROC analysis was conducted to calculate the average area under ROC curve obtained by each observer (AUCave), and a statistical test was performed using the multi-reader multi-case method. Comparable results were obtained with pairs of portable (AUCave: 0.949) and conventional radiographs (AUCave: 0.951). In a comparison between the same modality, there were no significant differences. In contrast, the ability to identify patients by comparing a portable and conventional radiograph (AUCave: 0.873) was lower than with the matching datasets (p=0.002 and p=0.004, respectively). In conclusion, the use of different imaging modalities reduces radiographers' ability to identify their patients.

Lesion type and reader experience affect the diagnostic accuracy of breast MRI: A multiple reader ROC study

PurposeTo evaluate the influence of lesion type (mass versus non-mass) and reader experience on the diagnostic performance of breast MRI (BMRI) in a non-screening setting. Materials and methodsConsecutive patients (mean age, 55±12 years) with breast lesions that were verified by biopsy or surgery, and who had had BMRI as part of their diagnostic workup, were eligible for this retrospective single-center study. Cancers diagnosed by biopsy before BMRI were excluded to eliminate biological and interpretation bias due to biopsy or chemotherapy effects (n=103). Six blinded readers (experience level, high (HE, n=2); intermediate (IE, n=2); and low (LE, n=2)) evaluated all examinations and assigned independent MRI BI-RADS ratings. Lesion type (mass, non-mass, focal) was noted. Receiver operating characteristics (ROC) and logistic regression analysis was performed to compare diagnostic accuracies. ResultsThere were 259 histologically verified lesions (123 malignant, 136 benign) investigated. There were 169 mass (103 malignant, 66 benign) and 48 non-mass lesions (19 malignant, 29 benign). Another 42 lesions that met the inclusion criteria were biopsied due to conventional findings (i.e., microcalcifications, architectural distortions), but did not enhance on MRI (41 benign, one DCIS). ROC analysis revealed a total area under the curve (AUC) between 0.834 (LE) and 0.935 (HI). Logistic regression identified a significant effect of non-mass lesions (P<0.0001) and reader experience (P=0.005) on diagnostic performance. ConclusionsNon-mass lesion type and low reader experience negatively affect the diagnostic performance of breast MRI in a non-screening setting.

Generalized Roe and Metz receiver operating characteristic model: analytic link between simulated decision scores and empirical AUC variances and covariances.

Modeling and simulation are often used to understand and investigate random quantities and estimators. In 1997, Roe and Metz introduced a simulation model to validate analysis methods for the popular endpoint in reader studies to evaluate medical imaging devices, the reader-averaged area under the receiver operating characteristic (ROC) curve. Here, we generalize the notation of the model to allow more flexibility in recognition that variances of ROC ratings depend on modality and truth state. We also derive and validate equations for computing population variances and covariances for reader-averaged empirical AUC estimates under the generalized model. The equations are one-dimensional integrals that can be calculated using standard numerical integration techniques. This work provides the theoretical foundation and validation for a Java application called iRoeMetz that can simulate multireader multicase ROC studies and numerically calculate the corresponding variances and covariances of the empirical AUC. The iRoeMetz application and source code can be found at the "iMRMC" project on the google code project hosting site. These results and the application can be used by investigators to investigate ROC endpoints, validate analysis methods, and plan future studies.

The average receiver operating characteristic curve in multireader multicase imaging studies.

In multireader, multicase (MRMC) receiver operating characteristic (ROC) studies for evaluating medical imaging systems, the area under the ROC curve (AUC) is often used as a summary metric. Owing to the limitations of AUC, plotting the average ROC curve to accompany the rigorous statistical inference on AUC is recommended. The objective of this article is to investigate methods for generating the average ROC curve from ROC curves of individual readers. We present both a non-parametric method and a parametric method for averaging ROC curves that produce a ROC curve, the area under which is equal to the average AUC of individual readers (a property we call area preserving). We use hypothetical examples, simulated data and a real-world imaging data set to illustrate these methods and their properties. We show that our proposed methods are area preserving. We also show that the method of averaging the ROC parameters, either the conventional bi-normal parameters (a, b) or the proper bi-normal parameters (c, da), is generally not area preserving and may produce a ROC curve that is intuitively not an average of multiple curves. Our proposed methods are useful for making plots of average ROC curves in MRMC studies as a companion to the rigorous statistical inference on the AUC end point. The software implementing these methods is freely available from the authors. METHODS for generating the average ROC curve in MRMC ROC studies are formally investigated. The area-preserving criterion we defined is useful to evaluate such methods.

Validation of multivariate screening methodology. Case study: Detection of food fraud

Multivariate screening methods are increasingly being implemented but there is no worldwide harmonized criterion for their validation. This study contributes to establish protocols for validating these methodologies. We propose the following strategy: (1) Establish the multivariate classification model and use receiver operating characteristic (ROC) curves to optimize the significance level (α) for setting the model’s boundaries. (2) Evaluate the performance parameter from the contingency table results and performance characteristic curves (PCC curves). The adulteration of hazelnut paste with almond paste and chickpea flour has been used as a case study. Samples were analyzed by infrared (IR) spectroscopy and the multivariate classification technique used was soft independent modeling of class analogies (SIMCA). The ROC study showed that the optimal α value for setting the SIMCA boundaries was 0.03 in both cases. The sensitivity value was 93%, specificity 100% for almond and 98% for chickpea, and efficiency 97% for almond and 93% for chickpea.

Diagnostic Efficacy of Conventional MRI Pulse Sequences in the Detection of Lesions Causing Internuclear Ophthalmoplegia in Multiple Sclerosis Patients

The purpose of this study was to investigate the diagnostic efficacy of a range of conventional magnetic resonance imaging (MRI) pulse sequences in the identification of internuclear ophthalmoplegia (INO) caused by medial longitudinal fasciculus (MLF) lesions in multiple sclerosis patients using a receiver-operating characteristic (ROC) methodology. A total of 15 clinically confirmed INO and 15 control subjects underwent conventional MRI at 1.5 T consisting of T2-weighted, proton density (PD)-weighted, and fluid-attenuated inversion recovery (FLAIR) sequences, following full institutional approval. A free-response, multiple-reader multiple-case design ROC study was used to evaluate the diagnostic efficacy of each sequence. All imaging sequences were evaluated by 10 board-certified neuroradiologists. Area under the curve (AUC), sensitivity, and specificity were analysed statistically for all three pulse sequences using repeated-measures analyses of variance and post-test analysis using Bonferroni's multiple comparison test of differences. No significant AUC differences were found between the three sequences (p = 0.0697), with T2 recording the highest AUC (0.8346). Sensitivity differences between PD (0.7927) and FLAIR (0.6329) were significant (p < 0.05). Non-significant differences were also evident between T2 and FLAIR (p = 0.0511). The specificity analysis revealed an overall difference (p = 0.0005), with specific inter-sequence differences shown between T2 and PD (p < 0.05) and PD and FLAIR (p < 0.001) with the PD values being lower than those provided with the other two sequences. T2-weighted axial imaging through the MLF region resulted in the greatest overall diagnostic efficacy when viewing a combination of mean AUC, sensitivity, and specificity, in terms of the identification of INO-causing lesions.

On the Shape of the Population ROC Curve

Human observers often do not produce empirical operating points near the northeast corner of the receiver operating characteristic (ROC) plot, and thus the local shape of the population ROC curve is unknown. We call attention to occult abnormalities and propose that considerations by human observers of the prior probability of occult abnormalities can cause the shape of the local population ROC curve to be convex, a straight line, or concave near the northeast corner of the ROC plot. We further conducted a set of simulated detection-task (without-search) experiments with human observers and, mathematically, with an ideal observer and a model observer. In the experiments, we used signals, pseudo-signals that were similar to signals, and random image noise. The relative frequency of occult signals was controlled in the experiments. In the simulated experiments, the population ROC curve of the ideal observer was always convex, but those of the model observer and of human observers were convex, a straight-line, or concave, depending on the relative frequency of occult signals. The population ROC curve for the model observer was identical to that for the ideal observer when knowing the relative frequency of occult signals was not important for the ideal observer, and it was similar to that for human observers otherwise. Observer consideration of the prior probability of occult abnormalities is important in ROC studies and could cause unexpected shapes of the local population ROC curve. Absence of empirical operating points near the northeast corner of the ROC plot may be caused by occult abnormalities.

Nonparametric receiver operating characteristic‐based evaluation for survival outcomes

For censored survival outcomes, it can be of great interest to evaluate the predictive power of individual markers or their functions. Compared with alternative evaluation approaches, approaches based on the time-dependent receiver operating characteristics (ROC) rely on much weaker assumptions, can be more robust, and hence are preferred. In this article, we examine evaluation of markers' predictive power using the time-dependent ROC curve and a concordance measure that can be viewed as a weighted area under the time-dependent area under the ROC curve profile. This study significantly advances from existing time-dependent ROC studies by developing nonparametric estimators of the summary indexes and, more importantly, rigorously establishing their asymptotic properties. It reinforces the statistical foundation of the time-dependent ROC-based evaluation approaches for censored survival outcomes. Numerical studies, including simulations and application to an HIV clinical trial, demonstrate the satisfactory finite-sample performance of the proposed approaches.

Measuring agreement between rating interpretations and binary clinical interpretations of images: a simulation study of methods for quantifying the clinical relevance of an observer performance paradigm

Laboratory receiver operating characteristic (ROC) studies, that are often used to evaluate medical imaging systems, differ from ‘live’ clinical interpretations in several respects which could compromise their clinical relevance. The aim was to develop methodology for quantifying the clinical relevance of a laboratory ROC study. A simulator was developed to generate ROC ratings data and binary clinical interpretations classified as correct or incorrect for a common set of images interpreted under clinical and laboratory conditions. The area under the trapezoidal ROC curve (AUC) was used as the laboratory figure-of-merit and the fraction of correct clinical decisions as the clinical figure-of-merit. Conventional agreement measures (Pearson, Spearman, Kendall and kappa) between the bootstrap-induced fluctuations of the two figures of merit were estimated. A jackknife pseudovalue transformation applied to the figures of merit was also investigated as a way to capture agreement existing at the individual image level that could be lost at the figure-of-merit level. It is shown that the pseudovalues define a relevance-ROC curve. The area under this curve (rAUC) measures the ability of the laboratory figure-of-merit-based pseudovalues to correctly classify incorrect versus correct clinical interpretations. Therefore, rAUC is a measure of the clinical relevance of an ROC study. The conventional measures and rAUC were compared under varying simulator conditions. It was found that design details of the ROC study, namely the number of bins, the difficulty level of the images, the ratio of disease-present to disease-absent images and the unavoidable difference between laboratory and clinical performance levels, can lead to serious underestimation of the agreement as indicated by conventional agreement measures, even for perfectly correlated data, while rAUC showed high agreement and was relatively immune to these details. At the same time rAUC was sensitive to factors such as intrinsic correlation between the laboratory and clinical decision variables and differences in reporting thresholds that are expected to influence agreement both at the individual image level and at the figure-of-merit level. Suggestions are made for how to conduct relevance-ROC studies aimed at assessing agreement between laboratory and clinical interpretations. The method could be used to evaluate the clinical relevance of alternative scalar figures of merit, such as the sensitivity at a predifined specificity.

Secretory Unit of Islet Transplant Objects (SUITO) Index Can Predict Severity of Hypoglycemic Episodes in Clinical Islet Cell Transplantation

One endpoint of clinical islet cell transplantation for type 1 diabetic patients is the elimination or reduction of hypoglycemia. We previously developed a simple tool to evaluate islet graft function: the secretory unit of islet transplant objects (SUITO) index. The aim of this study is to clarify the association between the SUITO index and hypoglycemic episodes. Data from 310 clinical evaluations of 11 islet recipients were included in this study. Fasting plasma C-peptide and glucose levels were measured at every evaluation. The SUITO index was calculated according to the following formula: 1500 × C-peptide level (ng/ml)/[blood glucose level (mg/dl) - 63]. The number of hypoglycemic events (<3.8 mmol/L) and severe hypoglycemic events (<2.2 mmol/L or hypoglycemic unawareness) was assessed on the basis of interviews and self-monitoring of blood glucose (SMBG). Receiver operating characteristic (ROC) analysis was performed to determine the cut-off values of the SUITO index for hypoglycemic events. Based on the ROC study, follow-up data after transplantations were divided into the following three groups: low-SUITO (SUITO index <10, n = 91), middle-SUITO (10 ≤SUITO index <26, n = 83), high-SUITO (SUITO index ≤26, n = 125). The frequency of total hypoglycemia in the high-SUITO group was significantly decreased when compared to the other groups (value with Kruskal-Wallis test p < 0.001). The frequency of total severe hypoglycemia was significantly decreased in the low-SUITO group compared to pretransplant status and further decreased in the middle- and high-SUITO group. Spearman correlation coefficients were -0.663 (p < 0.001) between the number of total hypoglycemic events per one month and the SUITO index and -0.521 (p < 0.001) between that of severe events and the SUITO index. The SUITO index could predict the severity of hypoglycemic episodes in type 1 diabetic patients who received islet cell transplantations.

Confidence intervals for performance assessment of linear observers

This work seeks to develop exact confidence interval estimators for figures of merit that describe the performance of linear observers, and to demonstrate how these estimators can be used in the context of x-ray computed tomography (CT). The figures of merit are the receiver operating characteristic (ROC) curve and associated summary measures, such as the area under the ROC curve. Linear computerized observers are valuable for optimization of parameters associated with image reconstruction algorithms and data acquisition geometries. They provide a means to perform assessment of image quality with metrics that account not only for shift-variant resolution and nonstationary noise but that are also task-based. We suppose that a linear observer with fixed template has been defined and focus on the problem of assessing the performance of this observer for the task of deciding if an unknown lesion is present at a specific location. We introduce a point estimator for the observer signal-to-noise ratio (SNR) and identify its sampling distribution. Then, we show that exact confidence intervals can be constructed from this distribution. The sampling distribution of our SNR estimator is identified under the following hypotheses: (i) the observer ratings are normally distributed for each class of images and (ii) the variance of the observer ratings is the same for each class of images. These assumptions are, for example, appropriate in CT for ratings produced by linear observers applied to low-contrast lesion detection tasks. Unlike existing approaches to the estimation of ROC confidence intervals, the new confidence intervals presented here have exactly known coverage probabilities when our data assumptions are satisfied. Furthermore, they are applicable to the most commonly used ROC summary measures, and they may be easily computed (a computer routine is supplied along with this article on the Medical Physics Website). The utility of our exact interval estimators is demonstrated through an image quality evaluation example using real x-ray CT images. Also, strong robustness is shown to potential deviations from the assumption that the ratings for the two classes of images have equal variance. Another aspect of our interval estimators is the fact that we can calculate their mean length exactly for fixed parameter values, which enables precise investigations of sampling effects. We demonstrate this aspect by exploring the potential reduction in statistical variability that can be gained by using additional images from one class, if such images are readily available. We find that when additional images from one class are used for an ROC study, the mean AUC confidence interval length for our estimator can decrease by as much as 35%. We have shown that exact confidence intervals can be constructed for ROC curves and for ROC summary measures associated with fixed linear computerized observers applied to binary discrimination tasks at a known location. Although our intervals only apply under specific conditions, we believe that they form a valuable tool for the important problem of optimizing parameters associated with image reconstruction algorithms and data acquisition geometries, particularly in x-ray CT.

ROC Curves in Non‐Parametric Location‐Scale Regression Models

Abstract. The receiver operating characteristic (ROC) curve is a tool of extensive use to analyse the discrimination capability of a diagnostic variable in medical studies. In certain situations, the presence of a covariate related to the diagnostic variable can increase the discriminating power of the ROC curve. In this article, we model the effect of the covariate over the diagnostic variable by means of non‐parametric location‐scale regression models. We propose a new non‐parametric estimator of the conditional ROC curve and study its asymptotic properties. We also present some simulations and an illustration to a data set concerning diagnosis of diabetes.

Use of likelihood ratios for comparisons of binary diagnostic tests: Underlying ROC curves

When comparing binary test results from two diagnostic systems, superiority in both "sensitivity" and "specificity" also implies differences in all conventional summary indices and locally in the underlying receiver operating characteristics (ROC) curves. However, when one of the two binary tests has higher sensitivity and lower specificity (or vice versa), comparisons of their performance levels are nontrivial and the use of different summary indices may lead to contradictory conclusions. A frequently used approach that is free of subjectivity associated with summary indices is based on the comparison of the underlying ROC curves that requires the collection of rating data using multicategory scales, whether natural or experimentally imposed. However, data for reliable estimation of ROC curves are frequently unavailable. The purpose of this article is to develop an approach of using "diagnostic likelihood ratios", namely, likelihood ratios of "positive" or "negative" responses, to make simple inferences regarding the underlying ROC curves and associated areas in the absence of reliable rating data or regarding the relative binary characteristics, when these are of primary interest. For inferences related to underlying curves, the authors exploit the assumption of concavity of the true underlying ROC curve to describe conditions under which these curves have to be different and under which the curves have different areas. For scenarios when the binary characteristics are of primary interest, the authors use characteristics of "chance performance" to demonstrate that the derived conditions provide strong evidence of superiority of one binary test as compared to another. By relating these derived conditions to hypotheses about the true likelihood ratios of two binary diagnostic tests being compared, the authors enable a straightforward statistical procedure for the corresponding inferences. The authors derived simple algebraic and graphical methods for describing the conditions for superiority of one of two diagnostic tests with respect to their binary characteristics, the underlying ROC curves, or the areas under the curves. The graphical regions are useful for identifying potential differences between two systems, which then have to be tested statistically. The simple statistical tests can be performed with well known methods for comparison of diagnostic likelihood ratios. The developed approach offers a solution for some of the more difficult to analyze scenarios, where diagnostic tests do not demonstrate concordant differences in terms of both sensitivity and specificity. In addition, the resulting inferences do not contradict the conclusions that can be obtained using conventional and reasonably defined summary indices. When binary diagnostic tests are of primary interest, the proposed approach offers an objective and powerful method for comparing two binary diagnostic tests. The significant advantage of this method is that it enables objective analyses when one test has higher sensitivity but lower specificity, while ensuring agreement with study conclusions based on other reasonable and widely acceptable summary indices. For truly multicategory diagnostic tests, the proposed method can help in concluding inferiority of one of the diagnostic tests based on binary data, thereby potentially saving the need for conducting a more expensive multicategory ROC study.