True Disease Status Research Articles

Non rate-controlled CT coronary angiography for the exclusion of obstructive coronary artery disease in the assessment of patients referred for transcatheter aortic valve implantation

Purpose: Optimal management of concomitant coronary artery disease (CAD) in patients undergoing transcatheter aortic valve implantation (TAVI) is contentious. This study aimed to assess the role of routine non rate-controlled CT coronary angiography (CTCA) in assessing CAD in patients referred for TAVI, thereby potentially precluding the need for invasive coronary angiography (ICA). CTCA in this cohort can be difficult due to the high burden of coronary calcium and the inability to perform standard pharmacological rate control. Methods: All patients underwent an ECG-gated 128-slice calcium score and CT angiogram to assess peripheral vasculature, aortic anatomy, aortic valve calcification and location of the coronary ostia. Coronary artery calcium scores (CACS) and CTCA images were scored by two independent, blinded observers and compared with ICA performed prior to TAVI. Significant coronary stenosis was defined as ≥70% and ICA was deemed the gold standard for true disease status. Sensitivity, specificity, positive (PPV) and negative predictive values (NPV) were calculated for CTCA at a patient level. Results: 70 consecutive patients referred for consideration of TAVI were studied: 40% male (n=28), mean age 78.9yrs (±7.91), 23% diabetic (n=16). Significant CAD was demonstrated in 30% of ICA studies (1 vessel disease (VD) 20% [n=14], 2VD 8.6% [n=6], 3VD 1.4% [n=1]). The sensitivity, specificity, PPV and NPV of CTCA for the detection of significant coronary stenoses were 57.9% (CI: 33.5% to 79.7%), 87.0% (CI: 75.1% to 94.6%), 61.1% (CI: 35.8% to 82.6%) and 85.4% (CI: 73.3% to 93.5%) respectively. The vast majority of false positive (5/6; 83%) and false negative (6/7; 86%) CTCAs were seen in patients in the highest quartile of CACS (>400). Conclusions: In patients undergoing CT assessment prior to TAVI, non rate-controlled CTCA shows poor sensitivity but good specificity and negative predictive value for the detection of significant CAD. A CAC score >400 may represent a threshold above which invasive angiography is mandated.

Finding the optimal cut-point for Gaussian and Gamma distributed biomarkers

Categorization is often needed for clinical decision making when dealing with diagnostic (prognostic) biomarkers and a binary outcome (true disease status). Four common methods used to dichotomize a continuous biomarker X are compared: the minimum P-value, the Youden index, the concordance probability and the point closest-to-(0, 1) corner in the ROC plane. These methods are compared from a theoretical point of view under Normal or Gamma biomarker distributions, showing whether or not they lead to the identification of the same true cut-point. The performance of the corresponding non-parametric estimators is then compared by simulation. Two motivating examples are presented. In all simulation scenarios, the point closest-to-(0, 1) corner in the ROC plane and concordance probability approaches outperformed the other methods. Both these methods showed good performance in the estimation of the optimal cut-point of a biomarker. However, when methods do not lead to the same optimal cut-point, scientists should focus on which one is truly what they want to estimate, and use it in practice. In addition, to improve communicability, the Youden index or the concordance probability associated to the estimated cut-point could be reported to summarize the associated classification accuracy. The use of the minimum P-value approach for cut-point finding is strongly not recommended because its objective function is computed under the null hypothesis of absence of association between the true disease status and X. This is in contrast with the presence of some discrimination potential of X that leads to the dichotomization issue.

Estimating confidence intervals for the difference in diagnostic accuracy with three ordinal diagnostic categories without a gold standard

With three ordinal diagnostic categories, the most commonly used measures for the overall diagnostic accuracy are the volume under the ROC surface (VUS) and partial volume under the ROC surface (PVUS), which are the extensions of the area under the ROC curve (AUC) and partial area under the ROC curve (PAUC), respectively. A gold standard (GS) test on the true disease status is required to estimate the VUS and PVUS. However, oftentimes it may be difficult, inappropriate, or impossible to have a GS because of misclassification error, risk to the subjects or ethical concerns. Therefore, in many medical research studies, the true disease status may remain unobservable. Under the normality assumption, a maximum likelihood (ML) based approach using the expectation–maximization (EM) algorithm for parameter estimation is proposed. Three methods using the concepts of generalized pivot and parametric/nonparametric bootstrap for confidence interval estimation of the difference in paired VUSs and PVUSs without a GS are compared. The coverage probabilities of the investigated approaches are numerically studied. The proposed approaches are then applied to a real data set of 118 subjects from a cohort study in early stage Alzheimer’s disease (AD) from the Washington University Knight Alzheimer’s Disease Research Center to compare the overall diagnostic accuracy of early stage AD between two different pairs of neuropsychological tests.

Empirical likelihood confidence regions for the evaluation of continuous‐scale diagnostic tests in the presence of verification bias

AbstractIn a continuous‐scale diagnostic test, when a cut‐off level is given, the performance of the test in distinguishing diseased subjects from non‐diseased subjects can be evaluated by its sensitivity and specificity. Joint inferences for sensitivity and specificity as well as cut‐off level play an important role in the assessment of the diagnostic accuracy of the test. Most current studies on this topic focus on complete data cases. However, in some studies, only a portion of subjects given their screening test results ultimately have their true disease status verified. In addition, the verification may depend on the test result and the subject's observed characteristics. Directly applying full data methods to verified subjects results in biased estimates, known as verification bias. In this paper, based on a general framework that combines empirical likelihood and general estimation equations with nuisance parameters, we propose various bias‐corrected joint empirical likelihood confidence regions for sensitivity and specificity with verification‐biased data. Thorough simulation studies are conducted to compare the finite sample performance of the proposed confidence regions in terms of coverage probabilities, and some suggestions are provided accordingly. Finally, an example is provided to illustrate the proposed methods. The Canadian Journal of Statistics 41: 398–420; 2013 © 2013 Statistical Society of Canada

Auxiliary Marker-Assisted Classification in the Absence of Class Identifiers

Constructing classification rules for accurate diagnosis of a disorder is an important goal in medical practice. In many clinical applications, there is no clinically significant anatomical or physiological deviation that exists to identify the gold standard disease status to inform development of classification algorithms. Despite the absence of perfect disease class identifiers, there are usually one or more disease-informative auxiliary markers along with feature variables that comprise known symptoms. Existing statistical learning approaches do not effectively draw information from auxiliary prognostic markers. We propose a large margin classification method, with particular emphasis on the support vector machine, assisted by available informative markers to classify disease without knowing a subject's true disease status. We view this task as statistical learning in the presence of missing data, and introduce a pseudo-Expectation-Maximization (EM) algorithm to the classification. A major difference between a regular EM algorithm and the algorithm proposed here is that we do not model the distribution of missing data given the observed feature variables either parametrically or semiparametrically. We also propose a sparse variable selection method embedded in the pseudo-EM algorithm. Theoretical examination shows that the proposed classification rule is Fisher consistent, and that under a linear rule, the proposed selection has an oracle variable selection property and the estimated coefficients are asymptotically normal. We apply the methods to build decision rules for including subjects in clinical trials of a new psychiatric disorder and present four applications to data available at the University of California, Irvine Machine Learning Repository.

A crossed random effects modeling approach for estimating diagnostic accuracy from ordinal ratings without a gold standard

In diagnostic studies without a gold standard, the assumption on the dependence structure of the multiple tests or raters plays an important role in model performance. In case of binary disease status, both conditional independence and crossed random effects structure have been proposed and their performance investigated. Less attention has been paid to the situation where the true disease status is ordinal. In this paper, we propose crossed subject-specific and rater-specific random effects to account for the dependence structure and assess the robustness of the proposed model to misspecification in the random effects distributions. We applied the models to data from the Physician Reliability Study, which focuses on assessing the diagnostic accuracy in a population of raters for the staging of endometriosis, a gynecological disorder in women. Using this new methodology, we estimate the probability of a correct classification and show that regional experts can more easily classify the intermediate stage than resident physicians.

Covariate Adjustment in Estimating the Area Under ROC Curve with Partially Missing Gold Standard

In ROC analysis, covariate adjustment is advocated when the covariates impact the magnitude or accuracy of the test under study. Meanwhile, for many large scale screening tests, the true condition status may be subject to missingness because it is expensive and/or invasive to ascertain the disease status. The complete-case analysis may end up with a biased inference, also known as "verification bias." To address the issue of covariate adjustment with verification bias in ROC analysis, we propose several estimators for the area under the covariate-specific and covariate-adjusted ROC curves (AUCx and AAUC). The AUCx is directly modeled in the form of binary regression, and the estimating equations are based on the U statistics. The AAUC is estimated from the weighted average of AUCx over the covariate distribution of the diseased subjects. We employ reweighting and imputation techniques to overcome the verification bias problem. Our proposed estimators are initially derived assuming that the true disease status is missing at random (MAR), and then with some modification, the estimators can be extended to the not missing at random (NMAR) situation. The asymptotic distributions are derived for the proposed estimators. The finite sample performance is evaluated by a series of simulation studies. Our method is applied to a data set in Alzheimer's disease research.

Association between microsatellite markers and bovine tuberculosis in Chadian Zebu cattle

Bovine tuberculosis (BTB) is a considerable threat to livestock keepers and public health in many developing and underdeveloped countries. We investigated associations between 20 microsatellite markers and three phenotypes for BTB in a sample of the Chadian cattle population. The phenotypes measured were: 1) single intra-dermal comparative cervical tuberculin test (SICCT) performed on live animals, 2) presence of lesion post-slaughter, 3) a bacteriological tissue culture test for Mycobacterium bovis using the samples with observed lesions and 4) a predicted Bayesian model (BM) estimate of a true BTB disease status using all tested animals. All traits were recorded in binary form and as either 1 = presence or 0 = absence. A total of 224 animals for SICCT, lesion and BM traits and 96 animals with bacteriological culture test were genotyped. Generalised linear models were fitted to the binary BTB phenotypes that consisted of age (covariate), sex (2 levels), breed (2 levels) and markers (alleles: 5 - 14 levels) as explanatory variables and implemented in R using glm with a logit link function. The model was fitted for each marker, separately. Six out of 20 markers tested were significantly associated with at least one trait considered; these were ILSTS005, ILSTS006, TGLA227, BM2113 and CSRM66. Genomic regions around these markers may serve as a basis for further functional investigations. This is the first study to report association of microsatellite markers with bovine tuberculosis traits in African or Chadian cattle population.

Risk factors for bovine tuberculosis in low incidence regions related to the movements of cattle

BackgroundBovine tuberculosis (bTB) remains difficult to eradicate from low incidence regions partly due to the imperfect sensitivity and specificity of routine intradermal tuberculin testing. Herds with unconfirmed reactors that are incorrectly classified as bTB-negative may be at risk of spreading disease, while those that are incorrectly classified as bTB-positive may be subject to costly disease eradication measures. This analysis used data from Scotland in the period leading to Officially Tuberculosis Free recognition (1) to investigate the risks associated with the movements of cattle from herds with different bTB risk classifications and (2) to identify herd demographic characteristics that may aid in the interpretation of tuberculin testing results.ResultsFrom 2002 to 2009, for every herd with confirmed bTB positive cattle identified through routine herd testing, there was an average of 2.8 herds with at least one unconfirmed positive reactor and 18.9 herds with unconfirmed inconclusive reactors. Approximately 75% of confirmed bTB positive herds were detected through cattle with no known movements outside Scotland. At the animal level, cattle that were purchased from Scottish herds with unconfirmed positive reactors and a recent history importing cattle from endemic bTB regions were significantly more likely to react positively on routine intradermal tuberculin tests, while cattle purchased from Scottish herds with unconfirmed inconclusive reactors were significantly more likely to react inconclusively. Case-case comparisons revealed few demographic differences between herds with confirmed positive, unconfirmed positive, and unconfirmed inconclusive reactors, which highlights the difficulty in determining the true disease status of herds with unconfirmed tuberculin reactors. Overall, the risk of identifying reactors through routine surveillance decreased significantly over time, which may be partly attributable to changes in movement testing regulations and the volume of cattle imported from endemic regions.ConclusionsAlthough the most likely source of bTB infections in Scotland was cattle previously imported from endemic regions, we found indirect evidence of transmission within Scottish cattle farms and cannot rule out the possibility of low level transmission between farms. Further investigation is needed to determine whether targeting herds with unconfirmed reactors and a history of importing cattle from high risk regions would benefit control efforts.

Evaluating accuracy of diagnostic tests with intermediate results in the absence of a gold standard

Intermediate test results often occur with diagnostic tests. When assessing diagnostic accuracy, it is important to properly report and account for these results. In the literature, these results are commonly discarded prior to analysis or treated as either a positive or a negative result. Although such adjustments allow sensitivity and specificity to be computed in the standard way, these forced decisions limit the interpretability and usefulness of the results. Estimation of diagnostic accuracy is further complicated when tests are evaluated without a gold standard. Although traditional latent class modeling can be readily applied to analyze these data and account for intermediate results, these models assume that tests are independent conditional on the true disease status, which is rarely valid in practice. We extend both the log-linear latent class model and the probit latent class model to accommodate the conditional dependence among tests while taking the intermediate results into consideration. We illustrate our methods using a simulation study and a published medical study on the detection of epileptiform activity in the brain.

Serodiagnosis of Tuberculosis in Asian Elephants (Elephas maximus) in Southern India: A Latent Class Analysis

Background Mycobacterium tuberculosis, a causative agent of chronic tuberculosis disease, is widespread among some animal species too. There is paucity of information on the distribution, prevalence and true disease status of tuberculosis in Asian elephants (Elephas maximus). The aim of this study was to estimate the sensitivity and specificity of serological tests to diagnose M. tuberculosis infection in captive elephants in southern India while simultaneously estimating sero-prevalence.Methodology/Principal FindingsHealth assessment of 600 elephants was carried out and their sera screened with a commercially available rapid serum test. Trunk wash culture of select rapid serum test positive animals yielded no animal positive for M. tuberculosis isolation. Under Indian field conditions where the true disease status is unknown, we used a latent class model to estimate the diagnostic characteristics of an existing (rapid serum test) and new (four in-house ELISA) tests. One hundred and seventy nine sera were randomly selected for screening in the five tests. Diagnostic sensitivities of the four ELISAs were 91.3–97.6% (95% Credible Interval (CI): 74.8–99.9) and diagnostic specificity were 89.6–98.5% (95% CI: 79.4–99.9) based on the model we assumed. We estimate that 53.6% (95% CI: 44.6–62.8) of the samples tested were free from infection with M. tuberculosis and 15.9% (97.5% CI: 9.8 - to 24.0) tested positive on all five tests.Conclusions/SignificanceOur results provide evidence for high prevalence of asymptomatic M. tuberculosis infection in Asian elephants in a captive Indian setting. Further validation of these tests would be important in formulating area-specific effective surveillance and control measures.

Efficiency of study designs in diagnostic randomized clinical trials

From the patients' management perspective, a good diagnostic test should contribute to both reflecting the true disease status and improving clinical outcomes. The diagnostic randomized clinical trial is designed to combine both diagnostic tests and therapeutic interventions. Evaluation of diagnostic tests is carried out with therapeutic outcomes as the primary endpoint rather than test accuracy. We lay out the probability framework for evaluating such trials. We compare two commonly referred designs-the two-arm design and the paired design-in a formal statistical hypothesis testing setup and identify the causal connection between the two tests. The paired design is shown to be more efficient than the two-arm design. The efficiency gains vary depending on the discordant rates of test results. We derive sample size formulas for both binary and continuous endpoints. We derive estimation of important quantities under the paired design and also conduct simulation studies to verify the theoretical results. We illustrate the method with an example of designing a randomized study on preoperative staging of bladder cancer.

Estimation of AUC or Partial AUC Under Test-Result-Dependent Sampling

The area under the receiver operating characteristic (ROC) curve (AUC) and partial area under the ROC curve (pAUC) are summary measures used to assess the accuracy of a biomarker in discriminating true disease status. The standard sampling approach used in biomarker validation studies is often inefficient and costly, especially when ascertaining the true disease status is costly and invasive. To improve efficiency and reduce the cost of biomarker validation studies, we consider a test-result-dependent sampling (TDS) scheme, in which subject selection for determining the disease state is dependent on the result of a biomarker assay. We first estimate the test-result distribution using data arising from the TDS design. With the estimated empirical test-result distribution, we propose consistent nonparametric estimators for AUC and pAUC and establish the asymptotic properties of the proposed estimators. Simulation studies show that the proposed estimators have good finite sample properties and that the TDS design yields more efficient AUC and pAUC estimates than a simple random sampling design. A data example based on an ongoing cancer clinical trial is provided to illustrate the TDS design and the proposed estimators. This work can find broad applications in the design and analysis of biomarker validation studies.

Maximizing an ROC‐type measure via linear combination of markers when the gold reference is continuous

Effectively combining many classification instruments or diagnostic measurements together to improve the classification accuracy of individuals is a common idea in disease diagnosis or classification. These ensemble-type diagnostic methods can be constructed with respect to different kinds of performance criterions. Among them, the receiver operating characteristic (ROC) curve is the most popular criterion, which, together with some indexes derived from it, is commonly used to evaluate and summarize the performance of a classification instrument, such as a biomarker or a classifier. However, the usefulness of ROC curve and its related indexes relies on the existence of a binary label for each individual subject. In many disease diagnosis situations, such a binary variable may not exist, but only the continuous measurement of the true disease status is available. This true disease status is often referred to as the 'gold standard'. The modified area under ROC curve (AUC)-type measure defined by Obuchowski is a method proposed to accommodate such a situation. However, there is still no method for finding the optimal combination of diagnostic measurements, with respect to such an index, to have better diagnostic power than that of each individual measurement. In this paper, we propose an algorithm for finding the optimal combination with respect to such an extended AUC-type measure such that the combined measurement can have more diagnostic power. We illustrate the performance of our algorithm by using some synthesized data and a diabetes data set.

Impact of imperfect test sensitivity on determining risk factors: the case of bovine tuberculosis.

BackgroundImperfect diagnostic testing reduces the power to detect significant predictors in classical cross-sectional studies. Assuming that the misclassification in diagnosis is random this can be dealt with by increasing the sample size of a study. However, the effects of imperfect tests in longitudinal data analyses are not as straightforward to anticipate, especially if the outcome of the test influences behaviour. The aim of this paper is to investigate the impact of imperfect test sensitivity on the determination of predictor variables in a longitudinal study.Methodology/Principal FindingsTo deal with imperfect test sensitivity affecting the response variable, we transformed the observed response variable into a set of possible temporal patterns of true disease status, whose prior probability was a function of the test sensitivity. We fitted a Bayesian discrete time survival model using an MCMC algorithm that treats the true response patterns as unknown parameters in the model. We applied our approach to epidemiological data of bovine tuberculosis outbreaks in England and investigated the effect of reduced test sensitivity in the determination of risk factors for the disease. We found that reduced test sensitivity led to changes to the collection of risk factors associated with the probability of an outbreak that were chosen in the ‘best’ model and to an increase in the uncertainty surrounding the parameter estimates for a model with a fixed set of risk factors that were associated with the response variable.Conclusions/SignificanceWe propose a novel algorithm to fit discrete survival models for longitudinal data where values of the response variable are uncertain. When analysing longitudinal data, uncertainty surrounding the response variable will affect the significance of the predictors and should therefore be accounted for either at the design stage by increasing the sample size or at the post analysis stage by conducting appropriate sensitivity analyses.

On ROC analysis with nonbinary reference standard

Statistical methods for the evaluation of the accuracy of diagnostic tests usually assume a binary true disease status. However, this assumption may not be realistic in practical settings in which "disease" is defined by dichotomizing continuous or ordinal categorical measures using a pre-specified threshold value. In this paper, we focus on the analysis of studies in which both the diagnostic test and the reference standard are reported as continuous measures. We propose a semiparametric model for estimating the sensitivity, specificity, and the ROC curve as functions of reference standard thresholds. Under suitable order restrictions on the mean of the test result variable, fitting is done via two alternative approaches: isotonic regression and monotone smoothing splines. The model provides the basis to assess the effect of varying reference standard threshold on the performance of a diagnostic test. An example to evaluate the ability of the maximal SUV-lean (standardized uptake value normalized to lean body mass) in predicting axillary node involvement in women diagnosed with breast cancer is presented.

A robust method using propensity score stratification for correcting verification bias for binary tests

Sensitivity and specificity are common measures of the accuracy of a diagnostic test. The usual estimators of these quantities are unbiased if data on the diagnostic test result and the true disease status are obtained from all subjects in an appropriately selected sample. In some studies, verification of the true disease status is performed only for a subset of subjects, possibly depending on the result of the diagnostic test and other characteristics of the subjects. Estimators of sensitivity and specificity based on this subset of subjects are typically biased; this is known as verification bias. Methods have been proposed to correct verification bias under the assumption that the missing data on disease status are missing at random (MAR), that is, the probability of missingness depends on the true (missing) disease status only through the test result and observed covariate information. When some of the covariates are continuous, or the number of covariates is relatively large, the existing methods require parametric models for the probability of disease or the probability of verification (given the test result and covariates), and hence are subject to model misspecification. We propose a new method for correcting verification bias based on the propensity score, defined as the predicted probability of verification given the test result and observed covariates. This is estimated separately for those with positive and negative test results. The new method classifies the verified sample into several subsamples that have homogeneous propensity scores and allows correction for verification bias. Simulation studies demonstrate that the new estimators are more robust to model misspecification than existing methods, but still perform well when the models for the probability of disease and probability of verification are correctly specified.

Computer-aided diagnosis of renal obstruction: utility of log-linear modeling versus standard ROC and kappa analysis

BackgroundThe accuracy of computer-aided diagnosis (CAD) software is best evaluated by comparison to a gold standard which represents the true status of disease. In many settings, however, knowledge of the true status of disease is not possible and accuracy is evaluated against the interpretations of an expert panel. Common statistical approaches to evaluate accuracy include receiver operating characteristic (ROC) and kappa analysis but both of these methods have significant limitations and cannot answer the question of equivalence: Is the CAD performance equivalent to that of an expert? The goal of this study is to show the strength of log-linear analysis over standard ROC and kappa statistics in evaluating the accuracy of computer-aided diagnosis of renal obstruction compared to the diagnosis provided by expert readers.MethodsLog-linear modeling was utilized to analyze a previously published database that used ROC and kappa statistics to compare diuresis renography scan interpretations (non-obstructed, equivocal, or obstructed) generated by a renal expert system (RENEX) in 185 kidneys (95 patients) with the independent and consensus scan interpretations of three experts who were blinded to clinical information and prospectively and independently graded each kidney as obstructed, equivocal, or non-obstructed.ResultsLog-linear modeling showed that RENEX and the expert consensus had beyond-chance agreement in both non-obstructed and obstructed readings (both p < 0.0001). Moreover, pairwise agreement between experts and pairwise agreement between each expert and RENEX were not significantly different (p = 0.41, 0.95, 0.81 for the non-obstructed, equivocal, and obstructed categories, respectively). Similarly, the three-way agreement of the three experts and three-way agreement of two experts and RENEX was not significantly different for non-obstructed (p = 0.79) and obstructed (p = 0.49) categories.ConclusionLog-linear modeling showed that RENEX was equivalent to any expert in rating kidneys, particularly in the obstructed and non-obstructed categories. This conclusion, which could not be derived from the original ROC and kappa analysis, emphasizes and illustrates the role and importance of log-linear modeling in the absence of a gold standard. The log-linear analysis also provides additional evidence that RENEX has the potential to assist in the interpretation of diuresis renography studies.

Linkage of patient records from disparate sources

We review ideas, approaches and progress in the field of record linkage. We point out that the latent class models used in probabilistic matching have been well developed and applied in a different context of diagnostic testing when the true disease status is unknown. The methodology developed in the diagnostic testing setting can be potentially translated and applied in record linkage. Although there are many methods for record linkage, a comprehensive evaluation of methods for a wide range of real-world data with different data characteristics and with true match status is absent due to lack of data sharing. However, the recent availability of generators of synthetic data with realistic characteristics renders such evaluations feasible.

Combining multiple continuous tests for the diagnosis of kidney impairment in the absence of a gold standard

Receiver operating characteristic (ROC) curves are commonly used to summarize the classification accuracy of diagnostic tests. It is not uncommon in medical practice that multiple diagnostic tests are routinely performed or multiple disease markers are available for the same individuals. When the true disease status is verified by a gold standard (GS) test, a variety of methods have been proposed to combine such potential correlated tests to increase the accuracy of disease diagnosis. In this article, we propose a method of combining multiple diagnostic tests in the absence of a GS. We assume that the test values and their classification accuracies are dependent on covariates. Simulation studies are performed to examine the performance of the combination method. The proposed method is applied to data from a population-based aging study to compare the accuracy of three screening tests for kidney function and to estimate the prevalence of moderate kidney impairment.