Abstract
BackgroundUsing Monte Carlo simulations, we compare different methods (maximizing Youden index, maximizing mutual information, and logistic regression) for their ability to determine optimum binary cut-off thresholds for a ratio-scaled diagnostic test variable. Special attention is given to the stability and precision of the results in dependence on the distributional characteristics as well as the pre-test probabilities of the diagnostic categories in the test population.MethodsFictitious data sets of a ratio-scaled diagnostic test with different distributional characteristics are generated for 50, 100 and 200 fictitious “individuals” with systematic variation of pre-test probabilities of two diagnostic categories. For each data set, optimum binary cut-off limits are determined employing different methods. Based on these optimum cut-off thresholds, sensitivities and specificities are calculated for the respective data sets. Mean values and SD of these variables are computed for 1000 repetitions each.ResultsOptimizations of cut-off limits using Youden index and logistic regression-derived likelihood ratio functions with correct adaption for pre-test probabilities both yield reasonably stable results, being nearly independent from pre-test probabilities actually used. Maximizing mutual information yields cut-off levels decreasing with increasing pre-test probability of disease. The most precise results (in terms of the smallest SD) are usually seen for the likelihood ratio method. With this parametric method, however, cut-off values show a significant positive bias and, hence, specificities are usually slightly higher, and sensitivities are consequently slightly lower than with the two non-parametric methods.ConclusionsIn terms of stability and bias, Youden index is best suited for determining optimal cut-off limits of a diagnostic variable. The results of Youden method and likelihood ratio method are surprisingly insensitive against distributional differences as well as pre-test probabilities of the two diagnostic categories. As an additional bonus of the parametric procedure, transfer of the likelihood ratio functions, obtained from logistic regression analysis, to other diagnostic scenarios with different pre-test probabilities is straightforward.Electronic supplementary materialThe online version of this article (doi:10.1186/s12911-014-0099-1) contains supplementary material, which is available to authorized users.
Highlights
Using Monte Carlo simulations, we compare different methods for their ability to determine optimum binary cut-off thresholds for a ratio-scaled diagnostic test variable
Best known is the analysis of simple diagnostic test situations which can be represented by means of a 2 × 2-contingency table: one dimension of such a table is defined by two diagnostic categories (e.g., “non-diseased” versus “diseased”), and the second dimension represents the dichotomous test result (e.g., “normal” versus “pathological”)
An interesting way of evaluating diagnostic tests is provided by information theory [5,6,7,8,9], and an alternative elegant way of dealing with multiple, variably scaled diagnostic variables, has been suggested in 1982 by Albert [10]: he demonstrated that logistic regression analysis can be employed to compute likelihood ratio functions which, in analogy to the well-known likelihood ratio obtained from a simple 2 × 2-contingency table, are useful to compute post-test probability functions of the diagnostic categories investigated
Summary
Using Monte Carlo simulations, we compare different methods (maximizing Youden index, maximizing mutual information, and logistic regression) for their ability to determine optimum binary cut-off thresholds for a ratio-scaled diagnostic test variable. An interesting way of evaluating diagnostic tests is provided by information theory [5,6,7,8,9], and an alternative elegant way of dealing with multiple, variably scaled diagnostic variables, has been suggested in 1982 by Albert [10]: he demonstrated that logistic regression analysis can be employed to compute likelihood ratio functions which, in analogy to the well-known likelihood ratio obtained from a simple 2 × 2-contingency table, are useful to compute post-test probability functions of the diagnostic categories investigated. A critical step when applying logistic regression results for the computation of likelihood ratio functions is a correction according to the pre-test probabilities of the diagnostic categories used for the regression procedure [10].
Sign-in/Register to view highlights & summary Sign-in/Register to access full text options 
Talk to us
Join us for a 30 min session where you can share your feedback and ask us any queries you have
Schedule a callDisclaimer: All third-party content on this website/platform is and will remain the property of their respective owners and is provided on "as is" basis without any warranties, express or implied. Use of third-party content does not indicate any affiliation, sponsorship with or endorsement by them. Any references to third-party content is to identify the corresponding services and shall be considered fair use under The CopyrightLaw.
