ROC Curve Approach Research Articles

Time-Dependent ROC Curve Analysis for Assessing the Capability of Radiation-Induced CD8 T-Lymphocyte Apoptosis to Predict Late Toxicities after Adjuvant Radiotherapy of Breast Cancer Patients.

Late fibrosis can occur in breast cancer patients treated with curative-intent radiotherapy. Predicting this toxicity is of clinical interest in order to adapt the irradiation dose delivered. Radiation-induced CD8 T-lymphocyte apoptosis (RILA) had been proven to be associated with less grade ≥2 late radiation-induced toxicities in patients with miscellaneous cancers. Tobacco smoking status and adjuvant hormonotherapy were also identified as potential factors related to late-breast-fibrosis-free survival. This article evaluates the predictive performance of the RILA using a ROC curve analysis that takes into account the dynamic nature of fibrosis occurrence. This time-dependent ROC curve approach is also applied to evaluate the ability of the RILA combined with the other previously identified factors. Our analysis includes a Monte Carlo cross-validation procedure and the calculation of an expected cost of misclassification, which provides more importance to patients who have no risk of late fibrosis in order to be able to treat them with the maximal irradiation dose. Performance evaluation was assessed at 12, 24, 36 and 50 months. At 36 months, our results were comparable to those obtained in a previous study, thus underlying the predictive power of the RILA. Based on specificity and cost, RILA alone seemed to be the most performant, while its association with the other factors had better negative predictive value results.

Industry 4.0 oriented predictive analytics of cardiovascular diseases using machine learning, hyperparameter tuning and ensemble techniques

PurposeThe world is shifting towards the fourth industrial revolution (Industry 4.0), symbolising the move to digital, fully automated habitats and cyber-physical systems. Industry 4.0 consists of innovative ideas and techniques in almost all sectors, including Smart health care, which recommends technologies and mechanisms for early prediction of life-threatening diseases. Cardiovascular disease (CVD), which includes stroke, is one of the world’s leading causes of sickness and deaths. As per the American Heart Association, CVDs are a leading cause of death globally, and it is believed that COVID-19 also influenced the health of cardiovascular and the number of patients increases as a result. Early detection of such diseases is one of the solutions for a lower mortality rate. In this work, early prediction models for CVDs are developed with the help of machine learning (ML), a form of artificial intelligence that allows computers to learn and improve on their own without requiring to be explicitly programmed.Design/methodology/approachThe proposed CVD prediction models are implemented with the help of ML techniques, namely, decision tree, random forest, k-nearest neighbours, support vector machine, logistic regression, AdaBoost and gradient boosting. To mitigate the effect of over-fitting and under-fitting problems, hyperparameter optimisation techniques are used to develop efficient disease prediction models. Furthermore, the ensemble technique using soft voting is also used to gain more insight into the data set and accurate prediction models.FindingsThe models were developed to help the health-care providers with the early diagnosis and prediction of heart disease patients, reducing the risk of developing severe diseases. The created heart disease risk evaluation model is built on the Jupyter Notebook Web application, and its performance is calculated using unbiased indicators such as true positive rate, true negative rate, accuracy, precision, misclassification rate, area under the ROC curve and cross-validation approach. The results revealed that the ensemble heart disease model outperforms the other proposed and implemented models.Originality/valueThe proposed and developed CVD prediction models aims at predicting CVDs at an early stage, thereby taking prevention and precautionary measures at a very early stage of the disease to abate the predictive maintenance as recommended in Industry 4.0. Prediction models are developed on algorithms’ default values, hyperparameter optimisations and ensemble techniques.

Identification of biomarker panels as predictors of severity in coronary artery disease.

Matrix metalloproteinases (MMPs) are implicated in atherosclerotic plaque rupture and recondition. Specific tissue inhibitors (TIMPs) control MMP functions. Both MMPs and TIMPs are potential biomarkers of plaque instability. Elevated Apo‐CII and CIII and Apo‐E levels are recognized as cardiovascular disease risk factors. We aimed to establish the best blood biomarker panel to evaluate the coronary artery disease (CAD) severity. Plasma levels of MMP‐3 and MMP‐9, TIMP‐1 and TIMP‐2, Apo‐CII, Apo‐CIII and Apo‐E were measured in 472 patients with CAD evaluated by coronary angiography and electrocardiography, and in 285 healthy controls. MMP‐3 and MMP‐9 plasma levels in CAD patients were significantly increased (P < 0.001) compared to controls (3.54‐ and 3.81‐fold, respectively). Furthermore, these increments are modulated by CAD severity as well as for Apo‐CII and Apo‐CIII levels (P < 0.001). TIMPs levels were decreased in CAD versus controls (P < 0.001) and in inverse correlation to MMPs. Standard ROC curve approach showed the importance of panels of biomarkers, including MMP‐3, MMP‐9, TIMP‐1, TIMP‐2, Apo‐CII and Apo‐CIII, for disease aggravation diagnosis. A high area under curve (AUC) value (0.995) was reached for the association of MMP‐9, TIMP‐2 and Apo‐CIII. The unbalance between MMPs and TIMPs in vascular wall and dyslipidaemia creates favourable conditions for plaque disruption. Our study suggests that the combination of MMP‐9, TIMP‐2 and Apo‐CIII values (‘CAD aggravation panel’) characterizes the severity of CAD, that is electrophysiological state, number of involved vessels, stent disposal and type of stent.

Unintentional exposure to pregabalin in ≤6-year-old children: a nationwide French Poison Control Center study

Introduction In France, pregabalin is widely prescribed in adults but still not approved for children. We aimed to investigate the incidence of pregabalin exposure in ≤6-year-old children, to describe the characteristics and outcome of ingestions involving pregabalin alone, and to estimate a clinically relevant toxic dose in this population. Methods Retrospective analysis of pregabalin exposures in ≤6-year-old children, collected by the French Poison Control Centers in 2004–2019. The incidence was estimated using pregabalin prescription data from the Health Improvement Network database (the French version of THIN). The poison severity score (PSS) was used to grade severity. Results We found 313 unintentional immediate-release pregabalin ingestions in ≤6-year-old children. The number of cases per 100,000 pregabalin-treated adults increased over time (p < 0.001). One hundred twenty-six cases involving pregabalin alone (age, 2 years [1.6–3.0] (median [25th–75th percentiles]); median ingested dose 6.4 mg/kg [3.6–10.9]) were analyzed. No child presented an underlying neurological/cardiac disease and/or took concomitant medications. Most of the children (77%) remained asymptomatic (PSS0) while 21% and 2% developed minor (PSS1) or moderate (PSS2) neurological symptoms, respectively. No severe complications/fatalities were reported. All symptomatic children recovered within 24 h. The ingested pregabalin dose was positively correlated with PSS (p < 0.0001). Using a ROC curve approach (area under the curve, 0.85; p < 0.001), ingestion of ≥19.4 mg/kg pregabalin was appropriate to recommend hospital referral (sensitivity, 39% [95% confidence interval (95% CI), 24–56], specificity, 100% [95% CI, 96–100], predictive positive value, 100% [95% CI, 64–100], and negative predictive value, 85% [95% CI, 82–89]). Symptomatic children who ingested <19.4 mg/kg pregabalin developed minor symptoms. Conclusion Despite increasing prescriptions in adults in France, unintentional pregabalin ingestions in ≤6-year-old children remain rare and cause minimal toxicity. Children with no underlying neurological/cardiac disease and concomitant medication ingesting <19.4 mg/kg immediate-release pregabalin alone can be safely observed at home.

P47 A PREDICTION MODEL OF HOSPITAL STAY AFTER ESOPHAGECTOMY WITHIN AN ENHANCED RECOVERY AFTER SURGERY (ERAS) PATHWAY

Abstract Background and aims ERAS principles have been applied even in oesophageal surgery, however low compliance and high rates of deviation from protocols are often reported. Aim of this study was to assess whether ERAS protocol deviation and other perioperative factors could be associated with a delayed discharge. Methods Patients who underwent esophagectomy at our Institution between 2014 and 2017 were included in the study. All patients were managed according to our perioperative ERAS protocol. In-hospital deaths were excluded. All data were prospectively entered in an electronic database. The 8th postoperative day (POD) was considered as the ideal day for discharge and therefore our outcome. A retrospective analysis of most relevant ERAS items and clinical features was conducted in order to identify variables positively or negatively associated to a delayed discharge. Variables associated with our outcome were exploited to build a score able to predict delayed discharge. An initial validation of the accuracy of the score was carried out on a separate patients’ sample. A χ2 test assessed association between outcome and categorical variables. Adapted standardized residuals of multilevel categorical variables allowed a dichotomous division. A new χ2 test confirmed the association. A ROC curve approach evaluated the role of continuous variables as markers of hospitalization longer than outcome. A χ2 test assessed the presence of association between the dichotomized variable and outcome. Backward logistic binary regression selected significant predictors with a selection criteria of p-value of &lt;.05. Results 286 esophagectomies were included in the study. ASA score *** 3, surgery duration *** 255 minutes and failure to mobilize within 24 hours were associated with a delayed discharge. “Hybrid” oesophagectomy was linked to a discharge within the target day. The logistic regression model was statistically significant (p &lt; 0,001) and correctly classified 81,9% of cases. Sensitivity was 96,6%. Positive predictive value was 83,6% and negative 45,4%. The predictive score system, once applied to control population, correctly identified 100% of those discharged after 8th POD. Conclusions Our results seem to be clinically meaningful. All these parameters can be retrieved within the first 24 hours. The initial validation revealed good predictive properties.

Prediction of the landslide susceptibility: Which algorithm, which precision?

Coupling machine learning algorithms with spatial analytical techniques for landslide susceptibility modeling is a worth considering issue. So, the current research intend to present the first comprehensive comparison among the performances of ten advanced machine learning techniques (MLTs) including artificial neural networks (ANNs), boosted regression tree (BRT), classification and regression trees (CART), generalized linear model (GLM), generalized additive model (GAM), multivariate adaptive regression splines (MARS), naïve Bayes (NB), quadratic discriminant analysis (QDA), random forest (RF), and support vector machines (SVM) for modeling landslide susceptibility and evaluating the importance of variables in GIS and R open source software. This study was carried out in the Ghaemshahr Region, Iran. The performance of MLTs has been evaluated using the area under ROC curve (AUC-ROC) approach. The results showed that AUC values for ten MLTs vary from 62.4 to 83.7%. It has been found that the RF (AUC=83.7%) and BRT (AUC=80.7%) have the best performances comparison to other MLTs.

Time-dependent ROC curve analysis in medical research: current methods and applications

BackgroundROC (receiver operating characteristic) curve analysis is well established for assessing how well a marker is capable of discriminating between individuals who experience disease onset and individuals who do not. The classical (standard) approach of ROC curve analysis considers event (disease) status and marker value for an individual as fixed over time, however in practice, both the disease status and marker value change over time. Individuals who are disease-free earlier may develop the disease later due to longer study follow-up, and also their marker value may change from baseline during follow-up. Thus, an ROC curve as a function of time is more appropriate. However, many researchers still use the standard ROC curve approach to determine the marker capability ignoring the time dependency of the disease status or the marker.MethodsWe comprehensively review currently proposed methodologies of time-dependent ROC curves which use single or longitudinal marker measurements, aiming to provide clarity in each methodology, identify software tools to carry out such analysis in practice and illustrate several applications of the methodology. We have also extended some methods to incorporate a longitudinal marker and illustrated the methodologies using a sequential dataset from the Mayo Clinic trial in primary biliary cirrhosis (PBC) of the liver.ResultsFrom our methodological review, we have identified 18 estimation methods of time-dependent ROC curve analyses for censored event times and three other methods can only deal with non-censored event times. Despite the considerable numbers of estimation methods, applications of the methodology in clinical studies are still lacking.ConclusionsThe value of time-dependent ROC curve methods has been re-established. We have illustrated the methods in practice using currently available software and made some recommendations for future research.

Ion mobility spectrometry nuisance alarm threshold analysis for illicit narcotics based on environmental background and a ROC-curve approach.

The discriminative potential of an ion mobility spectrometer (IMS) for trace detection of illicit narcotics relative to environmental background was investigated with a receiver operating characteristic (ROC) curve framework. The IMS response of cocaine, heroin, methamphetamine, 3,4-methylenedioxymethamphetamine (MDMA), and Δ(9)-tetrahydro-cannabinol (THC) was evaluated against environmental background levels derived from the screening of incoming delivery vehicles at a federal facility. Over 20 000 samples were collected over a multiyear period under two distinct sets of instrument operating conditions, a baseline mode and an increased desorption/drift tube temperature and sampling time mode. ROC curves provided a quantifiable representation of the interplay between sensitivity (true positive rate, TPR) and specificity (1 - false positive rate, FPR). A TPR of 90% and minimized FPR were targeted as the detection limits of IMS for the selected narcotics. MDMA, THC, and cocaine demonstrated single nanogram sensitivity at 90% TPR and <10% FPR, with improvements to both MDMA and cocaine in the elevated temperature/increased sampling mode. Detection limits in the tens of nanograms with poor specificity (FPR ≈ 20%) were observed for methamphetamine and heroin under baseline conditions. However, elevating the temperature reduced the background in the methamphetamine window, drastically improving its response (90% TPR and 3.8% FPR at 1 ng). On the contrary, the altered mode conditions increased the level of background for THC and heroin, partially offsetting observed enhancements to desorption. The presented framework demonstrated the significant effect environmental background distributions have on sensitivity and specificity.

Suspicious Prenasal Skin Thickness-to-Nasal Bone Length Ratio: Prevalence and Correlation with Other Markers in Second and Third Trimester Fetuses with Down Syndrome.

To assess the prevalence and value of a suspicious prenasal skin thickness-to-nasal bone length ratio (PT/NB ratio) in comparison to other established markers in a large population of Down syndrome (DS) fetuses. This was a retrospective study of 139 DS fetuses and 530 normal fetuses scanned after 14 + 0 weeks of gestation. To characterize diagnostic performance, we used the ROC curve approach. The presence or absence of a PT/NB ratio > 0.8 and 11 other markers were assessed in the group of DS fetuses. A correlation analysis was performed in order to investigate associations between PT/NB ratio and other markers. Among DS fetuses the median PT/NB ratio was 1.06 (IQR 0.729) and was significantly higher compared to normal fetuses with 0.62 (IQR 0.148), (p < 0.001). Gestational age had no influence on the PT/NB ratio. A PT/NB ratio > 0.8 had the highest prevalence of all markers with 89.2 % in the group of DS fetuses, 3 cases were negative for all markers and 3 cases were positive only for PT/NB ratio > 0.8. Marker-specific comparison between prevalences of a suspicious PT/NB ratio with respect to the presence or absence of other markers was statistically significant for hypoplastic NB and major anomalies (p < 0.05). Utilization of at least one of the following five markers was sufficient for detecting 136 out of 139 fetuses with trisomy 21: suspicious PT/NB ratio, hypoplastic NB, nuchal fold thickness, white spot, shortened femur. The PT/NB ratio is one of the most powerful indicators of DS in the second trimester. It is objective to interpret, easy to measure, and is reproducible.

What is a clinically relevant change on the HIT-6 questionnaire? An estimation in a primary-care population of migraine patients

To interpret questionnaire scores, clinicians and researchers need to know what change in score reflects a meaningful change in the condition of an individual patient, and what difference reflects a meaningful difference between groups. These values differ between different populations. We determined the within-person minimally important change (MIC) and the between-group minimally important difference (MID) of the Headache Impact Test-6 (HIT-6) questionnaire in a primary-care population of migraine patients. We included 490 patients who participated in a clinical trial on the treatment of migraine in primary care. We compared their change scores on the HIT-6 questionnaire between baseline and at three-months follow-up with the answers to two anchor questions according to the 'mean change approach' and the 'ROC curve approach'. The within-person MIC was estimated to be between -2.5 points (mean change approach) and -6 points (ROC curve approach). The choice for the within-person MIC value depends on the consequences of false positives and false negatives in a particular setting. The between-group MID was estimated at -1.5 points. We determined the within-person MIC and between-group MID for the HIT-6 in a primary-care population of migraine patients. We recommend the use of these values for clinical care and for research.

Comparing the estimates of ROC curves by modeling methods

We investigate parameter estimation problems using the ROC curve approach. We compare several parametric, semiparametric, and nonparametric estimates of ROC curves on the assumption that the model is binormal. Our comparison is based on the analysis of numerical examples: we use generalized least square method for parametric estimation, functional modeling for semiparametric, and sdf for nonparametric estimation.

Prediction of disease status: A regressive model approach for repeated measures

In this paper, regressive models are proposed for modeling a sequence of transitions in longitudinal data. These models are employed to predict the future status of the outcome variable of the individuals on the basis of their underlying background characteristics or risk factors. The estimation of parameters and also estimates of conditional and unconditional probabilities are shown for repeated measures. The goodness of fit tests are extended in this paper on the basis of the deviance and the Hosmer–Lemeshow procedures and generalized to repeated measures. In addition, to measure the suitability of the proposed models for predicting the disease status, we have extended the ROC curve approach to repeated measures. The procedure is shown for the conditional models for any order as well as for the unconditional model, to predict the outcome at the end of the study. The test procedures are also suggested. For testing the differences between areas under the ROC curves in subsequent follow-ups, two different test procedures are employed, one of which is based on permutation test. In this paper, an unconditional model is proposed on the basis of conditional models for the disease progression of depression among the elderly population in the USA on the basis of the Health and Retirement Survey data collected longitudinally. The illustration shows that the disease progression observed conditionally can be employed to predict the outcome and the role of selected variables and the previous outcomes can be utilized for predictive purposes. The results show that the percentage of correct predictions of a disease is quite high and the measures of sensitivity and specificity are also reasonably impressive. The extended measures of area under the ROC curve show that the models provide a reasonably good fit in terms of predicting the disease status during a long period of time. This procedure will have extensive applications in the field of longitudinal data analysis where the objective is to obtain estimates of unconditional probabilities on the basis of series of conditional transitional models.

ROC-curve approach for determining the detection limit of a field chemical sensor

The detection limit of a field chemical sensor under realistic operating conditions is determined by receiver operator characteristic (ROC) curves. The chemical sensor is an ion mobility spectrometry (IMS) device used to detect a chemical marker in diesel fuel. The detection limit is the lowest concentration of the marker in diesel fuel that obtains the desired true-positive probability (TPP) and false-positive probability (FPP). A TPP of 0.90 and a FPP of 0.10 were selected as acceptable levels for the field sensor in this study. The detection limit under realistic operating conditions is found to be between 2 to 4 ppm (w/w). The upper value is the detection limit under challenging conditions. The ROC-based detection limit is very reliable because it is determined from multiple and repetitive sensor analyses under realistic circumstances. ROC curves also clearly illustrate and gauge the effects data preprocessing and sampling environments have on the sensor's detection limit.

Virtual Screening Workflow Development Guided by the “Receiver Operating Characteristic” Curve Approach. Application to High-Throughput Docking on Metabotropic Glutamate Receptor Subtype 4

The "receiver operating characteristic" (ROC) curve method is a well-recognized metric used as an objective way to evaluate the ability of a given test to discriminate between two populations. This facilitates decision-making in a plethora of fields in which a wrong judgment may have serious consequences including clinical diagnosis, public safety, travel security, and economic strategies. When virtual screening is used to speed-up the drug discovery process in pharmaceutical research, taking the right decision upon selecting or discarding a molecule prior to in vitro evaluation is of paramount importance. Characterizing both the ability of a virtual screening workflow to select active molecules and the ability to discard inactive ones, the ROC curve approach is well suited for this critical decision gate. As a case study, the first virtual screening workflow focused on metabotropic glutamate receptor subtype 4 (mGlu4R) agonists is reported here. Six compounds out of 38 selected and tested in vitro were shown to have agonist activity on this target of therapeutic interest.

Large‐scale environmental correlates of forest tree distributions in Catalonia (NE Spain)

ABSTRACTAim To explore the environmental correlates of tree species distributions in Catalonia according to the chorological status of the species.Location The study area is the region of Catalonia, in north‐eastern Spain.Methods We used presence‐absence data for 24 species, sampled in random plots distributed throughout forests of Catalonia. A climate model for the Catalonia region provided environmental variables. We used classification tree analysis to explore the environmental correlates of the realized niches of tree species. The predictive accuracy of the models was assessed using the ROC curve approach. Potential distribution maps of tree species were generated for the whole Catalonia region.Results Models were ranked from low to high accuracy for the 24 species. Differences in accuracy among species were related to the chorological status of species. Zonal species, or species at the core of their range (Mediterranean and Sub‐Mediterranean species), were generally well predicted, while extrazonal species, or species at the edge of their range, were predicted only moderately well. Mediterranean species distributions showed good correlations with extreme temperatures and annual precipitation.Main conclusions The above trends confirmed the difficulty of identifying the realized niche of species at the edges of their ranges. In contrast, Mediterranean and Sub‐Mediterranean species, which were at the core of their range, were well‐predicted, confirming the importance of extremes of temperature and annual precipitation as effective surrogates for variables having more direct physiological roles in limiting the ability of plants to survive and grow. Maps of potential tree distributions allowed us to define suitable habitats and to highlight areas where species have been planted outside their natural distribution.

P48 Summary roc curve approach to assess the role of prognostic factors in liver cirrhosis

P48 Summary roc curve approach to assess the role of prognostic factors in liver cirrhosis

How to Correct for Chance Agreement in the Estimation of Sensitivity and Specificity of Diagnostic Tests

The traditional concept of describing the validity of a diagnostic test neglects the presence of chance agreement between test result and true (disease) status. Sensitivity and specificity, as the fundamental measures of validity, can thus only be considered in conjunction with each other to provide an appropriate basis for the evaluation of the capacity of the test to discriminate truly diseased from truly undiseased subjects. In this paper, chance-corrected analogues of sensitivity and specificity are presented as supplemental measures of validity, which pay attention to the problem of chance agreement and offer the opportunity to be interpreted separately. While recent proposals of chance-correction techniques, suggested by several authors in this context, lead to measures which are dependent on disease prevalence, our method does not share this major disadvantage. We discuss the extension of the conventional ROC-curve approach to chance-corrected measures of sensitivity and specificity. Furthermore, point and asymptotic interval estimates of the parameters of interest are derived under different sampling frameworks for validation studies. The small sample behavior of the estimates is investigated in a simulation study, leading to a logarithmic modification of the interval estimate in order to hold the nominal confidence level for small samples.

Multivariate analysis in solitary cold thyroid nodules for the diagnosis of malignancy

A stepwise logistic regression (SLR) was performed on 162 patients with a solitary and cold thyroid nodule in order to discriminate between malignant and benign lesions. Sixteen variables were recorded for each patient. The predictions of the logistic regression model were compared to the histological diagnoses in order to evaluate the accuracy of the classification. The value of the logistic function (LF) was calculated for each patient. Using the ROC curve approach, an optimum threshold value (OTV) corresponding to a 100% sensitivity was defined. The classification obtained with the OTV was validated using a cross-validation procedure (CVP). The significant variables selected by the SLR are (from the most significant to the least significant): cytologic result, sex, irregular margin on the Tc scintigraphy and homogeneity on the ultrasound examination. The OTV corresponds to a specificity of 73% for a sensitivity of 100%. The specificity and the sensitivity obtained with the CVP are 73% and 96% respectively. In conclusion, the classification of the patients according to the value of the LF is a highly accurate diagnostic procedure.

Progress in evaluation of human observer visual detection performance using the ROC curve approach

This perspective is focused on amide groups of peptides interacting with water. 2D IR spectroscopy has already enabled structural aspects of the peptide backbone to be determined through its ability to measure the coupling between different amide-I modes. Here we describe why nonlinear IR is emerging as the method of choice to examine the fast components of the water dynamics near peptides and how isotopically edited peptide links can be used to probe the local water at a residue level in proteins. This type of research necessarily involves an intimate mix of theory and experiment. The description of the results is underpinned by relatively well established quantum-statistical theories that describe the important manifestations of peptide vibrational frequency fluctuations.