Difference In Area Under The ROC Curve Research Articles

STREM-1: A Biomarker of Mortality in Severe Malaria Impacted by Acute Kidney Injury.

Malaria is an important cause of mortality in African children. Identification of biomarkers to identify children at risk of mortality has the potential to improve outcomes. We evaluated 11 biomarkers of host response in 592 children with severe malaria. The primary outcome was biomarker performance for predicting mortality. Biomarkers were evaluated using receiver operating characteristic (ROC) curve analysis comparing the area under the ROC curve (AUROC). Mortality was 7.3% among children in the study with 72% of deaths occurring within 24 hours of admission. Among the candidate biomarkers, soluble triggering receptor expressed on myeloid cells 1 (sTREM-1) had the highest AUROC (0.78 [95% confidence interval, .70-.86]), outperforming several other biomarkers including C-reactive protein and procalcitonin. sTREM-1 was the top-performing biomarker across prespecified subgroups (malaria definition, site, sex, nutritional status, age). Using established cutoffs, we evaluated mortality across sTREM-1 risk zones. Among children with acute kidney injury, 39.9% of children with a critical-risk sTREM-1 result had an indication for dialysis. When evaluated relative to a disease severity score, sTREM-1 improved mortality prediction (difference in AUROC, P = .016). sTREM-1 is a promising biomarker to guide rational allocation of clinical resources and should be integrated into clinical decision support algorithms, particularly when acute kidney injury is suspected.

The Accuracy of the NSQIP Universal Surgical Risk Calculator Compared to Operation-Specific Calculators.

To compare the performance of the ACS NSQIP "universal" risk calculator (N-RC) to operation-specific RCs. Resources have been directed toward building operation-specific RCs because of an implicit belief that they would provide more accurate risk estimates than the N-RC. However, operation-specific calculators may not provide sufficient improvements in accuracy to justify the costs in development, maintenance, and access. For the N-RC, a cohort of 5,020,713 NSQIP patient records were randomly divided into 80% for machine learning algorithm training and 20% for validation. Operation-specific risk calculators (OS-RC) and OS-RCs with operation-specific predictors (OSP-RC) were independently developed for each of 6 operative groups (colectomy, whipple pancreatectomy, thyroidectomy, abdominal aortic aneurysm (open), hysterectomy/myomectomy, and total knee arthroplasty) and 14 outcomes using the same 80%/20% rule applied to the appropriate subsets of the 5M records. Predictive accuracy was evaluated using the area under the receiver operating characteristic curve (AUROC), the area under the precision-recall curve (AUPRC), and Hosmer-Lemeshow (H-L) P values, for 13 binary outcomes, and mean squared error for the length of stay outcome. The N-RC was found to have greater AUROC (P = 0.002) and greater AUPRC (P < 0.001) compared to the OS-RC. No other statistically significant differences in accuracy, across the 3 risk calculator types, were found. There was an inverse relationship between the operation group sample size and magnitude of the difference in AUROC (r = -0.278; P = 0.014) and in AUPRC (r = -0.425; P < 0.001) between N-RC and OS-RC. The smaller the sample size, the greater the superiority of the N-RC. While operation-specific RCs might be assumed to have advantages over a universal RC, their reliance on smaller datasets may reduce their ability to accurately estimate predictor effects. In the present study, this tradeoff between operation specificity and accuracy, in estimating the effects of predictor variables, favors the N-R, though the clinical impact is likely to be negligible.

Development, multi-institutional external validation, and algorithmic audit of an artificial intelligence-based Side-specific Extra-Prostatic Extension Risk Assessment tool (SEPERA) for patients undergoing radical prostatectomy: a retrospective cohort study

Accurate prediction of side-specific extraprostatic extension (ssEPE) is essential for performing nerve-sparing surgery to mitigate treatment-related side-effects such as impotence and incontinence in patients with localised prostate cancer. Artificial intelligence (AI) might provide robust and personalised ssEPE predictions to better inform nerve-sparing strategy during radical prostatectomy. We aimed to develop, externally validate, and perform an algorithmic audit of an AI-based Side-specific Extra-Prostatic Extension Risk Assessment tool (SEPERA). Each prostatic lobe was treated as an individual case such that each patient contributed two cases to the overall cohort. SEPERA was trained on 1022 cases from a community hospital network (Trillium Health Partners; Mississauga, ON, Canada) between 2010 and 2020. Subsequently, SEPERA was externally validated on 3914 cases across three academic centres: Princess Margaret Cancer Centre (Toronto, ON, Canada) from 2008 to 2020; L'Institut Mutualiste Montsouris (Paris, France) from 2010 to 2020; and Jules Bordet Institute (Brussels, Belgium) from 2015 to 2020. Model performance was characterised by area under the receiver operating characteristic curve (AUROC), area under the precision recall curve (AUPRC), calibration, and net benefit. SEPERA was compared against contemporary nomograms (ie, Sayyid nomogram, Soeterik nomogram [non-MRI and MRI]), as well as a separate logistic regression model using the same variables included in SEPERA. An algorithmic audit was performed to assess model bias and identify common patient characteristics among predictive errors. Overall, 2468 patients comprising 4936 cases (ie, prostatic lobes) were included in this study. SEPERA was well calibrated and had the best performance across all validation cohorts (pooled AUROC of 0·77 [95% CI 0·75-0·78] and pooled AUPRC of 0·61 [0·58-0·63]). In patients with pathological ssEPE despite benign ipsilateral biopsies, SEPERA correctly predicted ssEPE in 72 (68%) of 106 cases compared with the other models (47 [44%] in the logistic regression model, none in the Sayyid model, 13 [12%] in the Soeterik non-MRI model, and five [5%] in the Soeterik MRI model). SEPERA had higher net benefit than the other models to predict ssEPE, enabling more patients to safely undergo nerve-sparing. In the algorithmic audit, no evidence of model bias was observed, with no significant difference in AUROC when stratified by race, biopsy year, age, biopsy type (systematic only vs systematic and MRI-targeted biopsy), biopsy location (academic vs community), and D'Amico risk group. According to the audit, the most common errors were false positives, particularly for older patients with high-risk disease. No aggressive tumours (ie, grade >2 or high-risk disease) were found among false negatives. We demonstrated the accuracy, safety, and generalisability of using SEPERA to personalise nerve-sparing approaches during radical prostatectomy. None.

Evaluation of a deep learning method for the automated detection of supraspinatus tears on MRI.

To evaluate if deep learning is a feasible approach for automated detection of supraspinatus tears on MRI. A total of 200 shoulder MRI studies performed between 2015 and 2019 were retrospectively obtained from our institutional database using a balanced random sampling of studies containing a full-thickness tear, partial-thickness tear, or intact supraspinatus tendon. A 3-stage pipeline was developed comprised of a slice selection network based on a pre-trained residual neural network (ResNet); a segmentation network based on an encoder-decoder network (U-Net); and a custom multi-input convolutional neural network (CNN) classifier. Binary reference labels were created following review of radiologist reports and images by a radiology fellow and consensus validation by two musculoskeletal radiologists. Twenty percent of the data was reserved as a holdout test set with the remaining 80% used for training and optimization under a fivefold cross-validation strategy. Classification and segmentation accuracy were evaluated using area under the receiver operating characteristic curve (AUROC) and Dice similarity coefficient, respectively. Baseline characteristics in correctly versus incorrectly classified cases were compared using independent sample t-test and chi-squared. Test sensitivity and specificity of the classifier at the optimal Youden's index were 85.0% (95% CI: 62.1-96.8%) and 85.0% (95% CI: 62.1-96.8%), respectively. AUROC was 0.943 (95% CI: 0.820-0.991). Dice segmentation accuracy was 0.814 (95% CI: 0.805-0.826). There was no significant difference in AUROC between 1.5T and 3.0T studies. Sub-analysis showed superior sensitivity on full-thickness (100%) versus partial-thickness (72.5%) subgroups. Deep learning is a feasible approach to detect supraspinatus tears on MRI.

Development and Evaluation of Nutrition Screening Tool in Patients with Hepatitis B-Related Cirrhosis: A Cross-Sectional Study.

AimMalnutrition is common in cirrhotic patients; however, there is no gold standard for nutritional assessment for this population. The aim of this study was to develop a novel nutritional screening tool based on objective indicators for cirrhotic patients chronically infected with hepatitis B virus (HBV).MethodsThis was a cross-sectional study. Patients with hepatitis B-related cirrhosis were recruited. Malnutrition was diagnosed by the presence of any of the following conditions: Nutrition Risk Screening 2002 score greater than 3 points, Subjective Global Assessment grade B or C, and body mass index (BMI) <18.5 kg/m2. Nomogram model and decision tree model were developed, and the area under the receiver operating characteristic curve (AUROC) was compared.ResultsAmong the 231 studied cases, 92 (39%) were malnourished. Malnourished patients had significantly lower serum albumin, BMI and hand grip strength levels, but higher serum creatinine level and Child–Pugh grade. Two models were developed based on these variables. The nomogram model had a sensitivity of 0.696, a specificity of 0.820 and an accuracy of 0.813. The AUROC of nomogram model was 0.813 (95% CI: 0.758–0.869, p <0.001). For the decision tree model, the sensitivity, specificity and accuracy are 0.761, 0.885 and 0.886, respectively, with an AUROC of 0.886 (95% CI: 0842–0.930, p <0.001). The difference in AUROC between these two models was not statistically significant (p <0.001).ConclusionThe nomogram model and decision tree model developed in this study may aid assessing nutritional status for cirrhotic patients with HBV.

Effect of Lipoprotein(a) on the Diagnosis of Familial Hypercholesterolemia: Does It Make a Difference in the Clinic?

Diagnostic tools for familial hypercholesterolemia (FH) rely on estimation of LDL cholesterol concentration. However, routine measurement or calculation of LDL cholesterol concentration using the Friedewald equation contains a cholesterol contribution from lipoprotein(a) [Lp(a)]. We investigated whether Lp(a) influences the phenotypic diagnosis of FH by commonly used clinical criteria. A cohort of 907 adult index patients attending a clinic were studied. The Dutch Lipid Clinic Network (DLCN) and Simon Broome (SB) diagnostic criteria were estimated before and after adjusting LDL cholesterol concentration for the cholesterol content (30%) of Lp(a). Diagnostic reclassification rates and area under the ROC (AUROC) curves in predicting an FH mutation were also compared. Seventy-four patients defined by DLCN criteria (8.2%) and 207 patients defined by SB criteria (22.8%) were reclassified to "unlikely" FH after adjusting LDL cholesterol for Lp(a) cholesterol. The proportion of FH patients defined by DLCN (probable/definite) and SB (possible/definite) criteria decreased significantly in patients with increased Lp(a) (>0.5 g/L; n = 330) after Lp(a) cholesterol adjustment (P < 0.01). The overall reclassification rate was significantly higher in patients with Lp(a) concentration >1.0 g/L (P < 0.001). The AUROC curve for LDL cholesterol concentration ≥191 mg/dL (≥5.0 mmol/L), DLCN criteria, and SB criteria in predicting an FH mutation increased significantly after adjustment (P < 0.001). There was no significant difference in AUROC curve before and after Lp(a) cholesterol adjustment at an LDL cholesterol concentration ≥251 mg/dL (≥6.5 mmol/L). Adjusting LDL cholesterol concentration for Lp(a) cholesterol improves the diagnostic accuracy of DLCN and SB criteria, especially with Lp(a) >1.0 g/L and LDL cholesterol <251 mg/dL (<6.5 mmol/L). Lp(a) should be measured in all patients suspected of having FH.

Predictive Accuracy of Sweep Frequency Impedance Technology in Identifying Conductive Conditions in Newborns.

Diagnosing conductive conditions in newborns is challenging for both audiologists and otolaryngologists. Although high-frequency tympanometry (HFT), acoustic stapedial reflex tests, and wideband absorbance measures are useful diagnostic tools, there is performance measure variability in their detection of middle ear conditions. Additional diagnostic sensitivity and specificity measures gained through new technology such as sweep frequency impedance (SFI) measures may assist in the diagnosis of middle ear dysfunction in newborns. The purpose of this study was to determine the test performance of SFI to predict the status of the outer and middle ear in newborns against commonly used reference standards. Automated auditory brainstem response (AABR), HFT (1000 Hz), transient evoked otoacoustic emission (TEOAE), distortion product otoacoustic emission (DPOAE), and SFI tests were administered to the study sample. A total of 188 neonates (98 males and 90 females) with a mean gestational age of 39.4 weeks were included in the sample. Mean age at the time of testing was 44.4 hr. Diagnostic accuracy of SFI was assessed in terms of its ability to identify conductive conditions in neonates when compared with nine different reference standards (including four single tests [AABR, HFT, TEOAE, and DPOAE] and five test batteries [HFT + DPOAE, HFT + TEOAE, DPOAE + TEOAE, DPOAE + AABR, and TEOAE + AABR]), using receiver operating characteristic (ROC) analysis and traditional test performance measures such as sensitivity and specificity. The test performance of SFI against the test battery reference standard of HFT + DPOAE and single reference standard of HFT was high with an area under the ROC curve (AROC) of 0.87 and 0.82, respectively. Although the HFT + DPOAE test battery reference standard performed better than the HFT reference standard in predicting middle ear conductive conditions in neonates, the difference in AROC was not significant. Further analysis revealed that the highest sensitivity and specificity for SFI (86% and 88%, respectively) was obtained when compared with the reference standard of HFT + DPOAE. Among the four single reference standards, SFI had the highest sensitivity and specificity (76% and 88%, respectively) when compared against the HFT reference standard. The high test performance of SFI against the HFT and HFT + DPOAE reference standards indicates that the SFI measure has appropriate diagnostic accuracy in detection of conductive conditions in newborns. Hence, the SFI test could be used as adjunct tool to identify conductive conditions in universal newborn hearing screening programs, and can also be used in diagnostic follow-up assessments.

Abstract PR147

Background & Objectives: Predicting long-term neurological outcomes after severe traumatic brain (TBI) is important, but which prognostic model has the best performance remains uncertain. Materials & Methods: This retrospective cohort study included all patients who had severe TBI requiring decompressive craniectomy, between 2004 and 2014, in two neurosurgical centres in Western Australia. Severe disability requiring help with daily living, vegetative state, or death were defined as unfavourable neurological outcomes. Area under the receiver-operating-characteristic curve (AUROC) and slope and intercept of the calibration curve were used to assess discrimination and calibration.of the CRASH (Corticosteroid-Randomisation-After-Significant-Head injury) and IMPACT (International-Mission-For-Prognosis-And-Clinical-Trial) models, respectively. Results: Of the 319 patients included in the study, 118 (37%) had unfavourable neurological outcomes at 18-month after decompressive craniectomy for severe TBI. Both CRASH (AUROC 0.86, 95% confidence interval 0.81–0.90) and IMPACT (AUROC 0.85, 95%CI 0.80–0.89) models were similarly good in discriminating between favourable and unfavourable neurological outcome at 18-month after surgery (p=0.690 for the difference in AUROC derived from the two models). Although both models tended to over-predict the risks of long-term unfavourable outcome, the IMPACT model had a slightly better calibration than the CRASH model (intercept of the calibration curve = -4.1 vs. -5.7, respectively), especially when the predicted risks of unfavourable outcome were <80%. Conclusion: Both CRASH and IMPACT prognostic models were good in discriminating between favourable and unfavourable long-term neurological outcome for patients with severe TBI requiring decompressive craniectomy, but the calibration of the IMPACT full-model was better than the CRASH model. These results suggest that both prognostic models can be used for risk adjustments in randomised controlled trials but the better calibration of the IMPACT model makes it as the preferred model for risk adjustment in observational studies and in providing objective prognostic outcome prediction. Disclosure of Interest: None declared

Predicting long-term neurological outcomes after severe traumatic brain injury requiring decompressive craniectomy: A comparison of the CRASH and IMPACT prognostic models

BackgroundPredicting long-term neurological outcomes after severe traumatic brain (TBI) is important, but which prognostic model in the context of decompressive craniectomy has the best performance remains uncertain. MethodsThis prospective observational cohort study included all patients who had severe TBI requiring decompressive craniectomy between 2004 and 2014, in the two neurosurgical centres in Perth, Western Australia. Severe disability, vegetative state, or death were defined as unfavourable neurological outcomes. Area under the receiver-operating-characteristic curve (AUROC) and slope and intercept of the calibration curve were used to assess discrimination and calibration of the CRASH (Corticosteroid-Randomisation-After-Significant-Head injury) and IMPACT (International-Mission-For-Prognosis-And-Clinical-Trial) models, respectively. ResultsOf the 319 patients included in the study, 119 (37%) had unfavourable neurological outcomes at 18-month after decompressive craniectomy for severe TBI. Both CRASH (AUROC 0.86, 95% confidence interval 0.81–0.90) and IMPACT full-model (AUROC 0.85, 95% CI 0.80–0.89) were similar in discriminating between favourable and unfavourable neurological outcome at 18-month after surgery (p=0.690 for the difference in AUROC derived from the two models). Although both models tended to over-predict the risks of long-term unfavourable outcome, the IMPACT model had a slightly better calibration than the CRASH model (intercept of the calibration curve=−4.1 vs. −5.7, and log likelihoods −159 vs. −360, respectively), especially when the predicted risks of unfavourable outcome were <80%. ConclusionsBoth CRASH and IMPACT prognostic models were good in discriminating between favourable and unfavourable long-term neurological outcome for patients with severe TBI requiring decompressive craniectomy, but the calibration of the IMPACT full-model was better than the CRASH model.

Association of common variants identified by recent genome-wide association studies with obesity in Chinese children: a case-control study.

BackgroundLarge-scale genome-wide association studies have identified multiple genetic variants that are associated with elevated body mass index (BMI) or the risk of obesity in Caucasian or Asian populations. We examined whether these variants are individually associated with obesity in Chinese children, and also assessed their cumulative effects and predictive value for obesity risk in Chinese children.MethodsWe genotyped 40 single nucleotide polymorphisms (SNPs) and conducted association analyses for 32/40 SNPs with an estimated minor allele frequency >1 % in 2 030 unrelated Chinese children, including 607 normal-weight, 718 overweight, and 705 obese individuals from two cross-sectional study groups. Logistic regression and linear regression under the additive model were used to examine associations, and the area under the receiver operating characteristic curve (AUCROC) was reported as prediction summary.ResultsWe identified obesity association for 6 SNPs near SEC16B, RBJ, CDKAL1, TFAP2B, MAP2K5 and FTO (odds ratios (ORs) ranged from 1.19 to 1.41, nominal two-sided P-values < 0.05). Association (Bonferroni corrected) of rs543874 near SEC16B and rs2241423 near MAP2K5 had presumably stronger effects on obesity in Chinese children than in Caucasian populations. Their risk alleles were also associated with BMI standard deviation score (BMI-SDS) variability. We demonstrated the cumulative effects of the 32 SNPs on obesity risk (per risk allele: OR = 1.06, 95 % CI: 1.03-1.11, P = 4.84 × 10-4) and BMI-SDS (β = 0.04, 95 % CI: 0.02-0.06, P = 3.69 × 10-7). The difference in AUCROC for a model with covariates (age, age square, sex and study group) and the model including covariates and all 32 SNPs was 2.8 % (P = 0.0002).ConclusionWhile six SNPs were individually associated with obesity in Chinese children, the 32 common variants identified by recent GWA studies had cumulative effects and resulted in a limited increase in the AUCROC predictive value for childhood obesity.Electronic supplementary materialThe online version of this article (doi:10.1186/s12881-016-0268-4) contains supplementary material, which is available to authorized users.

Prediction of cardiac arrest in critically ill patients presenting to the emergency department using a machine learning score incorporating heart rate variability compared with the modified early warning score

IntroductionA key aim of triage is to identify those with high risk of cardiac arrest, as they require intensive monitoring, resuscitation facilities, and early intervention. We aim to validate a novel machine learning (ML) score incorporating heart rate variability (HRV) for triage of critically ill patients presenting to the emergency department by comparing the area under the curve, sensitivity and specificity with the modified early warning score (MEWS).MethodsWe conducted a prospective observational study of critically ill patients (Patient Acuity Category Scale 1 and 2) in an emergency department of a tertiary hospital. At presentation, HRV parameters generated from a 5-minute electrocardiogram recording are incorporated with age and vital signs to generate the ML score for each patient. The patients are then followed up for outcomes of cardiac arrest or death.ResultsFrom June 2006 to June 2008 we enrolled 925 patients. The area under the receiver operating characteristic curve (AUROC) for ML scores in predicting cardiac arrest within 72 hours is 0.781, compared with 0.680 for MEWS (difference in AUROC: 0.101, 95% confidence interval: 0.006 to 0.197). As for in-hospital death, the area under the curve for ML score is 0.741, compared with 0.693 for MEWS (difference in AUROC: 0.048, 95% confidence interval: -0.023 to 0.119). A cutoff ML score ≥ 60 predicted cardiac arrest with a sensitivity of 84.1%, specificity of 72.3% and negative predictive value of 98.8%. A cutoff MEWS ≥ 3 predicted cardiac arrest with a sensitivity of 74.4%, specificity of 54.2% and negative predictive value of 97.8%.ConclusionWe found ML scores to be more accurate than the MEWS in predicting cardiac arrest within 72 hours. There is potential to develop bedside devices for risk stratification based on cardiac arrest prediction.