Area Under The Receiver Operating Characteristic Research Articles

Smoking Classification Using Novel Plasma Cytokines by implementing Machine Learning and Statistical Methods.

Smoking is a major cause of premature and preventable death. Tobacco exposure has a detrimental effect on many organs and contributes to multiple diseases including chronic obstructive pulmonary disease (COPD), cardiovascular disease, cancer, and diabetes. Cytokines are inflammatory biomarkers that are mechanistically associated with smoking. Machine Learning algorithms allow for the quantitative assessment of the contributions of individual cytokines to tobacco-related diseases. The mapping of cytokines to disease can facilitate and direct treatment modalities. By the application of k Nearest Neighbor (k-NN) and Random Forest machine learning algorithms on 63 plasma cytokines we have demonstrated the classification of smoking. To ensure optimal results, performance improvement techniques such as k-fold cross validation and hyper parameter tuning are employed. Separability efficiency achieved by the models is evaluated using the Area Under the Receiver Operating Characteristic (AUROC) metric. The most significant cytokines that enabled the classification are identified and presented. The statistically significant difference for AUROC score of k-NN and Random Forest has been ascertained using the 2-sample independent t test. A reasonably good classification performance was achieved by k-NN algorithm with an AUROC metric of .87, and a 95% CI of (.823,.917). Random forest exceeded k-NN algorithm's performance, with a perfect AUROC score of 1 and a 95% CI of (1,1). From among the ten most prominent cytokines that contributed to the classification, the ones common to both algorithms are: LIF, IL22, G-CSF/CSF-3, TRAIL. AUROC scores for k-NN and Random Forest are significantly different (p-value = 5.105e-16). The discovery and transference of biomarkers such as cytokines from the platform of molecular investigation to clinical practice, can facilitate precision medicine-based therapeutic interventions.

Utility of end-tidal carbon dioxide to guide resuscitation termination in prolonged out-of-hospital cardiac arrest

Study objectiveTo evaluate if the change in end-tidal carbon dioxide (ETCO2) over time has improved discriminatory value for determining resuscitation futility compared to a single ETCO2 value in prolonged, refractory non-shockable out-of-hospital cardiac arrest (OHCA). MethodsThis is a retrospective analysis of adult refractory non-shockable, non-traumatic OHCA patients in the Portland Cardiac Arrest Epidemiologic Registry (PDX Epistry) from 2018 to 2021. We defined refractory non-shockable OHCA cases as patients with lack of a shockable rhythm at any time or return of spontaneous circulation at any time prior to 30-min of on-scene resuscitation. We abstracted ETCO2 values first recorded after advanced airway placement and nearest to the 30-min mark of on-scene resuscitation (30 min-ETCO2) from EMS charts. The primary outcome was survival to hospital discharge. We compared 30 min-ETCO2 cutoffs of 10 mmHg and 20 mmHg to the trend (increasing or not) from initial to 30 min-ETCO2 (delta-ETCO2) using sensitivity, specificity, and area under the receiver operating curves (AUROC). ResultsOf 3837 adult OHCA, 2850 were initially non-shockable, and there were 617 (16.1%) cases of refractory non-shockable OHCA at 30-min. We excluded 320 cases without at least two ETCO2 recordings in the EMS chart, leaving 297 cases that met inclusion criteria. Of these, 176 (59.3%) were transported and 2 (0.7%) survived to discharge. Using absolute 30 min-ETCO2 cutoffs, both survivors were in the >10 mmHg group (sensitivity 100.0%, specificity 12.5%), whereas only one survivor was identified in the >20 mmHg group (sensitivity 50.0%, specificity 32.5%). Using delta-ETCO2, both survivors were in the increasing ETCO2 group (sensitivity 100.0%, specificity 60.7%). In comparing the two tests that did not misclassify survivors, the AUROC [95% CI] was higher when using delta-ETCO2 (0.803 [0.775–0.831]) compared to an absolute cutoff of 10 mmHg (0.563 [0.544–0.582]). ConclusionsNearly one-sixth of EMS-treated adult OHCA patients had refractory non-shockable arrests after at least 30 min of ongoing resuscitation. In this group, the ETCO2 trend following advanced airway placement may be more accurate in guiding termination of resuscitation than an absolute ETCO2 cutoff of 10 or 20 mmHg.

Fungal and bacterial gut microbiota differ between Clostridioides difficile colonization and infection.

Aim: The bacterial microbiota is well-recognized for its role in Clostridioides difficile colonization and infection, while fungi and yeasts remain understudied. The aim of this study was to analyze the predictive value of the mycobiota and its interactions with the bacterial microbiota in light of C. difficile colonization and infection. Methods: The mycobiota was profiled by ITS2 sequencing of fecal DNA from C. difficile infection (CDI) patients (n = 29), asymptomatically C. difficile colonization (CDC) patients (n = 38), and hospitalized controls with C. difficile negative stool culture (controls; n = 38). Previously published 16S rRNA gene sequencing data of the same cohort were used additionally for machine learning and fungal-bacterial network analysis. Results: CDI patients were characterized by a significantly higher abundance of Candida spp. (MD 0.270 ± 0.089, P = 0.002) and Candida albicans (MD 0.165 ± 0.082, P = 0.023) compared to controls. Additionally, they were deprived of Aspergillus spp. (MD -0.067 ± 0.026, P = 0.000) and Penicillium spp. (MD -0.118 ± 0.043, P = 0.000) compared to CDC patients. Network analysis revealed a positive association between several fungi and bacteria in CDI and CDC, although the analysis did not reveal a direct association between Clostridioides spp. and fungi. Furthermore, the microbiota machine learning model outperformed the models based on the mycobiota and the joint microbiota-mycobiota model. The microbiota classifier successfully distinguished CDI from CDC [Area Under the Receiver Operating Characteristic (AUROC) = 0.884] and CDI from controls (AUROC = 0.905). Blautia and Bifidobacterium were marker genera associated with CDC patients and controls. Conclusion: The gut mycobiota differs between CDI, CDC, and controls and may affect Clostridioides spp. through indirect interactions. The mycobiota data alone could not successfully discriminate CDC from controls or CDI patients and did not have additional predictive value to the bacterial microbiota data. The identification of bacterial marker genera associated with CDC and controls warrants further investigation.

Comparison of the diagnostic utility of CHOKAI, STONE and STONE PLUS scores in predicting ureteral stones larger than 5 mm

Objective: The objective of this study is to assess the diagnostic accuracy of CHOKAI, STONE, and STONE plus scores in detecting ureteral stones larger than 5 mm. Methods: This comparative diagnostic accuracy study was performed in a tertiary care emergency department (ED) and included consecutive patients who presented to the ED with flank pain over a 1-year period. The performance of these scoring scales in detecting ureteral stones measuring >5 mm was determined by area under the receiver operating characteristics (AUROC) comparison. Results: Ureteral stones were detected in 270 (70.3%) of 384 patients included in the study. While 146 patients (54.1%) had ureteral stones measuring <5 mm, 124 patients (54.1%) had stones measuring 5 mm or more in size. The performance of the CHOKAI score in predicting ureteral stones was superior to that of the STONE PLUS score; STONE PLUS score was also superior to STONE score ( p < 0.001). Analysis of the diagnostic value of CHOKAI score and STONE PLUS scores in predicting >5-mm stones showed no statistically significant difference ( p = 0.59). Conclusion: CHOKAI and STONE PLUS scores are useful in predicting >5-mm ureteral stones and have similar diagnostic utility to each other. STONE score does not provide accurate diagnostic information about ureteral stones larger than 5 mm. Level of evidence: Level 2b

Wi-Fi Fingerprint for Indoor Keyless Entry Systems with Ensemble Learning Regression-Classification Model

Keyless entry systems have made human life more comfortable because the possibility of someone leaving a key is non-existent. However, there is a research opportunity to use Wi-Fi fingerprint localization in an indoor keyless entry system. The use of Wi-Fi as a solution is low cost because of the already hdeployed Wi-Fi access points (AP) around buildings. This study uses ensemble learning to utilize Wi-Fi fingerprints for keyless entry systems in indoor environments. The first step of this research is to obtain the UJIIndoorLoc Data Set, an open-source Wi-Fi fingerprint dataset. The next step is to design and implement a keyless entry system based on Wi-Fi Fingerprint. We compared four ensemble learning methods: Random Forest, AdaBoost, gradient boosting, and XGBoost. We use several metrics to compare the four methods, namely r2, the area under the receiver operating curve (AUROC), and the equal error rate (EER). The test results show that for localization based on ensemble regression, XGBoost has the best performance compared to the other three ensemble methods for both longitude and latitude predictions. The r2values are 0.94 and 0.98, respectively. For authentication based on ensemble classification, the gradient-boosting model performance increases as the number of weak learners increases. The optimum number of weak learners is 60. With AUROC = 0.99, gradient boosting outperforms random forest, AdaBoost, and XGBoost on authentication. Finally, our authentication method has EER = 0.01, which outperforms other state-of-the-art keyless entry systems. This research focuses on local building map datasets for future work.

MBMF: Constructing memory banks of multi‐scale features for anomaly detection

AbstractIn industrial manufacturing, how to accurately classify defective products and locate the location of defects has always been a concern. Previous studies mainly measured similarity based on extracting single‐scale features of samples. However, only using the features of a single scale is hard to represent different sizes and types of anomalies. Therefore, the authors propose a set of memory banks of multi‐scale features (MBMF) to enrich feature representation and detect and locate various anomalies. To extract features of different scales, different aggregation functions are designed to produce the feature maps at different granularity. Based on the multi‐scale features of normal samples, the MBMF are constructed. Meanwhile, to better adapt to the feature distribution of the training samples, the authors proposed a new iterative updating method for the memory banks. Testing on the widely used and challenging dataset of MVTec AD, the proposed MBMF achieves competitive image‐level anomaly detection performance (Image‐level Area Under the Receiver Operator Curve (AUROC)) and pixel‐level anomaly segmentation performance (Pixel‐level AUROC). To further evaluate the generalisation of the proposed method, we also implement anomaly detection on the BeanTech AD dataset, a commonly used dataset in the field of anomaly detection, and the Fashion‐MNIST dataset, a widely used dataset in the field of image classification. The experimental results also verify the effectiveness of the proposed method.

Predicting Brain Amyloid Status Using the NIH Toolbox for Assessment of Neurological and Behavioral Function (NIHTB)

AbstractBackgroundAmyloid‐beta (Ab) plaque is a pathophysiological hallmark of Alzheimer’s disease (AD). As anti‐amyloid monoclonal antibodies enter the market, predicting brain amyloid status is critical to determine treatment eligibility. Here, we predicted brain amyloid status in the Advancing Reliable Measurement in Alzheimer’s Disease and Cognitive Aging (ARMADA) study using machine learning.MethodsARMADA, a multisite study, aimed to validate the National Institute of Health Toolbox for Assessment of Neurological and Behavioral Function (NIHTB) in older adults with different cognitive abilities (normal, MCI, early AD dementia). We used cognition, emotion, motor, sensation scores from NIHTB, and demographics to predict amyloid status measured by PET or CSF. Each site used its own criteria to define the amyloid PET positivity (Positive for AD biomarkers, Negative, Visual reading unavailable) or CSF result (Consistent with AD, Not consistent with AD, Borderline, Indeterminate). Amyloid positivity was defined as PET being “Positive for AD biomarkers” or CSF being “Consistent with AD.” We applied LASSO and random forest models and used the area under the receiver operating curve (AUROC) to evaluate the ability to identify amyloid positivity.ResultsOf 462 ARMADA participants, 199 had either PET or CSF information (mean age 76.3 ± 7.7, 51.3% women, 42.3% some or complete college education, 50.3% graduate education, 88.9% White, 33.2% with positive AD biomarkers). The random forest model reached AUROC of 0.74 with higher specificity than sensitivity (AUROC 95% CI:0.73 – 0.76, Sensitivity 0.50, Specificity 0.88) on the held‐out test set; higher than the LASSO model (0.68 (95% CI:0.68 – 0.69)). The 10 features with the highest importance from the random forest model are: picture sequence memory, cognition total composite, cognition fluid composite, list sorting working memory, words‐in‐noise test (hearing), pattern comparison processing speed, odor identification, 2‐minutes‐walk endurance, 4‐meter walk gait speed, and picture vocabulary. Overall, our model revealed the validity of measurements in cognition, motor, and sensation domains, in associating with AD biomarkers.ConclusionOur results support the utilization of the NIH toolbox as an efficient and standardizable AD biomarker measurement that is better at identifying amyloid negative (i.e., high specificity) than positive (i.e., low sensitivity) cases.

Facilitating Clinical Use of the Amsterdam Instrumental Activities of Daily Living Questionnaire: Normative Data and Diagnostic Cutoff Values

AbstractBackgroundThe Amsterdam Instrumental Activities of Daily Living Questionnaire(A‐IADL‐Q) measures impairments in everyday functioning in the context of dementia. To facilitate interpretation and clinical implementation of the A‐IADL‐Q, we aimed to generate demographically‐adjusted norm scores and a clinical cutoff value.MethodCross‐sectional data were used from Dutch Brain Research Registry(N = 1,064; mean (M) age = 62±11year; 69.5% female) European Medial Information Framework‐Alzheimer’s Disease(EMIF‐AD) 90+ (N = 63; Mage = 92±2year; 52.4% female), and European Prevention of Alzheimer’s Dementia Longitudinal Cohort Study(EPAD‐LCS)(N = 300; Mage = 62±7year; 71.7% female)(Table 1). Norm scores were generated using a I) linear regression model approach; and II) Box‐Cox Power Exponential(BCPE) model approach (the latter falling within the generalized model for location shape and size(GAMLSS) framework). For both methods, different combinations of the variables age, sex and education were tested (including polynomial age and age*education interaction) and the best model was selected based on the lowest Aikake Information Criteria(AIC) value. Next, diagnostic contrast was generated using an independent memory clinic sample (the Amsterdam Dementia Cohort(ADC)(N = 2,123, Mage = 64±8year, 46.1% female)(Table 1), including people with dementia (both Alzheimer’s Disease(AD) and non‐AD). The optimal cut‐off value for cognitively healthy versus dementia was determined, and corresponding diagnostic accuracy was calculated using area under the receiver operating curves(AUC‐ROC). This optimal cutoff was then applied to different diagnostic contrasts, including mild cognitive impairment(MCI) and subjective cognitive decline(SCD).ResultThe best suitable normative model was the linear regression model, including cubic polynomial for age, education, and age*education interaction. This model was considered superior to the best BCPE model, which had infinitely large residuals due to bad model fit. The optimal cutoff to distinguish normal functioning from dementia, based on bootstrap average estimates of the Youden Index, was 1.82 standard deviation(SD) below the population mean, resulting in an AUC of 0.97[95%CI 0.98‐0.98]). Normative data are made publicly available in an online tool(Fig.1).ConclusionWe established demographically adjusted norm scores for the A‐IADL‐Q using a large representative sample, and provided an optimal cutoff value for cognitively healthy versus dementia, with outstanding accuracy. Our results, captured in the online tool, can help clinicians to interpret A‐IADL‐Q scores more easily by allowing comparison of functioning against a reference population.

AUCReshaping: improved sensitivity at high-specificity

The evaluation of deep-learning (DL) systems typically relies on the Area under the Receiver-Operating-Curve (AU-ROC) as a performance metric. However, AU-ROC, in its holistic form, does not sufficiently consider performance within specific ranges of sensitivity and specificity, which are critical for the intended operational context of the system. Consequently, two systems with identical AU-ROC values can exhibit significantly divergent real-world performance. This issue is particularly pronounced in the context of anomaly detection tasks, a commonly employed application of DL systems across various research domains, including medical imaging, industrial automation, manufacturing, cyber security, fraud detection, and drug research, among others. The challenge arises from the heavy class imbalance in training datasets, with the abnormality class often incurring a considerably higher misclassification cost compared to the normal class. Traditional DL systems address this by adjusting the weighting of the cost function or optimizing for specific points along the ROC curve. While these approaches yield reasonable results in many cases, they do not actively seek to maximize performance for the desired operating point. In this study, we introduce a novel technique known as AUCReshaping, designed to reshape the ROC curve exclusively within the specified sensitivity and specificity range, by optimizing sensitivity at a predetermined specificity level. This reshaping is achieved through an adaptive and iterative boosting mechanism that allows the network to focus on pertinent samples during the learning process. We primarily investigated the impact of AUCReshaping in the context of abnormality detection tasks, specifically in Chest X-Ray (CXR) analysis, followed by breast mammogram and credit card fraud detection tasks. The results reveal a substantial improvement, ranging from 2 to 40%, in sensitivity at high-specificity levels for binary classification tasks.

Prognostic algorithms for post-discharge readmission and mortality among mother-infant dyads: an observational study protocol.

In low-income country settings, the first six weeks after birth remain a critical period of vulnerability for both mother and newborn. Despite recommendations for routine follow-up after delivery and facility discharge, few mothers and newborns receive guideline recommended care during this period. Prediction modelling of post-delivery outcomes has the potential to improve outcomes for both mother and newborn by identifying high-risk dyads, improving risk communication, and informing a patient-centered approach to postnatal care interventions. This study aims to derive post-discharge risk prediction algorithms that identify mother-newborn dyads who are at risk of re-admission or death in the first six weeks after delivery at a health facility. This prospective observational study will enroll 7,000 mother-newborn dyads from two regional referral hospitals in southwestern and eastern Uganda. Women and adolescent girls aged 12 and above delivering singletons and twins at the study hospitals will be eligible to participate. Candidate predictor variables will be collected prospectively by research nurses. Outcomes will be captured six weeks following delivery through a follow-up phone call, or an in-person visit if not reachable by phone. Two separate sets of prediction models will be built, one set of models for newborn outcomes and one set for maternal outcomes. Derivation of models will be based on optimization of the area under the receiver operator curve (AUROC) and specificity using an elastic net regression modelling approach. Internal validation will be conducted using 10-fold cross-validation. Our focus will be on the development of parsimonious models (5-10 predictor variables) with high sensitivity (>80%). AUROC, sensitivity, and specificity will be reported for each model, along with positive and negative predictive values. The current recommendations for routine postnatal care are largely absent of benefit to most mothers and newborns due to poor adherence. Data-driven improvements to postnatal care can facilitate a more patient-centered approach to such care. Increasing digitization of facility care across low-income settings can further facilitate the integration of prediction algorithms as decision support tools for routine care, leading to improved quality and efficiency. Such strategies are urgently required to improve newborn and maternal postnatal outcomes. https://clinicaltrials.gov/, identifier (NCT05730387).

Application of machine learning algorithms to construct and validate a prediction model for coronary heart disease risk in patients with periodontitis: a population-based study

BackgroundThe association between periodontitis and cardiovascular disease is increasingly recognized. In this research, a prediction model utilizing machine learning (ML) was created and verified to evaluate the likelihood of coronary heart disease in individuals affected by periodontitis.MethodsWe conducted a comprehensive analysis of data obtained from the National Health and Nutrition Examination Survey (NHANES) database, encompassing the period between 2009 and 2014.This dataset comprised detailed information on a total of 3,245 individuals who had received a confirmed diagnosis of periodontitis. Subsequently, the dataset was randomly partitioned into a training set and a validation set at a ratio of 6:4. As part of this study, we conducted weighted logistic regression analyses, both univariate and multivariate, to identify risk factors that are independent predictors for coronary heart disease in individuals who have periodontitis. Five different machine learning algorithms, namely Logistic Regression (LR), Gradient Boosting Machine (GBM), Support Vector Machine (SVM), K-Nearest Neighbor (KNN), and Classification and Regression Tree (CART), were utilized to develop the model on the training set. The evaluation of the prediction models’ performance was conducted on both the training set and validation set, utilizing metrics including AUC (Area under the receiver operating characteristic curve), Brier score, calibration plot, and decision curve analysis (DCA). Additionally, a graphical representation called a nomogram was created using logistic regression to visually depict the predictive model.ResultsThe factors that were found to independently contribute to the risk, as determined by both univariate and multivariate logistic regression analyses, encompassed age, race, presence of myocardial infarction, chest pain status, utilization of lipid-lowering medications, levels of serum uric acid and serum creatinine. Among the five evaluated machine learning models, the KNN model exhibited exceptional accuracy, achieving an AUC value of 0.977. The calibration plot and brier score illustrated the model's ability to accurately estimate probabilities. Furthermore, the model's clinical applicability was confirmed by DCA.ConclusionOur research showcases the effectiveness of machine learning algorithms in forecasting the likelihood of coronary heart disease in individuals with periodontitis, thereby aiding healthcare professionals in tailoring treatment plans and making well-informed clinical decisions.

A Prognostic Nomogram Survival Model for Newly Diagnosed Patients with AIDS-Related Diffuse Large B-Cell Lymphoma: A Multicenter Cohort Study in China

BACKGROUND The prognosis of patients with AIDS-related diffuse large B-cell lymphoma(AR-DLBCL) becomes very poor when the lymphoma relapses or is refractory to first line immunochemotherapy. We aimed to develop a novel ARDPI nomogram prognostic model for risk stratification so as to guide individualized treatment to achieve sustained remission and improve the overall prognosis for newly-diagnosed AR-DLBCL patients. METHODS We interrogated data from 306 patients with newly-diagnosed AR-DLBCL. We filtered variables using LASSO regression and Cox regression to identify prognostic co-variates and develop a survival model, we termed AR-DLBCL Prognostic Index (ARDPI). We evaluated model discrimination, calibration and clinical benefit by Area Under the Receiver-Operator Characteristic (AUROC), calibration plots and decision curve analysis (DCA). Next, we compared the ARDPI model discrimination, calibration and clinical benefit with the IPI and NCCN-IPI models using the same methods. Lastly, we stratified patients into three survival risk cohorts based on ARDPI model by X-tile selecting cutoff point. RESULTS 7 co-variates were independently correlated with survival and were used to develop the ARDPI model including age, lymphocyte monocyte ratio (LMR), CD5 expression on lymphoma cells, blood EBV-DNA copy number, CD4/CD8 ratio, central nervous system (CNS) involvement and anti-HIV therapy (ART). AUROCs of ARDPI model for 1-, 3-, and 5-year were 0.80 (95% Confidence Interval [CI], 0.72, 0.88), 0.78 (0.69, 0.87) and 0.77 (0.63, 0.91). Predicted and calibrated values were concordant. The DCA curve had higher net benefit using the ARDPI model. Prediction accuracy of the ARDPI model was better compared with the IPI and NCCN-IPI models. For example, 3-year survival AUROC in the ARDPI model was 0.78 (0.69, 0.87) compared with the IPI (0.53 [0.43, 0.63] P< 0.001) and the NCCN-IPI (0.52 [0.42, 0.62] P< 0.001). Using the ARDPI model, we identified 3 survival risk cohorts with 3-year survivals of 0.80 , 0.38 and 0.09 ( P<0.001). CONCLUSION The ARDPI has good survival prediction accuracy in newly-diagnosed persons with AIDS-related DLBCL and using it has clinical benefit.Accuracy is better than the IPI and NCCN-IPI models. Validation of our conclusions is needed.

Development of a Real-Time Dashboard Characterizing Acute Leukemia Clinical Trial Enrollment Diversity at the Practice, Investigator, and Individual Clinician Level

Introduction Adult acute leukemia clinical trials in the United States under-enroll persons from historically marginalized groups. Practices, investigators, and individual clinicians have identified the lack of multidimensional, real-time enrollment data as a barrier to remediating participation inequities. We sought to overcome this by developing a visual data interface (“dashboard”) for assessing enrollment diversity across an institution in real-time. Methods Real-time dashboard data sources were the electronic health record (EPIC) and the trial registration system (OnCore); Tableau software was used for data visualization. The primary objective was to create an integrated data source that accurately assigned individuals as patients of the correct clinical group, primary oncologist, and research protocol(s). The primary endpoint was an accurate assignment algorithm, defined as an area under the receiver operator curve (AUROC) of >=0.90, as compared to manual review. Trial enrollment was assigned via OnCore and was itself the reference standard, so algorithm classification was considered accurate if individuals were assigned correctly to the clinical group and primary oncologist. The AUROC of a basic assignment algorithm was 0.78, requiring a sample size of 528 to show improvement to 0.90 or above. Interim testing was performed to assess potential improvements in logic accuracy. Exploratory objectives were to assess associations between exposures (e.g., age, race, ethnicity, sex) and outcomes (treatment trial and biobank enrollment) among a one-year cohort of patients followed by the clinical group, as determined by the final algorithm. Due to small numbers in some race and ethnicity categories, a People of Color aggregate variable was generated and included those listed as Hispanic Black, Other, Unknown, White, and Non-Hispanic Asian, Black, and Other. Associations were determined using parametric or non-parametric bivariate tests, as appropriate; multiplicity was not considered. Results Interim testing was used to revise diagnostic criteria assignment and inpatient encounter algorithms. This improved classification AUROCs to 0.80 and then 0.85, after which the physician assignment algorithm was further revised and finalized. Final testing of 528 patients seen on randomly chosen dates resulted in an AUROC of 0.98 for the assignment algorithm against manual classification. Real-time data visualizations were then developed to display multidimensional enrollment diversity data grouped by patient panels for individual clinicians (Figure 1) and by trial for primary investigators, with additional views showing trends over time and enrollment process metrics (i.e., who is consented, eligible, on study). From 6/22/2022-6/21/2023 there were 432 unique patients followed by the clinical group, of which 124 (28.7%) participated in a treatment trial and 287 (66.4%) in a biobanking study. Patient characteristics and bivariate comparisons of participation by study type are shown in Table 1. Characteristics associated with treatment trial participation included lower Social Deprivation Index score (20 vs 27, p=0.05), residing outside Massachusetts (36.2% vs 25.5%, p=0.03), and having ALL (33.8% vs 25.8%, p=0.02). Characteristics associated with biobank study participation included lower Social Deprivation Index score (21 vs 25, p=0.01) and having ALL (33.8% vs 25.8%, p=0.02). While Persons of Color participated in treatment trials (22.4% vs 29.7%) and biobanks (62.1% vs 66.7%) numerically less than NHW, but differences were not significant (p>0.05). Conclusions An enrollment diversity dashboard can accurately provide real-time data on acute leukemia clinical research participation at a large referral center. In this cohort, enrollment rates were relatively high and associated with socioeconomic status, place of residence, and disease type but not race-ethnicity. The dashboard is now being revised to incorporate trial eligibility metrics and will be piloted as performance feedback for practices, investigators, and individual clinicians.

Use of Machine Learning to Predict 30-Day Reutilization of Care for Patients with Sickle Cell Disease Treated for Vaso-Occlusive Crisis

Background Sickle cell disease (SCD) is an inherited disorder of red blood cells affecting millions of people worldwide. SCD is characterized by recurrent episodes of severe pain attacks, also called vaso-occlusive crisis (VOC), and are the most common reason for hospitalization. Approximately 90% of the hospital admissions are for pain treatment and the hospital readmission rate is alarmingly high, as 30-50% are re-admitted within 30 days. We previously were successful in developing machine learning models to predict pain using inpatient vital signs data as well as Apple Watch data in patients with SCD hospitalized for VOC. We now aimed to predict reutilization of care within 30 days in patients with SCD treated for a VOC. Methods Patients with SCD aged 18 years and above, who were admitted for a VOC to the day hospital or Duke University Hospital between April and June 2022, were eligible for this study. Following informed consent, demographics, SCD genotype, details from the hospitalization such as length of stay, pain scores, as well as vital signs measured per standard of care were collected from the electronic medical records including blood pressure, pulse rate, respiratory rate, oxygen saturation and temperature. Baseline vital signs during regular visits were collected 6 months before and after the hospitalization. We used the values of nearest neighbors to fill in the empty entries. The vital signs over the whole hospital stay were averaged and those numbers were used as the predictor values for the model. The primary outcome was reutilization of care defined as readmission within 30 days to the day hospital and/or hospital. The predictors were used to fit 3 different machine learning classification models for the prediction of reutilization of care: random forest, logistic regression, gradient boosting. By fitting random forest model on the whole dataset, we were able to rank all the features using mean decrease in impurities. To avoid overfitting, we only used the best four predictors which were diastolic blood pressure, pulse rate, respiratory rate and pain score.The performance of the machine learning models was evaluated using the following metrics: accuracy, precision, recall, F1 score and area under the receiver-operating-curve (AUC). Results Eighteen participants with SCD were included in this study. The median age at inclusion was 30 years (IQR 22-34). The majority of the participants had SCD genotype HbSS (68%), and all were Black of African-American. There were 10 participants treated at day hospital (56%), while the other 8 participants were admitted to the hospital with a median length of stay of 7.5 days (IQR 2.5-10). After discharge, 15 participants sought medical care at least once within 30 days (83%); 8 were hospitalized (44%), and 13 were readmitted to the day hospital (72%). This pilot study consisted of 88 vital sign data points across the 18 patients. The metrics of our best-performing machine learning model, the random forest model, were: accuracy 70%, precision 0.94, recall 0.71, F1 score 0.81 and AUC 0.61 (Figure 1 and 2). The difference in precisions and recalls for all the models reflects the class imbalance in the dataset. To test how the model will perform for independent data sets, we used 5-fold cross-validation, and the cross-validation accuracy was 66% with a standard deviation of 7.7%. Conclusion In this pilot study, our machine learning model was able to accurately predict health care reutilization within 30 days following discharge, with real-time vital signs data collected during clinic visits and hospital admissions in participants with SCD. Prediction of reutilization may help healthcare providers identify those at high risk and allow considerations for inpatient and outpatient strategies for patient management. Our next efforts include prediction of re-utilization from data collected from consumer wearables and in a larger number of participants.

Harnessing Artificial Intelligence for Risk Stratification in Acute Myeloid Leukemia (AML): Evaluating the Utility of Longitudinal Electronic Health Record (EHR) Data Via Graph Neural Networks

Background: AML is a life-threatening disease, and to determine which patients need allogeneic stem cell transplantation, hematologists risk-stratify each case. However, standard risk stratification using the European LeukemiaNet (ELN) criteria is focused on baseline mutations and chromosomal aberrations, and the risk estimate is not updated during a patient's course. In other blood cancers, recalculating the risk with treatment response data can help guide the need for more intensive therapy (Kurtz, et al, Cell, 2019). Furthermore, deep learning graph neural networks (GNN) applied to EHR data have strong predictive power in a hematology context (Fouladvand, et al, J Biomed Inform, 2023). Thus, we evaluated the power of a GNN to predict survival in AML using longitudinal EHR data, specifically with labs and histological features that are not included in the ELN but may capture the treatment response. Methods: Patients who were seen at the Stanford Cancer Institute, had EHR data available within six months of diagnosis, and were diagnosed with AML between June 1998 and January 2021 were included in this retrospective analysis. The GNN was trained to predict survival at two years from diagnosis using the first six months of clinical data. Patients were excluded if they were lost to follow-up before two years or died before six months. Data were collected from structured databases associated with Stanford's EHR, except that diagnosis dates were from Stanford's Cancer Registry, and survival data was supplemented with other databases including the Social Security Death Index. Dysplasia, bone marrow cellularity, and bone marrow blast percentages from pathology reports (“pathology report data”) were extracted using text processing algorithms and weakly supervised machine learning (Ratner, et al, ArXiv, 2017). To represent time series information, we framed each patient's timeline as a network (or “graph”) of events. The primary GNN model was a heterogenous graph transformer classifier with two node types: complete blood count (CBC) data and pathology report data (Hu, et al, ArXiv, 2020). Data from the same week were assumed to be from the same timeframe and connected with bidirectional edges. Data separated by longer time periods were connected with unidirectional edges of a separate edge type. The independent test dataset consisted of patients whose ELN 2022 classification was available, and to train the model, the remaining data were divided into train/validation splits of 0.9/0.1. Results: Of the 2,535 patients with survival data, 1,029 met inclusion criteria. Table 1 summarizes the data available in the EHR for each variable, and nearly all patients had CBC and pathology report data. The area under the receiver operating characteristic (AUROC) using the ELN 2022 criteria for predicting survival in the test dataset was 0.79. The AUROC curve for the GNN model was comparable at 0.76, despite not using any variables from the ELN criteria, and the model effectively stratified patients' disease into high- and low-risk in the independent test dataset (hazard ratio [HR] 3.0, log-rank p = 0.0009). Interestingly, despite not having access to mutation or cytogenetic data, the high-risk cases were enriched in known high-risk mutations, like TP53 and RUNX1, and in high-risk chromosomal aberrations, like 5q deletion (Table 1). Although the model predictions correlated with the ELN criteria in some ways, they also stratified the ELN intermediate-risk AML cases into high and low risk (HR 6.1 for model-predicted high risk among ELN intermediate cases, p = 0.07). Conclusions: Risk stratification using artificial intelligence and longitudinal data from the EHR performed comparably to the ELN 2022 criteria and has the potential to further stratify the ELN categories. The model performed well despite only using histological features and lab values, which are more readily available and more frequently updated than next-generation sequencing results. In the future, this approach may further improve with a larger sample size and additional variables, such as measurable residual disease and treatment information. Given the heterogeneity and increasing complexity of AML classification, leveraging artificial intelligence to assist with classification will be crucial, and these results are a step towards a future where data are automatically extracted from the EHR and used for continuously updated risk stratification.

Artificial Intelligence: A Reliable Tool to Detect the Elongation of the Styloid Process.

Background Eagle's syndrome is characterized by the anomalous elongation of the styloid process. This condition is usually identified through the manual evaluation of orthopantomogram (OPG) images, which is time-consuming and can have interobserver variability. The application of Artificial intelligence (AI) in radiology is gaining importance and interest in recent years. The application of AI in detecting styloid process elongation is less explored, advocating for research in the same arena. Aim and objectives The study aimed to evaluate the accuracy of artificial intelligence in detecting styloid process elongation in digital OPGs and to compare the performance of the three different AI algorithms with that of the manual radiographic evaluation by the radiologist. Materials and methods A total of 400 digital OPGs were screened, and linear measurements of the styloid process length (ImageJ software (National Institute of Health, Maryland, USA)) were done for the identification of styloid process elongation by a single calibrated observer to finally include a processed image dataset including 169 images of the elongated styloid process and 200 images of the normal styloid process.A machine learning approach was used to detect the styloid process elongation using the three different AI models: logistic regression, neural network, and Naïve Bayes algorithms in Orange software (University of Ljubljana, Slovenia). Performance evaluation was done using the accuracy, sensitivity, specificity, precision, recall, F1 score, and AUC-ROC (area under the receiver operating characteristic) curve. Results Logistic regression and neural network algorithms depicted the highest accuracy of 100% with no false positives or false negatives, securing a score of 1.000 for all the metrics.However, the Naïve Bayes model demonstrated a fairly considerable accuracy, classifying 49 false positive images and 59 false negative images with an AUC (area under the curve) score of 78 %. Nevertheless, it performed better than random guessing. Conclusion Logistic regression and neural network algorithms accurately detected styloid process elongation similar to that of manual radiographic evaluation. The Naïve Bayes algorithm did not perform an accurate classification yet performed better than random guessing. AI holds a promising scope for its application in automatically detecting styloid process elongation in digital OPGs.

The Predictive Abilities of Machine Learning Algorithms in Patients with Thoracolumbar Spinal Cord Injuries

ObjectivesThe goal of this study is to implement machine learning (ML) algorithms to predict mortality, non-home discharges, prolonged length of stay (LOS), prolonged length of intensive care unit stay (ICU-LOS), and major complications in patients diagnosed with thoracolumbar spinal cord injury (tlSCI), while creating a publicly accessible online tool. MethodsThe American College of Surgeons Trauma Quality Program database was utilized to identify patients with tlSCI. Feature selection was performed with the LASSO algorithm. Five ML algorithms, including TabPFN, TabNet, XGBoost, LightGBM, and Random Forest, were used along with the Optuna optimization library for hyperparameter tuning. Results147,819 patients were included in the analysis. For each outcome, we determined the best model for deployment in our web application based on the area under the receiver operating characteristic (AUROC) values. The top performing algorithms were as follows: LightGBM for mortality with an AUROC of 0.885, TabPFN for non-home discharges with an AUROC of 0.801, LightGBM for prolonged LOS with an AUROC of 0.673, Random Forest for prolonged ICU-LOS with an AUROC of 0.664, LightGBM for major complications with an AUROC of 0.73. ConclusionsML models demonstrate good predictive ability for in-hospital mortality, and non-home discharges, fair predictive ability for major complications, and prolonged ICU-LOS, but poor predictive ability for prolonged LOS. We have developed a web application that allows these models to be accessed.

Optimizing Circulating Tumor DNA Limits of Detection for DLBCL during First Line Therapy

Background: Circulating tumor DNA (ctDNA) assessment is effective in diffuse large B-cell lymphoma (DLBCL) monitoring and risk stratification, with prognostic utility throughout first-line (1L) therapy. DLBCL ctDNA assays vary in analytical sensitivity, or limit of detection (LOD), which range from parts per thousand to below 1 part per million (1 in 10 6). With differing assays and time-points assessed in DLBCL studies, the relationship between analytical and clinical sensitivity for outcome prediction remains unclear. We assessed the prognostic ability of ctDNA at various LODs and used modeling strategies to project the efficacy of assays for minimal residual disease (MRD) detection. Methods: We previously reported a DLBCL dataset assessing plasma ctDNA before, during and after curative-intent 1L anthracycline-based treatment from multiple prospective studies, including RCHOP or EP[O]CH with acalabrutinib, lenalidomide, obinutuzumab, and polatuzumab (Roschewski et al. ASH 2022). In this study, ctDNA was evaluated by Phased Variant Enrichment Detection and Sequencing (PhasED-Seq, Foresight Diagnostics), a ctDNA MRD assay with LOD below 1 in 10 6. Samples from this study and a prospective DLBCL cohort treated with standard 1L therapy at Samsung Medical Center were considered in this analysis. We assessed the prognostic ability of ctDNA detection, considering LODs from 1 in 100 (10 2)to 1 in 10 6 to predict progression free survival (PFS) before, during and after 1L treatment. We assessed the desired sensitivity for ctDNA MRD during treatment by generating patient-specific log-linear models assuming exponential decay. We projected the distribution of variant allele frequencies (VAFs) during therapy from cycles 2 to 5 for patients who progressed to determine the minimal acceptable analytical sensitivity. Results: We included 230 patients consisting of 588 ctDNA samples, with 201 before therapy, 71 at C2D1, 101 at C3D1, 70 at C4D1 and 145 at end of therapy (EOT). Median follow-up was 17.5 months and 62 patients (27%) progressed. To evaluate the impact of LOD at treatment milestones, we applied a threshold for ctDNA positivity ranging from 1 in 10 2 to 1 in 10 6. Increased LOD had no effect on the performance of ctDNA before therapy (Figure A & B). At C2D1, if the LOD was at least 1 in 10 4, there was no difference in MRD prognostic performance. Starting at C3D1, improving the LOD for ctDNA positivity down to 1 in 10 6 showed superior predictive power for PFS at 24 months. The power of ctDNA to predict PFS at 24 months improved later in therapy, with area under the receiver operator curves (AUROCs) for PFS24 of 0.68, 0.73, 0.77, 0.88, and 0.86, at pretreatment, C2D1, C3D1, C4D1, and EOT time-points respectively (Figure A). To further understand the desired LOD for ctDNA MRD, we developed personalized models of ctDNA VAFs. Exponential models fit the data well in 43/44 cases with ≥ 3 MRD-detectable samples through C4D1, with a median correlation of 0.91, confirming their utility for modeling ctDNA. We generated log-linear models for 106 patients with ≥ 2 MRD-detectable samples, and used log-fold change in VAF per cycle (i.e. the slope) to compare patients by progression. Median log-fold change in VAF per cycle was worse for patients who progressed at -1.1 (IQR -1.4, -0.7) compared to those who remained disease-free with median -1.5 (IQR -2.0, -1.1) (p<0.0001). Patients with primary refractory DLBCL on EOT imaging had less robust log-fold change per cycle with median of -0.9 (IQR -1.3, -0.5) compared to those who relapsed after CR with median -1.3 (IQR -1.7, -0.9) (p=0.0004). We used the distribution of slopes to project VAFs for cycles 2 to 5 for patients who progressed to determine the LODs that provide acceptable clinical sensitivity. We found the analytical sensitivity for detecting DLBCL extended lower for each cycle, demonstrating the need for a LOD at least 1 in 10 6 for robust MRD detection at late time-points during 1L treatment. Conclusion: When using an ultrasensitive assay, MRD assessment at later timepoints better predicts PFS than at early timepoints. While the technical LOD does not affect disease burden assessment before therapy, using more sensitive assays during and after therapy improves disease detection and outcome prediction. Utilizing the most sensitive ctDNA MRD assays in 1L DLBCL therapy will maximize the efficacy of MRD-driven therapeutic strategies and MRD as a surrogate endpoint in future trials.

Predicting Major Bleeding in Patients with Venous Thromboembolism on Extended Anticoagulation Therapy Using Follow-up Data and Long Short-Term Memory Based Recurrent Neural Network

Background: It is often challenging for physicians to decide on the duration of anticoagulation treatment for patients with venous thromboembolism (VTE), as they need to weigh the risk of recurrent thrombosis and bleeding at the same time. Several clinical models have been developed to help physicians identify patients with high risk of bleeding. However, these tools use only the baseline clinical information and are not able to incorporate the clinical conditions and events that occur over time that may influence the risk of bleeding. Therefore, capturing the patterns and relationships in the continuously changing time series of clinical data could be a more powerful approach to developing predictor models than just relying on the baseline clinical information. Nonetheless, creating predictive models from follow-up clinical information is challenging given that time series clinical data are non-uniform, high-dimensional, multivariate observations. Here, we present the first attempt at creating a model that uses the time series follow-up information to predict major bleeding over time using a Recurrent Neural Network (RNN) with Long Short-Term Memory (LSTM) cells. Methods: 2542 patients diagnosed with VTE were enrolled in a prospective cohort study over 8 years. In addition to recording their clinical information at baseline, 6-month follow-up interviews were conducted using a standard script to monitor bleeding status and record clinical information. Major bleeding was defined by the International Society on Thrombosis and Haemostasis, with suspected bleeding events classified by an independent adjudication committee. Overall, 118 patients had major bleeding - a 4.6% incidence rate. The median and mode of the clinical variables were used to impute missing numerical and categorical values, respectively, and for patients who had no follow-up information, for whom bleeding occurred before the first follow-up, an artificial follow-up data point was generated from their corresponding baseline data. Thereafter, the data was divided into two stratified sets: 70% for training, and 30% for testing. Five supervised neural network-based machine learning models with different architectures were trained on the baseline dataset, or the follow-up dataset, or both to predict major bleeding. After training, these machine learning models were tested on the testing set and compared to the conventional clinical models, modified to make them compatible with the available predictor variables in our dataset, including the CHAP, the HAS-BLED, the VTE-BLEED, the RIETE, the ACCP, and the OBRI, which only use the baseline information. Results: Overall, the models that used the follow-up information had a higher area under the Receiver Operating Curve (AUROC) or c-statistic compared to the other models that only relied on the baseline dataset. In particular, the LSTM RNN model was able to achieve AUROC of 81.3% that is more than 10% higher compared to the best performing clinical model. We discovered that the LSTM RNN model mostly relied on features such as number of concomitant medications, years since baseline visit, use of specific antibiotics or antiplatelet agents, and presence of new hypertension to predict bleeding from the follow-up dataset. Furthermore, half of the bleeding events occurred within the first year after patients' baseline visits - a trend reflected in the predictions made by LSTM RNN model. Finally, the models that used both the baseline and the follow-up datasets showed different results depending on their architectures; that is, the simpler ensemble model achieved AUROC of 82.5% while the more complex model had AUROC of 70.8% due to overfitting. Conclusion: We have shown that using time series follow-up data can improve bleeding risk prediction in patients with VTE who are on extended anticoagulant therapy compared to just using the baseline data, and clinicians might benefit from using such an approach. Furthermore, our results indicate that LSTM RNN is a suitable architecture to model routine clinical follow-up data. Finally, we believe using time series data could improve the performance of the other clinical models that are currently based on one-time baseline measurements.

879. Host Response Classifiers Identify Infection and Illness Severity and Improve Antibiotic Decision-Making in Patients with Suspected Infections Presenting to the Emergency Department

Abstract Background Nonspecific presentation and limited diagnostic solutions of acute infections and sepsis in emergency departments (EDs) result in early antimicrobial administration at a cost to antimicrobial stewardship. We validated the host response classifiers, IMX-BVN-3b and SEV-3b, to determine bacterial and viral infection status and illness severity. We also assessed antibiotic over- and underuse. Methods We prospectively enrolled adult ED patients with suspected acute infections or sepsis with ≥1 vital sign abnormal across 7 sites. At enrollment, we surveyed treating physicians on infection probability and antibiotic prescribing. BVN/SEV-3b were calculated using NanoString nCounter® on PAXgene® blood samples. We compared BVN-3b likelihood of bacterial and viral infection to post-hoc clinical adjudication infection status and SEV-3b likelihood of severe outcomes to 7-day need for ICU care and 30-day mortality. Results Of 568 enrolled patients, 346 had consensus adjudications (131 bacterial, 52 viral, 1 coinfection, 162 noninfected). The BVN-3b area under the receiver operating curve (AUROC) for bacterial infections was 0.82 (95%CI 0.77-0.87), compared to 0.76 (95%CI 0.71-0.81) for procalcitonin (p = 0.011). BVN-3b AUROC for viral infections was 0.89 (95%CI 0.84-0.95). SEV-3b AUROC was 0.78 (95%CI 0.69-0.86) and 0.88 (95%CI 0.73-1) for 7-day ICU care and 30-day mortality, respectively. Of the 346, 226 patients had a pre-lab result physician questionnaire; of these, 110 patients received antibiotics. Of these, BVN-3b would have corrected 71% of antibiotic prescribing errors (34/48 overprescriptions and 13/18 underprescriptions). Conclusion BVN-3b and SEV-3b accurately identified bacterial and viral infections and risk status. If implemented in a rapid workflow, these tests may improve patient management and antibiotic prescribing, reducing healthcare costs in the ED. Disclosures Arianna Beltran, BS, Inflammatix: Internship Uan-I Chen, MS, Inflammatix: Stocks/Bonds Edward A. Michelson, MD, Inflamatix: Research expense reimbursement for sponsored clinical trial Alexandra Weissman, MD, Inflammatix Inc: Advisor/Consultant Evangelos J. Giamarellos-Bourboulis, PhD, D(ABMM), Abbot Product Operations AG: Grant/Research Support|Abbot Product Operations AG: Honoraria|bioMerieux: Grant/Research Support|bioMerieux: Honoraria|Horizon 2020 European Program: Grant/Research Support|Horizon Health European Program: Grant/Research Support|Sobi AB: Advisor/Consultant|Sobi AB: Grant/Research Support|Sobi AB: Honoraria Oliver Liesenfeld, MD, Inflammatix Inc.: Ownership Interest Natalie N. Whitfield, PhD, D(ABMM), GenMark Dx/Roche: Employee|GenMark Dx/Roche: Stocks/Bonds|Inflammatix: Employee|Inflammatix: Stocks/Bonds