Area Under The Precision-recall Curve Research Articles

Development and external validation of deep learning clinical prediction models using variable-length time series data.

To compare and externally validate popular deep learning model architectures and data transformation methods for variable-length time series data in 3 clinical tasks (clinical deterioration, severe acute kidney injury [AKI], and suspected infection). This multicenter retrospective study included admissions at 2 medical centers that spanned 2007-2022. Distinct datasets were created for each clinical task, with 1 site used for training and the other for testing. Three feature engineering methods (normalization, standardization, and piece-wise linear encoding with decision trees [PLE-DTs]) and 3 architectures (long short-term memory/gated recurrent unit [LSTM/GRU], temporal convolutional network, and time-distributed wrapper with convolutional neural network [TDW-CNN]) were compared in each clinical task. Model discrimination was evaluated using the area under the precision-recall curve (AUPRC) and the area under the receiver operating characteristic curve (AUROC). The study comprised 373825 admissions for training and 256128 admissions for testing. LSTM/GRU models tied with TDW-CNN models with both obtaining the highest mean AUPRC in 2 tasks, and LSTM/GRU had the highest mean AUROC across all tasks (deterioration: 0.81, AKI: 0.92, infection: 0.87). PLE-DT with LSTM/GRU achieved the highest AUPRC in all tasks. When externally validated in 3 clinical tasks, the LSTM/GRU model architecture with PLE-DT transformed data demonstrated the highest AUPRC in all tasks. Multiple models achieved similar performance when evaluated using AUROC. The LSTM architecture performs as well or better than some newer architectures, and PLE-DT may enhance the AUPRC in variable-length time series data for predicting clinical outcomes during external validation.

Interpretable machine learning in predicting drug-induced liver injury among tuberculosis patients: model development and validation study.

The objective of this research was to create and validate an interpretable prediction model for drug-induced liver injury (DILI) during tuberculosis (TB) treatment. A dataset of TB patients from Ningbo City was used to develop models employing the eXtreme Gradient Boosting (XGBoost), random forest (RF), and the least absolute shrinkage and selection operator (LASSO) logistic algorithms. The model's performance was evaluated through various metrics, including the area under the receiver operating characteristic curve (AUROC) and the area under the precision recall curve (AUPR) alongside the decision curve. The Shapley Additive exPlanations (SHAP) method was used to interpret the variable contributions of the superior model. A total of 7,071TB patients were identified from the regional healthcare dataset. The study cohort consisted of individuals with a median age of 47years, 68.0% of whom were male, and 16.3% developed DILI. We utilized part of the high dimensional propensity score (HDPS) method to identify relevant variables and obtained a total of 424 variables. From these, 37 variables were selected for inclusion in a logistic model using LASSO. The dataset was then split into training and validation sets according to a 7:3 ratio. In the validation dataset, the XGBoost model displayed improved overall performance, with an AUROC of 0.89, an AUPR of 0.75, an F1 score of 0.57, and a Brier score of 0.07. Both SHAP analysis and XGBoost model highlighted the contribution of baseline liver-related ailments such as DILI, drug-induced hepatitis (DIH), and fatty liver disease (FLD). Age, alanine transaminase (ALT), and total bilirubin (Tbil) were also linked to DILI status. XGBoost demonstrates improved predictive performance compared to RF and LASSO logistic in this study. Moreover, the introduction of the SHAP method enhances the clinical understanding and potential application of the model. For further research, external validation and more detailed feature integration are necessary.

Machine learning algorithms to predict colistin-induced nephrotoxicity from electronic health records in patients with multidrug-resistant Gram-negative infection

ObjectivesColistin-induced nephrotoxicity prolongs hospitalisation and increases mortality. The study aimed to construct machine learning models to predict colistin-induced nephrotoxicity in patients with multidrug-resistant Gram-negative infection. MethodsPatients receiving colistin from three hospitals in the Clinical Research Database were included. Data were divided into a derivation cohort (2011–2017) and a temporal validation cohort (2018–2020). Fifteen machine learning models were established by categorical boosting, light gradient boosting machine and random forest. Classifier performances were compared by the sensitivity, F1 score, Matthews correlation coefficient (MCC), area under the receiver operating characteristic (AUROC) curve, and area under the precision-recall curve (AUPRC). SHapley Additive exPlanations plots were drawn to understand feature importance and interactions. ResultsThe study included 1392 patients, with 360 (36.4%) and 165 (40.9%) experiencing nephrotoxicity in the derivation and temporal validation cohorts, respectively. The categorical boosting with oversampling achieved the highest performance with a sensitivity of 0.860, an F1 score of 0.740, an MCC of 0.533, an AUROC curve of 0.823, and an AUPRC of 0.737. The feature importance demonstrated that the days of colistin use, cumulative dose, daily dose, latest C-reactive protein, and baseline haemoglobin were the most important risk factors, especially for vulnerable patients. A cutoff colistin dose of 4.0 mg/kg body weight/d was identified for patients at higher risk of nephrotoxicity. ConclusionsMachine learning techniques can be an early identification tool to predict colistin-induced nephrotoxicity. The observed interactions suggest a modification in dose adjustment guidelines. Future geographic and prospective validation studies are warranted to strengthen the real-world applicability.

Predicting haemoglobin deferral using machine learning models: Can we use the same prediction model across countries?

Personalized donation strategies based on haemoglobin (Hb) prediction models may reduce Hb deferrals and hence costs of donation, meanwhile improving commitment of donors. We previously found that prediction models perform better in validation data with a high Hb deferral rate. We therefore investigate how Hb deferral prediction models perform when exchanged with other blood establishments. Donation data from the past 5 years from random samples of 10,000 donors from Australia, Belgium, Finland, the Netherlands and South Africa were used to fit random forest models for Hb deferral prediction. Trained models were exchanged between blood establishments. Model performance was evaluated using the area under the precision-recall curve (AUPR). Variable importance was assessed using SHapley Additive exPlanations (SHAP) values. Across the validation datasets and exchanged models, the AUPR ranged from 0.05 to 0.43. Exchanged models performed similarly within validation datasets, irrespective of the origin of the training data. Apart from subtle differences, the importance of most predictor variables was similar in all trained models. Our results suggest that Hb deferral prediction models trained in different blood establishments perform similarly within different validation datasets, regardless of the deferral rate of their training data. Models learn similar associations in different blood establishments.

Machine learning models predict the emergence of depression in Argentinean college students during periods of COVID-19 quarantine.

The COVID-19 pandemic has exacerbated mental health challenges, particularly depression among college students. Detecting at-risk students early is crucial but remains challenging, particularly in developing countries. Utilizing data-driven predictive models presents a viable solution to address this pressing need. 1) To develop and compare machine learning (ML) models for predicting depression in Argentinean students during the pandemic. 2) To assess the performance of classification and regression models using appropriate metrics. 3) To identify key features driving depression prediction. A longitudinal dataset (N = 1492 college students) captured T1 and T2 measurements during the Argentinean COVID-19 quarantine. ML models, including linear logistic regression classifiers/ridge regression (LogReg/RR), random forest classifiers/regressors, and support vector machines/regressors (SVM/SVR), are employed. Assessed features encompass depression and anxiety scores (at T1), mental disorder/suicidal behavior history, quarantine sub-period information, sex, and age. For classification, models' performance on test data is evaluated using Area Under the Precision-Recall Curve (AUPRC), Area Under the Receiver Operating Characteristic curve, Balanced Accuracy, F1 score, and Brier loss. For regression, R-squared (R2), Mean Absolute Error, and Mean Squared Error are assessed. Univariate analyses are conducted to assess the predictive strength of each individual feature with respect to the target variable. The performance of multi- vs univariate models is compared using the mean AUPRC score for classifiers and the R2 score for regressors. The highest performance is achieved by SVM and LogReg (e.g., AUPRC: 0.76, 95% CI: 0.69, 0.81) and SVR and RR models (e.g., R2 for SVR and RR: 0.56, 95% CI: 0.45, 0.64 and 0.45, 0.63, respectively). Univariate models, particularly LogReg and SVM using depression (AUPRC: 0.72, 95% CI: 0.64, 0.79) or anxiety scores (AUPRC: 0.71, 95% CI: 0.64, 0.78) and RR using depression scores (R2: 0.48, 95% CI: 0.39, 0.57) exhibit performance levels close to those of the multivariate models, which include all features. These findings highlight the relevance of pre-existing depression and anxiety conditions in predicting depression during quarantine, underscoring their comorbidity. ML models, particularly SVM/SVR and LogReg/RR, demonstrate potential in the timely detection of at-risk students. However, further studies are needed before clinical implementation.

VEpiNet: A multimodal interictal epileptiform discharge detection method based on video and electroencephalogram data

To enhance deep learning-based automated interictal epileptiform discharge (IED) detection, this study proposes a multimodal method, vEpiNet, that leverages video and electroencephalogram (EEG) data. Datasets comprise 24 931 IED (from 484 patients) and 166 094 non-IED 4-second video-EEG segments. The video data is processed by the proposed patient detection method, with frame difference and Simple Keypoints (SKPS) capturing patients’ movements. EEG data is processed with EfficientNetV2. The video and EEG features are fused via a multilayer perceptron. We developed a comparative model, termed nEpiNet, to test the effectiveness of the video feature in vEpiNet. The 10-fold cross-validation was used for testing. The 10-fold cross-validation showed high areas under the receiver operating characteristic curve (AUROC) in both models, with a slightly superior AUROC (0.9902) in vEpiNet compared to nEpiNet (0.9878). Moreover, to test the model performance in real-world scenarios, we set a prospective test dataset, containing 215 h of raw video-EEG data from 50 patients. The result shows that the vEpiNet achieves an area under the precision–recall curve (AUPRC) of 0.8623, surpassing nEpiNet’s 0.8316. Incorporating video data raises precision from 70% (95% CI, 69.8%–70.2%) to 76.6% (95% CI, 74.9%–78.2%) at 80% sensitivity and reduces false positives by nearly a third, with vEpiNet processing one-hour video-EEG data in 5.7 min on average. Our findings indicate that video data can significantly improve the performance and precision of IED detection, especially in prospective real clinic testing. It suggests that vEpiNet is a clinically viable and effective tool for IED analysis in real-world applications.

A novel electronic health record-based, machine-learning model to predict severe hypoglycemia leading to hospitalizations in older adults with diabetes: A territory-wide cohort and modeling study.

Older adults with diabetes are at high risk of severe hypoglycemia (SH). Many machine-learning (ML) models predict short-term hypoglycemia are not specific for older adults and show poor precision-recall. We aimed to develop a multidimensional, electronic health record (EHR)-based ML model to predict one-year risk of SH requiring hospitalization in older adults with diabetes. We adopted a case-control design for a retrospective territory-wide cohort of 1,456,618 records from 364,863 unique older adults (age ≥65 years) with diabetes and at least 1 Hong Kong Hospital Authority attendance from 2013 to 2018. We used 258 predictors including demographics, admissions, diagnoses, medications, and routine laboratory tests in a one-year period to predict SH events requiring hospitalization in the following 12 months. The cohort was randomly split into training, testing, and internal validation sets in a 7:2:1 ratio. Six ML algorithms were evaluated including logistic-regression, random forest, gradient boost machine, deep neural network (DNN), XGBoost, and Rulefit. We tested our model in a temporal validation cohort in the Hong Kong Diabetes Register with predictors defined in 2018 and outcome events defined in 2019. Predictive performance was assessed using area under the receiver operating characteristic curve (AUROC), area under the precision-recall curve (AUPRC) statistics, and positive predictive value (PPV). We identified 11,128 SH events requiring hospitalization during the observation periods. The XGBoost model yielded the best performance (AUROC = 0.978 [95% CI 0.972 to 0.984]; AUPRC = 0.670 [95% CI 0.652 to 0.688]; PPV = 0.721 [95% CI 0.703 to 0.739]). This was superior to an 11-variable conventional logistic-regression model comprised of age, sex, history of SH, hypertension, blood glucose, kidney function measurements, and use of oral glucose-lowering drugs (GLDs) (AUROC = 0.906; AUPRC = 0.085; PPV = 0.468). Top impactful predictors included non-use of lipid-regulating drugs, in-patient admission, urgent emergency triage, insulin use, and history of SH. External validation in the HKDR cohort yielded AUROC of 0.856 [95% CI 0.838 to 0.873]. Main limitations of this study included limited transportability of the model and lack of geographically independent validation. Our novel-ML model demonstrated good discrimination and high precision in predicting one-year risk of SH requiring hospitalization. This may be integrated into EHR decision support systems for preemptive intervention in older adults at highest risk.

Deep learning to predict rapid progression of Alzheimer's disease from pooled clinical trials: A retrospective study.

The rate of progression of Alzheimer's disease (AD) differs dramatically between patients. Identifying the most is critical because when their numbers differ between treated and control groups, it distorts the outcome, making it impossible to tell whether the treatment was beneficial. Much recent effort, then, has gone into identifying RPs. We pooled de-identified placebo-arm data of three randomized controlled trials (RCTs), EXPEDITION, EXPEDITION 2, and EXPEDITION 3, provided by Eli Lilly and Company. After processing, the data included 1603 mild-to-moderate AD patients with 80 weeks of longitudinal observations on neurocognitive health, brain volumes, and amyloid-beta (Aβ) levels. RPs were defined by changes in four neurocognitive/functional health measures. We built deep learning models using recurrent neural networks with attention mechanisms to predict RPs by week 80 based on varying observation periods from baseline (e.g., 12, 28 weeks). Feature importance scores for RP prediction were computed and temporal feature trajectories were compared between RPs and non-RPs. Our evaluation and analysis focused on models trained with 28 weeks of observation. The models achieved robust internal validation area under the receiver operating characteristic (AUROCs) ranging from 0.80 (95% CI 0.79-0.82) to 0.82 (0.81-0.83), and the area under the precision-recall curve (AUPRCs) from 0.34 (0.32-0.36) to 0.46 (0.44-0.49). External validation AUROCs ranged from 0.75 (0.70-0.81) to 0.83 (0.82-0.84) and AUPRCs from 0.27 (0.25-0.29) to 0.45 (0.43-0.48). Aβ plasma levels, regional brain volumetry, and neurocognitive health emerged as important factors for the model prediction. In addition, the trajectories were stratified between predicted RPs and non-RPs based on factors such as ventricular volumes and neurocognitive domains. Our findings will greatly aid clinical trialists in designing tests for new medications, representing a key step toward identifying effective new AD therapies.

CAFES: Chest X-ray Analysis using Federated Self-supervised Learning for Pediatric COVID-19 Detection.

Chest X-rays (CXRs) play a pivotal role in cost-effective clinical assessment of various heart and lung related conditions. The urgency of COVID-19 diagnosis prompted their use in identifying conditions like lung opacity, pneumonia, and acute respiratory distress syndrome in pediatric patients. We propose an AI-driven solution for binary COVID-19 versus non-COVID-19 classification in pediatric CXRs. We present a Federated Self-Supervised Learning (FSSL) framework to enhance Vision Transformer (ViT) performance for COVID-19 detection in pediatric CXRs. ViT's prowess in vision-related binary classification tasks, combined with self-supervised pre-training on adult CXR data, forms the basis of the FSSL approach. We implement our strategy on the Rhino Health Federated Computing Platform (FCP), which ensures privacy and scalability for distributed data. The chest X-ray analysis using the federated SSL (CAFES) model, utilizes the FSSL-pre-trained ViT weights and demonstrated gains in accurately detecting COVID-19 when compared with a fully supervised model. Our FSSL-pre-trained ViT showed an area under the precision-recall curve (AUPR) of 0.952, which is 0.231 points higher than the fully supervised model for COVID-19 diagnosis using pediatric data. Our contributions include leveraging vision transformers for effective COVID-19 diagnosis from pediatric CXRs, employing distributed federated learning-based self-supervised pre-training on adult data, and improving pediatric COVID-19 diagnosis performance. This privacy-conscious approach aligns with HIPAA guidelines, paving the way for broader medical imaging applications.

Explainable Machine Learning Model to Preoperatively Predict Postoperative Complications in Inpatients With Cancer Undergoing Major Operations

Explainable Machine Learning Model to Preoperatively Predict Postoperative Complications in Inpatients With Cancer Undergoing Major Operations

Clinical Features Predicting COVID-19 Severity Risk at the Time of Hospitalization.

The global spread of COVID-19 has led to significant mortality and morbidity worldwide. Early identification of COVID-19 patients who are at high risk of developing severe disease can help in improved patient management, care, and treatment, as well as in the effective allocation of hospital resources. The severity prediction at the time of hospitalization can be extremely helpful in deciding the treatment of COVID-19 patients. To this end, this study presents an interpretable artificial intelligence (AI) model, named COVID-19 severity predictor (CoSP) that predicts COVID-19 severity using the clinical features at the time of hospital admission. We utilized a dataset comprising 64 demographic and laboratory features of 7,416 confirmed COVID-19 patients that were collected at the time of hospital admission. The proposed hierarchical CoSP model performs four-class COVID severity risk prediction into asymptomatic, mild, moderate, and severe categories. CoSP yielded better performance with good interpretability, as observed via Shapley analysis on COVID severity prediction compared to the other popular ML methods, with an area under the received operating characteristic curve (AUC-ROC) of 0.95, an area under the precision-recall curve (AUPRC) of 0.91, and a weighted F1-score of 0.83. Out of 64 initial features, 19 features were inferred as predictive of the severity of COVID-19 disease by the CoSP model. Therefore, an AI model predicting COVID-19 severity may be helpful for early intervention, optimizing resource allocation, and guiding personalized treatments, potentially enabling healthcare professionals to save lives and allocate resources effectively in the fight against the pandemic.

A comparative evaluation of machine learning ensemble approaches for disease prediction using multiple datasets

PurposeMachine learning models are used to develop and improve various disease prediction systems. Ensemble learning is a machine learning technique that combines many classifiers to increase performance by making more accurate predictions than a single classifier. Although several researchers have employed ensemble techniques for disease prediction, a comprehensive comparative study of these techniques still needs to be provided.MethodsUsing 16 disease datasets from Kaggle and the UCI Machine Learning Repository, this study compares the performance of 15 variants of ensemble techniques for disease prediction. The comparison was performed using six performance measures: accuracy, precision, recall, F1 score, AUC (Area Under the receiver operating characteristics Curve) and AUPRC (Area Under the Precision-Recall Curve).ResultsStacking variant of Multi-level stacking showed superior disease prediction performance compared with other bagging and boosting variants, followed by another stacking variant (Classical stacking). Overall, stacking outperformed bagging and boosting for disease prediction. Logit Boost showed the worst performance.ConclusionThe findings of this study can help researchers select an appropriate ensemble approach for future studies focusing on accurate disease prediction.

Clinical performance of automated machine learning: A systematic review

Introduction: Automated machine learning (autoML) removes technical and technological barriers to building artificial intelligence models. We aimed to summarise the clinical applications of autoML, assess the capabilities of utilised platforms, evaluate the quality of the evidence trialling autoML, and gauge the performance of autoML platforms relative to conventionally developed models, as well as each other. Method: This review adhered to a prospectively registered protocol (PROSPERO identifier CRD42022344427). The Cochrane Library, Embase, MEDLINE and Scopus were searched from inception to 11 July 2022. Two researchers screened abstracts and full texts, extracted data and conducted quality assessment. Disagreement was resolved through discussion and as if required, arbitration by a third researcher. Results: There were 26 distinct autoML platforms featured in 82 studies. Brain and lung disease were the most common fields of study of 22 specialties. AutoML exhibited variable performance: area under the receiver operator characteristic curve (AUCROC) 0.35–1.00, F1-score 0.16–0.99, area under the precision-recall curve (AUPRC) 0.51–1.00. AutoML exhibited the highest AUCROC in 75.6% trials; the highest F1-score in 42.3% trials; and the highest AUPRC in 83.3% trials. In autoML platform comparisons, AutoPrognosis and Amazon Rekognition performed strongest with unstructured and structured data, respectively. Quality of reporting was poor, with a median DECIDE-AI score of 14 of 27. Conclusion: A myriad of autoML platforms have been applied in a variety of clinical contexts. The performance of autoML compares well to bespoke computational and clinical benchmarks. Further work is required to improve the quality of validation studies. AutoML may facilitate a transition to data-centric development, and integration with large language models may enable AI to build itself to fulfil user-defined goals.

Predicting Successful Weaning from Mechanical Ventilation by Reduction in Positive End-expiratory Pressure Level Using Machine Learning.

Weaning patients from mechanical ventilation (MV) is a critical and resource intensive process in the Intensive Care Unit (ICU) that impacts patient outcomes and healthcare expenses. Weaning methods vary widely among providers. Prolonged MV is associated with adverse events and higher healthcare expenses. Predicting weaning readiness is a non-trivial process in which the positive end-expiratory pressure (PEEP), a crucial component of MV, has potential to be indicative but has not yet been used as the target. We aimed to predict successful weaning from mechanical ventilation by targeting changes in the PEEP-level using a supervised machine learning model. This retrospective study included 12,153 mechanically ventilated patients from Medical Information Mart for Intensive Care (MIMIC-IV) and eICU collaborative research database (eICU-CRD). Two machine learning models (Extreme Gradient Boosting and Logistic Regression) were developed using a continuous PEEP reduction as target. The data is splitted into 80% as training set and 20% as test set. The model's predictive performance was reported using 95% confidence interval (CI), based on evaluation metrics such as area under the receiver operating characteristic (AUROC), area under the precision-recall curve (AUPRC), F1-Score, Recall, positive predictive value (PPV), and negative predictive value (NPV). The model's descriptive performance was reported as the variable ranking using SHAP (SHapley Additive exPlanations) algorithm. The best model achieved an AUROC of 0.84 (95% CI 0.83-0.85) and an AUPRC of 0.69 (95% CI 0.67-0.70) in predicting successful weaning based on the PEEP reduction. The model demonstrated a Recall of 0.85 (95% CI 0.84-0.86), F1-score of 0.86 (95% CI 0.85-0.87), PPV of 0.87 (95% CI 0.86-0.88), and NPV of 0.64 (95% CI 0.63-0.66). Most of the variables that SHAP algorithm ranked to be important correspond with clinical intuition, such as duration of MV, oxygen saturation (SaO2), PEEP, and Glasgow Coma Score (GCS) components. This study demonstrates the potential application of machine learning in predicting successful weaning from MV based on continuous PEEP reduction. The model's high PPV and moderate NPV suggest that it could be a useful tool to assist clinicians in making decisions regarding ventilator management.

Feature selection strategies: a comparative analysis of SHAP-value and importance-based methods

In the context of high-dimensional credit card fraud data, researchers and practitioners commonly utilize feature selection techniques to enhance the performance of fraud detection models. This study presents a comparison in model performance using the most important features selected by SHAP (SHapley Additive exPlanations) values and the model’s built-in feature importance list. Both methods rank features and choose the most significant ones for model assessment. To evaluate the effectiveness of these feature selection techniques, classification models are built using five classifiers: XGBoost, Decision Tree, CatBoost, Extremely Randomized Trees, and Random Forest. The Area under the Precision-Recall Curve (AUPRC) serves as the evaluation metric. All experiments are executed on the Kaggle Credit Card Fraud Detection Dataset. The experimental outcomes and statistical tests indicate that feature selection methods based on importance values outperform those based on SHAP values across classifiers and various feature subset sizes. For models trained on larger datasets, it is recommended to use the model’s built-in feature importance list as the primary feature selection method over SHAP. This suggestion is based on the rationale that computing SHAP feature importance is a distinct activity, while models naturally provide built-in feature importance as part of the training process, requiring no additional effort. Consequently, opting for the model’s built-in feature importance list can offer a more efficient and practical approach for larger datasets and more intricate models.

Abstract 890: Multimodal modeling of digitized histopathology slides improves risk stratification in hormone receptor-positive breast cancer patients

Abstract In early-stage hormone receptor-positive breast cancer, genomic risk scores identify patients who stand to benefit from up-front chemotherapy but introduce financial and logistical hurdles to care. We assembled a cohort of 5,244 patients with 11,671 corresponding whole-side images of breast tumors stained with hematoxylin and eosin. We developed a multimodal machine learning model to infer risk of distal metastatic recurrence from routine clinical data. Specifically, the model interprets text from the pathologist’s report using a large language model and uses self-supervised vision transformers to interpret the corresponding whole-slide image. Tensor fusion joins the modalities to infer Genomic Health’s Oncotype DX recurrence score. Inferred recurrence score from the multimodal model correlated with measured score with a concordance correlation coefficient of 0.64 (95% C.I. 0.59 - 0.69) in the withheld test set, compared to 0.55 (95% C.I. 0.49 - 0.61) and 0.56 (95% C.I. 0.52 - 0.60) for the linguistic and visual unimodal models, respectively. The multimodal model attains an area under the precision-recall curve (AUPRC) of 0.69 (AUROC=0.88) for identifying high-risk disease in the full-information setting (when images and pathology reports with quantitative hormone receptor status and grade are available) in a withheld test set, compared to AUPRC of 0.61 and 0.66 for the linguistic and visual models, respectively. By comparison, in the same full-information setting, the clinical nomogram introduced by Orucevic et al. in 2019 achieves an AUPRC of 0.48. We suggest the operating point at which precision is 94.4% and recall is 33.3%. Digitized whole-slide images of routine breast biopsies and their associated synoptic pathology reports contain much of the information necessary to stratify patients by risk of distal metastatic recurrence, when modeled appropriately. Our model could enable hospitals to rapidly triage the need for genomic risk testing, possibly precluding one third of orders without loss of accuracy. This helps allocate scarce resources for genomic tests and valuable weeks prior to beginning therapy while maintaining the standard of precision oncology. Citation Format: Kevin M. Boehm, Antonio Marra, Jorge S. Reis-Filho, Sarat Chandarlapaty, Fresia Pareja, Sohrab P. Shah. Multimodal modeling of digitized histopathology slides improves risk stratification in hormone receptor-positive breast cancer patients [abstract]. In: Proceedings of the American Association for Cancer Research Annual Meeting 2024; Part 1 (Regular Abstracts); 2024 Apr 5-10; San Diego, CA. Philadelphia (PA): AACR; Cancer Res 2024;84(6_Suppl):Abstract nr 890.

Accurate prediction of pure uric acid urinary stones in clinical context via a combination of radiomics and machine learning.

Oral chemolysis is an effective and non-invasive treatment for uric acid urinary stones. This study aimed to classify urinary stones into either pure uric acid (pUA) or other composition (Others) using non-contrast-enhanced computed tomography scans (NCCTs). Instances managed at our institution from 2019 to 2021 were screened. They were labeled as either pUA or Others based upon composition analyses, and randomly split into training or testing data set. Several instances contained multiple NCCTs which were all collected. In each of NCCTs, individual urinary stone was treated as individual sample. From manually drawn volumes of interest, we extracted original and wavelet radiomics features for each sample. The most important features were then selected via the Least Absolute Shrinkage and Selection Operator for building the final model on a Support Vector Machine. Performance on the testing set was evaluated via accuracy, sensitivity, specificity, and area under the precision-recall curve (AUPRC). There were 302 instances, of which 118 had pUA urinary stones, generating 576 samples in total. From 851 original and wavelet radiomics features extracted for each sample, 10 most important features were ultimately selected. On the testing data set, accuracy, sensitivity, specificity, and AUPRC were 93.9%, 97.9%, 92.2%, and 0.958, respectively, for per-sample prediction, and 90.8%, 100%, 87.5%, and 0.902, respectively, for per-instance prediction. The machine learning algorithm trained with radiomics features from NCCTs can accurately predict pUA urinary stones. Our work suggests a potential assisting tool for stone disease treatment selection.

Development and validation of ‘Patient Optimizer’ (POP) algorithms for predicting surgical risk with machine learning

BackgroundPre-operative risk assessment can help clinicians prepare patients for surgery, reducing the risk of perioperative complications, length of hospital stay, readmission and mortality. Further, it can facilitate collaborative decision-making and operational planning.ObjectiveTo develop effective pre-operative risk assessment algorithms (referred to as Patient Optimizer or POP) using Machine Learning (ML) that predict the development of post-operative complications and provide pilot data to inform the design of a larger prospective study.MethodsAfter institutional ethics approval, we developed a base model that encapsulates the standard manual approach of combining patient-risk and procedure-risk. In an automated process, additional variables were included and tested with 10-fold cross-validation, and the best performing features were selected. The models were evaluated and confidence intervals calculated using bootstrapping. Clinical expertise was used to restrict the cardinality of categorical variables (e.g. pathology results) by including the most clinically relevant values. The models were created with logistic regression (LR) and extreme gradient-boosted trees using XGBoost (Chen and Guestrin, 2016). We evaluated performance using the area under the receiver operating characteristic curve (AUROC) and the area under the precision-recall curve (AUPRC). Data was obtained from a metropolitan university teaching hospital from January 2015 to July 2020. Data collection was restricted to adult patients undergoing elective surgery.ResultsA total of 11,475 adult admissions were included. The performance of XGBoost and LR was very similar across endpoints and metrics. For predicting the risk of any post-operative complication, kidney failure and length-of-stay (LOS), POP with XGBoost achieved an AUROC (95%CI) of 0.755 (0.744, 0.767), 0.869 (0.846, 0.891) and 0.841 (0.833, 0.847) respectively and AUPRC of 0.651 (0.632, 0.669), 0.336 (0.282, 0.390) and 0.741 (0.729, 0.753) respectively. For 30-day readmission and in-patient mortality, POP with XGBoost achieved an AUROC (95%CI) of 0.610 (0.587, 0.635) and 0.866 (0.777, 0.943) respectively and AUPRC of 0.116 (0.104, 0.132) and 0.031 (0.015, 0.072) respectively.ConclusionThe POP algorithms effectively predicted any post-operative complication, kidney failure and LOS in the sample population. A larger study is justified to improve the algorithm to better predict complications and length of hospital stay. A larger dataset may also improve the prediction of additional specific complications, readmission and mortality.

Synthesizing class labels for highly imbalanced credit card fraud detection data

Acquiring labeled datasets often incurs substantial costs primarily due to the requirement of expert human intervention to produce accurate and reliable class labels. In the modern data landscape, an overwhelming proportion of newly generated data is unlabeled. This paradigm is especially evident in domains such as fraud detection and datasets for credit card fraud detection. These types of data have their own difficulties associated with being highly class imbalanced, which poses its own challenges to machine learning and classification. Our research addresses these challenges by extensively evaluating a novel methodology for synthesizing class labels for highly imbalanced credit card fraud data. The methodology uses an autoencoder as its underlying learner to effectively learn from dataset features to produce an error metric for use in creating new binary class labels. The methodology aims to automatically produce new labels with minimal expert input. These class labels are then used to train supervised classifiers for fraud detection. Our empirical results show that the synthesized labels are of high enough quality to produce classifiers that significantly outperform a baseline learner comparison when using area under the precision-recall curve (AUPRC). We also present results of varying levels of positive-labeled instances and their effect on classifier performance. Results show that AUPRC performance improves as more instances are labeled positive and belong to the minority class. Our methodology thereby effectively addresses the concerns of high class imbalance in machine learning by creating new and effective class labels.

Identifying predictors of the tooth loss phenotype in a large periodontitis patient cohort using a machine learning approach

ObjectivesThis study aimed to identify predictors associated with the tooth loss phenotype in a large periodontitis patient cohort in the university setting. MethodsInformation on periodontitis patients and nineteen factors identified at the initial visit was extracted from electronic health records. The primary outcome is tooth loss phenotype (presence or absence of tooth loss). Prediction models were built on significant factors (single or combinatory) selected by the RuleFit algorithm, and these factors were further adopted by regression models. Model performance was evaluated by Area Under the Receiver Operating Characteristic Curve (AUROC) and Area Under the Precision-Recall Curve (AUPRC). Associations between predictors and the tooth loss phenotype were also evaluated by classical statistical approaches to validate the performance of machine learning models. ResultsIn total, 7840 patients were included. The machine learning model predicting the tooth loss phenotype achieved AUROC of 0.71 and AUPRC of 0.66. Age, periodontal diagnosis, number of missing teeth at baseline, furcation involvement, and tooth mobility were associated with the tooth loss phenotype in both machine learning and classical statistical models. ConclusionsThe rule-based machine learning approach improves model explainability compared to classical statistical methods. However, the model's generalizability needs to be further validated by external datasets. Clinical SignificancePredictors identified by the current machine learning approach using the RuleFit algorithm had clinically relevant thresholds in predicting the tooth loss phenotype in a large and diverse periodontitis patient cohort. The results of this study will assist clinicians in performing risk assessment for periodontitis at the initial visit.