Precision-recall Curve Research Articles

Deep learning to predict rapid progression of Alzheimer's disease from pooled clinical trials: A retrospective study.

The rate of progression of Alzheimer's disease (AD) differs dramatically between patients. Identifying the most is critical because when their numbers differ between treated and control groups, it distorts the outcome, making it impossible to tell whether the treatment was beneficial. Much recent effort, then, has gone into identifying RPs. We pooled de-identified placebo-arm data of three randomized controlled trials (RCTs), EXPEDITION, EXPEDITION 2, and EXPEDITION 3, provided by Eli Lilly and Company. After processing, the data included 1603 mild-to-moderate AD patients with 80 weeks of longitudinal observations on neurocognitive health, brain volumes, and amyloid-beta (Aβ) levels. RPs were defined by changes in four neurocognitive/functional health measures. We built deep learning models using recurrent neural networks with attention mechanisms to predict RPs by week 80 based on varying observation periods from baseline (e.g., 12, 28 weeks). Feature importance scores for RP prediction were computed and temporal feature trajectories were compared between RPs and non-RPs. Our evaluation and analysis focused on models trained with 28 weeks of observation. The models achieved robust internal validation area under the receiver operating characteristic (AUROCs) ranging from 0.80 (95% CI 0.79-0.82) to 0.82 (0.81-0.83), and the area under the precision-recall curve (AUPRCs) from 0.34 (0.32-0.36) to 0.46 (0.44-0.49). External validation AUROCs ranged from 0.75 (0.70-0.81) to 0.83 (0.82-0.84) and AUPRCs from 0.27 (0.25-0.29) to 0.45 (0.43-0.48). Aβ plasma levels, regional brain volumetry, and neurocognitive health emerged as important factors for the model prediction. In addition, the trajectories were stratified between predicted RPs and non-RPs based on factors such as ventricular volumes and neurocognitive domains. Our findings will greatly aid clinical trialists in designing tests for new medications, representing a key step toward identifying effective new AD therapies.

Predicting cytogenetic risk in multiple myeloma using conventional whole-body MRI, spinal dynamic contrast-enhanced MRI, and spinal diffusion-weighted imaging

ObjectivesCytogenetic abnormalities are predictors of poor prognosis in multiple myeloma (MM). This paper aims to build and validate a multiparametric conventional and functional whole-body MRI-based prediction model for cytogenetic risk classification in newly diagnosed MM.MethodsPatients with newly diagnosed MM who underwent multiparametric conventional whole-body MRI, spinal dynamic contrast-enhanced (DCE-)MRI, spinal diffusion-weighted MRI (DWI) and had genetic analysis were retrospectively included (2011–2020/Ghent University Hospital/Belgium). Patients were stratified into standard versus intermediate/high cytogenetic risk groups. After segmentation, 303 MRI features were extracted. Univariate and model-based methods were evaluated for feature and model selection. Testing was performed using receiver operating characteristic (ROC) and precision-recall curves. Models comparing the performance for genetic risk classification of the entire MRI protocol and of all MRI sequences separately were evaluated, including all features. Four final models, including only the top three most predictive features, were evaluated.ResultsThirty-one patients were enrolled (mean age 66 ± 7 years, 15 men, 13 intermediate-/high-risk genetics). None of the univariate models and none of the models with all features included achieved good performance. The best performing model with only the three most predictive features and including all MRI sequences reached a ROC-area-under-the-curve of 0.80 and precision-recall-area-under-the-curve of 0.79. The highest statistical performance was reached when all three MRI sequences were combined (conventional whole-body MRI + DCE-MRI + DWI). Conventional MRI always outperformed the other sequences. DCE-MRI always outperformed DWI, except for specificity.ConclusionsA multiparametric MRI-based model has a better performance in the noninvasive prediction of high-risk cytogenetics in newly diagnosed MM than conventional MRI alone.Critical relevance statementAn elaborate multiparametric MRI-based model performs better than conventional MRI alone for the noninvasive prediction of high-risk cytogenetics in newly diagnosed multiple myeloma; this opens opportunities to assess genetic heterogeneity thus overcoming sampling bias.Key points• Standard genetic techniques in multiple myeloma patients suffer from sampling bias due to tumoral heterogeneity.• Multiparametric MRI noninvasively predicts genetic risk in multiple myeloma.• Combined conventional anatomical MRI, DCE-MRI, and DWI had the highest statistical performance to predict genetic risk.• Conventional MRI alone always outperformed DCE-MRI and DWI separately to predict genetic risk. DCE-MRI alone always outperformed DWI separately, except for the parameter specificity to predict genetic risk.• This multiparametric MRI-based genetic risk prediction model opens opportunities to noninvasively assess genetic heterogeneity thereby overcoming sampling bias in predicting genetic risk in multiple myeloma.Graphical

An all-inclusive model for predicting invasive bacterial infection in febrile infants age 7-60 days.

Invasive bacterial infections (IBIs) in febrile infants are rare but potentially devastating. We aimed to derive and validate a predictive model for IBI among febrile infants age 7-60 days. Data were abstracted retrospectively from electronic records of 37 emergency departments (EDs) for infants with a measured temperature >=100.4 F who underwent an ED evaluation with blood and urine cultures. Models to predict IBI were developed and validated respectively using a random 80/20 dataset split, including 10-fold cross-validation. We used precision recall curves as the classification metric. Of 4411 eligible infants with a mean age of 37 days, 29% had characteristics that would likely have excluded them from existing risk stratification protocols. There were 196 patients with IBI (4.4%), including 43 (1.0%) with bacterial meningitis. Analytic approaches varied in performance characteristics (precision recall range 0.04-0.29, area under the curve range 0.5-0.84), with the XGBoost model demonstrating the best performance (0.29, 0.84). The five most important variables were serum white blood count, maximum temperature, absolute neutrophil count, absolute band count, and age in days. A machine learning model (XGBoost) demonstrated the best performance in predicting a rare outcome among febrile infants, including those excluded from existing algorithms. Several models for the risk stratification of febrile infants have been developed. There is a need for a preferred comprehensive model free from limitations and algorithm exclusions that accurately predicts IBIs. This is the first study to derive an all-inclusive predictive model for febrile infants aged 7-60 days in a community ED sample with IBI as a primary outcome. This machine learning model demonstrates potential for clinical utility in predicting IBI.

Predicting anomalies in computer networks using autoencoder-based representation learning

<span>Recent improvements in the internet of things (IoT), cloud services, and network data variety have increased the demand for complex anomaly detection algorithms in network intrusion detection systems (IDSs) capable of dealing with sophisticated network threats. Academics are interested in deep and machine learning (ML) breakthroughs because they have the potential to address complex challenges such as zero-day attacks. In comparison to firewalls, IDS are the initial line of network security. This study suggests merging supervised and unsupervised learning in identification systems IDS. Support vector machine (SVM) is an anomaly-based classification classifier. Deep autoencoder (DAE) lowers dimensionality. DAE are compared to principal component analysis (PCA) in this study, and hyper-parameters for F-1 micro score and balanced accuracy are specified. We have an uneven set of data classes. precision-recall curves, average precision (AP) score, train-test times, t-SNE, grid search, and L1/L2 regularization methods are used. KDDTrain+ and KDDTest+ datasets will be used in our model. For classification and performance, the DAE+SVM neural network technique is successful. Autoencoders outperformed linear PCA in terms of capturing valuable input attributes using t-SNE to embed high dimensional inputs on a two-dimensional plane. Our neural system outperforms solo SVM and PCA encoded SVM in multi-class scenarios.</span>

CAFES: Chest X-ray Analysis using Federated Self-supervised Learning for Pediatric COVID-19 Detection.

Chest X-rays (CXRs) play a pivotal role in cost-effective clinical assessment of various heart and lung related conditions. The urgency of COVID-19 diagnosis prompted their use in identifying conditions like lung opacity, pneumonia, and acute respiratory distress syndrome in pediatric patients. We propose an AI-driven solution for binary COVID-19 versus non-COVID-19 classification in pediatric CXRs. We present a Federated Self-Supervised Learning (FSSL) framework to enhance Vision Transformer (ViT) performance for COVID-19 detection in pediatric CXRs. ViT's prowess in vision-related binary classification tasks, combined with self-supervised pre-training on adult CXR data, forms the basis of the FSSL approach. We implement our strategy on the Rhino Health Federated Computing Platform (FCP), which ensures privacy and scalability for distributed data. The chest X-ray analysis using the federated SSL (CAFES) model, utilizes the FSSL-pre-trained ViT weights and demonstrated gains in accurately detecting COVID-19 when compared with a fully supervised model. Our FSSL-pre-trained ViT showed an area under the precision-recall curve (AUPR) of 0.952, which is 0.231 points higher than the fully supervised model for COVID-19 diagnosis using pediatric data. Our contributions include leveraging vision transformers for effective COVID-19 diagnosis from pediatric CXRs, employing distributed federated learning-based self-supervised pre-training on adult data, and improving pediatric COVID-19 diagnosis performance. This privacy-conscious approach aligns with HIPAA guidelines, paving the way for broader medical imaging applications.

Prediction model for spinal cord injury in spinal tuberculosis patients using multiple machine learning algorithms: a multicentric study

Spinal cord injury (SCI) is a prevalent and serious complication among patients with spinal tuberculosis (STB) that can lead to motor and sensory impairment and potentially paraplegia. This research aims to identify factors associated with SCI in STB patients and to develop a clinically significant predictive model. Clinical data from STB patients at a single hospital were collected and divided into training and validation sets. Univariate analysis was employed to screen clinical indicators in the training set. Multiple machine learning (ML) algorithms were utilized to establish predictive models. Model performance was evaluated and compared using receiver operating characteristic (ROC) curves, area under the curve (AUC), calibration curve analysis, decision curve analysis (DCA), and precision-recall (PR) curves. The optimal model was determined, and a prospective cohort from two other hospitals served as a testing set to assess its accuracy. Model interpretation and variable importance ranking were conducted using the DALEX R package. The model was deployed on the web by using the Shiny app. Ten clinical characteristics were utilized for the model. The random forest (RF) model emerged as the optimal choice based on the AUC, PRs, calibration curve analysis, and DCA, achieving a test set AUC of 0.816. Additionally, MONO was identified as the primary predictor of SCI in STB patients through variable importance ranking. The RF predictive model provides an efficient and swift approach for predicting SCI in STB patients.

A novel nonparametric time-dependent precision-recall curve estimator for right-censored survival data.

In order to assess prognostic risk for individuals in precision health research, risk prediction models are increasingly used, in which statistical models are used to estimate the risk of future outcomes based on clinical and nonclinical characteristics. The predictive accuracy of a risk score must be assessed before it can be used in routine clinical decision making, where the receiver operator characteristic curves, precision-recall curves, and their corresponding area under the curves are commonly used metrics to evaluate the discriminatory ability of a continuous risk score. Among these the precision-recall curves have been shown to be more informative when dealing with unbalanced biomarker distribution between classes, which is common in rare event, even though except one, all existing methods are proposed for classic uncensored data. This paper is therefore to propose a novel nonparametric estimation approach for the time-dependent precision-recall curve and its associated area under the curve for right-censored data. A simulation is conducted to show the better finite sample property of the proposed estimator over the existing method and a real-world data from primary biliary cirrhosis trial is used to demonstrate the practical applicability of the proposedestimator.

Explainable Machine Learning Model to Preoperatively Predict Postoperative Complications in Inpatients With Cancer Undergoing Major Operations

Explainable Machine Learning Model to Preoperatively Predict Postoperative Complications in Inpatients With Cancer Undergoing Major Operations

A deep-learning approach to predict bleeding risk over time in patients on extended anticoagulation therapy

BackgroundThus far, all the clinical models developed to predict major bleeding in patients on extended anticoagulation therapy use the baseline predictors to stratify patients into different risk groups. Therefore, these models do not account for the clinical changes and events that occur after the baseline visit, which can modify risk of bleeding. However, it is difficult to develop predictive models from the routine follow-up clinical interviews, which are irregular sequences of multivariate time series data. ObjectivesTo demonstrate that deep learning can incorporate patient time series follow-up data to improve prediction of major bleeding. MethodsWe used the baseline and follow-up data that were collected over 8 years in a longitudinal cohort study of 2542 patients, of whom 118 had major bleeding. Four supervised neural network–based machine-learning models were trained on the baseline, follow-up, or both datasets using 70% of the data. The performance of these models was evaluated, along with modified versions of 6 previously developed clinical models, on the remaining 30% of the data. ResultsAn ensemble of feedforward and recurrent neural networks that used the baseline and follow-up data was the best-performing model, achieving a sensitivity and a specificity of 61% and 82%, respectively, in identifying major bleeding, and it outperformed the previously developed clinical models in terms of area under the receiver operating characteristic curve (82%) and area under the precision–recall curve (14%). ConclusionTime series follow-up data can improve major bleeding prediction in patients on extended anticoagulation therapy.

Early prediction of end-stage kidney disease using electronic health record data: a machine learning approach with a 2-year horizon.

In the United States, end-stage kidney disease (ESKD) is responsible for high mortality and significant healthcare costs, with the number of cases sharply increasing in the past 2 decades. In this study, we aimed to reduce these impacts by developing an ESKD model for predicting its occurrence in a 2-year period. We developed a machine learning (ML) pipeline to test different models for the prediction of ESKD. The electronic health record was used to capture several kidney disease-related variables. Various imputation methods, feature selection, and sampling approaches were tested. We compared the performance of multiple ML models using area under the ROC curve (AUCROC), area under the Precision-Recall curve (PR-AUC), and Brier scores for discrimination, precision, and calibration, respectively. Explainability methods were applied to the final model. Our best model was a gradient-boosting machine with feature selection and imputation methods as additional components. The model exhibited an AUCROC of 0.97, a PR-AUC of 0.33, and a Brier score of 0.002 on a holdout test set. A chart review analysis by expert physicians indicated clinical utility. An ESKD prediction model can identify individuals at risk for ESKD and has been successfully deployed within our health system.

757 Machine Learning-Driven Web Application for Enhanced Clinical Decision Support in Thoracolumbar Spinal Cord Injuries

INTRODUCTION: Thoracolumbar spinal cord injuries (tlSCI) can result in negative short-term in-hospital outcomes, underscoring the importance of precise prediction tools. METHODS: This retrospective cohort study included 147,826 patients from the American College of Surgeons Trauma Quality Program database. XGBoost, LightGBM, CatBoost, and Random Forest algorithms were utilized in combination with the Optuna optimization library for hyperparameter tuning. Models were evaluated graphically with receiver operating characteristic (ROC) curves, precision-recall curves, and numerically with precision, recall, F1 score, accuracy, area under ROC (AUROC), and area under PRC. The best-performing models were integrated into an open-access web application for individualized predictions. RESULTS: The best-performing models for each outcome were as follows: for predicting mortality, Random Forest achieved the highest AUROC (0.865, 95% CI 0.848-0.879); in predicting non-home discharges, LightGBM had the highest AUROC (0.801, 95% CI 0.794-0.804); and for predicting major complications, Random Forest performed best with an AUROC of 0.733 (95% CI 0.642-0.808). Algorithms' receiver operator curves for the outcomes: (A) mortality, (B) non-home discharges, and (C) major complications. CONCLUSIONS: This study successfully developed ML models to predict critical in-hospital outcomes for patients with tlSCI. By incorporating the top-performing algorithms into a user-friendly web application, these models have the potential to improve clinical decision-making and management strategies for tlSCI patients, ultimately enhancing patient care and outcomes.

Deep learning and capsule endoscopy: Automatic multi-brand and multi-device panendoscopic detection of vascular lesions.

Background and study aims Capsule endoscopy (CE) is commonly used as the initial exam for suspected mid-gastrointestinal bleeding after normal upper and lower endoscopy. Although the assessment of the small bowel is the primary focus of CE, detecting upstream or downstream vascular lesions may also be clinically significant. This study aimed to develop and test a convolutional neural network (CNN)-based model for panendoscopic automatic detection of vascular lesions during CE. Patients and methods A multicentric AI model development study was based on 1022 CE exams. Our group used 34655 frames from seven types of CE devices, of which 11091 were considered to have vascular lesions (angiectasia or varices) after triple validation. We divided data into a training and a validation set, and the latter was used to evaluate the model's performance. At the time of division, all frames from a given patient were assigned to the same dataset. Our primary outcome measures were sensitivity, specificity, accuracy, positive predictive value (PPV), negative predictive value (NPV), and an area under the precision-recall curve (AUC-PR). Results Sensitivity and specificity were 86.4% and 98.3%, respectively. PPV was 95.2%, while the NPV was 95.0%. Overall accuracy was 95.0%. The AUC-PR value was 0.96. The CNN processed 115 frames per second. Conclusions This is the first proof-of-concept artificial intelligence deep learning model developed for pan-endoscopic automatic detection of vascular lesions during CE. The diagnostic performance of this CNN in multi-brand devices addresses an essential issue of technological interoperability, allowing it to be replicated in multiple technological settings.

Personalizing patient risk of a life-altering event: An application of machine learning to hemiarch surgery

ObjectiveThe study objective was to assess a machine learning model’s ability to predict the occurrence of life-altering events in hemiarch surgery and determine contributing patient characteristics and intraoperative factors. MethodsIn total, 602 patients who underwent hemiarch replacement at a high-volume aortic center from 2009 to 2022 were included. Patients were randomly divided into training (80%) and testing (20%) sets with various eXtreme gradient boosting candidate models constructed to predict the risk of experiencing life-altering events, including stroke, mortality, or new renal replacement therapy requirement. A total of 64 input parameters from the index hospitalization were identified, including 24 demographic characteristics as well as 8 preoperative and 32 intraoperative variables. A SHapley Additive exPlanation beeswarm plot was generated to identify and interpret the impact of individual features on the predictions of the final model. ResultsA life-altering event was noted in 15% (90/602) of patients who underwent hemiarch replacement, including urgent/emergency cases and dissections. The final eXtreme Gradient Boosting model demonstrated a cross-validation accuracy of 88% on the testing set and was well calibrated as evidenced by a low Brier score of 0.12. The best performing model achieved an area under the receiver operating characteristic curve of 0.76 and an area under the precision recall curve of 0.55. The SHapley Additive exPlanation beeswarm plot provided insights into key features that significantly influenced model prediction. ConclusionsMachine learning demonstrated superior accuracy in predicting hemiarch patients who would experience a life-altering event. This model may help to guide patients and clinicians in stratifying risk on an individual basis, which may in turn influence clinical decision-making.

Clinical Features Predicting COVID-19 Severity Risk at the Time of Hospitalization.

The global spread of COVID-19 has led to significant mortality and morbidity worldwide. Early identification of COVID-19 patients who are at high risk of developing severe disease can help in improved patient management, care, and treatment, as well as in the effective allocation of hospital resources. The severity prediction at the time of hospitalization can be extremely helpful in deciding the treatment of COVID-19 patients. To this end, this study presents an interpretable artificial intelligence (AI) model, named COVID-19 severity predictor (CoSP) that predicts COVID-19 severity using the clinical features at the time of hospital admission. We utilized a dataset comprising 64 demographic and laboratory features of 7,416 confirmed COVID-19 patients that were collected at the time of hospital admission. The proposed hierarchical CoSP model performs four-class COVID severity risk prediction into asymptomatic, mild, moderate, and severe categories. CoSP yielded better performance with good interpretability, as observed via Shapley analysis on COVID severity prediction compared to the other popular ML methods, with an area under the received operating characteristic curve (AUC-ROC) of 0.95, an area under the precision-recall curve (AUPRC) of 0.91, and a weighted F1-score of 0.83. Out of 64 initial features, 19 features were inferred as predictive of the severity of COVID-19 disease by the CoSP model. Therefore, an AI model predicting COVID-19 severity may be helpful for early intervention, optimizing resource allocation, and guiding personalized treatments, potentially enabling healthcare professionals to save lives and allocate resources effectively in the fight against the pandemic.

An Appearance-Semantic Descriptor with Coarse-to-Fine Matching for Robust VPR.

In recent years, semantic segmentation has made significant progress in visual place recognition (VPR) by using semantic information that is relatively invariant to appearance and viewpoint, demonstrating great potential. However, in some extreme scenarios, there may be semantic occlusion and semantic sparsity, which can lead to confusion when relying solely on semantic information for localization. Therefore, this paper proposes a novel VPR framework that employs a coarse-to-fine image matching strategy, combining semantic and appearance information to improve algorithm performance. First, we construct SemLook global descriptors using semantic contours, which can preliminarily screen images to enhance the accuracy and real-time performance of the algorithm. Based on this, we introduce SemLook local descriptors for fine screening, combining robust appearance information extracted by deep learning with semantic information. These local descriptors can address issues such as semantic overlap and sparsity in urban environments, further improving the accuracy of the algorithm. Through this refined screening process, we can effectively handle the challenges of complex image matching in urban environments and obtain more accurate results. The performance of SemLook descriptors is evaluated on three public datasets (Extended-CMU Season, Robot-Car Seasons v2, and SYNTHIA) and compared with six state-of-the-art VPR algorithms (HOG, CoHOG, AlexNet_VPR, Region VLAD, Patch-NetVLAD, Forest). In the experimental comparison, considering both real-time performance and evaluation metrics, the SemLook descriptors are found to outperform the other six algorithms. Evaluation metrics include the area under the curve (AUC) based on the precision-recall curve, Recall@100%Precision, and Precision@100%Recall. On the Extended-CMU Season dataset, SemLook descriptors achieve a 100% AUC value, and on the SYNTHIA dataset, they achieve a 99% AUC value, demonstrating outstanding performance. The experimental results indicate that introducing global descriptors for initial screening and utilizing local descriptors combining both semantic and appearance information for precise matching can effectively address the issue of location recognition in scenarios with semantic ambiguity or sparsity. This algorithm enhances descriptor performance, making it more accurate and robust in scenes with variations in appearance and viewpoint.

Artificial intelligence auxiliary diagnosis and treatment system for breast cancer in developing countries.

In many developing countries, a significant number of breast cancer patients are unable to receive timely treatment due to a large population base, high patient numbers, and limited medical resources. This paper proposes a breast cancer assisted diagnosis system based on electronic medical records. The goal of this system is to address the limitations of existing systems, which primarily rely on structured electronic records and may miss crucial information stored in unstructured records. The proposed approach is a breast cancer assisted diagnosis system based on electronic medical records. The system utilizes breast cancer enhanced convolutional neural networks with semantic initialization filters (BC-INIT-CNN). It extracts highly relevant tumor markers from unstructured medical records to aid in breast cancer staging diagnosis and effectively utilizes the important information present in unstructured records. The model's performance is assessed using various evaluation metrics. Such as accuracy, ROC curves, and Precision-Recall curves. Comparative analysis demonstrates that the BC-INIT-CNN model outperforms several existing methods in terms of accuracy and computational efficiency. The proposed breast cancer assisted diagnosis system based on BC-INIT-CNN showcases the potential to address the challenges faced by developing countries in providing timely treatment to breast cancer patients. By leveraging unstructured medical records and extracting relevant tumor markers, the system enables accurate staging diagnosis and enhances the utilization of valuable information.

A comparative evaluation of machine learning ensemble approaches for disease prediction using multiple datasets

PurposeMachine learning models are used to develop and improve various disease prediction systems. Ensemble learning is a machine learning technique that combines many classifiers to increase performance by making more accurate predictions than a single classifier. Although several researchers have employed ensemble techniques for disease prediction, a comprehensive comparative study of these techniques still needs to be provided.MethodsUsing 16 disease datasets from Kaggle and the UCI Machine Learning Repository, this study compares the performance of 15 variants of ensemble techniques for disease prediction. The comparison was performed using six performance measures: accuracy, precision, recall, F1 score, AUC (Area Under the receiver operating characteristics Curve) and AUPRC (Area Under the Precision-Recall Curve).ResultsStacking variant of Multi-level stacking showed superior disease prediction performance compared with other bagging and boosting variants, followed by another stacking variant (Classical stacking). Overall, stacking outperformed bagging and boosting for disease prediction. Logit Boost showed the worst performance.ConclusionThe findings of this study can help researchers select an appropriate ensemble approach for future studies focusing on accurate disease prediction.

Clinical performance of automated machine learning: A systematic review

Introduction: Automated machine learning (autoML) removes technical and technological barriers to building artificial intelligence models. We aimed to summarise the clinical applications of autoML, assess the capabilities of utilised platforms, evaluate the quality of the evidence trialling autoML, and gauge the performance of autoML platforms relative to conventionally developed models, as well as each other. Method: This review adhered to a prospectively registered protocol (PROSPERO identifier CRD42022344427). The Cochrane Library, Embase, MEDLINE and Scopus were searched from inception to 11 July 2022. Two researchers screened abstracts and full texts, extracted data and conducted quality assessment. Disagreement was resolved through discussion and as if required, arbitration by a third researcher. Results: There were 26 distinct autoML platforms featured in 82 studies. Brain and lung disease were the most common fields of study of 22 specialties. AutoML exhibited variable performance: area under the receiver operator characteristic curve (AUCROC) 0.35–1.00, F1-score 0.16–0.99, area under the precision-recall curve (AUPRC) 0.51–1.00. AutoML exhibited the highest AUCROC in 75.6% trials; the highest F1-score in 42.3% trials; and the highest AUPRC in 83.3% trials. In autoML platform comparisons, AutoPrognosis and Amazon Rekognition performed strongest with unstructured and structured data, respectively. Quality of reporting was poor, with a median DECIDE-AI score of 14 of 27. Conclusion: A myriad of autoML platforms have been applied in a variety of clinical contexts. The performance of autoML compares well to bespoke computational and clinical benchmarks. Further work is required to improve the quality of validation studies. AutoML may facilitate a transition to data-centric development, and integration with large language models may enable AI to build itself to fulfil user-defined goals.

Predicting Successful Weaning from Mechanical Ventilation by Reduction in Positive End-expiratory Pressure Level Using Machine Learning.

Weaning patients from mechanical ventilation (MV) is a critical and resource intensive process in the Intensive Care Unit (ICU) that impacts patient outcomes and healthcare expenses. Weaning methods vary widely among providers. Prolonged MV is associated with adverse events and higher healthcare expenses. Predicting weaning readiness is a non-trivial process in which the positive end-expiratory pressure (PEEP), a crucial component of MV, has potential to be indicative but has not yet been used as the target. We aimed to predict successful weaning from mechanical ventilation by targeting changes in the PEEP-level using a supervised machine learning model. This retrospective study included 12,153 mechanically ventilated patients from Medical Information Mart for Intensive Care (MIMIC-IV) and eICU collaborative research database (eICU-CRD). Two machine learning models (Extreme Gradient Boosting and Logistic Regression) were developed using a continuous PEEP reduction as target. The data is splitted into 80% as training set and 20% as test set. The model's predictive performance was reported using 95% confidence interval (CI), based on evaluation metrics such as area under the receiver operating characteristic (AUROC), area under the precision-recall curve (AUPRC), F1-Score, Recall, positive predictive value (PPV), and negative predictive value (NPV). The model's descriptive performance was reported as the variable ranking using SHAP (SHapley Additive exPlanations) algorithm. The best model achieved an AUROC of 0.84 (95% CI 0.83-0.85) and an AUPRC of 0.69 (95% CI 0.67-0.70) in predicting successful weaning based on the PEEP reduction. The model demonstrated a Recall of 0.85 (95% CI 0.84-0.86), F1-score of 0.86 (95% CI 0.85-0.87), PPV of 0.87 (95% CI 0.86-0.88), and NPV of 0.64 (95% CI 0.63-0.66). Most of the variables that SHAP algorithm ranked to be important correspond with clinical intuition, such as duration of MV, oxygen saturation (SaO2), PEEP, and Glasgow Coma Score (GCS) components. This study demonstrates the potential application of machine learning in predicting successful weaning from MV based on continuous PEEP reduction. The model's high PPV and moderate NPV suggest that it could be a useful tool to assist clinicians in making decisions regarding ventilator management.

Validation of a Multivariable Model to Predict Suicide Attempt in a Mental Health Intake Sample

Given that suicide rates have been increasing over the past decade and the demand for mental health care is at an all-time high, targeted prevention efforts are needed to identify individuals seeking to initiate mental health outpatient services who are at high risk for suicide. Suicide prediction models have been developed using outpatient mental health encounters, but their performance among intake appointments has not been directly examined. To assess the performance of a predictive model of suicide attempts among individuals seeking to initiate an episode of outpatient mental health care. This prognostic study tested the performance of a previously developed machine learning model designed to predict suicide attempts within 90 days of any mental health outpatient visit. All mental health intake appointments scheduled between January 1, 2012, and April 1, 2022, at Kaiser Permanente Northern California, a large integrated health care delivery system serving over 4.5 million patients, were included. Data were extracted and analyzed from August 9, 2022, to July 31, 2023. Suicide attempts (including completed suicides) within 90 days of the appointment, determined by diagnostic codes and government databases. All predictors were extracted from electronic health records. The study included 1 623 232 scheduled appointments from 835 616 unique patients. There were 2800 scheduled appointments (0.17%) followed by a suicide attempt within 90 days. The mean (SD) age across appointments was 39.7 (15.8) years, and most appointments were for women (1 103 184 [68.0%]). The model had an area under the receiver operating characteristic curve of 0.77 (95% CI, 0.76-0.78), an area under the precision-recall curve of 0.02 (95% CI, 0.02-0.02), an expected calibration error of 0.0012 (95% CI, 0.0011-0.0013), and sensitivities of 37.2% (95% CI, 35.5%-38.9%) and 18.8% (95% CI, 17.3%-20.2%) at specificities of 95% and 99%, respectively. The 10% of appointments at the highest risk level accounted for 48.8% (95% CI, 47.0%-50.6%) of the appointments followed by a suicide attempt. In this prognostic study involving mental health intakes, a previously developed machine learning model of suicide attempts showed good overall classification performance. Implementation research is needed to determine appropriate thresholds and interventions for applying the model in an intake setting to target high-risk cases in a manner that is acceptable to patients and clinicians.