Mean Area Under The Receiver Operating Characteristic Curve Research Articles

Machine learning models on a web application to predict short-term postoperative outcomes following anterior cervical discectomy and fusion

BackgroundThe frequency of anterior cervical discectomy and fusion (ACDF) has increased up to 400% since 2011, underscoring the need to preoperatively anticipate adverse postoperative outcomes given the procedure’s expanding use. Our study aims to accomplish two goals: firstly, to develop a suite of explainable machine learning (ML) models capable of predicting adverse postoperative outcomes following ACDF surgery, and secondly, to embed these models in a user-friendly web application, demonstrating their potential utility.MethodsWe utilized data from the National Surgical Quality Improvement Program database to identify patients who underwent ACDF surgery. The outcomes of interest were four short-term postoperative adverse outcomes: prolonged length of stay (LOS), non-home discharges, 30-day readmissions, and major complications. We utilized five ML algorithms - TabPFN, TabNET, XGBoost, LightGBM, and Random Forest - coupled with the Optuna optimization library for hyperparameter tuning. To bolster the interpretability of our models, we employed SHapley Additive exPlanations (SHAP) for evaluating predictor variables’ relative importance and used partial dependence plots to illustrate the impact of individual variables on the predictions generated by our top-performing models. We visualized model performance using receiver operating characteristic (ROC) curves and precision-recall curves (PRC). Quantitative metrics calculated were the area under the ROC curve (AUROC), balanced accuracy, weighted area under the PRC (AUPRC), weighted precision, and weighted recall. Models with the highest AUROC values were selected for inclusion in a web application.ResultsThe analysis included 57,760 patients for prolonged LOS [11.1% with prolonged LOS], 57,780 for non-home discharges [3.3% non-home discharges], 57,790 for 30-day readmissions [2.9% readmitted], and 57,800 for major complications [1.4% with major complications]. The top-performing models, which were the ones built with the Random Forest algorithm, yielded mean AUROCs of 0.776, 0.846, 0.775, and 0.747 for predicting prolonged LOS, non-home discharges, readmissions, and complications, respectively.ConclusionsOur study employs advanced ML methodologies to enhance the prediction of adverse postoperative outcomes following ACDF. We designed an accessible web application to integrate these models into clinical practice. Our findings affirm that ML tools serve as vital supplements in risk stratification, facilitating the prediction of diverse outcomes and enhancing patient counseling for ACDF.

Prediction models for post-discharge mortality among under-five children with suspected sepsis in Uganda: A multicohort analysis.

In many low-income countries, over five percent of hospitalized children die following hospital discharge. The lack of available tools to identify those at risk of post-discharge mortality has limited the ability to make progress towards improving outcomes. We aimed to develop algorithms designed to predict post-discharge mortality among children admitted with suspected sepsis. Four prospective cohort studies of children in two age groups (0-6 and 6-60 months) were conducted between 2012-2021 in six Ugandan hospitals. Prediction models were derived for six-months post-discharge mortality, based on candidate predictors collected at admission, each with a maximum of eight variables, and internally validated using 10-fold cross-validation. 8,810 children were enrolled: 470 (5.3%) died in hospital; 257 (7.7%) and 233 (4.8%) post-discharge deaths occurred in the 0-6-month and 6-60-month age groups, respectively. The primary models had an area under the receiver operating characteristic curve (AUROC) of 0.77 (95%CI 0.74-0.80) for 0-6-month-olds and 0.75 (95%CI 0.72-0.79) for 6-60-month-olds; mean AUROCs among the 10 cross-validation folds were 0.75 and 0.73, respectively. Calibration across risk strata was good: Brier scores were 0.07 and 0.04, respectively. The most important variables included anthropometry and oxygen saturation. Additional variables included: illness duration, jaundice-age interaction, and a bulging fontanelle among 0-6-month-olds; and prior admissions, coma score, temperature, age-respiratory rate interaction, and HIV status among 6-60-month-olds. Simple prediction models at admission with suspected sepsis can identify children at risk of post-discharge mortality. Further external validation is recommended for different contexts. Models can be digitally integrated into existing processes to improve peri-discharge care as children transition from the hospital to the community.

Development and external validation of deep learning clinical prediction models using variable-length time series data.

To compare and externally validate popular deep learning model architectures and data transformation methods for variable-length time series data in 3 clinical tasks (clinical deterioration, severe acute kidney injury [AKI], and suspected infection). This multicenter retrospective study included admissions at 2 medical centers that spanned 2007-2022. Distinct datasets were created for each clinical task, with 1 site used for training and the other for testing. Three feature engineering methods (normalization, standardization, and piece-wise linear encoding with decision trees [PLE-DTs]) and 3 architectures (long short-term memory/gated recurrent unit [LSTM/GRU], temporal convolutional network, and time-distributed wrapper with convolutional neural network [TDW-CNN]) were compared in each clinical task. Model discrimination was evaluated using the area under the precision-recall curve (AUPRC) and the area under the receiver operating characteristic curve (AUROC). The study comprised 373825 admissions for training and 256128 admissions for testing. LSTM/GRU models tied with TDW-CNN models with both obtaining the highest mean AUPRC in 2 tasks, and LSTM/GRU had the highest mean AUROC across all tasks (deterioration: 0.81, AKI: 0.92, infection: 0.87). PLE-DT with LSTM/GRU achieved the highest AUPRC in all tasks. When externally validated in 3 clinical tasks, the LSTM/GRU model architecture with PLE-DT transformed data demonstrated the highest AUPRC in all tasks. Multiple models achieved similar performance when evaluated using AUROC. The LSTM architecture performs as well or better than some newer architectures, and PLE-DT may enhance the AUPRC in variable-length time series data for predicting clinical outcomes during external validation.

A supervised, externally validated machine learning model for artifact and drainage detection in high-resolution intracranial pressure monitoring data.

In neurocritical care, data from multiple biosensors are continuously measured, but only sporadically acknowledged by the attending physicians. In contrast, machine learning (ML) tools can analyze large amounts of data continuously, taking advantage of underlying information. However, the performance of such ML-based solutions is limited by different factors, for example, by patient motion, manipulation, or, as in the case of external ventricular drains (EVDs), the drainage of CSF to control intracranial pressure (ICP). The authors aimed to develop an ML-based algorithm that automatically classifies normal signals, artifacts, and drainages in high-resolution ICP monitoring data from EVDs, making the data suitable for real-time artifact removal and for future ML applications. In their 2-center retrospective cohort study, the authors used labeled ICP data from 40 patients in the first neurocritical care unit (University Hospital Zurich) for model development. The authors created 94 descriptive features that were used to train the model. They compared histogram-based gradient boosting with extremely randomized trees after building pipelines with principal component analysis, hyperparameter optimization via grid search, and sequential feature selection. Performance was measured with nested 5-fold cross-validation and multiclass area under the receiver operating characteristic curve (AUROC). Data from 20 patients in a second, independent neurocritical care unit (Charité - Universitätsmedizin Berlin) were used for external validation with bootstrapping technique and AUROC. In cross-validation, the best-performing model achieved a mean AUROC of 0.945 (95% CI 0.92-0.969) on the development dataset. On the external validation dataset, the model performed with a mean AUROC of 0.928 (95% CI 0.908-0.946) in 100 bootstrapping validation cycles to classify normal signals, artifacts, and drainages. Here, the authors developed a well-performing supervised model with external validation that can detect normal signals, artifacts, and drainages in ICP signals from patients in neurocritical care units. For future analyses, this is a powerful tool to discard artifacts or to detect drainage events in ICP monitoring signals.

Machine Learning-Based Prediction of Suicidality in Adolescents With Allergic Rhinitis: Derivation and Validation in 2 Independent Nationwide Cohorts.

Given the additional risk of suicide-related behaviors in adolescents with allergic rhinitis (AR), it is important to use the growing field of machine learning (ML) to evaluate this risk. This study aims to evaluate the validity and usefulness of an ML model for predicting suicide risk in patients with AR. We used data from 2 independent survey studies, Korea Youth Risk Behavior Web-based Survey (KYRBS; n=299,468) for the original data set and Korea National Health and Nutrition Examination Survey (KNHANES; n=833) for the external validation data set, to predict suicide risks of AR in adolescents aged 13 to 18 years, with 3.45% (10,341/299,468) and 1.4% (12/833) of the patients attempting suicide in the KYRBS and KNHANES studies, respectively. The outcome of interest was the suicide attempt risks. We selected various ML-based models with hyperparameter tuning in the discovery and performed an area under the receiver operating characteristic curve (AUROC) analysis in the train, test, and external validation data. The study data set included 299,468 (KYRBS; original data set) and 833 (KNHANES; external validation data set) patients with AR recruited between 2005 and 2022. The best-performing ML model was the random forest model with a mean AUROC of 84.12% (95% CI 83.98%-84.27%) in the original data set. Applying this result to the external validation data set revealed the best performance among the models, with an AUROC of 89.87% (sensitivity 83.33%, specificity 82.58%, accuracy 82.59%, and balanced accuracy 82.96%). While looking at feature importance, the 5 most important features in predicting suicide attempts in adolescent patients with AR are depression, stress status, academic achievement, age, and alcohol consumption. This study emphasizes the potential of ML models in predicting suicide risks in patients with AR, encouraging further application of these models in other conditions to enhance adolescent health and decrease suicide rates.

Differentiating IDH-mutant astrocytomas and 1p19q-codeleted oligodendrogliomas using DSC-PWI: high performance through cerebral blood volume and percentage of signal recovery percentiles.

Presurgical differentiation between astrocytomas and oligodendrogliomas remains an unresolved challenge in neuro-oncology. This research aims to provide a comprehensive understanding of each tumor's DSC-PWI signatures, evaluate the discriminative capacity of cerebral blood volume (CBV) and percentage of signal recovery (PSR) percentile values, and explore the synergy of CBV and PSR combination for pre-surgical differentiation. Patients diagnosed with grade 2 and 3 IDH-mutant astrocytomas and IDH-mutant 1p19q-codeleted oligodendrogliomas were retrospectively retrieved (2010-2022). 3D segmentations of each tumor were conducted, and voxel-level CBV and PSR were extracted to compute mean, minimum, maximum, and percentile values. Statistical comparisons were performed using the Mann-Whitney U test and the area under the receiver operating characteristic curve (AUC-ROC). Lastly, the five most discriminative variables were combined for classification with internal cross-validation. The study enrolled 52 patients (mean age 45-year-old, 28 men): 28 astrocytomas and 24 oligodendrogliomas. Oligodendrogliomas exhibited higher CBV and lower PSR than astrocytomas across all metrics (e.g., mean CBV = 2.05 and 1.55, PSR = 0.68 and 0.81 respectively). The highest AUC-ROCs and the smallest p values originated from CBV and PSR percentiles (e.g., PSRp70 AUC-ROC = 0.84 and p value = 0.0005, CBVp75 AUC-ROC = 0.8 and p value = 0.0006). The mean, minimum, and maximum values yielded lower results. Combining the best five variables (PSRp65, CBVp70, PSRp60, CBVp75, and PSRp40) achieved a mean AUC-ROC of 0.87 for differentiation. Oligodendrogliomas exhibit higher CBV and lower PSR than astrocytomas, traits that are emphasized when considering percentiles rather than mean or extreme values. The combination of CBV and PSR percentiles results in promising classification outcomes. The combination of histogram-derived percentile values of cerebral blood volume and percentage of signal recovery from DSC-PWI enhances the presurgical differentiation between astrocytomas and oligodendrogliomas, suggesting that incorporating these metrics into clinical practice could be beneficial. • The unsupervised selection of percentile values for cerebral blood volume and percentage of signal recovery enhances presurgical differentiation of astrocytomas and oligodendrogliomas. • Oligodendrogliomas exhibit higher cerebral blood volume and lower percentage of signal recovery than astrocytomas. • Cerebral blood volume and percentage of signal recovery combined provide a broader perspective on tumor vasculature and yield promising results for this preoperative classification.

Wearable Biosensing to Predict Imminent Aggressive Behavior in Psychiatric Inpatient Youths With Autism

Aggressive behavior is a prevalent and challenging issue in individuals with autism. To investigate whether changes in peripheral physiology recorded by a wearable biosensor and machine learning can be used to predict imminent aggressive behavior before it occurs in inpatient youths with autism. This noninterventional prognostic study used data collected from March 2019 to March 2020 from 4 primary care psychiatric inpatient hospitals. Enrolled participants were 86 psychiatric inpatients with confirmed diagnoses of autism exhibiting operationally defined self-injurious behavior, emotion dysregulation, or aggression toward others; 16 individuals were not included (18.6%) because they would not wear the biosensor (8 individuals) or were discharged before an observation could be made (8 individuals). Data were analyzed from March 2020 through October 2023. Research staff performed live behavioral coding of aggressive behavior while inpatient study participants wore a commercially available biosensor that recorded peripheral physiological signals (cardiovascular activity, electrodermal activity, and motion). Logistic regression, support vector machines, neural networks, and domain adaptation were used to analyze time-series features extracted from biosensor data. Area under the receiver operating characteristic curve (AUROC) values were used to evaluate the performance of population- and person-dependent models. There were 70 study participants (mean [range; SD] age, 11.9 [5-19; 3.5] years; 62 males [88.6%]; 1 Asian [1.4%], 5 Black [7.1%], 1 Native Hawaiian or Other Pacific Islander [1.4%], and 63 White [90.0%]; 5 Hispanic [7.5%] and 62 non-Hispanic [92.5%] among 67 individuals with ethnicity data). Nearly half of the population (32 individuals [45.7%]) was minimally verbal, and 30 individuals (42.8%) had an intellectual disability. Participant length of inpatient hospital stay ranged from 8 to 201 days, and the mean (SD) length was 37.28 (33.95) days. A total of 429 naturalistic observational coding sessions were recorded, totaling 497 hours, wherein 6665 aggressive behaviors were documented, including self-injury (3983 behaviors [59.8%]), emotion dysregulation (2063 behaviors [31.0%]), and aggression toward others (619 behaviors [9.3%]). Logistic regression was the best-performing overall classifier across all experiments; for example, it predicted aggressive behavior 3 minutes before onset with a mean AUROC of 0.80 (95% CI, 0.79-0.81). This study replicated and extended previous findings suggesting that machine learning analyses of preceding changes in peripheral physiology may be used to predict imminent aggressive behaviors before they occur in inpatient youths with autism. Further research will explore clinical implications and the potential for personalized interventions.

Development of Deep Ensembles to Screen for Autism and Symptom Severity Using Retinal Photographs

ImportanceScreening for autism spectrum disorder (ASD) is constrained by limited resources, particularly trained professionals to conduct evaluations. Individuals with ASD have structural retinal changes that potentially reflect brain alterations, including visual pathway abnormalities through embryonic and anatomic connections. Whether deep learning algorithms can aid in objective screening for ASD and symptom severity using retinal photographs is unknown.ObjectiveTo develop deep ensemble models to differentiate between retinal photographs of individuals with ASD vs typical development (TD) and between individuals with severe ASD vs mild to moderate ASD.Design, Setting, and ParticipantsThis diagnostic study was conducted at a single tertiary-care hospital (Severance Hospital, Yonsei University College of Medicine) in Seoul, Republic of Korea. Retinal photographs of individuals with ASD were prospectively collected between April and October 2022, and those of age- and sex-matched individuals with TD were retrospectively collected between December 2007 and February 2023. Deep ensembles of 5 models were built with 10-fold cross-validation using the pretrained ResNeXt-50 (32×4d) network. Score-weighted visual explanations for convolutional neural networks, with a progressive erasing technique, were used for model visualization and quantitative validation. Data analysis was performed between December 2022 and October 2023.ExposuresAutism Diagnostic Observation Schedule–Second Edition calibrated severity scores (cutoff of 8) and Social Responsiveness Scale–Second Edition T scores (cutoff of 76) were used to assess symptom severity.Main Outcomes and MeasuresThe main outcomes were participant-level area under the receiver operating characteristic curve (AUROC), sensitivity, and specificity. The 95% CI was estimated through the bootstrapping method with 1000 resamples.ResultsThis study included 1890 eyes of 958 participants. The ASD and TD groups each included 479 participants (945 eyes), had a mean (SD) age of 7.8 (3.2) years, and comprised mostly boys (392 [81.8%]). For ASD screening, the models had a mean AUROC, sensitivity, and specificity of 1.00 (95% CI, 1.00-1.00) on the test set. These models retained a mean AUROC of 1.00 using only 10% of the image containing the optic disc. For symptom severity screening, the models had a mean AUROC of 0.74 (95% CI, 0.67-0.80), sensitivity of 0.58 (95% CI, 0.49-0.66), and specificity of 0.74 (95% CI, 0.67-0.82) on the test set.Conclusions and RelevanceThese findings suggest that retinal photographs may be a viable objective screening tool for ASD and possibly for symptom severity. Retinal photograph use may speed the ASD screening process, which may help improve accessibility to specialized child psychiatry assessments currently strained by limited resources.

Artificial intelligence for the detection of sacroiliitis on magnetic resonance imaging in patients with axial spondyloarthritis.

Magnetic resonance imaging (MRI) is important for the early detection of axial spondyloarthritis (axSpA). We developed an artificial intelligence (AI) model for detecting sacroiliitis in patients with axSpA using MRI. This study included MRI examinations of patients who underwent semi-coronal MRI scans of the sacroiliac joints owing to chronic back pain with short tau inversion recovery (STIR) sequences between January 2010 and December 2021. Sacroiliitis was defined as a positive MRI finding according to the ASAS classification criteria for axSpA. We developed a two-stage framework. First, the Faster R-CNN network extracted regions of interest (ROIs) to localize the sacroiliac joints. Maximum intensity projection (MIP) of three consecutive slices was used to mimic the reading of two adjacent slices. Second, the VGG-19 network determined the presence of sacroiliitis in localized ROIs. We augmented the positive dataset six-fold. The sacroiliitis classification performance was measured using the sensitivity, specificity, and area under the receiver operating characteristic curve (AUROC). The prediction models were evaluated using three-round three-fold cross-validation. A total of 296 participants with 4,746 MRI slices were included in the study. Sacroiliitis was identified in 864 MRI slices of 119 participants. The mean sensitivity, specificity, and AUROC for the detection of sacroiliitis were 0.725 (95% CI, 0.705-0.745), 0.936 (95% CI, 0.924-0.947), and 0.830 (95%CI, 0.792-0.868), respectively, at the image level and 0.947 (95% CI, 0.912-0.982), 0.691 (95% CI, 0.603-0.779), and 0.816 (95% CI, 0.776-0.856), respectively, at the patient level. In the original model, without using MIP and dataset augmentation, the mean sensitivity, specificity, and AUROC were 0.517 (95% CI, 0.493-0.780), 0.944 (95% CI, 0.933-0.955), and 0.731 (95% CI, 0.681-0.780), respectively, at the image level and 0.806 (95% CI, 0.729-0.883), 0.617 (95% CI, 0.523-0.711), and 0.711 (95% CI, 0.660-0.763), respectively, at the patient level. The performance was improved by MIP techniques and data augmentation. An AI model was developed for the detection of sacroiliitis using MRI, compatible with the ASAS criteria for axSpA, with the potential to aid MRI application in a wider clinical setting.

Metabolomic Biomarker Signatures for Bipolar and Unipolar Depression.

Bipolar disorder (BD) is frequently misdiagnosed as major depressive disorder (MDD) because of overlapping symptoms and the lack of objective diagnostic tools. To identify a reproducible metabolomic biomarker signature in patient dried blood spots (DBSs) that differentiates BD from MDD during depressive episodes and assess its added value when combined with self-reported patient information. This diagnostic analysis used samples and data from the Delta study, conducted in the UK between April 27, 2018, and February 6, 2020. The primary objective was to identify BD in patients with a recent (within the past 5 years) diagnosis of MDD and current depressive symptoms (Patient Health Questionnaire-9 score of 5 or more). Participants were recruited online through voluntary response sampling. The analysis was carried out between February 2022 and July 2023. Patient data were collected using a purpose-built online questionnaire (n = 635 questions). DBS metabolites (n = 630) were analyzed using a targeted mass spectrometry-based platform. Mood disorder diagnoses were established using the Composite International Diagnostic Interview. Of 241 patients in the discovery cohort, 170 (70.5%) were female; 67 (27.8%) were subsequently diagnosed with BD and 174 (72.2%) were confirmed as having MDD; and the mean (SD) age was 28.1 (7.1) years. Of 30 participants in the validation cohort, 16 (53%) were female; 9 (30%) were diagnosed with BD and 21 (70%) with MDD; and the mean (SD) age was 25.4 (6.3) years. DBS metabolite levels were assessed in 241 patients with depressive symptoms with a recent diagnosis of MDD, of whom 67 were subsequently diagnosed with BD by the Composite International Diagnostic Interview and 174 were confirmed as having MDD. The identified 17-biomarker panel provided a mean (SD) cross-validated area under the receiver operating characteristic curve (AUROC) of 0.71 (SD, 0.12; P < .001), with ceramide d18:0/24:1 emerging as the strongest biomarker. Combining biomarker data with patient-reported information significantly enhanced diagnostic performance of models based on extensive demographic data, PHQ-9 scores, and the outcomes from the Mood Disorder Questionnaire. The identified biomarkers were correlated primarily with lifetime manic symptoms and were validated in a separate group of patients who received a new clinical diagnosis of MDD (n = 21) or BD (n = 9) during the study's 1-year follow-up period, with a mean (SD) AUROC of 0.73 (0.06; P < .001). This study provides a proof of concept for developing an accessible biomarker test to facilitate the differential diagnosis of BD and MDD and highlights the potential involvement of ceramides in the pathophysiological mechanisms of mood disorders.

Generation of a Melanoma and Nevus Data Set From Unstandardized Clinical Photographs on the Internet

Artificial intelligence (AI) training for diagnosing dermatologic images requires large amounts of clean data. Dermatologic images have different compositions, and many are inaccessible due to privacy concerns, which hinder the development of AI. To build a training data set for discriminative and generative AI from unstandardized internet images of melanoma and nevus. In this diagnostic study, a total of 5619 (CAN5600 data set) and 2006 (CAN2000 data set; a manually revised subset of CAN5600) cropped lesion images of either melanoma or nevus were semiautomatically annotated from approximately 500 000 photographs on the internet using convolutional neural networks (CNNs), region-based CNNs, and large mask inpainting. For unsupervised pretraining, 132 673 possible lesions (LESION130k data set) were also created with diversity by collecting images from 18 482 websites in approximately 80 countries. A total of 5000 synthetic images (GAN5000 data set) were generated using the generative adversarial network (StyleGAN2-ADA; training, CAN2000 data set; pretraining, LESION130k data set). The area under the receiver operating characteristic curve (AUROC) for determining malignant neoplasms was analyzed. In each test, 1 of the 7 preexisting public data sets (total of 2312 images; including Edinburgh, an SNU subset, Asan test, Waterloo, 7-point criteria evaluation, PAD-UFES-20, and MED-NODE) was used as the test data set. Subsequently, a comparative study was conducted between the performance of the EfficientNet Lite0 CNN on the proposed data set and that trained on the remaining 6 preexisting data sets. The EfficientNet Lite0 CNN trained on the annotated or synthetic images achieved higher or equivalent mean (SD) AUROCs to the EfficientNet Lite0 trained using the pathologically confirmed public data sets, including CAN5600 (0.874 [0.042]; P = .02), CAN2000 (0.848 [0.027]; P = .08), and GAN5000 (0.838 [0.040]; P = .31 [Wilcoxon signed rank test]) and the preexisting data sets combined (0.809 [0.063]) by the benefits of increased size of the training data set. The synthetic data set in this diagnostic study was created using various AI technologies from internet images. A neural network trained on the created data set (CAN5600) performed better than the same network trained on preexisting data sets combined. Both the annotated (CAN5600 and LESION130k) and synthetic (GAN5000) data sets could be shared for AI training and consensus between physicians.

Self-supervised machine learning using adult inpatient data produces effective models for pediatric clinical prediction tasks.

Development of electronic health records (EHR)-based machine learning models for pediatric inpatients is challenged by limited training data. Self-supervised learning using adult data may be a promising approach to creating robust pediatric prediction models. The primary objective was to determine whether a self-supervised model trained in adult inpatients was noninferior to logistic regression models trained in pediatric inpatients, for pediatric inpatient clinical prediction tasks. This retrospective cohort study used EHR data and included patients with at least one admission to an inpatient unit. One admission per patient was randomly selected. Adult inpatients were 18years or older while pediatric inpatients were more than 28 days and less than 18 years. Admissions were temporally split into training (January 1, 2008 to December 31, 2019), validation (January 1, 2020 to December 31, 2020), and test (January 1, 2021 to August 1, 2022) sets. Primary comparison was a self-supervised model trained in adult inpatients versus count-based logistic regression models trained in pediatric inpatients. Primary outcome was mean area-under-the-receiver-operating-characteristic-curve (AUROC) for 11 distinct clinical outcomes. Models were evaluated in pediatric inpatients. When evaluated in pediatric inpatients, mean AUROC of self-supervised model trained in adult inpatients (0.902) was noninferior to count-based logistic regression models trained in pediatric inpatients (0.868) (mean difference = 0.034, 95% CI=0.014-0.057; P < .001 for noninferiority and P = .006 for superiority). Self-supervised learning in adult inpatients was noninferior to logistic regression models trained in pediatric inpatients. This finding suggests transferability of self-supervised models trained in adult patients to pediatric patients, without requiring costly model retraining.

Radiomics-based Machine Learning Methods for Volume Doubling Time Prediction of Pulmonary Ground-glass Nodules With Baseline Chest Computed Tomography.

Reliable prediction of volume doubling time (VDT) is essential for the personalized management of pulmonary ground-glass nodules (GGNs). We aimed to determine the optimal VDT prediction method by comparing different machine learning methods only based on the baseline chest computed tomography (CT) images. Seven classical machine learning methods were evaluated in terms of their stability and performance for VDT prediction. The VDT, calculated by the preoperative and baseline CT, was divided into 2 groups with a cutoff value of 400 days. A total of 90 GGNs from 3 hospitals constituted the training set, and 86 GGNs from the fourth hospital served as the external validation set. The training set was used for feature selection and model training, and the validation set was used to evaluate the predictive performance of the model independently. The eXtreme Gradient Boosting showed the highest predictive performance (accuracy: 0.890±0.128 and area under the ROC curve (AUC): 0.896±0.134), followed by the neural network (NNet) (accuracy: 0.865±0.103 and AUC: 0.886±0.097). While regarding stability, the NNet showed the highest robustness against data perturbation (relative SDs [%] of mean AUC: 10.9%). Therefore, the NNet was chosen as the final model, achieving high accuracy of 0.756 in the external validation set. The NNet is a promising machine learning method to predict the VDT of GGNs, which would assist in the personalized follow-up and treatment strategies for GGNs reducing unnecessary follow-up and radiation dose.

#6569 PREDICTION OF ALL-CAUSE MORTALITY FOR CHRONIC KIDNEY DISEASE PATIENTS USING FOUR MODELS OF MACHINE LEARNING

Abstract Background and Aims Prediction tools developed from general population data to predict all-cause mortality are not adapted to patients with chronic kidney disease (CKD), as this population has a higher risk of mortality. This study aimed to create a clinical prediction tool with good predictive performance to predict 2-year all-cause mortality in patients with stage 4 or 5 CKD using an innovative approach with machine learning models and a synthetic population. Method The national, observational, descriptive and prospective Photo-Graphe 3 study was used to create the learning database. Four models (i) logistic regression; (ii) deep learning; (iii) random forest and (iv) Bayesian network were used to create four prediction tools. The performance of each model, including the area under the receiver operating characteristic curve (AUC-ROC) value, accuracy, sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV), was evaluated and compared using 10-fold cross-validation. The prediction tool with the best performance was selected and optimized using a synthetic population and the explanatory variables most related to mortality. The synthetic population was created by the Bayesian imputation method. The variables most associated with 2-year all-cause mortality were determined by compromising the number of variables selected and the AUC-ROC value when successively adding the variable according to its percentage variance value. The performance of the optimized prediction tool in predicting 2-year mortality was then evaluated by 10-fold cross validation. Results All prediction tools except the one developed with the random forest model showed satisfactory discrimination (AUC-ROC ≥ 0.70). Overall, the prediction tools developed using the Bayesian network and logistic regression tended to have better performance. Although not significantly different from logistic regression, the prediction tool developed using the Bayesian network was chosen for further development because of its advantages. From the 534 patients in the study population, a synthetic population of 2000 patients (survivor:death ratio = 1:1) was created. The seven most informative variables ranked in descending order were: age, ESA, CV history, smoking status, 25-OH vitamin D level, PTH level, and ferritin level (Figure 1). The optimized clinical prediction tool had satisfactory internal performance. The mean accuracy was 73.8% (SD = 3.6), the mean AUC-ROC was 0.81 (SD = 0.03), the mean sensitivity was 71.0% (SD = 5.4), the mean specificity was 76.5% (SD = 3.0), the mean PPV was 75.1% (SD = 3.2), and the mean NPV was 72.6% (SD = 4.1). Conclusion Bayesian network model was used to create a seven-variable prediction tool to predict the 2-year all-cause mortality in patients with stage 4–5 CKD. This prediction tool has a satisfactory performance. Prior to external validation, the proposed prediction tool can be used at: https://bit.ly/3JPhrkh for research purpose. This is the first time that a synthetic population has been applied to create predictive models. In this study, the synthetic population showed its advantage in dealing with sample size issues in developing predictive models and in improving the performance of the prediction tool in terms of sensitivity and PPV.

Deep learning-based subtyping of gastric cancer histology predicts clinical outcome: a multi-institutional retrospective study

IntroductionThe Laurén classification is widely used for Gastric Cancer (GC) histology subtyping. However, this classification is prone to interobserver variability and its prognostic value remains controversial. Deep Learning (DL)-based assessment of hematoxylin and eosin (H&E) stained slides is a potentially useful tool to provide an additional layer of clinically relevant information, but has not been systematically assessed in GC.ObjectiveWe aimed to train, test and externally validate a deep learning-based classifier for GC histology subtyping using routine H&E stained tissue sections from gastric adenocarcinomas and to assess its potential prognostic utility.MethodsWe trained a binary classifier on intestinal and diffuse type GC whole slide images for a subset of the TCGA cohort (N = 166) using attention-based multiple instance learning. The ground truth of 166 GC was obtained by two expert pathologists. We deployed the model on two external GC patient cohorts, one from Europe (N = 322) and one from Japan (N = 243). We assessed classification performance using the Area Under the Receiver Operating Characteristic Curve (AUROC) and prognostic value (overall, cancer specific and disease free survival) of the DL-based classifier with uni- and multivariate Cox proportional hazard models and Kaplan–Meier curves with log-rank test statistics.ResultsInternal validation using the TCGA GC cohort using five-fold cross-validation achieved a mean AUROC of 0.93 ± 0.07. External validation showed that the DL-based classifier can better stratify GC patients' 5-year survival compared to pathologist-based Laurén classification for all survival endpoints, despite frequently divergent model-pathologist classifications. Univariate overall survival Hazard Ratios (HRs) of pathologist-based Laurén classification (diffuse type versus intestinal type) were 1.14 (95% Confidence Interval (CI) 0.66–1.44, p-value = 0.51) and 1.23 (95% CI 0.96–1.43, p-value = 0.09) in the Japanese and European cohorts, respectively. DL-based histology classification resulted in HR of 1.46 (95% CI 1.18–1.65, p-value < 0.005) and 1.41 (95% CI 1.20–1.57, p-value < 0.005), in the Japanese and European cohorts, respectively. In diffuse type GC (as defined by the pathologist), classifying patients using the DL diffuse and intestinal classifications provided a superior survival stratification, and demonstrated statistically significant survival stratification when combined with pathologist classification for both the Asian (overall survival log-rank test p-value < 0.005, HR 1.43 (95% CI 1.05–1.66, p-value = 0.03) and European cohorts (overall survival log-rank test p-value < 0.005, HR 1.56 (95% CI 1.16–1.76, p-value < 0.005)).ConclusionOur study shows that gastric adenocarcinoma subtyping using pathologist’s Laurén classification as ground truth can be performed using current state of the art DL techniques. Patient survival stratification seems to be better by DL-based histology typing compared with expert pathologist histology typing. DL-based GC histology typing has potential as an aid in subtyping. Further investigations are warranted to fully understand the underlying biological mechanisms for the improved survival stratification despite apparent imperfect classification by the DL algorithm.

Deep learning-based ensemble model using hematoxylin and eosin (H&amp;E) whole slide images (WSIs) for the prediction of MET mutations in non-small cell lung cancer (NSCLC).

e13578 Background: The MET proto-oncogene is involved in tumorigenesis and progression and has emerged as an attractive targetable genomic alteration in NSCLC. However, it has not been actively tested in NSCLC due to the lack of a cost-effective screening tool. We developed a deep-learning based model to predict genomic profiles from H&amp;E WSI and applied it to MET pathologic mutations in NSCLC. The prediction model is based on an ensemble model trained not solely on H&amp;E images but also on multi-layered semantic contents that were produced by a pre-trained AI analyzer, Lunit SCOPE IO. Methods: The model was trained on the MET aberration (exon 14 skipping, n=191; pathogenic mutation, n=109) and paired wild-type dataset (n=500). To extract deep features from raw H&amp;E images, a self-supervised vision transformer was used, and an AI-based pathology profiling analyzer extracted semantic contents such as the spatial information of tumor cells, lymphocytes, cancer epithelium, and cancer stroma. A set of classifiers was trained based on the two features, and the ensemble of these features was used to improve robustness. Following cross-validation, the model was applied to an independent clinical dataset with MET sequencing results from The Cancer Genome Atlas (TCGA) LUAD and LUSC datasets (n=914), Samsung Medical Center (SMC, n=361), and Chonnam National University Hospital (CNUH, n=54). Results: The best cross-validation performances of the models predicting MET aberration measured by mean area under the receiver operating characteristic curve (AUROC) were 0.772 when trained by only H&amp;E images (HE-only), 0.788 by AI semantic content with MLP classifier (AISC-MLP), and 0.803 by AISC with random forest (AI-RF), respectively. An ensemble of the three models showed an increased AUROC of 0.837 in the training dataset by cross-validation. These models were applied to the external validation dataset (n=1,329), including 20 (1.5%) MET exon 14 skipping and known pathologic mutations. The mean AUROC to predict MET aberration by the ensemble model was 0.817 with 95% sensitivity, 64.7% specificity. The AUROC of TCGA, SMC, and CNUH cohorts were 0.815, 0.802, and 0.812, respectively. Conclusions: An AI-based ensemble model combining H&amp;E images with semantic contents extracted from pre-developed AI models significantly improved the accuracy and robustness of MET pathogenic mutation prediction using an H&amp;E sample in NSCLC. These findings allow for cost-effective screening for MET alterations.

O085 Estimating postoperative mortality in colorectal surgery- a systematic review of risk prediction models

Abstract Introduction Risk prediction models are frequently used to support decision-making in colorectal surgery but can be inaccurate. Machine learning (ML) is becoming increasingly popular, and its application may increase predictive accuracy. We compared conventional risk prediction models for postoperative mortality (based on regression analysis) with ML models to determine the benefit of the latter approach. Methods The study was registered in PROSPERO(CRD42022364753). Following the PRISMA guidelines, a systematic search of three databases (MEDLINE, EMBASE, WoS) was conducted (from 1/1/2000 to 29/09/2022). Studies were included if they reported the development of a risk model to estimate short-term postoperative mortality for patients undergoing colorectal surgery. Discrimination and calibration performance metrics were compared. Studies were evaluated against CHARMS and TRIPOD criteria. Results 3,052 articles were screened, and 45 studies were included. The total sample size was 1,356,058 patients. Six studies used ML techniques for model development. Most studies (n=42) reported the area under the receiver operating characteristic curve (AUROC) as a measure of discrimination. There was no significant difference in the mean AUROC values between regression models (0.833 s.d.±0.52) and ML (0.846 s.d.±0.55), p=0.539. Calibration statistics, which measure the agreement between predicted estimates and observed outcomes, were less consistent. Risk of bias assessment found most concerns in the data handling and analysis domains of eligible studies. Conclusion Our study showed comparable predictive performance between regression and ML methods in colorectal surgery. Integration of ML in colorectal risk prediction is promising but further refinement of the models is required to support routine clinical adoption.

Deep learning-based prediction of osseointegration for dental implant using plain radiography

BackgroundIn this study, we investigated whether deep learning-based prediction of osseointegration of dental implants using plain radiography is possible.MethodsPanoramic and periapical radiographs of 580 patients (1,206 dental implants) were used to train and test a deep learning model. Group 1 (338 patients, 591 dental implants) included implants that were radiographed immediately after implant placement, that is, when osseointegration had not yet occurred. Group 2 (242 patients, 615 dental implants) included implants radiographed after confirming successful osseointegration. A dataset was extracted using random sampling and was composed of training, validation, and test sets. For osseointegration prediction, we employed seven different deep learning models. Each deep-learning model was built by performing the experiment 10 times. For each experiment, the dataset was randomly separated in a 60:20:20 ratio. For model evaluation, the specificity, sensitivity, accuracy, and AUROC (Area under the receiver operating characteristic curve) of the models was calculated.ResultsThe mean specificity, sensitivity, and accuracy of the deep learning models were 0.780–0.857, 0.811–0.833, and 0.799–0.836, respectively. Furthermore, the mean AUROC values ranged from to 0.890–0.922. The best model yields an accuracy of 0.896, and the worst model yields an accuracy of 0.702.ConclusionThis study found that osseointegration of dental implants can be predicted to some extent through deep learning using plain radiography. This is expected to complement the evaluation methods of dental implant osseointegration that are currently widely used.

Development and validation of machine learning models for predicting prognosis and guiding individualized postoperative chemotherapy: A real-world study of distal cholangiocarcinoma.

Distal cholangiocarcinoma (dCCA), originating from the common bile duct, is greatly associated with a dismal prognosis. A series of different studies based on cancer classification have been developed, aimed to optimize therapy and predict and improve prognosis. In this study, we explored and compared several novel machine learning models that might lead to an improvement in prediction accuracy and treatment options for patients with dCCA. In this study, 169 patients with dCCA were recruited and randomly divided into the training cohort (n = 118) and the validation cohort (n = 51), and their medical records were reviewed, including survival outcomes, laboratory values, treatment strategies, pathological results, and demographic information. Variables identified as independently associated with the primary outcome by least absolute shrinkage and selection operator (LASSO) regression, the random survival forest (RSF) algorithm, and univariate and multivariate Cox regression analyses were introduced to establish the following different machine learning models and canonical regression model: support vector machine (SVM), SurvivalTree, Coxboost, RSF, DeepSurv, and Cox proportional hazards (CoxPH). We measured and compared the performance of models using the receiver operating characteristic (ROC) curve, integrated Brier score (IBS), and concordance index (C-index) following cross-validation. The machine learning model with the best performance was screened out and compared with the TNM Classification using ROC, IBS, and C-index. Finally, patients were stratified based on the model with the best performance to assess whether they benefited from postoperative chemotherapy through the log-rank test. Among medical features, five variables, including tumor differentiation, T-stage, lymph node metastasis (LNM), albumin-to-fibrinogen ratio (AFR), and carbohydrate antigen 19-9 (CA19-9), were used to develop machine learning models. In the training cohort and the validation cohort, C-index achieved 0.763 vs. 0.686 (SVM), 0.749 vs. 0.692 (SurvivalTree), 0.747 vs. 0.690 (Coxboost), 0.745 vs. 0.690 (RSF), 0.746 vs. 0.711 (DeepSurv), and 0.724 vs. 0.701 (CoxPH), respectively. The DeepSurv model (0.823 vs. 0.754) had the highest mean area under the ROC curve (AUC) than other models, including SVM (0.819 vs. 0.736), SurvivalTree (0.814 vs. 0.737), Coxboost (0.816 vs. 0.734), RSF (0.813 vs. 0.730), and CoxPH (0.788 vs. 0.753). The IBS of the DeepSurv model (0.132 vs. 0.147) was lower than that of SurvivalTree (0.135 vs. 0.236), Coxboost (0.141 vs. 0.207), RSF (0.140 vs. 0.225), and CoxPH (0.145 vs. 0.196). Results of the calibration chart and decision curve analysis (DCA) also demonstrated that DeepSurv had a satisfactory predictive performance. In addition, the performance of the DeepSurv model was better than that of the TNM Classification in C-index, mean AUC, and IBS (0.746 vs. 0.598, 0.823 vs. 0.613, and 0.132 vs. 0.186, respectively) in the training cohort. Patients were stratified and divided into high- and low-risk groups based on the DeepSurv model. In the training cohort, patients in the high-risk group would not benefit from postoperative chemotherapy (p = 0.519). In the low-risk group, patients receiving postoperative chemotherapy might have a better prognosis (p = 0.035). In this study, the DeepSurv model was good at predicting prognosis and risk stratification to guide treatment options. AFR level might be a potential prognostic factor for dCCA. For the low-risk group in the DeepSurv model, patients might benefit from postoperative chemotherapy.

Association between the extent of diffusion restriction on brain diffusion-weighted imaging and neurological outcomes after an out-of-hospital cardiac arrest

BackgroundThis study evaluated the association between the extent of diffusion restriction on brain diffusion-weighted imaging (DWI) and neurological outcomes in patients who underwent targeted temperature management (TTM) after an out-of-hospital cardiac arrest (OHCA). MethodsPatients who underwent brain magnetic resonance imaging within 10 days of OHCA between 2012 and 2021 were analysed. The extent of diffusion restriction was described according to the modified DWI Alberta Stroke Program Early Computed Tomography Score (DWI-ASPECTS). The 35 predefined brain regions were assigned a score if diffuse signal changes were concordantly present in DWI scans and apparent diffusion coefficient maps. The primary outcome was an unfavourable neurological outcome at 6 months. The sensitivity, specificity, and receiver operating characteristic (ROC) curves for the measured parameters were analysed. Cut-off values were determined to predict the primary outcome. The predictive cut-off DWI-ASPECTS was internally validated using five-fold cross-validation. ResultsOf the 301 patients, 108 (35.9%) had 6-month favourable neurological outcomes. Patients with unfavourable outcomes had higher whole-brain DWI-ASPECTS (median, 31 [26–33] vs. 0 [0–1], P < 0.001) than those with favourable outcomes. The area under the ROC curve (AUROC) of whole-brain DWI-ASPECTS was 0.957 (95% confidence interval [CI] 0.928–0.977). A cut-off value of ≥8 for unfavourable neurological outcomes had specificity and sensitivity of 100% (95% CI 96.6–100) and 89.6% (95% CI 84.4–93.6), respectively. The mean AUROC was 0.956. ConclusionMore extensive diffusion restriction on DWI-ASPECTS in patients with OHCA who underwent TTM was associated with 6-month unfavourable neurological outcomes.Running title: Diffusion restriction and neurological outcomes after cardiac arrest.