Area Under The Precision-recall Curve Research Articles

Sequential autoencoders for feature engineering and pretraining in major depressive disorder risk prediction.

We evaluated autoencoders as a feature engineering and pretraining technique to improve major depressive disorder (MDD) prognostic risk prediction. Autoencoders can represent temporal feature relationships not identified by aggregate features. The predictive performance of autoencoders of multiple sequential structures was evaluated as feature engineering and pretraining strategies on an array of prediction tasks and compared to a restricted Boltzmann machine (RBM) and random forests as a benchmark. We study MDD patients from Vanderbilt University Medical Center. Autoencoder models with Attention and long-short-term memory (LSTM) layers were trained to create latent representations of the input data. Predictive performance was evaluated temporally by fitting random forest models to predict future outcomes with engineered features as input and using autoencoder weights to initialize neural network layers. We evaluated area under the precision-recall curve (AUPRC) trends and variation over the study population's treatment course. The pretrained LSTM model improved predictive performance over pretrained Attention models and benchmarks in 3 of 4 outcomes including self-harm/suicide attempt (AUPRCs, LSTM pretrained = 0.012, Attention pretrained = 0.010, RBM = 0.009, random forest = 0.005). The use of autoencoders for feature engineering had varied results, with benchmarks outperforming LSTM and Attention encodings on the self-harm/suicide attempt outcome (AUPRCs, LSTM encodings = 0.003, Attention encodings = 0.004, RBM = 0.009, random forest = 0.005). Improvement in prediction resulting from pretraining has the potential for increased clinical impact of MDD risk models. We did not find evidence that the use of temporal feature encodings was additive to predictive performance in the study population. This suggests that predictive information retained by model weights may be lost during encoding. LSTM pretrained model predictive performance is shown to be clinically useful and improves over state-of-the-art predictors in the MDD phenotype. LSTM model performance warrants consideration of use in future related studies. LSTM models with pretrained weights from autoencoders were able to outperform the benchmark and a pretrained Attention model. Future researchers developing risk models in MDD may benefit from the use of LSTM autoencoder pretrained weights.

Detecting changes in the performance of a clinical machine learning tool over time

Excessive use of blood cultures (BCs) in Emergency Departments (EDs) results in low yields and high contamination rates, associated with increased antibiotic use and unnecessary diagnostics. Our team previously developed and validated a machine learning model to predict BC outcomes and enhance diagnostic stewardship. While the model showed promising initial results, concerns over performance drift due to evolving patient demographics, clinical practices, and outcome rates warrant continual monitoring and evaluation of such models. A real-time evaluation of the model's performance was conducted between October 2021 and September 2022. The model was integrated into Amsterdam UMC's Electronic Health Record system, predicting BC outcomes for all adult patients with BC draws in real time. The model's performance was assessed monthly using metrics including the Area Under the Curve (AUC), Area Under the Precision-Recall Curve (AUPRC), and Brier scores. Statistical Process Control (SPC) charts were used to monitor variation over time. Across 3.035 unique adult patient visits, the model achieved an average AUC of 0.78, AUPRC of 0.41, and a Brier score of 0.10 for predicting the outcome of BCs drawn in the ED. While specific population characteristics changed over time, no statistical points outside the statistical control range were detected in the AUC, AUPRC, and Brier scores, indicating stable model performance. The average BC positivity rate during the study period was 13.4%. Despite significant changes in clinical practice, our BC stewardship tool exhibited stable performance, suggesting its robustness to changing environments. Using SPC charts for various metrics enables simple and effective monitoring of potential performance drift. The assessment of the variation of outcome rates and population changes may guide the specific interventions, such as intercept correction or recalibration, that may be needed to maintain a stable model performance over time. This study suggested no need to recalibrate or correct our BC stewardship tool. No funding to disclose.

Managing RT Schedules of Early-Stage Breast Cancer Patients with a Genetic-Dosimetric Validated Model for Late Fibrosis

Define a multifactorial risk prediction model for RT-induced fibrosis and investigate the benefit of a personalized approach for breast cancer (BC) patients (pts) treated with whole breast RT. In a previous study, we confirmed the predictive role of 30 SNPs from the literature and built an interaction aware Polygenic Risk Score (PRS, following the methods from Franco RO 2021) for Late Fibrosis (FG2+) on a cohort of 1500 pts from the REQUITE EU/USA prospective observational study. The PRS weights the radiosensitive (RS) and radioresistant (RR) genetic components and can be included in NTCP models. In a subgroup from the same cohort (390 pts), we have also confirmed an NTCP model based on biologically Equivalent Uniform Dose (BEUD) from PTV DVHs for pts treated at 40-50 Gy and no RT boost. Here, we combine PRS and BEUD into a sigmoid model allowing PRS to modulate BEUD50 (BEUD leading to 50% FG2+), i.e., we permitted a personalized BEUD50. We can also consider this as translating the PRS into a personalized equivalent BEUD, which is added/subtracted to the treatment BEUD. We evaluated model performances through ROC-AUC, calibration plot and Precision-Recall AUC. A total of 381 pts had complete dosimetric/genetic data, prescribed dose 40-50 Gy, and no fibrotic alteration at RT start. We scored FG2+ in 87 pts (23%). PRS ranged from -13 (more RR pts) to 7 (more RS), and a unit in PRS corresponds to 5.3 Gy BEUD or 3 Gy in EQ EUD2 Gy. Table 1 summarizes model performances, with details for subgroups below/above the quartiles I/III of the BEUD distribution. The PRS-only model correctly describes the toxicity rates in the whole population (calibration slope/offset = 0/1). Still, it overestimates/underestimates the absolute risks in the low/high dose ranges. The integrated model improves AUC-ROC and AUC-PRC by 5% and 10% and guarantees a better calibration in pts receiving low/high BEUD to the PTV. We developed a multifactorial model for FG2+ based on two previously validated models and reported the improvement against single-factor models. The BEUD+PRS model is suitable for assisting clinicians in managing early-stage BC pts. The number of fractions or the daily dose could be reduced for RS pts. The integrated model resulted in a possible quantitative tool for driving the planning decision process. Also, it showed a better performance in the high BEUD region, suggesting the potential value of its extension toward RT including boost or ultra hypofractionation. We are testing this extension in the whole REQUITE cohort.

Advancing polytrauma care: developing and validating machine learning models for early mortality prediction

BackgroundRapid identification of high-risk polytrauma patients is crucial for early intervention and improved outcomes. This study aimed to develop and validate machine learning models for predicting 72 h mortality in adult polytrauma patients using readily available clinical parameters.MethodsA retrospective analysis was conducted on polytrauma patients from the Dryad database and our institution. Missing values pertinent to eligible individuals within the Dryad database were compensated for through the k-nearest neighbor algorithm, subsequently randomizing them into training and internal validation factions on a 7:3 ratio. The patients of our institution functioned as external validation cohorts. The predictive efficacy of random forest (RF), neural network, and XGBoost models was assessed through an exhaustive suite of performance indicators. The SHapley Additive exPlanations (SHAP) and Local Interpretable Model-Agnostic Explanations (LIME) methods were engaged to explain the supreme-performing model. Conclusively, restricted cubic spline analysis and multivariate logistic regression were employed as sensitivity analyses to verify the robustness of the findings.ResultsParameters including age, body mass index, Glasgow Coma Scale, Injury Severity Score, pH, base excess, and lactate emerged as pivotal predictors of 72 h mortality. The RF model exhibited unparalleled performance, boasting an area under the receiver operating characteristic curve (AUROC) of 0.87 (95% confidence interval [CI] 0.84–0.89), an area under the precision-recall curve (AUPRC) of 0.67 (95% CI 0.61–0.73), and an accuracy of 0.83 (95% CI 0.81–0.86) in the internal validation cohort, paralleled by an AUROC of 0.98 (95% CI 0.97–0.99), an AUPRC of 0.88 (95% CI 0.83–0.93), and an accuracy of 0.97 (95% CI 0.96–0.98) in the external validation cohort. It provided the highest net benefit in the decision curve analysis in relation to the other models. The outcomes of the sensitivity examinations were congruent with those inferred from SHAP and LIME.ConclusionsThe RF model exhibited the best performance in predicting 72 h mortality in adult polytrauma patients and has the potential to aid clinicians in identifying high-risk patients and guiding clinical decision-making.

Performance and limitations of a supervised deep learning approach for the histopathological Oxford Classification of glomeruli with IgA nephropathy

Background and ObjectiveThe Oxford Classification for IgA nephropathy is the most successful example of an evidence-based nephropathology classification system. The aim of our study was to replicate the glomerular components of Oxford scoring with an end-to-end deep learning pipeline that involves automatic glomerular segmentation followed by classification for mesangial hypercellularity (M), endocapillary hypercellularity (E), segmental sclerosis (S) and active crescents (C). MethodsA total number of 1056 periodic acid–Schiff (PAS) whole slide images (WSIs), coming from 386 kidney biopsies, were annotated. Several detection models for glomeruli, based on the Mask R-CNN architecture, were trained on 587 WSIs, validated on 161 WSIs, and tested on 127 WSIs. For the development of segmentation models, 20,529 glomeruli were annotated, of which 16,571 as training and 3958 as validation set. The test set of the segmentation module comprised of 2948 glomeruli. For the Oxford classification, 6206 expert-annotated glomeruli from 308 PAS WSIs were labelled for M, E, S, C and split into a training set of 4298 glomeruli from 207 WSIs, and a test set of 1908 glomeruli. We chose the best-performing models to construct an end-to-end pipeline, which we named MESCnn (MESC classification by neural network), for the glomerular Oxford classification of WSIs. ResultsInstance segmentation yielded excellent results with an AP50 ranging between 78.2–80.1 % (79.4 ± 0.7 %) on the validation and 75.1–77.7 % (76.5 ± 0.9 %) on the test set. The aggregated Jaccard Index was between 73.4–75.9 % (75.0 ± 0.8 %) on the validation and 69.1–73.4 % (72.2 ± 1.4 %) on the test set. At granular glomerular level, Oxford Classification was best replicated for M with EfficientNetV2-L with a mean ROC-AUC of 90.2 % and a mean precision/recall area under the curve (PR-AUC) of 81.8 %, best for E with MobileNetV2 (ROC-AUC 94.7 %) and ResNet50 (PR-AUC 75.8 %), best for S with EfficientNetV2-M (mean ROC-AUC 92.7 %, mean PR-AUC 87.7 %), best for C with EfficientNetV2-L (ROC-AUC 92.3 %) and EfficientNetV2-S (PR-AUC 54.7 %). At biopsy-level, correlation between expert and deep learning labels fulfilled the demands of the Oxford Classification. ConclusionWe designed an end-to-end pipeline for glomerular Oxford Classification on both a granular glomerular and an entire biopsy level. Both the glomerular segmentation and the classification modules are freely available for further development to the renal medicine community.

Prediction of American Society of Anesthesiologists Physical Status Classification from preoperative clinical text narratives using natural language processing

BackgroundElectronic health records (EHR) contain large volumes of unstructured free-form text notes that richly describe a patient’s health and medical comorbidities. It is unclear if perioperative risk stratification can be performed directly from these notes without manual data extraction. We conduct a feasibility study using natural language processing (NLP) to predict the American Society of Anesthesiologists Physical Status Classification (ASA-PS) as a surrogate measure for perioperative risk. We explore prediction performance using four different model types and compare the use of different note sections versus the whole note. We use Shapley values to explain model predictions and analyze disagreement between model and human anesthesiologist predictions.MethodsSingle-center retrospective cohort analysis of EHR notes from patients undergoing procedures with anesthesia care spanning all procedural specialties during a 5 year period who were not assigned ASA VI and also had a preoperative evaluation note filed within 90 days prior to the procedure. NLP models were trained for each combination of 4 models and 8 text snippets from notes. Model performance was compared using area under the receiver operating characteristic curve (AUROC) and area under the precision recall curve (AUPRC). Shapley values were used to explain model predictions. Error analysis and model explanation using Shapley values was conducted for the best performing model.ResultsFinal dataset includes 38,566 patients undergoing 61,503 procedures with anesthesia care. Prevalence of ASA-PS was 8.81% for ASA I, 31.4% for ASA II, 43.25% for ASA III, and 16.54% for ASA IV-V. The best performing models were the BioClinicalBERT model on the truncated note task (macro-average AUROC 0.845) and the fastText model on the full note task (macro-average AUROC 0.865). Shapley values reveal human-interpretable model predictions. Error analysis reveals that some original ASA-PS assignments may be incorrect and the model is making a reasonable prediction in these cases.ConclusionsText classification models can accurately predict a patient’s illness severity using only free-form text descriptions of patients without any manual data extraction. They can be an additional patient safety tool in the perioperative setting and reduce manual chart review for medical billing. Shapley feature attributions produce explanations that logically support model predictions and are understandable to clinicians.

LDS-CNN: a deep learning framework for drug-target interactions prediction based on large-scale drug screening.

Drug-target interaction (DTI) is a vital drug design strategy that plays a significant role in many processes of complex diseases and cellular events. In the face of challenges such as extensive protein data and experimental costs, it is suggested to apply bioinformatics approaches to exploit potential interactions to design new targeted medications. Different data and interaction types bring difficulties to study involving incompatible and heterology formats. The analysis of drug-target interactions in a comprehensive and unified model is a significant challenge. Here, we propose a general method for predicting interactions between small-molecule drugs and protein targets, Large-scale Drug target Screening Convolutional Neural Network (LDS-CNN), which used unified encoding to achieve the calculation of the different data formats in an integrated model to realize feature abstraction and potential object prediction. On 898,412 interaction data involving 1683 small-molecule compounds and 14,350 human proteins from 8.8 billion records, the proposed method achieved an area under the curve (AUC) of 0.96, an area under the precision-recall curve (AUPRC) of 0.95, and an accuracy of 90.13%. The experimental results illustrated that the proposed method attained high accuracy on the test set, indicating its high predictive ability in drug-target interaction prediction. LDS-CNN is effective for the prediction of large-scale datasets and datasets composed of data with different formats. In this study, we propose a DTI prediction method to solve the problems of unified encoding of large-scale data in multiple formats. It provides a feasible way to efficiently abstract the features among different types of drug-related data, thus reducing experimental costs and time consumption. The proposed method can be used to identify potential drug targets and candidates for the treatment of complex diseases. This work provides a reference for DTI to process large-scale data and different formats with deep learning methods and provides certain suggestions for future research.

Distilling Knowledge From an Ensemble of Vision Transformers for Improved Classification of Breast Ultrasound

To develop a deep learning model for the automated classification of breast ultrasound images as benign or malignant. More specifically, the application of vision transformers, ensemble learning, and knowledge distillation is explored for breast ultrasound classification. Single view, B-mode ultrasound images were curated from the publicly available Breast Ultrasound Image (BUSI) dataset, which has categorical ground truth labels (benign vs malignant) assigned by radiologists and malignant cases confirmed by biopsy. The performance of vision transformers (ViT) is compared to convolutional neural networks (CNN), followed by a comparison between supervised, self-supervised, and randomly initialized ViT. Subsequently, the ensemble of 10 independently trained ViT, where the ensemble model is the unweighted average of the output of each individual model is compared to the performance of each ViT alone. Finally, we train a single ViT to emulate the ensembled ViT using knowledge distillation. On this dataset that was trained using five-fold cross validation, ViT outperforms CNN, while self-supervised ViT outperform supervised and randomly initialized ViT. The ensemble model achieves an area under the receiver operating characteristics curve (AuROC) and area under the precision recall curve (AuPRC) of 0.977 and 0.965 on the test set, outperforming the average AuROC and AuPRC of the independently trained ViTs (0.958±0.05 and 0.931±0.016). The distilled ViT achieves an AuROC and AuPRC of 0.972 and 0.960. Both transfer learning and ensemble learning can each offer increased performance independently and can be sequentially combined to collectively improve the performance of the final model. Furthermore, a single vision transformer can be trained to match the performance of an ensemble of a set of vision transformers using knowledge distillation.

NGCICM: A Novel Deep Learning-Based Method for Predicting circRNA-miRNA Interactions.

The circRNAs and miRNAs play an important role in the development of human diseases, and they can be widely used as biomarkers of diseases for disease diagnosis. In particular, circRNAs can act as sponge adsorbers for miRNAs and act together in certain diseases. However, the associations between the vast majority of circRNAs and diseases and between miRNAs and diseases remain unclear. Computational-based approaches are urgently needed to discover the unknown interactions between circRNAs and miRNAs. In this paper, we propose a novel deep learning algorithm based on Node2vec and Graph ATtention network (GAT), Conditional Random Field (CRF) layer and Inductive Matrix Completion (IMC) to predict circRNAs and miRNAs interactions (NGCICM). We construct a GAT-based encoder for deep feature learning by fusing the talking-heads attention mechanism and the CRF layer. The IMC-based decoder is also constructed to obtain interaction scores. The Area Under the receiver operating characteristic Curve (AUC) of the NGCICM method is 0.9697, 0.9932 and 0.9980, and the Area Under the Precision-Recall curve (AUPR) is 0.9671, 0.9935 and 0.9981, respectively, using 2-fold, 5-fold and 10-fold Cross-Validation (CV) as the benchmark. The experimental results confirm the effectiveness of the NGCICM algorithm in predicting the interactions between circRNAs and miRNAs.

Factors associated with the quality of death certification in Brazilian municipalities: A data-driven non-linear model.

Studies evaluating the local quality of death certification in Brazil focused on completeness of death reporting or inappropriate coding of causes of death, with few investigating missing data. We aimed to use missing and unexpected values in core topics to assess the quality of death certification in Brazilian municipalities, to evaluate its correlation with the percentage of garbage codes, and to employ a data-driven approach with non-linear models to investigate the association of the socioeconomic and health infrastructure context with quality of death statistics among municipalities. This retrospective study used data from the Mortality Information System (2010-2017), and municipal data regarding healthcare infrastructure, socioeconomic characteristics, and death rates. Quality of death certification was assessed by missing or unexpected values in the following core topics: dates of occurrence, registration, and birth, place of occurrence, certifier, sex, and marital status. Models were fit to classify municipalities according to the quality of death certification (poor quality defined as death records with missing or unexpected values in core topics ≥ 80%). Municipalities with poor quality of death certification (43.9%) presented larger populations, lower death rates, lower socioeconomic index, healthcare infrastructure with fewer beds and physicians, and higher proportion of public healthcare facilities. The correlation coefficients between quality of death certification assessed by missing or unexpected values and the proportion of garbage codes were weak (0.11-0.49), but stronger for municipalities with lower socioeconomic scores. The model that best fitted the data was the random forest classifier (ROC AUC = 0.76; precision-recall AUC = 0.78). This innovative way of assessing the quality of death certification could help quality improvement initiatives to include the correctness of essential fields, in addition to garbage coding or completeness of records, especially in municipalities with lower socioeconomic status where garbage coding and the correctness of core topics appear to be related issues.

Convolutional Neural Networks (CNNs) for Pneumonia Classification on Pediatric Chest Radiographs.

Pneumonia is an infectious disease that is especially harmful to those with weak immune systems, such as children under the age of 5. While radiologists' diagnosis of pediatric pneumonia on chest radiographs (CXRs) is often accurate, subtle findings can be missed due to the subjective nature of the diagnosis process. Artificial intelligence (AI) techniques, such as convolutional neural networks (CNNs), can help make the process more objective and precise. However, off-the-shelf CNNs may perform poorly if they are not tuned to their appropriate hyperparameters. Our study aimed to identify the CNNs and their hyperparameter combinations (dropout, batch size, and optimizer) that optimize model performance. Sixty models based on five CNNs (VGG 16, VGG 19, DenseNet 121, DenseNet 169, and InceptionResNet V2) and 12 hyperparameter combinations were tested. Adam, Root Mean Squared Propagation (RmsProp), and Mini-Batch Stochastic Gradient Descent (SGD) optimizers were used. Two batch sizes, 32 and 64, were utilized. A dropout rate of either 0.5 or 0.7 was used in all dropout layers. We used a deidentified CXR dataset of 4200 pneumonia (Figure 1a) and 1600 normal images (Figure 1b).Seventy percent of the CXRs in the dataset were used for training the model, 20% were used for validating the model, and 10% were used for testing the model. All CNNs were trained first on the ImageNet dataset. They were then trained, with frozen weights, on the CXR-containing dataset. Results:Among the 60 models, VGG-19 (dropout of 0.5, batch size of 32, and Adam optimizer) was the most accurate. This model achieved an accuracy of 87.9%. A dropout of 0.5 consistently gave higher accuracy, area under the receiver operating characteristics curve (AUROC), and area under the precision-recall curve (AUPRC) compared to a dropout of 0.7. The CNNs InceptionResNet V2, DenseNet 169, VGG 16, and VGG 19 significantly outperformed the DenseNet121 CNN in accuracy and AUROC. The Adam and RmsProp optimizer had improved AUROC and AUPRC compared to the SGD optimizer. The batch size had no statistically significant effect on model performance. We recommend using low dropout rates (0.5) and RmsProp or Adam optimizer for pneumonia-detecting CNNs. Additionally, we discourage using the DenseNet121 CNN when other CNNs are available. Finally, the batch size may be set to any value, dependent on computational resources.

CT-based radiomics model for the prediction of genomic alterations in renal cell carcinoma (RCC)

Abstract Background Radiogenomics is an emerging tool with applications in screening for molecular biomarkers in diagnostic and prognostic assessment through the extraction of quantitative data from medical images (Shui et al., Front Oncol 2021). In this study, we aim to develop machine learning (ML) models to predict the most common genomic alterations present in our subset of renal cell carcinoma patients (pts). Methods Retrospectively, pts with tissue genomic testing from CT-guided biopsy samples were identified. Genomic testing was done via GEM ExTra assay, a CAP-accredited, CLIA-certified test encompassing tumor whole exome sequencing and whole transcriptome sequencing (TGen; Phoenix, AZ). Biopsy sample collection sites were identified from pre-biopsy contrast CT images, and the lesions were segmented with ITK-SNAP software, from which 510 radiomic features were extracted with Pyradiomics. The Least Absolute Shrinkage and Selection Operator (LASSO) regression was used to select the most relevant features. Logistic regression (LR) and support vector machine (SVM) classifiers were built for the prediction of PBRM1, VHL, and SETD2 gene alterations. Multiple metrics were used to evaluate the predictive performance via leave-one-out cross-validation, including the area under the receiver operating characteristic (AUROC) and the area under the precision-recall curve (AUPRC). Feature importance was evaluated with Shapley additive explanations (SHAP) method. Results A total of 14 RCC pts (10:4 M:F) with genomic testing from CT-guided biopsies were identified. The majority of pts were White (85.7%) and had clear cell histology (71.4%). The most common locations for the CT-guided biopsy were lung (18.2%), soft tissue (13.6%), kidney (9.1%), and bone (9.1%). The most common alterations were seen in PBRM1 (50%), VHL (43.9%), and SETD2 (35.7%) genes. The PBRM1 gene was predicted with the highest AUROC (0.84) and AUPRC (0.88) with the SVM classifier, followed by the SETD2 gene (AUROC=0.78 and AUPRC=0.66) with LR classifier and VHL gene (AUROC=0.56 and AUPRC=0.65) with SVM classifier. Notably, all three models showed good sensitivity in classifying gene mutation status (PBRM1: 0.86; VHL: 0.88; SETD2: 0.89). Among all radiomic features, first-order features, Gray Level Size Zone Matrix features, and Gray Level Dependence Matrix features were found to be the most important features for predictions. Conclusions Using a CT-based radiomics analysis of the biopsy area, we showed that SVM and LR prediction models could predict PBRM1, VHL, and SETD2 mutations with high accuracy. These models may assist in identifying potentially actionable alterations and yield ease in treatment selection for RCC. Further extensive studies are warranted to validate our findings and improve our model. CDMRP DOD Funding: no

Random Forest Modeling of Acute Toxicity in Anal Cancer: Effects of Peritoneal Cavity Contouring Approaches on Model Performance

Our purpose was to analyze the effect on gastrointestinal (GI) toxicity models when their dose-volume metrics predictors are derived from segmentations of the peritoneal cavity after different contouring approaches. A random forest machine learning approach was used to predict acute grade ≥3 GI toxicity from dose-volume metrics and clinicopathologic factors for 246 patients (toxicity incidence=9.5%) treated with definitive chemoradiation for squamous cell carcinoma of the anus. Three types of random forest models were constructed based on different bowel bag segmentation approaches: (1) physician-delineated after Radiation Therapy Oncology Group (RTOG) guidelines, (2) autosegmented by a deep learning model (nnU-Net) following RTOG guidelines, and (3) autosegmented but spanning the entire bowel space. Each model type was evaluated using repeated cross-validation (100 iterations; 50%/50% training/test split). The performance of the models was assessed using area under the precision-recall curve (AUPRC) and the receiver operating characteristic curve (AUROCC), as well as optimal F1 score. When following RTOG guidelines, the models based on the nnU-Net auto segmentations (mean values: AUROCC, 0.71 ± 0.07; AUPRC, 0.42 ± 0.09; F1 score, 0.46 ± 0.08) significantly outperformed (P < .001) those based on the physician-delineated contours (mean values: AUROCC, 0.67 ± 0.07; AUPRC, 0.34 ± 0.08; F1 score, 0.36 ± 0.07). When spanning the entire bowel space, the performance of the autosegmentation models improved considerably (mean values: AUROCC, 0.87 ± 0.05; AUPRC, 0.70 ± 0.09; F1 score, 0.68 ± 0.09). Random forest models were superior at predicting acute grade ≥3 GI toxicity when based on RTOG-defined bowel bag autosegmentations rather than physician-delineated contours. Models based on autosegmentations spanning the entire bowel space show further considerable improvement in model performance. The results of this study should be further validated using an external data set.

Prediction of gestational diabetes mellitus in Asian women using machine learning algorithms

This study developed a machine learning algorithm to predict gestational diabetes mellitus (GDM) using retrospective data from 34,387 pregnancies in multi-centers of South Korea. Variables were collected at baseline, E0 (until 10 weeks’ gestation), E1 (11–13 weeks’ gestation) and M1 (14–24 weeks’ gestation). The data set was randomly divided into training and test sets (7:3 ratio) to compare the performances of light gradient boosting machine (LGBM) and extreme gradient boosting (XGBoost) algorithms, with a full set of variables (original). A prediction model with the whole cohort achieved area under the receiver operating characteristics curve (AUC) and area under the precision-recall curve (AUPR) values of 0.711 and 0.246 at baseline, 0.720 and 0.256 at E0, 0.721 and 0.262 at E1, and 0.804 and 0.442 at M1, respectively. Then comparison of three models with different variable sets were performed: [a] variables from clinical guidelines; [b] selected variables from Shapley additive explanations (SHAP) values; and [c] Boruta algorithms. Based on model [c] with the least variables and similar or better performance than the other models, simple questionnaires were developed. The combined use of maternal factors and laboratory data could effectively predict individual risk of GDM using a machine learning model.

A machine learning-based model for predicting distant metastasis in patients with rectal cancer.

Distant metastasis from rectal cancer usually results in poorer survival and quality of life, so early identification of patients at high risk of distant metastasis from rectal cancer is essential. The study used eight machine-learning algorithms to construct a machine-learning model for the risk of distant metastasis from rectal cancer. We developed the models using 23867 patients with rectal cancer from the Surveillance, Epidemiology, and End Results (SEER) database between 2010 and 2017. Meanwhile, 1178 rectal cancer patients from Chinese hospitals were selected to validate the model performance and extrapolation. We tuned the hyperparameters by random search and tenfold cross-validation to construct the machine-learning models. We evaluated the models using the area under the receiver operating characteristic curves (AUC), the area under the precision-recall curve (AUPRC), decision curve analysis, calibration curves, and the precision and accuracy of the internal test set and external validation cohorts. In addition, Shapley's Additive explanations (SHAP) were used to interpret the machine-learning models. Finally, the best model was applied to develop a web calculator for predicting the risk of distant metastasis in rectal cancer. The study included 23,867 rectal cancer patients and 2,840 patients with distant metastasis. Multiple logistic regression analysis showed that age, differentiation grade, T-stage, N-stage, preoperative carcinoembryonic antigen (CEA), tumor deposits, perineural invasion, tumor size, radiation, and chemotherapy were-independent risk factors for distant metastasis in rectal cancer. The mean AUC value of the extreme gradient boosting (XGB) model in ten-fold cross-validation in the training set was 0.859. The XGB model performed best in the internal test set and external validation set. The XGB model in the internal test set had an AUC was 0.855, AUPRC was 0.510, accuracy was 0.900, and precision was 0.880. The metric AUC for the external validation set of the XGB model was 0.814, AUPRC was 0.609, accuracy was 0.800, and precision was 0.810. Finally, we constructed a web calculator using the XGB model for distant metastasis of rectal cancer. The study developed and validated an XGB model based on clinicopathological information for predicting the risk of distant metastasis in patients with rectal cancer, which may help physicians make clinical decisions. rectal cancer, distant metastasis, web calculator, machine learning algorithm, external validation.

Prehospital Early Warning Scores to Predict Mortality in Patients Using Ambulances

Early warning scores (EWSs) are designed for in-hospital use but are widely used in the prehospital field, especially in select groups of patients potentially at high risk. To be useful for paramedics in daily prehospital clinical practice, evaluations are needed of the predictive value of EWSs based on first measured vital signs on scene in large cohorts covering unselected patients using ambulance services. To validate EWSs' ability to predict mortality and intensive care unit (ICU) stay in an unselected cohort of adult patients who used ambulances. This prognostic study conducted a validation based on a cohort of adult patients (aged ≥18 years) who used ambulances in the North Denmark Region from July 1, 2016, to December 31, 2020. EWSs (National Early Warning Score 2 [NEWS2], modified NEWS score without temperature [mNEWS], Quick Sepsis Related Organ Failure Assessment [qSOFA], Rapid Emergency Triage and Treatment System [RETTS], and Danish Emergency Process Triage [DEPT]) were calculated using first vital signs measured by ambulance personnel. Data were analyzed from September 2022 through May 2023. The primary outcome was 30-day-mortality. Secondary outcomes were 1-day-mortality and ICU admission. Discrimination was assessed using area under the receiver operating characteristic curve (AUROC) and area under the precision recall curve (AUPRC). There were 107 569 unique patients (52 650 females [48.9%]; median [IQR] age, 65 [45-77] years) from the entire cohort of 219 323 patients who used ambulance services, among whom 119 992 patients (54.7%) had called the Danish national emergency number. NEWS2, mNEWS, RETTS, and DEPT performed similarly concerning 30-day mortality (AUROC range, 0.67 [95% CI, 0.66-0.68] for DEPT to 0.68 [95% CI, 0.68-0.69] for mNEWS), while qSOFA had lower performance (AUROC, 0.59 [95% CI, 0.59-0.60]; P vs other scores < .001). All EWSs had low AUPRCs, ranging from 0.09 (95% CI, 0.09-0.09) for qSOFA to 0.14 (95% CI, 0.13-0.14) for mNEWS.. Concerning 1-day mortality and ICU admission NEWS2, mNEWS, RETTS, and DEPT performed similarly, with AUROCs ranging from 0.72 (95% CI, 0.71-0.73) for RETTS to 0.75 (95% CI, 0.74-0.76) for DEPT in 1-day mortality and 0.66 (95% CI, 0.65-0.67) for RETTS to 0.68 (95% CI, 0.67-0.69) for mNEWS in ICU admission, and all EWSs had low AUPRCs. These ranged from 0.02 (95% CI, 0.02-0.03) for qSOFA to 0.04 (95% CI, 0.04-0.04) for DEPT in 1-day mortality and 0.03 (95% CI, 0.03-0.03) for qSOFA to 0.05 (95% CI, 0.04-0.05) for DEPT in ICU admission. This study found that EWSs in daily clinical use in emergency medical settings performed moderately in the prehospital field among unselected patients who used ambulances when assessed based on initial measurements of vital signs. These findings suggest the need of appropriate triage and early identification of patients at low and high risk with new and better EWSs also suitable for prehospital use.

Predicting 7-day unplanned readmission in elderly patients with coronary heart disease using machine learning.

Short-term unplanned readmission is always neglected, especially for elderly patients with coronary heart disease (CHD). However, tools to predict unplanned readmission are lacking. This study aimed to establish the most effective predictive model for the unplanned 7-day readmission in elderly CHD patients using machine learning (ML) algorithms. The detailed clinical data of elderly CHD patients were collected retrospectively. Five ML algorithms, including extreme gradient boosting (XGB), random forest, multilayer perceptron, categorical boosting, and logistic regression, were used to establish predictive models. We used the area under the receiver operating characteristic curve (AUC), accuracy, precision, recall, the F1 value, the Brier score, the area under the precision-recall curve (AUPRC), and the calibration curve to evaluate the performance of ML models. The SHapley Additive exPlanations (SHAP) value was used to interpret the best model. The final study included 834 elderly CHD patients, whose average age was 73.5 ± 8.4 years, among whom 426 (51.08%) were men and 139 had 7-day unplanned readmissions. The XGB model had the best performance, exhibiting the highest AUC (0.9729), accuracy (0.9173), F1 value (0.9134), and AUPRC (0.9766). The Brier score of the XGB model was 0.08. The calibration curve of the XGB model showed good performance. The SHAP method showed that fracture, hypertension, length of stay, aspirin, and D-dimer were the most important indicators for the risk of 7-day unplanned readmissions. The top 10 variables were used to build a compact XGB, which also showed good predictive performance. In this study, five ML algorithms were used to predict 7-day unplanned readmissions in elderly patients with CHD. The XGB model had the best predictive performance and potential clinical application perspective.

Learning from Highly Imbalanced Big Data with Label Noise

This study explores the effects of class label noise on detecting fraud within three highly imbalanced healthcare fraud data sets containing millions of claims and minority class sizes as small as 0.1%. For each data set, 29 noise distributions are simulated by varying the level of class noise and the distribution of noise between the fraudulent and non-fraudulent classes. Four popular machine learning algorithms are evaluated on each noise distribution using six rounds of five-fold cross-validation. Performance is measured using the area under the precision-recall curve (AUPRC), true positive rate (TPR), and true negative rate (TNR) in order to understand the effect of the noise level, noise distribution, and their interactions. AUPRC results show that negative class noise, i.e. fraudulent samples incorrectly labeled as non-fraudulent, is the most detrimental to model performance. TPR and TNR results show that there are significant trade-offs in class-wise performance as noise transitions between the positive and the negative class. Finally, results reveal how overfitting negatively impacts the classification performance of some learners, and how simple regularization can be used to combat this overfitting and improve classification performance across all noise distributions.

Histology-Based Prediction of Therapy Response to Neoadjuvant Chemotherapy for Esophageal and Esophagogastric Junction Adenocarcinomas Using Deep Learning.

Quantifying treatment response to gastroesophageal junction (GEJ) adenocarcinomas is crucial to provide an optimal therapeutic strategy. Routinely taken tissue samples provide an opportunity to enhance existing positron emission tomography-computed tomography (PET/CT)-based therapy response evaluation. Our objective was to investigate if deep learning (DL) algorithms are capable of predicting the therapy response of patients with GEJ adenocarcinoma to neoadjuvant chemotherapy on the basis of histologic tissue samples. This diagnostic study recruited 67 patients with I-III GEJ adenocarcinoma from the multicentric nonrandomized MEMORI trial including three German university hospitals TUM (University Hospital Rechts der Isar, Munich), LMU (Hospital of the Ludwig-Maximilians-University, Munich), and UME (University Hospital Essen, Essen). All patients underwent baseline PET/CT scans and esophageal biopsy before and 14-21 days after treatment initiation. Treatment response was defined as a ≥35% decrease in SUVmax from baseline. Several DL algorithms were developed to predict PET/CT-based responders and nonresponders to neoadjuvant chemotherapy using digitized histopathologic whole slide images (WSIs). The resulting models were trained on TUM (n = 25 pretherapy, n = 47 on-therapy) patients and evaluated on our internal validation cohort from LMU and UME (n = 17 pretherapy, n = 15 on-therapy). Compared with multiple architectures, the best pretherapy network achieves an area under the receiver operating characteristic curve (AUROC) of 0.81 (95% CI, 0.61 to 1.00), an area under the precision-recall curve (AUPRC) of 0.82 (95% CI, 0.61 to 1.00), a balanced accuracy of 0.78 (95% CI, 0.60 to 0.94), and a Matthews correlation coefficient (MCC) of 0.55 (95% CI, 0.18 to 0.88). The best on-therapy network achieves an AUROC of 0.84 (95% CI, 0.64 to 1.00), an AUPRC of 0.82 (95% CI, 0.56 to 1.00), a balanced accuracy of 0.80 (95% CI, 0.65 to 1.00), and a MCC of 0.71 (95% CI, 0.38 to 1.00). Our results show that DL algorithms can predict treatment response to neoadjuvant chemotherapy using WSI with high accuracy even before therapy initiation, suggesting the presence of predictive morphologic tissue biomarkers.

Predicting Herb-disease Associations Through Graph Convolutional Network

Background: In recent years, herbs have become very popular worldwide as a form of complementary and alternative medicine (CAM). However, there are many types of herbs and diseases, whose associations are impossible to be fully revealed. Identifying new therapeutic indications of herbs, that is drug repositioning, is a critical supplement for new drug development. Considering that exploring the associations between herbs and diseases by wet-lab techniques is time-consuming and laborious, there is an urgent need for reliable computational methods to fill this gap. : In this study, we first preprocessed the herbs and their indications in the TCM-Suit database, a comprehensive, accurate, and integrated traditional Chinese medicine database, to obtain the herb-disease association network. We then proposed a novel model based on a graph convolution network (GCN) to infer potential new associations between herbs and diseases. Methods: In our method, the effective features of herbs and diseases were extracted through multi-layer GCN, then the layer attention mechanism was introduced to combine the features learned from multiple GCN layers, and jump connections were added to reduce the over-smoothing phenomenon caused by multi-layer GCN stacking. Finally, the recovered herb-disease association network was generated by the bilinear decoder. We applied our model together with four other methods (including SCMFDD, BNNR, LRMCMDA, and DRHGCN) to predict herb-disease associations. Compared with all other methods, our model showed the highest area under the receiver operating characteristic curve (AUROC), the area under the precision-recall curve (AUPRC), as well as the highest recall in the five-fold cross-validation. Conclusion: We further used our model to predict the candidate herbs for Alzheimer's disease and found the compounds mediating herbs and diseases through the herb-compound-gene-disease network. The relevant literature also confirmed our findings.