Development of a machine learning-based prediction model: Identifying high-risk households and child maltreatment risk

For the implementation of valuable prediction models in clinical practice, properly conducted and well reported studies on early stages of model development and validation are essential. We systematically reviewed the adherence of 152 studies on machine learning-based prediction models to the 22-item checklist with the minimum standards for high quality reporting, TRIPOD. Overall, articles adhered to a median of 38.7% of applicable TRIPOD items. We identified that TRIPOD requires new items to cover AI-related aspects. We systematically reviewed the included studies for 15 spin practices and 11 poor reporting standards. A considerable number of studies lack a pre-specified protocol, make claims of clinical applicability (without further validation), and limitations are neither reported nor discussed in the context of previously developed models. Given that a first approach to spin evaluation using a classification scheme for studies on prognostic factors proved to be inefficient, we present SPIN-PM, a new framework for spin identification tailored to studies on prediction models. We provide a detailed overview of the study design, modelling strategies, and performance measures reported in studies on machine learning-based prediction models. Most studies reported only the development of prediction models and focused on binary outcomes. Within the 152 studies, we evaluated 522 models (on average 9.4 models per study), in which the most common modelling algorithms used were support vector machine and random forest. We comprehensively reviewed the methodological quality and risk of bias of studies on prediction models developed using machine learning techniques. We applied PROBAST to 152 studies on model development and 19 external validations. Of these 171 analyses, 148 were rated at high risk of bias. Efforts to improve the design, conduct, reporting, and validation of such studies are necessary to boost the introduction of machine learning based prediction models into clinical practice. We compared the absolute risk probabilities of three different modelling techniques: logistic regression, random forest, and support vector machine. For the last two techniques, we applied two different implementation methods within the package ‘caret’ in the statistical software R. Using logistic regression as benchmark, we showed that risk probabilities for deep venous thrombosis vary substantially between modelling techniques and implementation methods. TRIPOD and PROBAST were published to facilitate the critical appraisal of studies on diagnostic and prognostic prediction models. We described the five stages for the development of both extensions to machine learning-based models. The systematic reviews presented in this thesis compromised stage one. A survey using the Delphi methodology constitute stage 2 and the results are briefly discussed in Chapter 12. In conclusion, we have thoroughly evaluated the methodological conduct and reporting of studies on machine learning-based prediction models. The findings will contribute to the development of both PROBAST-AI and TRIPOD-AI. Furthermore, we have proposed a framework for spin identification in studies on prediction models, SPIN-PM. Overall, we upgraded the current guidance for quality evaluation and interpretation of findings in studies on prediction models, potentially helping reduce vague and biased research outputs.

BackgroundThe machine learning models with dose factors and the deep learning models with dose distribution matrix have been used to building lung toxics models for radiotherapy and achieve promising results. However, few studies have integrated clinical features into deep learning models. This study aimed to explore the role of three-dimension dose distribution and clinical features in predicting radiation pneumonitis (RP) in esophageal cancer patients after radiotherapy and designed a new hybrid deep learning network to predict the incidence of RP.MethodsA total of 105 esophageal cancer patients previously treated with radiotherapy were enrolled in this study. The three-dimension (3D) dose distributions within the lung were extracted from the treatment planning system, converted into 3D matrixes and used as inputs to predict RP with ResNet. In total, 15 clinical factors were normalized and converted into one-dimension (1D) matrixes. A new prediction model (HybridNet) was then built based on a hybrid deep learning network, which combined 3D ResNet18 and 1D convolution layers. Machine learning-based prediction models, which use the traditional dosiomic factors with and without the clinical factors as inputs, were also constructed and their predictive performance compared with that of HybridNet using tenfold cross validation. Accuracy and area under the receiver operator characteristic curve (AUC) were used to evaluate the model effect. DeLong test was used to compare the prediction results of the models.ResultsThe deep learning-based model achieved superior prediction results compared with machine learning-based models. ResNet performed best in the group that only considered dose factors (accuracy, 0.78 ± 0.05; AUC, 0.82 ± 0.25), whereas HybridNet performed best in the group that considered both dose factors and clinical factors (accuracy, 0.85 ± 0.13; AUC, 0.91 ± 0.09). HybridNet had higher accuracy than that of Resnet (p = 0.009).ConclusionBased on prediction results, the proposed HybridNet model could predict RP in esophageal cancer patients after radiotherapy with significantly higher accuracy, suggesting its potential as a useful tool for clinical decision-making. This study demonstrated that the information in dose distribution is worth further exploration, and combining multiple types of features contributes to predict radiotherapy response.

Early identification of noninvasive respiratory support (NIRS) failure in acute respiratory failure (ARF) is clinically relevant, as delayed intubation is associated with worse outcomes. Machine learning-based prediction models have been proposed to support escalation decisions, but their performance and reliability remain uncertain. To systematically evaluate the discriminative performance of machine learning-based models for predicting NIRS failure in adults with ARF. We conducted a systematic review and meta-analysis following PRISMA 2020 guidelines and registered the protocol in PROSPERO (CRD420251167330). PubMed, Web of Science, and Scopus were searched from January 2010 to the final search date. Cohort studies developing or validating machine learning models to predict NIRS failure, primarily defined as endotracheal intubation, were included. Discrimination was assessed using the area under the receiver operating characteristic curve (AUC). Logit-transformed AUCs were synthesized using random-effects models with restricted maximum likelihood estimation and Hartung-Knapp confidence intervals. Risk of bias and certainty of evidence were assessed using PROBAST-AI and GRADE, respectively. Fourteen cohort studies comprising 34,500 patients were included. The descriptive pooled AUC was 0.84 (95% CI, 0.78-0.89) with extreme heterogeneity (I2 = 99.5%) and wide prediction intervals. Subgroup analyses showed no statistically significant differences by validation strategy or type of noninvasive respiratory support. All studies were rated at high risk of bias, and the certainty of evidence was very low. Machine learning-based models demonstrate moderate discrimination; however, extreme heterogeneity, high risk of bias, and very low certainty of evidence preclude clinical implementation. https://www.crd.york.ac.uk/PROSPERO/view/CRD420251167330.

Empirical evaluation of performance degradation of machine learning-based predictive models – A case study in healthcare information systems

Simple SummaryMulti-gene assays (MGAs), such as Oncotype DX and Mammaprint, are used to provide predictive and prognostic values in treatment of ER+HER2− breast cancer. However, their accessibility is restricted due to their high cost in some countries. For this reason, many studies have been conducted to develop the tests that can replace the multi-gene assays, but practicality is still insufficient. The aim of our study is to develop a highly accessible machine learning-based model for predicting the result of MGA. Our accurate and affordable machine learning-based predictive model may serve as a cost-effective alternative to the expensive multi-gene assays.This study aimed to develop a machine learning-based prediction model for predicting multi-gene assay (MGA) risk categories. Patients with estrogen receptor-positive (ER+)/HER2− breast cancer who had undergone Oncotype DX (ODX) or MammaPrint (MMP) were used to develop the prediction model. The development cohort consisted of a total of 2565 patients including 2039 patients tested with ODX and 526 patients tested with MMP. The MMP risk prediction model utilized a single XGBoost model, and the ODX risk prediction model utilized combined LightGBM, CatBoost, and XGBoost models through soft voting. Additionally, the ensemble (MMP + ODX) model combining MMP and ODX utilized CatBoost and XGBoost through soft voting. Ten random samples, corresponding to 10% of the modeling dataset, were extracted, and cross-validation was performed to evaluate the accuracy on each validation set. The accuracy of our predictive models was 84.8% for MMP, 87.9% for ODX, and 86.8% for the ensemble model. In the ensemble cohort, the sensitivity, specificity, and precision for predicting the low-risk category were 0.91, 0.66, and 0.92, respectively. The prediction accuracy exceeded 90% in several subgroups, with the highest prediction accuracy of 95.7% in the subgroup that met Ki-67 <20 and HG 1~2 and premenopausal status. Our machine learning-based predictive model has the potential to complement existing MGAs in ER+/HER2− breast cancer.

Perinatal mental health and risk of child maltreatment: A systematic review and meta-analysis

Development of learning-based predictive models for radiation-induced atrial fibrillation in non-small cell lung cancer patients by integrating patient-specific clinical, dosimetry, and diagnostic information

Precise prediction of hospital stay duration is essential for maximizing resource utilization during surgery. Existing lumbar spinal stenosis (LSS) surgery prediction models lack accuracy and generalizability. Machine learning can improve accuracy by considering preoperative factors. This study aimed to develop and validate a machine learning-based model for estimating hospital stay duration following decompression surgery for LSS. Data from 848 patients who underwent decompression surgery for LSS at three hospitals were examined. Twelve prediction models, using 79 preoperative variables, were developed for postoperative hospital stay estimation. The top five models were chosen. Fourteen models predicted prolonged hospital stay (≥14 days), and the most accurate model was chosen. Models were validated using a randomly divided training sample (70%) and testing cohort (30%). The top five models showed moderate linear correlations (0.576-0.624) between predicted and measured values in the testing sample. The ensemble of these models had moderate prediction accuracy for final length of stay (linear correlation 0.626, absolute mean error 2.26 days, standard deviation 3.45 days). The c5.0 decision tree model was the top predictor for prolonged hospital stay, with accuracies of 89.63% (training) and 87.2% (testing). Key predictors for longer stay included JOABPEQ social life domain, facility, history of vertebral fracture, diagnosis, and Visual Analogue Scale (VAS) of low back pain. A machine learning-based model was developed to predict postoperative hospital stay after LSS decompression surgery, using data from multiple hospital settings. Numerical prediction of length of stay was not very accurate, although favorable prediction of prolonged stay was accomplished using preoperative factors. The JOABPEQ social life domain score was the most important predictor.

There is increasing interest in machine learning-based prediction models in inflammatory bowel diseases [IBD]. We synthesised and critically appraised studies comparing machine learning vs traditional statistical models, using routinely available clinical data for risk prediction in IBD. Through a systematic review till January 1, 2021, we identified cohort studies that derived and/or validated machine learning models, based on routinely collected clinical data in patients with IBD, to predict the risk of harbouring or developing adverse clinical outcomes, and reported its predictive performance against a traditional statistical model for the same outcome. We appraised the risk of bias in these studies using the Prediction model Risk of Bias ASsessment [PROBAST] tool. We included 13 studies on machine learning-based prediction models in IBD, encompassing themes of predicting treatment response to biologics and thiopurines and predicting longitudinal disease activity and complications and outcomes in patients with acute severe ulcerative colitis. The most common machine learning models used were tree-based algorithms, which are classification approaches achieved through supervised learning. Machine learning models outperformed traditional statistical models in risk prediction. However, most models were at high risk of bias, and only one was externally validated. Machine learning-based prediction models based on routinely collected data generally perform better than traditional statistical models in risk prediction in IBD, though frequently have high risk of bias. Future studies examining these approaches are warranted, with special focus on external validation and clinical applicability.

We compared the prediction performance of machine learning-based undiagnosed diabetes prediction models with that of traditional statistics-based prediction models. We used the 2014–2020 Korean National Health and Nutrition Examination Survey (KNHANES) (N = 32,827). The KNHANES 2014–2018 data were used as training and internal validation sets and the 2019–2020 data as external validation sets. The receiver operating characteristic curve area under the curve (AUC) was used to compare the prediction performance of the machine learning-based and the traditional statistics-based prediction models. Using sex, age, resting heart rate, and waist circumference as features, the machine learning-based model showed a higher AUC (0.788 vs. 0.740) than that of the traditional statistical-based prediction model. Using sex, age, waist circumference, family history of diabetes, hypertension, alcohol consumption, and smoking status as features, the machine learning-based prediction model showed a higher AUC (0.802 vs. 0.759) than the traditional statistical-based prediction model. The machine learning-based prediction model using features for maximum prediction performance showed a higher AUC (0.819 vs. 0.765) than the traditional statistical-based prediction model. Machine learning-based prediction models using anthropometric and lifestyle measurements may outperform the traditional statistics-based prediction models in predicting undiagnosed diabetes.

Parental personality disorder and child maltreatment: A systematic review and meta-analysis

Documenting Opportunity for Systematic Identification and Mitigation of Risk for Child Maltreatment.

Most screening tests for T2DM in use today were developed using multivariate regression methods that are often further simplified to allow transformation into a scoring formula. The increasing volume of electronically collected data opened the opportunity to develop more complex, accurate prediction models that can be continuously updated using machine learning approaches. This study compares machine learning-based prediction models (i.e. Glmnet, RF, XGBoost, LightGBM) to commonly used regression models for prediction of undiagnosed T2DM. The performance in prediction of fasting plasma glucose level was measured using 100 bootstrap iterations in different subsets of data simulating new incoming data in 6-month batches. With 6 months of data available, simple regression model performed with the lowest average RMSE of 0.838, followed by RF (0.842), LightGBM (0.846), Glmnet (0.859) and XGBoost (0.881). When more data were added, Glmnet improved with the highest rate (+ 3.4%). The highest level of variable selection stability over time was observed with LightGBM models. Our results show no clinically relevant improvement when more sophisticated prediction models were used. Since higher stability of selected variables over time contributes to simpler interpretation of the models, interpretability and model calibration should also be considered in development of clinical prediction models.

Dementia is among the most distressing and burdensome health challenges in aging populations. Treatment efficacy is limited; however, early diagnosis can delay or prevent disease progression. Previous machine learning-based prediction models have limitations (e.g., they are based on clinical parameters or are not generalizable). Thus, in this study, prediction models were developed for current and future dementia solely based on demographic, socioeconomic, and health-related features. Demographic, socioeconomic, and health-related variables collected from the Korean Longitudinal Study of Ageing (KLoSA) were used to develop machine learning-based prediction models for current and future dementia with various algorithms. Two sampling strategies were used for feature selection, one based on domain knowledge and the other based on statistical testing. Hyperparameter tuning was performed using grid search with cross-validation on the training set, and model evaluation was conducted on a separate test set. In the initial no-follow-up dataset, 92 of 6,898 participants exhibited dementia. Among 6,207 participants without dementia initially, 69 developed dementia within 2 years. Linear support vector machine (SVM) and radial bias function SVM exhibited the best sensitivity for current and future dementia (79.4% and 77.7%, respectively). The SHAP (SHapley Additive exPlanations) approach improved the transparency of the model by highlighting the top ten features most strongly associated with increased dementia risk. We achieved reasonably accurate prediction results for dementia using only non-clinical features.

Development and Validation of a Machine Learning-Based Prediction Model for Detection of Biliary Atresia.