Model Performance Measures Research Articles

Diagnostic models to differentiate Takotsubo syndrome from acute coronary syndrome: A systematic review and meta-analysis.

Differentiation between patients with Takotsubo syndrome and acute coronary syndrome (ACS) remains a challenge. We performed a systematic review to identify and evaluate diagnostic predictive models to distinguish both conditions. We performed an electronic search in PubMed, EMBASE, and Scopus until January 2024. Observational studies that developed and/or validated multivariable diagnostic models to differentiate Takotsubo syndrome from ACS were included. The risk of bias was assessed using the Prediction model Risk Of Bias ASsessment Tool (PROBAST). We conducted a narrative synthesis of the performance measures of the diagnostic models evaluated in each study. In addition, a random-effects meta-analysis of the c-statistic with its 95% confidence interval (CI) of the InterTAK model was performed. Of 1015 articles, a total of 11 studies (n = 4552) were included. We identified eight new diagnostic models and eight were external validation of existing models. The most frequent model was InterTAK (n = 4). The reported c-statistic ranged from 0.77 to 0.97 across all models. Calibration plots were reported only for two models. The summary c-statistic was 0.89 (95% confidence interval 0.73-0.96) for the InterTAK model. The risk of bias was high for all models and the applicability was of low (50%) or unclear (50%) concern. Our review identified multiple diagnostic models to diagnose Takotsubo syndrome. Although most models showed acceptable-to-good discriminative performance, calibration measures were almost unreported and the risk of bias was a concern in most studies.

Identify the underlying true model from other models for clinical practice using model performance measures

ObjectiveTo assess whether the outcome generation true model could be identified from other candidate models for clinical practice with current conventional model performance measures considering various simulation scenarios and a CVD risk prediction as exemplar.Study design and settingThousands of scenarios of true models were used to simulate clinical data, various candidate models and true models were trained on training datasets and then compared on testing datasets with 25 conventional use model performance measures. This consists of univariate simulation (179.2k simulated datasets and over 1.792 million models), multivariate simulation (728k simulated datasets and over 8.736 million models) and a CVD risk prediction case analysis.ResultsTrue models had overall C statistic and 95% range of 0.67 (0.51, 0.96) across all scenarios in univariate simulation, 0.81 (0.54, 0.98) in multivariate simulation, 0.85 (0.82, 0.88) in univariate case analysis and 0.85 (0.82, 0.88) in multivariate case analysis. Measures showed very clear differences between the true model and flip-coin model, little or none differences between the true model and candidate models with extra noises, relatively small differences between the true model and proxy models missing causal predictors.ConclusionThe study found the true model is not always identified as the “outperformed” model by current conventional measures for binary outcome, even though such true model is presented in the clinical data. New statistical approaches or measures should be established to identify the casual true model from proxy models, especially for those in proxy models with extra noises and/or missing causal predictors.

Protocol for the development and validation of machine-learning models for predicting the risk of hypertriglyceridemia in critically ill patients receiving propofol sedation using retrospective data.

Propofol is a widely used sedative-hypnotic agent for critically-ill patients requiring invasive mechanical ventilation (IMV). Despite its clinical benefits, propofol is associated with increased risks of hypertriglyceridemia. Early identification of patients at risk for propofol-associated hypertriglyceridemia is crucial for optimizing sedation strategies and preventing adverse outcomes. Machine learning (ML) models offer a promising approach for predicting individualized patient risks of propofol-associated hypertriglyceridemia. We propose the development of a ML model aimed at predicting the risk of propofol-associated hypertriglyceridemia in ICU patients receiving IMV. The study will utilize retrospective data from four Mayo Clinic sites. Nested cross-validation (CV) will be employed, with a 10-fold inner CV loop for model tuning and selection as well as an outer loop using leave-one-site-out CV for external validation. Feature selection will be conducted using Boruta and LASSO-penalized logistic regression. Data preprocessing steps include missing data imputation, feature scaling, and dimensionality reduction techniques. Six ML algorithms will be tuned and evaluated. Bayesian optimization will be used for hyperparameter selection. Global model explainability will be assessed using permutation importance, and local model explainability will be assessed using SHapley Additive exPlanations (SHAP). The proposed ML model aims to provide a reliable and interpretable tool for clinicians to predict the risk of propofol-associated hypertriglyceridemia in ICU patients. The final model will be deployed in a web-based clinical risk calculator. The model development process and performance measures obtained during nested cross-validation will be described in a study publication to be disseminated in a peer-reviewed journal. The proposed study has received ethics approval from the Mayo Clinic Institutional Review Board (IRB #23-007416). Robust external validation using a nested cross-validation (CV) framework will help assess the generalizability of models produced from the modeling pipeline across different hospital settings.A diverse set of machine learning (ML) algorithms and advanced hyperparameter tuning techniques will be employed to identify the most optimal model configuration.Integration of feature explainability will enhance the clinical applicability of the ML models by providing transparency in predictions, which can improve clinician trust and encourage adoption.Reliance on retrospective data may introduce biases due to inconsistent or erroneous data collection, and the computational intensity of the validation approach may limit replication and future model expansion in resource-constrained settings.

Development and External Validation of a Machine Learning Model to Predict Restriction from Spirometry.

Though European Respiratory Society and American Thoracic Society (ERS/ATS) guidelines for pulmonary function test (PFT) interpretation recommend the use of the forced vital capacity (FVC) lower limit of normal (LLN) to exclude restriction, recent data suggest that the negative predictive value (NPV) of the FVC LLN is lower than has been accepted, particularly among non-Hispanic Black patients. Using a machine learning (ML) model-rather than the FVC LLN-to exclude restriction may improve the accuracy and equity of PFT interpretation. We sought to develop and externally validate a ML model to predict restriction from spirometry and to assess the potential impact of this model on PFT interpretation. We included PFTs with both static and dynamic lung volume measurements for patients between 18 and 80 years of age tested at pulmonary diagnostic labs within two health systems. We used PFTs from one health system to train logistic regression, random forest, and boosted tree models to predict restriction using demographic, anthropometric, and spirometric data. We used PFTs from the second health system to externally validate these models. The primary measure of model performance was the NPV. Model equity was assessed by comparing the NPV among non-Hispanic Black and non-Hispanic White patients. A total of 42 462 PFTs were used for model development and 24 524 for external validation. The prevalence of restriction was 29.8% in the development dataset and 39.6% in the validation dataset. Performance was similar across the three ML models with the best performance seen with the random forest model. The overall NPV of the random forest model (88.3%, 95% confidence interval [CI] 87.8% to 88.9%) was significantly greater than that of the FVC LLN (72.7%, 95% CI 72.1% to 73.3%). The NPV of the random forest model was greater than that of the FVC LLN among both non-Hispanic Black (74.6% [95% CI 72.5% to 76.6%] versus 49.5% [95% CI 47.8% to 51.2%]) and non-Hispanic White (90.9% [95% CI 90.3% to 91.5%] versus 79.6% [95% CI 78.9% to 80.3%]) patients. Using a ML model to exclude restriction from spirometry improves both the accuracy and equity of PFT interpretation.

Prediction models for outcomes in people with low back pain receiving conservative treatment: a systematic review

To identify, critically appraise and evaluate the performance measures of the available prediction models for outcomes in people with low back pain (LBP) receiving conservative treatment. In this systematic review, literature searches were conducted in Embase, Medline, CINAHL from their inception until February/2024. Studies containing follow-up assessment (e.g., prospective cohort studies, registry-based studies) investigating prediction models of outcomes (e.g., pain intensity and disability) for people with LBP receiving conservative treatment were included. Two independent reviewers performed the study selection, the data extraction using the Checklist for critical Appraisal and data extraction for systematic Reviews of prediction Modelling Studies (CHARMS), and risk of bias assessment using the Prediction model Risk of Bias Assessment (PROBAST). Findings of individual studies were reported narratively taking into account the discrimination and calibration measures of the prediction models. Seventy-five studies developing or investigating the validity of 216 models were included in this review. Most prediction models investigated people receiving physiotherapy treatment and most models included socio-demographic variables, clinical features, and self-reported measures as predictors. The discriminatory capacity of the internal validity of the 27 prediction models for pain intensity varied greatly showing a c-statistic ranging from 0.48 to 0.94. Similarly, the discriminatory capacity for 31 models for disability had the same pattern showing a c-statistic ranging from 0.48 to 0.86. The calibration measures of the internal validity of the prediction models predicting pain intensity and disability showed to be adequate. Only one of three studies testing the external validity of models to predict pain intensity and disability and reported both discrimination and calibration measures, which showed to be inadequate. The prediction models predicting the secondary outcomes (e.g., self-reported recovery, quality of life, return to work) showed varied performance measures for internal validity, and only two studies tested the external validity of models although they did not provide performance the performance measures. Several prediction models have been developed for people with LBP receiving conservative treatment, however most show inadequate discriminatory validity. A few studies externally validated the prediction models and future studies should focus on testing this before implementing in clinical practice.

Clinical analogy resolution performance for foundation language models

Using extensive data sources to create foundation language models has revolutionized the performance of deep learning-based architectures. This remarkable improvement has led to state-of-the-art results for various downstream NLP tasks, including clinical tasks. However, more research is needed to measure model performance intrinsically, especially in the clinical domain. We revisit the use of analogy questions as an effective method to measure the intrinsic performance of language models for the clinical domain in English. We tested multiple Transformers-based language models over analogy questions constructed from the Unified Medical Language System (UMLS), a massive knowledge graph of clinical concepts. Our results show that large language models are significantly more performant for analogy resolution than small language models. Similarly, domain-specific language models perform better than general domain language models. We also found a correlation between intrinsic and extrinsic performance, validated through PubMedQA extrinsic task. Creating clinical-specific and language-specific language models is essential for advancing biomedical and clinical NLP and will ensure a valid application in clinical practice. Finally, given that our proposed intrinsic test is based on a term graph available in multiple languages, the dataset can be built to measure the performance of models in languages other than English.

Melt pool super solution reconstruction based on dual path deep learning for laser powder bed fusion monitoring

Melt pool monitoring in laser powder bed fusion (L-PBF) is an important foundation for process control and melting dynamics research. However, due to hardware limitations and the intense metallurgical phenomena during the melting process, the quality of the melt pool monitoring images cannot be guaranteed. This paper proposes a deep learning-based melt pool image super-resolution (SR) reconstruction method based on the particularity of melt pool images. It is an innovative dual-path structure. The first branch utilizes residual-in-residual structures and the efficient channel attention-Net attention mechanism to achieve the adaptive feature extraction of high-frequency information, thereby capturing effective melt pool boundary information. The second branch uses the U-Net structure to address the low utilization of overall characteristics of the melt pool in the chain-based SR network. In addition to the universal peak signal-to-noise ratio and structural similarity index measure metrics, the reconstruction accuracy of melt pool contour features is used to measure model performance, which intuitively reflects the significance of this work for melt pool monitoring. The results demonstrate that the proposed SR reconstruction method enhances the resolution and clarity of melt pool images. Furthermore, SR reconstruction of the melt pool effectively reduces errors in extracting melt pool features. This method provides a network paradigm for high-precision L-PBF monitoring. It integrates important boundary information and overall morphology features of the melt pool through a dual path structure, thereby achieving reliable SR reconstruction. This research will contribute to low-cost in situ monitoring of L-PBF and subsequent process control.

Six-year performance of risk-based selection for lung cancer screening in the Manchester Lung Health Check cohort

ObjectiveRisk prediction models are used to determine eligibility for targeted lung cancer screening. However, prospective data regarding model performance in this setting are limited. Here we report the performance of...

Advanced Data Analytics and Machine Learning Driven Fraud Detection and Data Loss Prevention for Automated Incident Response in the US Healthcare Corporations

Fraud detection and data loss prevention are critical challenges facing US healthcare corporations as they strive to safeguard sensitive patient information and comply with stringent data protection regulations. The integration of advanced data analytics and machine learning has emerged as a powerful approach to enhance the efficiency and accuracy of detecting fraudulent activities and preventing data breaches. This study explores the application of machine learning-driven solutions in automating incident response for healthcare data security. The research begins by examining the current landscape of data analytics and machine learning in fraud detection, emphasizing the limitations of traditional methods. Through an extensive literature review and analysis of case studies within the US healthcare industry, the paper identifies key areas where advanced technologies can bridge existing gaps. The methods section outlines the data collection process, the machine learning algorithms implemented, and the evaluation metrics used to measure model performance. Results demonstrate the enhanced detection accuracy and prompt response capabilities of machine learning models compared to conventional techniques. The discussion delves into the implications of these findings, showcasing the transformative potential of automated incident response systems in reducing response times and mitigating data loss risks. Although promising, the study acknowledges limitations in data variability and model generalizability, suggesting avenues for further research. The paper concludes with strategic recommendations for adopting machine learning solutions in healthcare security protocols. By highlighting best practices and policy recommendations, this research aims to provide a roadmap for healthcare corporations seeking to strengthen their data protection frameworks. The insights presented underscore the pivotal role of advanced data analytics and machine learning in fortifying healthcare data security against evolving cyber threats.

Empowering Data Sharing in Neuroscience: A Deep Learning De-identification Method for Pediatric Brain MRIs.

Privacy concerns, such as identifiable facial features within brain scans, have hindered the availability of pediatric neuroimaging datasets for research. Consequently, pediatric neuroscience research lags adult counterparts, particularly in rare disease and under-represented populations. The removal of face regions (image defacing) can mitigate this, however existing defacing tools often fail with pediatric cases and diverse image types, leaving a critical gap in data accessibility. Given recent NIH data sharing mandates, novel solutions are a critical need. To develop an AI-powered tool for automatic defacing of pediatric brain MRIs, deep learning methodologies (nnU-Net) were employed using a large, diverse multi-institutional dataset of clinical radiology images. This included multi-parametric MRIs (T1w, T1w-contrast enhanced, T2w, T2w-FLAIR) with 976 total images from 208 brain tumor patients (Children's Brain Tumor Network, CBTN) and 36 clinical control patients (Scans with Limited Imaging Pathology, SLIP) ranging in age from 7 days to 21 years old. Face and ear removal accuracy for withheld testing data was the primary measure of model performance. Potential influences of defacing on downstream research usage were evaluated with standard image processing and AI-based pipelines. Group-level statistical trends were compared between original (non-defaced) and defaced images. Across image types, the model had high accuracy for removing face regions (mean accuracy, 98%; N=98 subjects/392 images), with lower performance for removal of ears (73%). Analysis of global and regional brain measures (SLIP cohort) showed minimal differences between original and defaced outputs (mean r S=0.93, all p < 0.0001). AI-generated whole brain and tumor volumes (CBTN cohort) and temporalis muscle metrics (volume, cross-sectional area, centile scores; SLIP cohort) were not significantly affected by image defacing (all r S>0.9, p<0.0001). The defacing model demonstrates efficacy in removing facial regions across multiple MRI types and exhibits minimal impact on downstream research usage. A software package with the trained model is freely provided for wider use and further development (pediatric-auto-defacer; https://github.com/d3b-center/pediatric-auto-defacer-public). By offering a solution tailored to pediatric cases and multiple MRI sequences, this defacing tool will expedite research efforts and promote broader adoption of data sharing practices within the neuroscience community. AI = artificial intelligence; CBTN = Children's Brain Tumor Network; CSA = cross-sectional area; SLIP = Scans with Limited Imaging Pathology; TMT = temporalis muscle thickness; NIH = National Institute of Health; LH = left hemisphere; RH = right hemisphere.

Experimental Investigation of Indirect Tensile Strength of Hot Mix Asphalt with Varying Hydrated Lime Content at Low Temperatures and Prediction with Soft-Computing Models

If asphalt pavements are exposed to cold weather conditions and high humidity for long periods of time, cracking of the pavement is an inevitable consequence. In such cases, it would be a good decision to focus on the filler material, which plays an important role in the performance variation in the hot asphalt mixtures used in the pavement. Although the use of hydrated lime as a filler material in hot asphalt mixtures is a common method frequently recommended to eliminate the adverse effects of low temperature and to keep moisture sensitivity under control in asphalt pavements, the sensitivity of the quantities of the material cannot be ignored. Therefore, in this study, an amount of filler in the mixture was replaced with hydrated lime (HL) filler additive at different rates of 0%, 1%, 2%, 3% and 4%. These asphalt briquettes, designed according to the Marshall method, have optimum asphalt contents for samples with specified HL content. In this study, where the temperature effect was examined at five different levels of −10 °C, −5 °C, 0 °C, 5 °C and 25 °C, the samples were produced in two different groups, conditioned and unconditioned, in order to examine the effect of water. The indirect tensile strength (ITS) test was applied on the produced samples. Experimental study showed that HL additive strengthened the material at low temperatures and made it more resistant to cold weather conditions and humidity. In the second part of the study, two different prediction models with varying configurations were introduced using nonlinear regression and feed-forward neural networks (FFNNs) and the best prediction performance among these was investigated. Examination of the performance measures of the prediction models indicated that ITS can be accurately predicted using both methods. As a result of comparing the developed models with the experimental data, the model provides significant contributions to the evaluation of the relationship between the ITS values obtained with the specified conditioning, temperature changes and HL contents.

Data-Driven Approach to Development of a Risk Score for Periprosthetic Joint Infections in Total Joint Arthroplasty Using Electronic Health Records

BackgroundPeriprosthetic joint infection (PJI) is an uncommon, but serious complication in total joint arthroplasty. Personalized risk prediction and risk factor management may allow better preoperative assessment and improved outcomes. We evaluated different data-driven approaches to develop surgery-specific PJI prediction models using large-scale data from the electronic health records (EHRs). MethodsA large institutional arthroplasty registry was leveraged to collect data from 58,574 procedures of 41,844 patients who underwent at least one primary and/or revision hip and/or knee arthroplasty between 2000 and 2019. The registry dataset was augmented with additional clinical, procedural, and laboratory data from the EHRs for more than 100 potential predictor variables. The main outcome was PJI within the first year after surgery. We implemented both traditional and machine learning methods for model development (lasso regression, relaxed lasso regression, ridge regression, random forest, stepwise regression, extreme gradient boosting, neural network) and used 10-fold cross-validation to calculate measures of model performance in terms of discrimination (c-statistic) and calibration. ResultsAll models discriminated similarly in predicting PJI risk, with negligible differences of less than 0.08 between the best- and worst-performing models. The relaxed and fully relaxed lasso models using the Cox model structure outperformed the other models with concordances of 0.787 in primary hip arthroplasty and 0.722 in revision hip arthroplasty, with the number of predictors ranging from nine to 41. The concordances with the relaxed lasso models were 0.681 in primary and 0.699 in revision knee arthroplasty, with a higher number of predictors in the models. Predictors included in the models varied substantially across the four surgical groups. ConclusionsThe incorporation of additional data from the EHRs offers limited improvement in PJI risk stratification. Furthermore, improvement in PJI risk prediction was modest with the machine learning approaches and may not justify the added complexity.

Electroencephalogram Emotion Recognition via AUC Maximization

Imbalanced datasets pose significant challenges in areas including neuroscience, cognitive science, and medical diagnostics, where accurately detecting minority classes is essential for robust model performance. This study addressed the issue of class imbalance, using the ‘liking’ label in the DEAP dataset as an example. Such imbalances were often overlooked by prior research, which typically focused on the more balanced arousal and valence labels and predominantly used accuracy metrics to measure model performance. To tackle this issue, we adopted numerical optimization techniques aimed at maximizing the area under the curve (AUC), thus enhancing the detection of underrepresented classes. Our approach, which began with a linear classifier, was compared against traditional linear classifiers, including logistic regression and support vector machines (SVMs). Our method significantly outperformed these models, increasing recall from 41.6% to 79.7% and improving the F1-score from 0.506 to 0.632. These results underscore the effectiveness of AUC maximization methods in neuroscience research by offering a robust solution for managing imbalanced datasets, developing more precise diagnostic tools and interventions for detecting critical minority classes in real-world scenarios.

Machine Learning-Based Detection of Endometriosis: A Retrospective Study in A Population of Iranian Female Patients.

Endometriosis, is a prevalent condition among women of childbearing age, characterized by the presence of ectopic endometrial glands. It is associated with pelvic pain and infertility. Unfortunately, the diagnosis of endometriosis is often delayed in many patients. While laparoscopic investigation is required for a definitive diagnosis, physical examination combined with ultrasonography can provide reasonably accurate detection. Machine learning (ML) techniques have shown promise tools in medical imaging and diagnostics. However, there is a lack of sufficient ML studies focusing on Iranian endometriosis female patients. In this study, we aimed to compare the diagnostic accuracy of different ML algorithms for endometriosis detection. In this retrospective study, our objective was to assess the diagnostic accuracy of different ML algorithms in classifying suspicious cases of endometriosis using ultrasonographic signs. Our data set consisted of 505 patients, among which 149 were confirmed cases of endometriosis. We divided the data set into training and test sets to train and evaluate the performance of the ML models. To ensure robust evaluation, we employed stratified 5-fold cross-validation and calculated the area under the receiver operating characteristic curve (AUC) as a measure of model performance. In the test set, a total of 37 out of 127 patients (29.1%) were diagnosed with endometriosis, while in the training set, 112 out of 378 patients (29.6%) were confirmed to have the condition. Sensitivities ranged from 59.5 to 75.7%, and specificities ranged from 71.7 to 83.3%. Notably, the SVM, Random Forest, Extra-Trees, and Gradient Boosting models exhibited the highest performance, with AUCs of 0.76. Our study supports the use of ML models for the screening and diagnosis of endometriosis. The superior performance of the SVM, Random Forest, Extra-Trees, and Gradient Boosting models, as indicated by their high AUCs, suggests their potential as valuable tools in improving the accuracy of endometriosis detection.

Quantifying Climate Change Variability for the Better Management of Water Resources: The Case of Kobo Valley, Danakil Basin, Ethiopia

Alterations in the hydrological cycle due to climate change are one of the key threats to the future accessibility of natural resources. This study used 12 GCM climate models from CMIP6 to evaluate future climate change scenarios by applying model performance measures and trend analysis in Kobo Valley, Ethiopia. The models were ranked based on their ability to analyze the historical datasets. The result of this study showed that the outputs of the FIO-ESM-2-0 CIMP6 model had a good overall ranking for both precipitation and temperature. After bias correction of the model-based projections with the observed data, the average annual precipitation in the average scenario (SSP2-4.5) decreased by 4.4% and 13% in 2054 and 2084, respectively. Similarly, in the worst-case scenario (SSP5-8.5), by the end of 2054 and 2084, decreases of 4% and 12.8%, respectively, were predicted. The average annual maximum temperature under the SSP2-4.5 scenario increased by 1.5 °C in 2054 and by 2.1 °C in 2084. The average annual maximum temperature under the worst-case (SSP5-8.5) scenario increased by 1.7 °C in 2054 and by 3.2 °C in 2084. In the middle scenario (SSP4.5), the average annual minimum temperature increased by 2.2 °C in 2054 and by 3 °C in 2084. The average annual minimum temperature under the worst-case (SSP5-8.5) scenario increased by 2.6 °C in 2054 and by 4.3 °C in 2084. The seasonal variability in precipitation in the studied valley will decrease in the winter and increase in the summer. A decrease in precipitation combined with an increase in temperature will strengthen the risk of drought events in the future.

Multicentre Validation of the RESECT-90 Prediction Model for 90-Day Mortality After Lung Resection

BackgroundThe RESECT-90 model was developed to predict 90-day mortality for patients undergoing lung resection but hasn't been externally validated. The aim of this study was to validate the RESECT-90 clinical prediction model using multicentre patient data from across the United Kingdom (UK). Materials and MethodsData from 12 UK thoracic surgery centers for patients undergoing lung resection between 2016 and 2020 with available 90-day mortality status were used to externally validate the RESECT-90 model. Measures of discrimination (area under the receiving operator characteristic curve [AUC]) and calibration (calibration slope, calibration intercept and flexible calibration plot) were assessed as measures of model performance. Model recalibration was also performed by updating the original model intercept and coefficients. ResultsA total of 12,241 patients were included. Overall 90-day mortality was 2.9% (n = 360). Acceptable model discrimination was demonstrated (AUC 0.74 [0.73, 0.75]). Calibration varied between centers with some evidence of overall model miscalibration (calibration slope 0.80 [0.66, 0.95] and calibration intercept 0.40 [0.29, 0.52]) despite acceptable appearances of the flexible calibration plot. The model was subsequently recalibrated, after which all measures of calibration indicated excellent performance. ConclusionsAfter external validation and recalibration using a large contemporary cohort of patients undergoing surgery in multiple geographical locations across the UK, the RESECT-90 model demonstrated satisfactory statistical performance for the prediction of 90-day mortality after lung resection. Whilst the recalibrated model will require ongoing validation, the results of this study suggest that routine use of the RESECT-90 model in UK thoracic surgery practice should be considered.

The performance of prognostic models depended on the choice of missing value imputation algorithm: a simulation study

ObjectivesThe development of clinical prediction models is often impeded by the occurrence of missing values in the predictors. Various methods for imputing missing values before modeling have been proposed. Some of them are based on variants of multiple imputations by chained equations, while others are based on single imputation. These methods may include elements of flexible modeling or machine learning algorithms, and for some of them user-friendly software packages are available. The aim of this study was to investigate by simulation if some of these methods consistently outperform others in performance measures of clinical prediction models. Study Design and SettingWe simulated development and validation cohorts by mimicking observed distributions of predictors and outcome variable of a real data set. In the development cohorts, missing predictor values were created in 36 scenarios defined by the missingness mechanism and proportion of noncomplete cases. We applied three imputation algorithms that were available in R software (R Foundation for Statistical Computing, Vienna, Austria): mice, aregImpute, and missForest. These algorithms differed in their use of linear or flexible models, or random forests, the way of sampling from the predictive posterior distribution, and the generation of a single or multiple imputed data set. For multiple imputation, we also investigated the impact of the number of imputations. Logistic regression models were fitted with the simulated development cohorts before (full data analysis) and after missing value generation (complete case analysis), and with the imputed data. Prognostic model performance was measured by the scaled Brier score, c-statistic, calibration intercept and slope, and by the mean absolute prediction error evaluated in validation cohorts without missing values. Performance of full data analysis was considered as ideal. ResultsNone of the imputation methods achieved the model's predictive accuracy that would be obtained in case of no missingness. In general, complete case analysis yielded the worst performance, and deviation from ideal performance increased with increasing percentage of missingness and decreasing sample size. Across all scenarios and performance measures, aregImpute and mice, both with 100 imputations, resulted in highest predictive accuracy. Surprisingly, aregImpute outperformed full data analysis in achieving calibration slopes very close to one across all scenarios and outcome models. The increase of mice's performance with 100 compared to five imputations was only marginal. The differences between the imputation methods decreased with increasing sample sizes and decreasing proportion of noncomplete cases. ConclusionIn our simulation study, model calibration was more affected by the choice of the imputation method than model discrimination. While differences in model performance after using imputation methods were generally small, multiple imputation methods as mice and aregImpute that can handle linear or nonlinear associations between predictors and outcome are an attractive and reliable choice in most situations.

Accurately Predicting Barrier Heights for Radical Reactions in Solution Using Deep Graph Networks.

Quantitative estimates of reaction barriers and solvent effects are essential for developing kinetic mechanisms and predicting reaction outcomes. Here, we create a new data set of 5,600 unique elementary radical reactions calculated using the M06-2X/def2-QZVP//B3LYP-D3(BJ)/def2-TZVP level of theory. A conformer search is done for each species using TPSS/def2-TZVP. Gibbs free energies of activation and of reaction for these radical reactions in 40 common solvents are obtained using COSMO-RS for solvation effects. These balanced reactions involve the elements H, C, N, O, and S, contain up to 19 heavy atoms, and have atom-mapped SMILES. All transition states are verified by an intrinsic reaction coordinate calculation. We next train a deep graph network to directly estimate the Gibbs free energy of activation and of reaction in both gas and solution phases using only the atom-mapped SMILES of the reactant and product and the SMILES of the solvent. This simple input representation avoids computationally expensive optimizations for the reactant, transition state, and product structures during inference, making our model well-suited for high-throughput predictive chemistry and quickly providing information for (retro-)synthesis planning tools. To properly measure model performance, we report results on both interpolative and extrapolative data splits and also compare to several baseline models. During training and testing, the data set is augmented by including the reverse direction of each reaction and variants with different resonance structures. After data augmentation, we have around 2 million entries to train the model, which achieves a testing set mean absolute error of 1.16 kcal mol-1 for the Gibbs free energy of activation in solution. We anticipate this model will accelerate predictions for high-throughput screening to quickly identify relevant reactions in solution, and our data set will serve as a benchmark for future studies.

Ensemble machine learning framework for predicting maternal health risk during pregnancy

Maternal health risks can cause a range of complications for women during pregnancy. High blood pressure, abnormal glucose levels, depression, anxiety, and other maternal health conditions can all lead to pregnancy complications. Proper identification and monitoring of risk factors can assist to reduce pregnancy complications. The primary goal of this research is to use real-world datasets to identify and predict Maternal Health Risk (MHR) factors. As a result, we developed and implemented the Quad-Ensemble Machine Learning framework to predict Maternal Health Risk Classification (QEML-MHRC). The methodology used a vacxsriety of Machine Learning (ML) models, which then integrated with four ensemble ML techniques to improve prediction. The dataset collected from various maternity hospitals and clinics subjected to nineteen training and testing tests. According to the exploratory data analysis, the most significant risk factors for pregnant women include high blood pressure, low blood pressure, and high blood sugar levels. The study proposed a novel approach to dealing with high-risk factors linked to maternal health. Dealing with class-specific performance elaborated further to properly understand the distinction between high, low, and medium risks. All tests yielded outstanding results when predicting the amount of risk during pregnancy. In terms of class performance, the dataset associated with the “HR” class outperformed the others, predicting 90% correctly. GBT with ensemble stacking outperformed and demonstrated remarkable performance for all evaluation measure (0.86) across all classes in the dataset. The key success of the models used in this work is the ability to measure model performance using a class-wise distribution. The proposed approach can help medical experts assess maternal health risks, saving lives and preventing complications throughout pregnancy. The prediction approach presented in this study can detect high-risk pregnancies early on, allowing for timely intervention and treatment. This study’s development and findings have the potential to raise public awareness of maternal health issues.

An Intelligent Fishery Detection Method Based on Cross-Domain Image Feature Fusion

Target detection technology plays a crucial role in fishery ecological monitoring, fishery diversity research, and intelligent aquaculture. Deep learning, with its distinct advantages, provides significant convenience to the fishery industry. However, it still faces various challenges in practical applications, such as significant differences in image species and image blurring. To address these issues, this study proposes a multi-scale, multi-level, and multi-stage cross-domain feature fusion model. In order to train the model more effectively, a new data set called Fish52 (multi-scene fish data set, a data set containing 52 fish species) was constructed, on which the model achieved an mAP (mean average precision is a key measure of model performance) of 82.57%. Furthermore, we compared prevalent one-stage and two-stage detection methods on the Lahatan (single-scene fish data set) and Fish30 data set (a data set containing 30 fish species) and tested them on the F4k (Fish4Knowledge (F4K) is a data set focused on fish detection and identification) and FishNet data set (it is a data set containing 94,532 images from 17,357 aquatic species). The mAP of our proposed model on the Fish30, Lahatan, F4k, and FishNet data sets reaches 91.72%, 98.7%, 88.6%, and 81.5%, respectively, outperforming existing mainstream models. Comprehensive empirical analysis indicates that our model possesses a high generalization ability and reaches advanced performance levels. In this study, the depth of the model backbone is deepened, a novel neck structure is proposed, and a new module is embedded therein. To enhance the fusion ability of the model, a new attention mechanism module is introduced. In addition, in the adaptive decoupling detection head module, introducing classes with independent parameters and regression adapters reduces interaction between different tasks. The proposed model can better monitor fishery resources and enhance aquaculture efficiency. It not only provides an effective approach for fish detection but also has certain reference significance for the identification of similar targets in other environments and offers assistance for the construction of smart fisheries and digital fisheries.