Estimates Of Model Performance Research Articles

Some combinatorics of data leakage induced by clusters

AbstractData leakage is a common issue that can lead to misleading generalisation error estimation and incorrect hyperparameter tuning. However, its mechanisms are not always well understood. In this work, we consider the case of clustered data and investigate the distribution of the number of elements in leakage when the data set is uniformly split. For both the validation and test sets, the first and second moments of the number of elements in leakage are derived analytically. Modelling consequences are investigated and exemplified on simulated data. In addition, the case of an actual agronomic feasibility study is presented. We demonstrate how data leakage can distort model performance estimation when an inadequate data splitting strategy is used. We provide an understanding of data leakage in the context of clustered data by quantifying its role in predictive modelling. This sheds light on related challenges that may impact the practice in agronomy and beyond.

Revealing the underlying patterns: Investigating dataset similarity, performance, and generalization

Supervised deep learning models require significant amount of labeled data to achieve an acceptable performance on a specific task. However, when tested on unseen data, the models may not perform well. Therefore, the models need to be trained with additional and varying labeled data to improve the generalization. In this work, our goal is to understand the models, their performance and generalization. We establish image-image, dataset-dataset, and image-dataset distances to gain insights into the model’s behavior. Our proposed distance metric when combined with model performance can help in selecting an appropriate model/architecture from a pool of candidate architectures. We have shown that the generalization of these models can be improved by only adding a small number of unseen images (say 1, 3 or 7) into the training set. Our proposed approach reduces training and annotation costs while providing an estimate of model performance on unseen data in dynamic environments.

ProteoMutaMetrics: machine learning approaches for solute carrier family 6 mutation pathogenicity prediction.

The solute carrier transporter family 6 (SLC6) is of key interest for their critical role in the transport of small amino acids or amino acid-like molecules. Their dysfunction is strongly associated with human diseases such as including schizophrenia, depression, and Parkinson's disease. Linking single point mutations to disease may support insights into the structure-function relationship of these transporters. This work aimed to develop a computational model for predicting the potential pathogenic effect of single point mutations in the SLC6 family. Missense mutation data was retrieved from UniProt, LitVar, and ClinVar, covering multiple protein-coding transcripts. As encoding approach, amino acid descriptors were used to calculate the average sequence properties for both original and mutated sequences. In addition to the full-sequence calculation, the sequences were cut into twelve domains. The domains are defined according to the transmembrane domains of the SLC6 transporters to analyse the regions' contributions to the pathogenicity prediction. Subsequently, several classification models, namely Support Vector Machine (SVM), Logistic Regression (LR), Random Forest (RF), and Extreme Gradient Boosting (XGBoost) with the hyperparameters optimized through grid search were built. For estimation of model performance, repeated stratified k-fold cross-validation was used. The accuracy values of the generated models are in the range of 0.72 to 0.80. Analysis of feature importance indicates that mutations in distinct regions of SLC6 transporters are associated with an increased risk for pathogenicity. When applying the model on an independent validation set, the performance in accuracy dropped to averagely 0.6 with high precision but low sensitivity scores.

Discovery Precision: An effective metric for evaluating performance of machine learning model for explorative materials discovery

The evaluation of machine learning (ML) models in identifying novel materials with superior Figure of Merit (FOM) compared to known materials is of utmost importance for exploring their potential applications. A well-designed evaluation should distinguish between good and poor models, avoid bias in the validation and testing set, and offer straightforward interpretable. To fulfill these requirements, a novel metric called Discovery Precision (DP) was proposed to assess the efficiency of ML models for material discovery in terms of probability. To validate the applicability of DP, we have established a comprehensive testing framework for comparing various validation methods and metrics. Through this framework, the DP metric demonstrates excellent performance in material discovery. Additionally, the simulation results indicate that DP effectively assists in model selection and provides an unbiased estimation of model performance.

Processing of Polymers Stress Relaxation Curves Using Machine Learning Methods

Currently, one of the topical areas of application of machine learning methods is the prediction of material characteristics. The aim of this work is to develop machine learning models for determining the rheological properties of polymers from experimental stress relaxation curves. The paper presents an overview of the main directions of metaheuristic approaches (local search, evolutionary algorithms) to solving combinatorial optimization problems. Metaheuristic algorithms for solving some important combinatorial optimization problems are described, with special emphasis on the construction of decision trees. A comparative analysis of algorithms for solving the regression problem in CatBoost Regressor has been carried out. The object of the study is the generated data sets obtained on the basis of theoretical stress relaxation curves. Tables of initial data for training models for all samples are presented, a statistical analysis of the characteristics of the initial data sets is carried out. The total number of numerical experiments for all samples was 346020 variations. When developing the models, CatBoost artificial intelligence methods were used, regularization methods (Weight Decay, Decoupled Weight Decay Regularization, Augmentation) were used to improve the accuracy of the model, and the Z-Score method was used to normalize the data. As a result of the study, intelligent models were developed to determine the rheological parameters of polymers included in the generalized non-linear Maxwell-Gurevich equation (initial relaxation viscosity, velocity modulus) using generated data sets for the EDT-10 epoxy binder as an example. Based on the results of testing the models, the quality of the models was assessed, graphs of forecasts for trainees and test samples, graphs of forecast errors were plotted. Intelligent models are based on the CatBoost algorithm and implemented in the Jupyter Notebook environment in Python. The constructed models have passed the quality assessment according to the following metrics: MAE, MSE, RMSE, MAPE. The maximum value of model error predictions was 0.86 for the MAPE metric, and the minimum value of model error predictions was 0.001 for the MSE metric. Model performance estimates obtained during testing are valid.

NIMG-26. DIFFERENTIATION OF IDH-WILDTYPE GLIOBLASTOMA AND TUMEFACTIVE MULTIPLE SCLEROSIS USING PRE-OPERATIVE BRAIN MRI AND DEEP LEARNING: RIGOROUS AND UNBIASED EXPERIMENTAL DESIGN AND ANALYTICAL APPROACH

Abstract Clinical management of IDH wildtype glioblastoma (GBM) and tumefactive multiple sclerosis (tMS) is drastically different. GBM requires maximal safe resection followed by chemoradiation, while tMS outcome is worsened by surgery and radiotherapy. Misdiagnosis of brain lesions may expose patients to unnecessary anxiety, surgery, or radiotherapy. Noninvasive methods to help with accurate diagnosis of tumor and non-tumor etiologies are needed. tMS subjects were identified by the Mayo Clinic Center for Multiple Sclerosis and Autoimmune Neurology (CMSAN). GBM subjects were matched to tMS by age at diagnosis, index MRI date, and sex. Inclusion criteria included one cm minimal lesion size and pre-operative post-contrast T1 and T2 images available for analysis. To identify potential experimental design bias, MRI technology and clinical characteristics were compared across groups. A 3D-DenseNet121 was used to develop a classification model using prespecified parameters: 600 epochs, batch size 16, learning rate 10-3, cross entropy loss, and AdamW optimizer. The stopping rule was defined as three sequential epoch cross entropy loss < 0.02. Five-fold cross validation (CV) was utilized to obtain an unbiased estimate of predictive performance (overfitting) and sensitivity of the 3D-DenseNet121 algorithm was evaluated. 224 subjects (114 GBM, 110 tMS) were analyzed: 40% male, median age 46. After one round of CV, the average unbiased estimate of sensitivity and specificity for predicting GBM from the final classification model was 0.86 and 0.73, respectively. To assess the reproducibility of the deep learning algorithm (3D-DenseNet121), the exact same CV folds were rerun resulting in average sensitivity and specificity of 0.82 and 0.75, respectively. Obtaining unbiased estimates of model performance using internal validation is critical before evaluating a final model on an external dataset. Understanding sensitivity of a deep learning algorithm is important for model validity. Differentiation of GBM and tMS appears reasonable with MRI; external validation is in progress.

316 Awardee Talk: Development and Assessment of Predictive Models for Improved Swine Farming

Abstract Prediction of outcomes is critical in swine breeding and management. This necessitates the development of predictive models that address challenges in swine farming. For predictive modeling, there have been significant advances in machine learning. Nevertheless, there are needs to adapt predictive models for specific swine farming problems including genomic prediction and behavior analysis. Furthermore, there is not yet a clear guideline on how to validate novel models in these fields. The overarching goal of this work was to validate a collection of predictive models for improved swine farming with applications to precision management, phenotyping, and breeding. The first study addressed the pig genomic prediction problem. Differential evolution algorithm was utilized to optimize hyperparameters of deep learning (DL) models, which affected the predictive performance of DL. Performance of optimized DL was compared with “best practice” DL architectures selected from literature and baseline DL models with randomly specified hyperparameters. Optimized models showed clear improvement. Despite the success of genomic prediction, phenotyping has become a bottleneck in breeding programs as it is still time-consuming and labor-intensive. The second study aimed at collecting a video dataset to study agonistic behavior of pigs and adapting a state-of-the-art DL pipeline to classify agonistic behaviors of pigs through video analysis. The pipeline was validated through various training-validation data partitions, where the training data were used for model development and the validation data were used for model evaluation. Results showed that splitting the training and validation sets at random led to over-optimistic estimates of model performance. Social behavior continues to be an important topic in animal behavior studies. However, less attention was paid to the modeling of animal social behavior. The last study focused on developing and validating a statistical model for the analysis of social interactions of pigs. Generalized linear mixed models were fitted, and a Bayesian framework was used for parameter estimation and posterior predictive model checking. The predictive performance of the models varied depending on the validation strategy, where three strategies were defined: random cross-validation, block-by-social-group cross-validation, and block-by-focal-animals validation. Collectively, the research items presented how state-of-the-art predictive models can be adapted for and validated in swine farming applications. The findings are also informative to livestock farming in general, as genomic prediction and animal social behavior (of group-housed animals) are popular topics in livestock species.

A machine learning approach for accelerated design of magnesium alloys. Part B: Regression and property prediction

Machine learning (ML) models provide great opportunities to accelerate novel material development, offering a virtual alternative to laborious and resource-intensive empirical methods. In this work, the second of a two-part study, an ML approach is presented that offers accelerated digital design of Mg alloys. A systematic evaluation of four ML regression algorithms was explored to rationalise the complex relationships in Mg-alloy data and to capture the composition-processing-property patterns. Cross-validation and hold-out set validation techniques were utilised for unbiased estimation of model performance. Using atomic and thermodynamic properties of the alloys, feature augmentation was examined to define the most descriptive representation spaces for the alloy data. Additionally, a graphical user interface (GUI) webtool was developed to facilitate the use of the proposed models in predicting the mechanical properties of new Mg alloys. The results demonstrate that random forest regression model and neural network are robust models for predicting the ultimate tensile strength and ductility of Mg alloys, with accuracies of ∼80% and 70% respectively. The developed models in this work are a step towards high-throughput screening of novel candidates for target mechanical properties and provide ML-guided alloy design.

A two-stage evolutionary algorithm for noisy bi-objective optimization

For noisy bi-objective optimization problems, the algorithm is affected by noise differently in different optimization stages. Generally speaking, only when the impact of the noise is non-negligible, the noise may lead to errors in the quality evaluation of the solutions, and then affect the performance of the algorithm. Therefore, we propose an adaptive switch strategy to make the algorithm switch adaptively among different noise treatments depending on the noise impact to solve noisy bi-objective optimization problems. In addition, a data selection strategy and a model performance estimation method are proposed to enhance the modeling-based denoising method. Furthermore, an output solution selection is used to select more reliable non-dominated solutions as the final output at the late stage of optimization. The performance of the proposed algorithm is verified on the DTLZ and WFG problems. From the experimental results, it can be found that the proposed algorithm is very competitive with other existing algorithms designed for noisy bi-objective optimization problems.

Sample Size Analysis for Machine Learning Clinical Validation Studies.

Before integrating new machine learning (ML) into clinical practice, algorithms must undergo validation. Validation studies require sample size estimates. Unlike hypothesis testing studies seeking a p-value, the goal of validating predictive models is obtaining estimates of model performance. There is no standard tool for determining sample size estimates for clinical validation studies for machine learning models. Our open-source method, Sample Size Analysis for Machine Learning (SSAML) was described and was tested in three previously published models: brain age to predict mortality (Cox Proportional Hazard), COVID hospitalization risk prediction (ordinal regression), and seizure risk forecasting (deep learning). Minimum sample sizes were obtained in each dataset using standardized criteria. SSAML provides a formal expectation of precision and accuracy at a desired confidence level. SSAML is open-source and agnostic to data type and ML model. It can be used for clinical validation studies of ML models.

The impact of inconsistent human annotations on AI driven clinical decision making

In supervised learning model development, domain experts are often used to provide the class labels (annotations). Annotation inconsistencies commonly occur when even highly experienced clinical experts annotate the same phenomenon (e.g., medical image, diagnostics, or prognostic status), due to inherent expert bias, judgments, and slips, among other factors. While their existence is relatively well-known, the implications of such inconsistencies are largely understudied in real-world settings, when supervised learning is applied on such ‘noisy’ labelled data. To shed light on these issues, we conducted extensive experiments and analyses on three real-world Intensive Care Unit (ICU) datasets. Specifically, individual models were built from a common dataset, annotated independently by 11 Glasgow Queen Elizabeth University Hospital ICU consultants, and model performance estimates were compared through internal validation (Fleiss’ κ = 0.383 i.e., fair agreement). Further, broad external validation (on both static and time series datasets) of these 11 classifiers was carried out on a HiRID external dataset, where the models’ classifications were found to have low pairwise agreements (average Cohen’s κ = 0.255 i.e., minimal agreement). Moreover, they tend to disagree more on making discharge decisions (Fleiss’ κ = 0.174) than predicting mortality (Fleiss’ κ = 0.267). Given these inconsistencies, further analyses were conducted to evaluate the current best practices in obtaining gold-standard models and determining consensus. The results suggest that: (a) there may not always be a “super expert” in acute clinical settings (using internal and external validation model performances as a proxy); and (b) standard consensus seeking (such as majority vote) consistently leads to suboptimal models. Further analysis, however, suggests that assessing annotation learnability and using only ‘learnable’ annotated datasets for determining consensus achieves optimal models in most cases.

Analysis of longitudinal data using constrained repeated random sampling-cross validation (CORRS-CV) and partial least squares

Longitudinal data constitutes a very important source of information to study systems or individuals over time. Analysis of this type of data is often performed using multivariate models to assess the association between high-dimensionality data and an independent variable for e.g., the identification of biomarkers of disease, building accurate classifiers, or the identification of descriptors of a time-dependent process. Cross validation (CV) is frequently used for model selection and development, but current CV strategies such as random k-fold CV or individual-based CV can provide overly optimistic model performance estimates because the independence between the samples included in the train and test subsets is often not ensured. To overcome this potential pitfall, here we show with help of simulated data how the use of constrained repeated random subsampling – cross validation (CORRS-CV) improves the independence between train and test subsets during CV, thus providing an accurate estimation of the model performance and facilitating the identification of informative variables.

Integrated data-driven modeling and experimental optimization of granular hydrogel matrices

Integrated data-driven modeling and experimental optimization of granular hydrogel matrices

Effects of randomness in the development of machine learning models in neuroimaging studies of schizophrenia

Numerous studies have used machine learning with neuroimaging data for identifying individuals with a schizophrenia diagnosis. However, inconsistent results have limited the ability of the psychiatric community to objectively judge and accept the value of this approach. One factor that has contributed to the inconsistency, but has long been ignored, is randomness in the practice of machine learning. This is manifest when executing the same machine learning pipeline multiple times on the same dataset but getting different results. In the current study, a dataset of anatomical MRI scans from 158 patients with first-episode medication-naïve schizophrenia and 166 matched controls was used to investigate the effect of randomness on classifier performance estimates under different algorithm complexity and data splitting ratios. The maximum discriminatory accuracy that could be reached was 62.6 % ± 4.7 % (43.5 %–79.3 %) obtained when using extra-trees classifiers without feature normalization. Regions contributing to discrimination were located at bilateral temporal lobes and right frontal lobe. The results show that randomness has a significant impact on the precision of model performance estimates, especially when the size of test set is small. Current neuroimaging feature engineering combined with machine learning still falls short of being able to make diagnoses in the clinical context, but has value in revealing patterns of regional brain alteration associated with the illness. The current results indicate that effects of randomness on model performance should be reported and considered in interpreting model utility and it is necessary to evaluate models on large test sets to obtain valid estimates of model performance.

Statistical approaches to identifying significant differences in predictive performance between machine learning and classical statistical models for survival data.

Research that seeks to compare two predictive models requires a thorough statistical approach to draw valid inferences about comparisons between the performance of the two models. Researchers present estimates of model performance with little evidence on whether they reflect true differences in model performance. In this study, we apply two statistical tests, that is, the 5 × 2-fold cv paired t-test, and the combined 5 × 2-fold cv F-test to provide statistical evidence on differences in predictive performance between the Fine-Gray (FG) and random survival forest (RSF) models for competing risks. These models are trained on different scenarios of low-dimensional simulated survival data to determine whether the differences in their predictive performance that exist are indeed significant. Each simulation was repeated one hundred times on ten different seeds. The results indicate that the RSF model is superior in predictive performance in the presence of complex relationships (quadratic and interactions) between the outcome and its predictors. The two statistical tests show that the differences in performance are significant in quadratic simulation but not significant in interaction simulations. The study has also revealed that the FG model is superior in predictive performance in linear simulations and its differences in predictive performance compared to the RSF model are significant. The combined 5 × 2-fold cv F-test has lower type I error rates compared to the 5 × 2-fold cv paired t-test.

Representative random sampling: an empirical evaluation of a novel bin stratification method for model performance estimation

High-dimensional cancer data can be burdensome to analyze, with complex relationships between molecular measurements, clinical diagnostics, and treatment outcomes. Data-driven computational approaches may be key to identifying relationships with potential clinical or research use. To this end, reliable comparison of feature engineering approaches in their ability to support machine learning survival modeling is crucial. With the limited number of cases often present in multi-omics datasets (“big p, little n,” or many features, few subjects), a resampling approach such as cross validation (CV) would provide robust model performance estimates at the cost of flexibility in intermediate assessments and exploration in feature engineering approaches. A holdout (HO) estimation approach, however, would permit this flexibility at the expense of reliability. To provide more reliable HO-based model performance estimates, we propose a novel sampling procedure: representative random sampling (RRS). RRS is a special case of continuous bin stratification which minimizes significant relationships between random HO groupings (or CV folds) and a continuous outcome. Monte Carlo simulations used to evaluate RRS on synthetic molecular data indicated that RRS-based HO (RRHO) yields statistically significant reductions in error and bias when compared with standard HO. Similarly, more consistent reductions are observed with RRS-based CV. While resampling approaches are the ideal choice for performance estimation with limited data, RRHO can enable more reliable exploratory feature engineering than standard HO.

Insights into performance evaluation of compound-protein interaction prediction methods.

Machine-learning-based prediction of compound-protein interactions (CPIs) is important for drug design, screening and repurposing. Despite numerous recent publication with increasing methodological sophistication claiming consistent improvements in predictive accuracy, we have observed a number of fundamental issues in experiment design that produce overoptimistic estimates of model performance. We systematically analyze the impact of several factors affecting generalization performance of CPI predictors that are overlooked in existing work: (i) similarity between training and test examples in cross-validation; (ii) synthesizing negative examples in absence of experimentally verified negative examples and (iii) alignment of evaluation protocol and performance metrics with real-world use of CPI predictors in screening large compound libraries. Using both state-of-the-art approaches by other researchers as well as a simple kernel-based baseline, we have found that effective assessment of generalization performance of CPI predictors requires careful control over similarity between training and test examples. We show that, under stringent performance assessment protocols, a simple kernel-based approach can exceed the predictive performance of existing state-of-the-art methods. We also show that random pairing for generating synthetic negative examples for training and performance evaluation results in models with better generalization in comparison to more sophisticated strategies used in existing studies. Our analyses indicate that using proposed experiment design strategies can offer significant improvements for CPI prediction leading to effective target compound screening for drug repurposing and discovery of putative chemical ligands of SARS-CoV-2-Spike and Human-ACE2 proteins. Code and supplementary material available at https://github.com/adibayaseen/HKRCPI. Supplementary data are available at Bioinformatics online.

Assessing the Added Value of Vital Signs Extracted from Electronic Health Records in Healthcare Risk Adjustment Models.

PurposePatient vital signs are related to specific health risks and outcomes but are underutilized in the prediction of health-care utilization and cost. To measure the added value of electronic health record (EHR) extracted Body Mass Index (BMI) and blood pressure (BP) values in improving healthcare risk and utilization predictions.Patients and MethodsA sample of 12,820 adult outpatients from the Johns Hopkins Health System (JHHS) were identified between 2016 and 2017, having high data quality and recorded values for BMI and BP. We evaluated the added value of BMI and BP in predicting health-care utilization and cost through a retrospective cohort design. BMI, mean arterial pressure (MAP), systolic and diastolic BPs were summarized as annual aggregated values. Concurrent annual BMI and MAP changes were quantified as the difference between maximum and minimum recorded values. Model performance estimates consisted of repeated 10-fold cross validation, compared to base model point estimates for demographic and diagnostic, coded events: (1) patient age and sex, (2) age, sex, and the Charlson weighted index, (3) age, sex and the Johns Hopkins ACG system’s DxPM risk score.ResultsBoth categorical BMI and BP were progressively indicative of disease comorbidity, but not uniformly related to health-care utilization or cost. Annual change in BMI and MAP improved predictions for most concurrent year outcomes when compared to base models.ConclusionWhen a healthcare system lacks relevant diagnostic or risk assessment information for a patient, vital signs may be useful for a simple estimation of disease risk, cost and utilization.

Impact of Regressand Stratification in Dataset Shift Caused by Cross-Validation

Data that have not been modeled cannot be correctly predicted. Under this assumption, this research studies how k-fold cross-validation can introduce dataset shift in regression problems. This fact implies data distributions in the training and test sets to be different and, therefore, a deterioration of the model performance estimation. Even though the stratification of the output variable is widely used in the field of classification to reduce the impacts of dataset shift induced by cross-validation, its use in regression is not widespread in the literature. This paper analyzes the consequences for dataset shift of including different regressand stratification schemes in cross-validation with regression data. The results obtained show that these allow for creating more similar training and test sets, reducing the presence of dataset shift related to cross-validation. The bias and deviation of the performance estimation results obtained by regression algorithms are improved using the highest amounts of strata, as are the number of cross-validation repetitions necessary to obtain these better results.

Trends in species distribution modelling in context of rare and endemic plants: a systematic review

Abstract Background Many research papers have utilized Species Distribution Models to estimate a species’ current and future geographic distribution and environmental niche. This study aims to (a) understand critical features of SDMs used to model endemic and rare species and (b) to identify possible constraints with the collected data. The present systematic review examined how SDMs are used on endemic and rare plant species to identify optimal practices for future research. Results The evaluated literature (79 articles) was published between January 2010 and December 2020. The number of papers grew considerably over time. The studies were primarily conducted in Asia (41%), Europe (24%), and Africa (2%). The bulk of the research evaluated (38%) focused on theoretical ecology, climate change impacts (19%), and conservation policy and planning (22%). Most of the papers were published in publications devoted to biodiversity conservation, ecological or multidisciplinary fields. The degree of uncertainty was not disclosed in most studies (81%). Conclusion This systematic review provides a broad overview of the emerging trends and gaps in the SDMs research. The majority of studies failed to present uncertainties and error estimates. However, when model performance estimates are given, the model results will be highly effective, allowing for more assurance in the predictions they make. Furthermore, based on our systematic review, we recommend that in the future rare and endemic SDMs should represent uncertainty levels and estimates of errors in the modelling process.