Traditional Cross-validation Research Articles

Optimized Random Forest Method for 3D Evaluation of Coalbed Methane Content Using Geophysical Logging Data.

Accurate evaluation of coalbed methane (CBM) content is crucial for effective exploration and development. Traditional gas content measurement methods based on laboratory analysis of drill core samples are costly, whereas geophysical logging methods offer a cost-effective alternative by providing continuous high-resolution profiles of rock layer physical properties. However, the relationship between CBM content and geophysical logging data is complex and nonlinear, necessitating an advanced prediction method. This study focuses on the No. 3 coal seam in the Shizhuang South Block of the Qinshui Basin, utilizing geophysical logging data and 148 sets of laboratory core samples. We employed the Random Forest (RF) method optimized with a simulated annealing-genetic algorithm (SA-GA) to develop the SA-GA-RF model for evaluating CBM content. The model's performance was validated using test data and new CBM well data, and it was applied to calculate the vertical gas content profiles of No. 3 coal seam across 128 wells. The SA-GA-RF model demonstrated an average relative error of 13.13% in the test data set, outperforming Backpropagation Neural Network (BPNN), Least Squares Support Vector Machine (LSSVM), Extreme Learning Machine (ELM), and multivariate regression (MR) methods. The model also exhibited strong generalizability in new wells and improved model-building efficiency compared to traditional cross-validation grid search methods. The construction of a three-dimensional CBM content model, incorporating well coordinates and elevation data, allowed for detailed identification of high gas content areas and layers. This three-dimensional model offers a more precise characterization than traditional two-dimensional isopleth maps, providing valuable insights for CBM exploration, reserve evaluation, and production optimization.

Explicit solutions for the asymptotically optimal bandwidth in cross-validation

Abstract We show that least-squares cross-validation methods share a common structure that has an explicit asymptotic solution, when the chosen kernel is asymptotically separable in bandwidth and data. For density estimation with a multivariate Student-t(ν) kernel, the cross-validation criterion becomes asymptotically equivalent to a polynomial of only three terms. Our bandwidth formulae are simple and noniterative, thus leading to very fast computations, their integrated squared-error dominates traditional cross-validation implementations, they alleviate the notorious sample variability of cross-validation and overcome its breakdown in the case of repeated observations. We illustrate our method with univariate and bivariate applications, of density estimation and nonparametric regressions, to a large dataset of Michigan State University academic wages and experience.

Phantom oscillations in principal component analysis.

Principal component analysis (PCA) is a dimensionality reduction method that is known for being simple and easy to interpret. Principal components are often interpreted as low-dimensional patterns in high-dimensional space. However, this simple interpretation fails for timeseries, spatial maps, and other continuous data. In these cases, nonoscillatory data may have oscillatory principal components. Here, we show that two common properties of data cause oscillatory principal components: smoothness and shifts in time or space. These two properties implicate almost all neuroscience data. We show how the oscillations produced by PCA, which we call "phantom oscillations," impact data analysis. We also show that traditional cross-validation does not detect phantom oscillations, so we suggest procedures that do. Our findings are supported by a collection of mathematical proofs. Collectively, our work demonstrates that patterns which emerge from high-dimensional data analysis may not faithfully represent the underlying data.

Cross-Validation: What Does It Estimate and How Well Does It Do It?

Cross-validation is a widely used technique to estimate prediction error, but its behavior is complex and not fully understood. Ideally, one would like to think that cross-validation estimates the prediction error for the model at hand, fit to the training data. We prove that this is not the case for the linear model fit by ordinary least squares; rather it estimates the average prediction error of models fit on other unseen training sets drawn from the same population. We further show that this phenomenon occurs for most popular estimates of prediction error, including data splitting, bootstrapping, and Mallow’s C p . Next, the standard confidence intervals for prediction error derived from cross-validation may have coverage far below the desired level. Because each data point is used for both training and testing, there are correlations among the measured accuracies for each fold, and so the usual estimate of variance is too small. We introduce a nested cross-validation scheme to estimate this variance more accurately, and show empirically that this modification leads to intervals with approximately correct coverage in many examples where traditional cross-validation intervals fail. Lastly, our analysis also shows that when producing confidence intervals for prediction accuracy with simple data splitting, one should not refit the model on the combined data, since this invalidates the confidence intervals. Supplementary materials for this article are available online.

Randomized Subensembles: An Approach to Reduce the Risk of Divergence in an Ensemble Kalman Filter Using Cross Validation

Abstract In an ensemble Kalman filter, when the analysis update of an ensemble member is computed using error statistics estimated from an ensemble that includes the background of the member being updated, the spread of the resulting ensemble systematically underestimates the uncertainty of the ensemble mean analysis. This problem can largely be avoided by applying cross validation: using an independent subset of ensemble members for updating each member. However, in some circumstances cross validation can lead to the divergence of one or more ensemble members from observations. This can culminate in catastrophic filter divergence in which the analyzed or forecast states become unrealistic in the diverging members. So far, such instabilities have been reported only in the context of highly nonlinear low-dimensional models. The first known manifestation of catastrophic filter divergence caused by the use of cross validation in an NWP context is reported here. To reduce the risk of such filter divergence, a modification to the traditional cross-validation approach is proposed. Instead of always assigning the ensemble members to the same subensembles, the members forming each subensemble are randomly chosen at every analysis step. It is shown that this new approach can prevent filter divergence and also brings a cycling ensemble data assimilation system containing divergent members back to a state consistent with Gaussianity. The randomized subensemble approach was implemented in the operational global ensemble prediction system at Environment and Climate Change Canada on 1 December 2021.

DWT-CV: Dense weight transfer-based cross validation strategy for model selection in biomedical data analysis

Model selection for deep learning algorithms is an extremely important step in the process of extracting knowledge from limited data, especially in biomedical data. The common approach is to adopt cross-validation techniques to randomly divide a small subset of the training set as the validation data for parameter tuning and model selection. However, this method may choose a sub-optimal model due to insufficient data utilization, and the process, such as k-fold cross-validation, is cumbersome and time-consuming. In this study, we propose a dense weight transfer-based cross validation (DWT-CV) strategy for biomedical data analysis and use this strategy to improve the generalization of deep learning algorithms with reduced training time using weight transfer learning. DWT-CV utilizes a dense weight aggregation and weight transfer mechanism to make the model more general and converge faster during the cross validation. The effectiveness of the proposed strategy is evaluated on multiple experiments with three different domains including biomedical image classification, drug–target affinity prediction, and medical image segmentation. Extensive experimental results demonstrate that our proposed DWT-CV strategy can make several deep learning benchmark methods perform better on multiple biomedical datasets, which implies that it may be an alternative to the traditional cross validation criterion for model selection.

Improving prediction accuracy of high-performance materials via modified machine learning strategy

In general, machine learning models trained by traditional cross validation evaluation strategies always have excellent interpolation ability in the known data space, but weak extrapolation ability in the unknown data space. However, in the field of material research, discovering new materials with higher performance always needs high prediction accuracy on the unknown data space outside the existing data range. In order to solve this challenge, we propose a modified evaluation strategy of machine learning modeling to promote the extrapolation ability by sorting the existing data according to the property value before the training/testing data set partition. It is demonstrated that the extrapolation and prediction abilities of machine learning model can be significantly improved by using this strategy. Machine learning models with lower prediction error and higher efficiency of discovering new materials are obtained on three materials data sets (bulk metallic glasses dataset, high-entropy alloys dataset, and piezoelectric materials dataset). The model errors are reduced from 23.84%, 9.79% and 14.47% of the traditional strategy to 2.44%, 7.67% and 8.07% of the modified strategy, respectively. It is indicated that this strategy can provide a solution for machine learning modeling in material field with requirements of discovering new materials with higher performance.

Automated hyperparameter selection for the PC algorithm

• The performance of the PC algorithm depends heavily on the choice of the Type I error rate α . • We propose to automatically tune α for a user chosen metric using a procedure called AutoPC. • AutoPC double checks the output of PC by rerunning the algorithm on restricting conditioning sets. • AutoPC provably maximizes the chosen metric and recovers the true CPDAG in the sample limit. • The algorithm also easily achieves state-of-the-art accuracy across multiple metrics with both synthetic and real data. The PC algorithm infers causal relations using conditional independence tests that require a pre-specified Type I α level. PC is however unsupervised, so we cannot tune α using traditional cross-validation. We therefore propose AutoPC, a fast procedure that optimizes α directly for a user chosen metric. We in particular force PC to double check its output by executing a second run on the recovered graph. We choose the final output as the one which maximizes stability between the two runs. AutoPC consistently outperforms the state of the art across multiple metrics.

Comparing Methods of Exploratory Data Analysis for the Moral Foundations Questionnaire with a Small Sample

When applied in a principled exploratory manner Data Mining and Machine Learning (DM/ML) can generate new insights and inform future research. This paper compares traditional regression with DM/ML methods to determine the best method for interpreting a small data set and develop insights to health professional’s moral foundations. Forward, Backward, Hierarchical, and Elastic-Net regression with k-fold Cross Validation and Bootstrapping were compared. Elastic-Net regression outperformed the traditional methods, Cross Validation and Bootstrapping produced comparable outcomes. Elastic-Net was used to determine a model of moral foundations for health professional students. DM/ML methods are appropriate for use with a psychological measure, small sample size, and present new opportunities for psychological research. Determination of sample size, all data exclusions, all manipulations, and all measures in the study are reported.

Abstract 15: Imputing a more targeted therapy for triple-negative breast cancers

Abstract Triple-negative breast cancer (TNBC) is a clinically defined subset of breast cancers that are inherently difficult to treat since, unlike the other breast cancer subtypes, they are defined by their lack of a distinct molecular target. Previously our lab developed a computational framework to impute drug response in patients that accurately mirrored the observed patient response. In short, this methodology involves building unsupervised linear regression models between cell line baseline gene expression values and the respective drug sensitivity metric (EC50 or AUC), which allows us to generate imputed sensitivity scores for cell line or patient tumor samples for which we have gene expression data. For this study, we applied this methodology to prospectively identify agents that are predicted to be more effective specifically in triple-negative breast cancer. We began by expanding the drug imputation training dataset to the Broad’s Cell Therapeutics Response Portal, which contains 545 drugs and over 1,000 cancer cell lines (CCLs) and imputed sensitivity in The Cancer Genome Atlas’s (TCGA’s) breast cancer cohort. Our imputation data indicated that TNBC tumors were expected to be comparatively sensitive to cell cycle pathway inhibitors. Among these compounds, the Wee1 inhibitor MK1775 (aka adavosertib and AZD1775) was identified for having the most significant predicted effect in TNBC patients compared to the receptor-positive patients. We were able to validate the accuracy of the MK1775 imputation model using traditional cross-validation approaches as well as in independent cell line dataset and showed that the measured and our predicted response to Wee1 inhibition strongly correlated. Additionally, we were able to uses our predicted MK1775 response scores to recapitulate the significant association between MK1775 response and p53 mutation status. We then looked to validate the efficacy of MK1775’s efficacy in TNBC cell line and xenograft experiments. We observed strong inhibition of cancer growth by MK1775 alone but also identified a combination effect between MK1775 and the TNBC standard-of-care agent paclitaxel as assayed by both in vitro viability experiments as well as in vivo tumor growth assays. We believe these results indicate that this methodology could generally be applied as a hypothesis-generating tool for identifying targeted agents for other disease subtypes as well as supporting the use of MK1775, especially in combination with paclitaxel, for TNBC. Citation Format: Robert F. Gruener, Geoffrey L. Greene, R. Stephanie Huang. Imputing a more targeted therapy for triple-negative breast cancers [abstract]. In: Proceedings of the AACR Special Conference on Advancing Precision Medicine Drug Development: Incorporation of Real-World Data and Other Novel Strategies; Jan 9-12, 2020; San Diego, CA. Philadelphia (PA): AACR; Clin Cancer Res 2020;26(12_Suppl_1):Abstract nr 15.

Adaptive Hybrid Soft-Sensor Model of Grinding Process Based on Regularized Extreme Learning Machine and Least Squares Support Vector Machine Optimized by Golden Sine Harris Hawk Optimization Algorithm

Soft-sensor technology plays a vital role in tracking and monitoring the key production indicators of the grinding and classifying process. Least squares support vector machine (LSSVM), as a soft-sensor model with strong generalization ability, can be used to predict key production indicators in complex grinding processes. The traditional crossvalidation method cannot obtain the ideal structure parameters of LSSVM. In order to improve the prediction accuracy of LSSVM, a golden sine Harris Hawk optimization (GSHHO) algorithm was proposed to optimize the structure parameters of LSSVM models with linear kernel, sigmoid kernel, polynomial kernel, and radial basis kernel, and the influences of GSHHO algorithm on the prediction accuracy under these LSSVM models were studied. In order to deal with the problem that the prediction accuracy of the model decreases due to changes of industrial status, this paper adopts moving window (MW) strategy to adaptively revise the LSSVM (MW-LSSVM), which greatly improves the prediction accuracy of the LSSVM. The prediction accuracy of the regularized extreme learning machine with MW strategy (MW-RELM) is higher than that of MW-LSSVM at some moments. Based on the training errors of LSSVM and RELM within the window, this paper proposes an adaptive hybrid soft-sensing model that switches between LSSVM and RELM. Compared with the previous MW-LSSVM, MW-neural network trained with extended Kalman filter(MW-KNN), and MW-RELM, the prediction accuracy of the hybrid model is further improved. Simulation results show that the proposed hybrid adaptive soft-sensor model has good generalization ability and prediction accuracy.

Scalable and Generalizable Social Bot Detection through Data Selection

Efficient and reliable social bot classification is crucial for detecting information manipulation on social media. Despite rapid development, state-of-the-art bot detection models still face generalization and scalability challenges, which greatly limit their applications. In this paper we propose a framework that uses minimal account metadata, enabling efficient analysis that scales up to handle the full stream of public tweets of Twitter in real time. To ensure model accuracy, we build a rich collection of labeled datasets for training and validation. We deploy a strict validation system so that model performance on unseen datasets is also optimized, in addition to traditional cross-validation. We find that strategically selecting a subset of training data yields better model accuracy and generalization than exhaustively training on all available data. Thanks to the simplicity of the proposed model, its logic can be interpreted to provide insights into social bot characteristics.

Efficient Hyper-Parameter Selection in Total Variation-Penalised XCT Reconstruction Using Freund and Shapire’s Hedge Approach

This paper studies the problem of efficiently tuning the hyper-parameters in penalised least-squares reconstruction for XCT. Discovered through the lens of the Compressed Sensing paradigm, penalisation functionals such as Total Variation types of norms, form an essential tool for enforcing structure in inverse problems, a key feature in the case where the number of projections is small as compared to the size of the object to recover. In this paper, we propose a novel hyper-parameter selection approach for total variation (TV)-based reconstruction algorithms, based on a boosting type machine learning procedure initially proposed by Freund and Shapire and called Hedge. The proposed approach is able to select a set of hyper-parameters producing better reconstruction than the traditional Cross-Validation approach, with reduced computational effort. Traditional reconstruction methods based on penalisation can be made more efficient using boosting type methods from machine learning.

Evaluation of knowledge graph embedding approaches for drug-drug interaction prediction in realistic settings

BackgroundCurrent approaches to identifying drug-drug interactions (DDIs), include safety studies during drug development and post-marketing surveillance after approval, offer important opportunities to identify potential safety issues, but are unable to provide complete set of all possible DDIs. Thus, the drug discovery researchers and healthcare professionals might not be fully aware of potentially dangerous DDIs. Predicting potential drug-drug interaction helps reduce unanticipated drug interactions and drug development costs and optimizes the drug design process. Methods for prediction of DDIs have the tendency to report high accuracy but still have little impact on translational research due to systematic biases induced by networked/paired data. In this work, we aimed to present realistic evaluation settings to predict DDIs using knowledge graph embeddings. We propose a simple disjoint cross-validation scheme to evaluate drug-drug interaction predictions for the scenarios where the drugs have no known DDIs.ResultsWe designed different evaluation settings to accurately assess the performance for predicting DDIs. The settings for disjoint cross-validation produced lower performance scores, as expected, but still were good at predicting the drug interactions. We have applied Logistic Regression, Naive Bayes and Random Forest on DrugBank knowledge graph with the 10-fold traditional cross validation using RDF2Vec, TransE and TransD. RDF2Vec with Skip-Gram generally surpasses other embedding methods. We also tested RDF2Vec on various drug knowledge graphs such as DrugBank, PharmGKB and KEGG to predict unknown drug-drug interactions. The performance was not enhanced significantly when an integrated knowledge graph including these three datasets was used.ConclusionWe showed that the knowledge embeddings are powerful predictors and comparable to current state-of-the-art methods for inferring new DDIs. We addressed the evaluation biases by introducing drug-wise and pairwise disjoint test classes. Although the performance scores for drug-wise and pairwise disjoint seem to be low, the results can be considered to be realistic in predicting the interactions for drugs with limited interaction information.

Cross-Validation With Confidence

Cross-validation is one of the most popular model and tuning parameter selection methods in statistics and machine learning. Despite its wide applicability, traditional cross-validation methods tend to overfit, due to the ignorance of the uncertainty in the testing sample. We develop a novel statistically principled inference tool based on cross-validation that takes into account the uncertainty in the testing sample. This method outputs a set of highly competitive candidate models containing the optimal one with guaranteed probability. As a consequence, our method can achieve consistent variable selection in a classical linear regression setting, for which existing cross-validation methods require unconventional split ratios. When used for tuning parameter selection, the method can provide an alternative trade-off between prediction accuracy and model interpretability than existing variants of cross-validation. We demonstrate the performance of the proposed method in several simulated and real data examples. Supplemental materials for this article can be found online.

Evaluating explorative prediction power of machine learning algorithms for materials discovery using [formula omitted]-fold forward cross-validation

The materials discovery problem usually aims to identify novel “outlier” materials with extremely low or high property values outside of the scope of all known materials. It can be mapped as an explorative prediction problem. However, currently the performance of machine learning algorithms for materials property prediction is usually evaluated via k-fold cross-validation (CV) or holdout-test, which tend to over-estimate their explorative prediction performance in discovering novel materials. We propose k-fold-m-step forward cross-validation (kmFCV) as a new way for evaluating exploration performance in materials property prediction and conducted a comprehensive benchmark evaluation on the exploration performance of a variety of prediction models on materials property (including formation energy, band gap, and superconducting critical temperature) prediction with different materials representation and machine learning algorithms. Our results show that even though current machine learning models can achieve good results when evaluated with traditional CV, their explorative power is actually very low as shown by our proposed kmFCV evaluation method and the proposed exploration accuracy. More advanced explorative machine learning algorithms are strongly needed for new materials discovery.

Multilevel network data facilitate statistical inference for curved ERGMs with geometrically weighted terms

Multilevel network data provide two important benefits for ERG modeling. First, they facilitate estimation of the decay parameters in geometrically weighted terms for degree and triad distributions. Estimating decay parameters from a single network is challenging, so in practice they are typically fixed rather than estimated. Multilevel network data overcome that challenge by leveraging replication. Second, such data make it possible to assess out-of-sample performance using traditional cross-validation techniques. We demonstrate these benefits by using a multilevel network sample of classroom networks from Poland. We show that estimating the decay parameters improves in-sample performance of the model and that the out-of-sample performance of our best model is strong, suggesting that our findings can be generalized to the population of interest.

CID Models on Real-world Social Networks and Goodness of Fit Measurements

Assessing the model fit quality of statistical models for network data is an ongoing and under-examined topic in statistical network analysis. Traditional metrics for evaluating model fit on tabular data such as the Bayesian Information Criterion are not suitable for models specialized for network data. We propose a novel self-developed goodness of fit (GOF) measure, the `stratified-sampling cross-validation' (SCV) metric, that uses a procedure similar to traditional cross-validation via stratified-sampling to select dyads in the network's adjacency matrix to be removed. SCV is capable of intuitively expressing different models' ability to predict on missing dyads. Using SCV on real-world social networks, we identify the appropriate statistical models for different network structures and generalize such patterns. In particular, we focus on conditionally independent dyad (CID) models such as the Erdos Renyi model, the stochastic block model, the sender-receiver model, and the latent space model.

Construction of prediction model of neural network railway bulk cargo floating price based on random forest regression algorithm

In order to improve the prediction accuracy and modeling speed of railway freight volume, this paper combines the cargo floating price prediction model with the neural network algorithm (hereinafter referred to as NNA) to establish a prediction model. The railway cargo floating price based on neural network (hereinafter referred to as PMBCFP) relies on the random forest regression algorithm (hereinafter referred to as RFRA). Through the neural network operator in the prediction model of the floating price of goods, the randomness of the original sequence is weakened, and the implicit rules in the series are mined. The characteristics of neural networks are computationally very simple. In addition, a random forest regression algorithm is applied to the optimization, RFRA’s choice. Case studies of China’s rail freight volume show that RMSE and other indicators are faster. The MAE, MPE and Tell inequality coefficients obtained from this model were 0.0628, 0.0523, 0.0162 and 0.0107, respectively. This model has good prediction results. The time to search for the best parameters of RFRA using the NNA algorithm is 55.656 s, which is 10.462 s less than the time required for traditional cross-validation methods. Therefore, it is suitable for short-term forecasting of railway freight volume.

The impact of different sources of heterogeneity on loss of accuracy from genomic prediction models.

Cross-study validation (CSV) of prediction models is an alternative to traditional cross-validation (CV) in domains where multiple comparable datasets are available. Although many studies have noted potential sources of heterogeneity in genomic studies, to our knowledge none have systematically investigated their intertwined impacts on prediction accuracy across studies. We employ a hybrid parametric/non-parametric bootstrap method to realistically simulate publicly available compendia of microarray, RNA-seq, and whole metagenome shotgun microbiome studies of health outcomes. Three types of heterogeneity between studies are manipulated and studied: (i) imbalances in the prevalence of clinical and pathological covariates, (ii) differences in gene covariance that could be caused by batch, platform, or tumor purity effects, and (iii) differences in the "true" model that associates gene expression and clinical factors to outcome. We assess model accuracy, while altering these factors. Lower accuracy is seen in CSV than in CV. Surprisingly, heterogeneity in known clinical covariates and differences in gene covariance structure have very limited contributions in the loss of accuracy when validating in new studies. However, forcing identical generative models greatly reduces the within/across study difference. These results, observed consistently for multiple disease outcomes and omics platforms, suggest that the most easily identifiable sources of study heterogeneity are not necessarily the primary ones that undermine the ability to accurately replicate the accuracy of omics prediction models in new studies. Unidentified heterogeneity, such as could arise from unmeasured confounding, may be more important.