Traditional Cross-validation Methods Research Articles

Profile electoral college cross-validation

Cross-validation (CV), while being extensively used for model selection, may have three major weaknesses. The regular 10-fold CV, for instance, is often unstable in its choice of the best model among the candidates. Secondly, the CV outcome of singling out one candidate based on the total prediction errors over the different folds does not convey any sensible information on how much one can trust the apparent winner. Lastly, when only one data splitting ratio is considered, regardless of its choice, it may work very poorly for some situations. In this work, to address these shortcomings, we propose a new averaging-voting based version of cross-validation for better comparison results. Simulations and real data are used to illustrate the superiority of the new approach over traditional CV methods.

Adaptive Hybrid Soft-Sensor Model of Grinding Process Based on Regularized Extreme Learning Machine and Least Squares Support Vector Machine Optimized by Golden Sine Harris Hawk Optimization Algorithm

Soft-sensor technology plays a vital role in tracking and monitoring the key production indicators of the grinding and classifying process. Least squares support vector machine (LSSVM), as a soft-sensor model with strong generalization ability, can be used to predict key production indicators in complex grinding processes. The traditional crossvalidation method cannot obtain the ideal structure parameters of LSSVM. In order to improve the prediction accuracy of LSSVM, a golden sine Harris Hawk optimization (GSHHO) algorithm was proposed to optimize the structure parameters of LSSVM models with linear kernel, sigmoid kernel, polynomial kernel, and radial basis kernel, and the influences of GSHHO algorithm on the prediction accuracy under these LSSVM models were studied. In order to deal with the problem that the prediction accuracy of the model decreases due to changes of industrial status, this paper adopts moving window (MW) strategy to adaptively revise the LSSVM (MW-LSSVM), which greatly improves the prediction accuracy of the LSSVM. The prediction accuracy of the regularized extreme learning machine with MW strategy (MW-RELM) is higher than that of MW-LSSVM at some moments. Based on the training errors of LSSVM and RELM within the window, this paper proposes an adaptive hybrid soft-sensing model that switches between LSSVM and RELM. Compared with the previous MW-LSSVM, MW-neural network trained with extended Kalman filter(MW-KNN), and MW-RELM, the prediction accuracy of the hybrid model is further improved. Simulation results show that the proposed hybrid adaptive soft-sensor model has good generalization ability and prediction accuracy.

Cross-Validation With Confidence

Cross-validation is one of the most popular model and tuning parameter selection methods in statistics and machine learning. Despite its wide applicability, traditional cross-validation methods tend to overfit, due to the ignorance of the uncertainty in the testing sample. We develop a novel statistically principled inference tool based on cross-validation that takes into account the uncertainty in the testing sample. This method outputs a set of highly competitive candidate models containing the optimal one with guaranteed probability. As a consequence, our method can achieve consistent variable selection in a classical linear regression setting, for which existing cross-validation methods require unconventional split ratios. When used for tuning parameter selection, the method can provide an alternative trade-off between prediction accuracy and model interpretability than existing variants of cross-validation. We demonstrate the performance of the proposed method in several simulated and real data examples. Supplemental materials for this article can be found online.

Construction of prediction model of neural network railway bulk cargo floating price based on random forest regression algorithm

In order to improve the prediction accuracy and modeling speed of railway freight volume, this paper combines the cargo floating price prediction model with the neural network algorithm (hereinafter referred to as NNA) to establish a prediction model. The railway cargo floating price based on neural network (hereinafter referred to as PMBCFP) relies on the random forest regression algorithm (hereinafter referred to as RFRA). Through the neural network operator in the prediction model of the floating price of goods, the randomness of the original sequence is weakened, and the implicit rules in the series are mined. The characteristics of neural networks are computationally very simple. In addition, a random forest regression algorithm is applied to the optimization, RFRA’s choice. Case studies of China’s rail freight volume show that RMSE and other indicators are faster. The MAE, MPE and Tell inequality coefficients obtained from this model were 0.0628, 0.0523, 0.0162 and 0.0107, respectively. This model has good prediction results. The time to search for the best parameters of RFRA using the NNA algorithm is 55.656 s, which is 10.462 s less than the time required for traditional cross-validation methods. Therefore, it is suitable for short-term forecasting of railway freight volume.

Optimal smoothing in nonparametric conditional quantile derivative function estimation

Marginal effect in nonparametric quantile regression is of special interest as it quantitatively measures how one unit change in explanatory variable heterogeneously affects dependent variable ceteris paribus at distinct quantiles. In this paper, we propose a data-driven bandwidth selection procedure based on the gradient of an unknown quantile regression function. Our method delivers the bandwidth with the oracle property in the sense that it is asymptotically equivalent to the optimal bandwidth if the true gradient were known. The results of Monte Carlo simulations are reported, and the finite sample performance of our proposed method confirms our theoretical analysis. An empirical application is also provided, showing that our proposed method delivers more reasonable and reliable quantile derivative estimates than traditional cross validation method.

Cross-validation in PCA models with the element-wise k-fold (ekf) algorithm: Practical aspects

This is the second paper of a series devoted to provide theoretical and practical results and new algorithms for the selection of the number of Principal Components (PCs) in Principal Component Analysis (PCA) using cross-validation. The study is especially focused on the element-wise k-fold (ekf), which is among the most used algorithms for that purpose. In this paper, a taxonomy of PCA applications is proposed and it is argued that cross-validatory algorithms computing the prediction error in observable variables, like ekf, are only suited for a class of applications. A number of cross-validation methods, several of which are original, are compared in two applications of this class: missing data imputation and compression. The results show that the ekf is especially suited for missing data applications while other traditional cross-validation methods, those by Wold and Eastment and Krzanowski, are not found to provide useful outcomes in any of the two applications. These results are of special value considering that the methods investigated are computed in the main commercial software packets for chemometrics. Finally, the choice of the missing data algorithm within ekf is also investigated.

Semiparametric estimation of volatility: some models and complexity choice in the adaptive functional-coefficient class

In this paper, semiparametric methods are applied to estimate multivariate volatility functions, using a residual approach as in [J. Fan and Q. Yao, Efficient estimation of conditional variance functions in stochastic regression, Biometrika 85 (1998), pp. 645–660; F.A. Ziegelmann, Nonparametric estimation of volatility functions: The local exponential estimator, Econometric Theory 18 (2002), pp. 985–991; F.A. Ziegelmann, A local linear least-absolute-deviations estimator of volatility, Comm. Statist. Simulation Comput. 37 (2008), pp. 1543–1564], among others. Our main goal here is two-fold: (1) describe and implement a number of semiparametric models, such as additive, single-index and (adaptive) functional-coefficient, in volatility estimation, all motivated as alternatives to deal with the curse of dimensionality present in fully nonparametric models; and (2) propose the use of a variation of the traditional cross-validation method to deal with model choice in the class of adaptive functional-coefficient models, choosing simultaneously the bandwidth, the number of covariates in the model and also the single-index smoothing variable. The modified cross-validation algorithm is able to tackle the computational burden caused by the model complexity, providing an important tool in semiparametric volatility estimation. We briefly discuss model identifiability when estimating volatility as well as nonnegativity of the resulting estimators. Furthermore, Monte Carlo simulations for several underlying generating models are implemented and applications to real data are provided.

Adaptive local kernel-based learning for soft sensor modeling of nonlinear processes

Soft sensor techniques have been widely used to estimate product quality or other key indices which cannot be measured online by hardware sensors. Unfortunately, their estimation performance would deteriorate under certain circumstances, e.g., the change of the process characteristics, especially for global learning approaches. Meanwhile, local learning methods always only utilize input information to select relevant instances, which may lead to a waste of output information and inaccurate sample selection. To overcome these disadvantages, a new local modeling algorithm, adaptive local kernel-based learning scheme (ALKL) is proposed. First, a new similarity measurement using both input and output information is proposed and utilized in a supervised locality preserving projection technique to select relevant samples. Second, an adaptive weighted least squares support vector regression (AW-LSSVR) is employed to establish a local model and predict output indices for each query data. In AW-LSSVR, instead of using traditional cross-validation methods, the trade-off parameters are adjusted iteratively and the local model is updated recursively, which reduces the computational complexity a lot. The proposed ALKL is applied to an online crude oil endpoint prediction in an industrial fluidized catalytic cracking unit (FCCU) process. The experimental results demonstrate the high precision of our ALKL approach.