Least-squares Cross-validation Research Articles

Explicit solutions for the asymptotically optimal bandwidth in cross-validation

Abstract We show that least-squares cross-validation methods share a common structure that has an explicit asymptotic solution, when the chosen kernel is asymptotically separable in bandwidth and data. For density estimation with a multivariate Student-t(ν) kernel, the cross-validation criterion becomes asymptotically equivalent to a polynomial of only three terms. Our bandwidth formulae are simple and noniterative, thus leading to very fast computations, their integrated squared-error dominates traditional cross-validation implementations, they alleviate the notorious sample variability of cross-validation and overcome its breakdown in the case of repeated observations. We illustrate our method with univariate and bivariate applications, of density estimation and nonparametric regressions, to a large dataset of Michigan State University academic wages and experience.

Kernel Averaging Estimators

The issue of bandwidth selection is a fundamental model selection problem stemming from the uncertainty about the smoothness of the regression. In this article, we advocate a model averaging approach to circumvent the problem caused by this uncertainty. Our new approach involves averaging across a series of Nadaraya-Watson kernel estimators each under a different bandwidth, with weights for these different estimators chosen such that a least-squares cross-validation criterion is minimized. We prove that the resultant combined-kernel estimator achieves the smallest possible asymptotic aggregate squared error. The superiority of the new estimator over estimators based on widely accepted conventional bandwidth choices in finite samples is demonstrated in a simulation study and a real data example.

Application of near-infrared spectroscopy technology in the complex fermentation system to achieve high-efficiency production

The fermentation process is dynamically changing, and the metabolic status can be grasped through real-time monitoring of environmental parameters. In this study, a real-time and on-line monitoring experiment platform for substrates and products detection was developed based on non-contact type near-infrared (NIR) spectroscopy technology. The prediction models for monitoring the fermentation process of lactic acid, sophorolipids (SLs) and sodium gluconate (SG) were established based on partial least-squares regression and internal cross-validation methods. Through fermentation verification, the accuracy and precision of the NIR model for the complex fermentation environments, different rheological properties (uniform system and multi-phase inhomogeneous system) and different parameter types (substrate, product and nutrients) have good applicability, and R2 was greater than 0.98, exhibiting a good linear relationship. The root mean square error of prediction shows that the model has high credibility. Through the control of appropriate glucose concentration in SG fermentation as well as glucose and oil concentrations SLs fermentation by NIR model, the titers of SG and SLs were increased to 11.8% and 26.8%, respectively. Although high cost of NIR spectrometer is a key issue for its wide application in an industrial scale. This work provides a basis for the application of NIR spectroscopy in complex fermentation systems.

Compilation method of CNC lathe cutting force spectrum based on kernel density estimation of G-SCE

The cutting force spectrum of the CNC lathe is the basic data for the reliability design, reliability test, and reliability evaluation of the CNC lathe and its components. Due to the complex and changeable turning conditions and different cutting processes, the cutting load presents multi-peak characteristics. At the same time, grouping the counted load cycles when parameter modeling will produce certain errors. As a result, the parameter distribution model cannot meet the modeling requirements. Thus, a compilation method based on kernel density estimation (KDE) of goodness-smoothness comprehensive evaluation (G-SCE) is proposed. The KDE is used to establish the dynamic cutting force distribution of the CNC lathe in which grouping the counted load cycles is not needed. For the bandwidth-determining methods, the rule of thumb method (ROT) and the least-squares cross-validation method (LCV) do not take into account the influence of different bandwidths on the goodness estimation and the smoothness of the estimated curve, and the G-CSE for KDE is proposed to determining the optimal bandwidth. It includes the estimation accuracy test method based on multiple goodness-of-fit tests and the smoothness test method based on the envelope curve, and the entropy method is used to comprehensively weights the estimated goodness index and the smoothness index to determine the optimal bandwidth. The results of the case analysis indicate that the method proposed can solve the problem of too large estimation error of parameter distribution for multimodal distribution. At the same time, it can better comprehensively evaluate the KDE under different bandwidths. In short, a new method of optimal bandwidth selection is proposed in the original method.

Nonparametric density estimation and bandwidth selection with B-spline bases: A novel Galerkin method

A general and efficient nonparametric density estimation procedure for local bases, including B-splines, is proposed, which employs a novel statistical Galerkin method combined with basis duality theory. To select the bandwidth, an efficient cross-validation procedure is introduced, based on closed-form expressions in terms of the primal and dual B-spline basis. By utilizing a closed-form expression for the dual basis, the least-squares cross validation formula is calculated in closed-form, enabling an efficient estimation of the optimal bandwidth. The full computational procedure achieves optimal complexity, and is very accurate in comparison with existing estimation procedures, including state-of-the-art kernel density estimators. The presented theoretical results are supported by extensive numerical experiments, which demonstrate the efficiency and accuracy of the new methodology. This new approach provides a complete and optimally efficient framework for density estimation with a B-spline basis, based on simple and elegant closed-form estimators with theoretical convergence results that are substantiated in numerical experiments.

Nonparametric Density Estimation and Bandwidth Selection with B-spline bases: a Novel Galerkin Method

We study a general and efficient nonparametric density estimation procedure for local bases, including B-splines, using a novel statistical Galerkin method, combined with basis duality theory. We provide an efficient cross-validation procedure to select the bandwidth, based on closed-form expressions in terms of the primal and dual basis. By utilizing a closed-form expression for the dual basis, we are able to compute the least-squares cross validation formula in closed-form, which allows us to effectively determine the bandwidth. The full computational procedure achieves optimal complexity, and is very accurate in comparisons with existing estimation procedures, including state-of-the-art kernel density estimators. Our theoretical results are supported by extensive numerical experiments, which demonstrate the efficiency and accuracy of the new methodology.

Reptiles on the wrong track? Moving beyond traditional estimators with dynamic Brownian Bridge Movement Models

BackgroundAnimal movement expressed through home ranges or space-use can offer insights into spatial and habitat requirements. However, different classes of estimation methods are currently instinctively applied to answer home range, space-use or movement-based research questions regardless of their widely varying outputs, directly impacting conclusions. Recent technological advances in animal tracking (GPS and satellite tags), have enabled new methods to quantify animal space-use and movement pathways, but so far have primarily targeted mammal and avian species.MethodsMost reptile spatial ecology studies only make use of two older home range estimation methods: Minimum Convex Polygons (MCP) and Kernel Density Estimators (KDE), particularly with the Least Squares Cross Validation (LSCV) and reference (href) bandwidth selection algorithms. These methods are frequently applied to answer space-use and movement-based questions. Reptile movement patterns are unique (e.g., low movement frequency, long stop-over periods), prompting investigation into whether newer movement-based methods –such as dynamic Brownian Bridge Movement Models (dBBMMs)– apply to Very High Frequency (VHF) radio-telemetry tracking data. We simulated movement data for three archetypical reptile species: a highly mobile active hunter, an ambush predator with long-distance moves and long-term sheltering periods, and an ambush predator with short-distance moves and short-term sheltering periods. We compared traditionally used estimators, MCP and KDE, with dBBMMs, across eight feasible VHF field sampling regimes for reptiles, varying from one data point every four daylight hours, to once per month.ResultsAlthough originally designed for GPS tracking studies, dBBMMs outperformed MCPs and KDE href across all tracking regimes in accurately revealing movement pathways, with only KDE LSCV performing comparably at some higher frequency sampling regimes. However, the LSCV algorithm failed to converge with these high-frequency regimes due to high site fidelity, and was unstable across sampling regimes, making its use problematic for species exhibiting long-term sheltering behaviours. We found that dBBMMs minimized the effect of individual variation, maintained low error rates balanced between omission (false negative) and commission (false positive), and performed comparatively well even under low frequency sampling regimes (e.g., once a month).ConclusionsWe recommend dBBMMs as a valuable alternative to MCP and KDE methods for reptile VHF telemetry data, for research questions associated with space-use and movement behaviours within the study period: they work under contemporary tracking protocols and provide more stable estimates. We demonstrate for the first time that dBBMMs can be applied confidently to low-resolution tracking data, while improving comparisons across regimes, individuals, and species.

Seabed modelling with a least-squares support-vector machine and sample cross-validation

This paper reports on a study to model seabed surfaces using the least-squares support-vector machine algorithm with a sample cross-validation (CV) method. It starts with a brief overview of the sample selection method of the algorithm and gives two important characteristics of the algorithm. It then focuses on the theory of sample CV and the steps of sample selection using this theory. Finally, to verify the validity of the sample CV method in sample selection for the algorithm, the measured multi-beam bathymetric data are selected to calculate and analyse. It is concluded that the sample CV method can reasonably screen out the sounding training samples with a large contribution to the function model, making the constructed function model more reasonable.

Bandwidth selection for kernel density estimation: a Hermite series-based direct plug-in approach

In this paper we propose a new class of Hermite series-based direct plug-in bandwidth selectors for kernel density estimation and we describe their asymptotic and finite sample behaviours. Unlike the direct plug-in bandwidth selectors considered in the literature, the proposed methodology does not involve multistage strategies and reference distributions are no longer needed. The new bandwidth selectors show a good finite sample performance when the underlying probability density function presents not only ‘easy-to-estimate’ but also ‘hard-to-estimate’ distribution features. This quality, that is not shared by other widely used bandwidth selectors as the classical plug-in or the least-square cross-validation methods, is the most significant aspect of the Hermite series-based direct plug-in approach to bandwidth selection.

Kernel smoothed probability mass functions for ordered datatypes

We propose a kernel function for ordered categorical data that overcomes limitations present in ordered kernel functions appearing in the literature on the estimation of probability mass functions for multinomial ordered data. Some limitations arise from assumptions made about the support of the underlying random variable. Furthermore, many existing ordered kernel functions lack a particularly appealing property, namely the ability to deliver discrete uniform probability estimates for some value of the smoothing parameter. We propose an asymmetric empirical support kernel function that adapts to the data at hand and possesses certain desirable features. There are no difficulties arising from zero counts caused by gaps in the data while it encompasses both the empirical proportions and the discrete uniform probabilities at the lower and upper boundaries of the smoothing parameter. We propose likelihood and least-squares cross-validation for smoothing parameter selection and study their asymptotic and finite-sample behaviour.

Robust kernels for kernel density estimation

The likelihood cross validation (LCV) and the least square cross validation (LSCV) are two commonly used methods of bandwidth selection in kernel density estimation. The LCV is generally more efficient but sensitive to tail-heaviness; in contrast, the LSCV fares well for heavy-tailed distributions but tends to undersmooth and is generally more variable. In this study, we propose two novel kernel functions that are robust against heavy-tailed distributions and at the same time adaptive with respect to the sample tail-heaviness in a data-driven manner. The proposed method is simple to implement. Our simulations show that it performs similarly to the LCV for regular- and thin-tailed distributions and outperforms the LSCV for heavy-tailed distributions, suggesting that it can be an overall competitive alternative. An empirical application to income distribution estimation is provided.

Kernel density estimation model for wind speed probability distribution with applicability to wind energy assessment in China

A kernel density estimation (KDE) model for the probability distribution of wind speed (PDWS) is proposed in this paper for application to wind energy assessment (WEA) in China. Four bandwidth selectors, including normal scale (NS), plug in, biased cross-validation, and least-square cross validation, are proposed for the KDE model. Popular parametric distribution models are also introduced for comparisons with the KDE models. Based on five-year day-average wind speed data from 698 nationwide wind stations in China, the performance and robustness of both parametric and KDE models were evaluated comprehensively on the regional scale. Wind power density (WPD) and wind turbine power output (WTPO), which are the priorities in WEA, are subsequently calculated based on the estimated PDWS models. The ranking results of four individual metrics and one comprehensive metric indicate that the four KDE models outperform the parametric models in fitting the PDWS. The KDE-NS model performs the best compared to the other three KDE models. In addition to the KDE models, the generalized gamma and generalized extreme values were considered as better parametric models in fitting the PDWS. KDE models also performed well in WPD estimation, especially the KDE-NS model with a mean absolute percentage error (MAPE) value as low as 2%. Some parametric models, i.e., Johnson SB and Wakeby, which are not outstanding in PDWS fitting, however perform well in WPD estimation, and their MAPE values can be controlled to remain within 3%. This indicates that the result of the PDWS is not completely equivalent to that of WPD estimation. The WPD and WTPO in most of China's interior areas are less than 40W/m2 and 1.2 GWh, respectively. In the eastern coastal areas, middle and eastern Inner Mongolia, and some western provinces, the WPD and WTPO are relatively higher, and can reach or exceed 240W/m2 and 3.5 GWh, respectively.

A bootstrap approach for bandwidth selection in estimating conditional efficiency measures

Conditional efficiency measures are needed when the production process does not depend only on the inputs and outputs, but may be influenced by external factors and/or environmental variables (Z). They are estimated by means of a nonparametric estimator of the conditional distribution function of the inputs and outputs, conditionally on values of Z. For doing this, smoothing procedures and smoothing parameters, the bandwidths, are involved. So far, Least Squares Cross Validation (LSCV) methods have been used, which have been proven to provide bandwidths with optimal rates for estimating conditional distributions. In efficiency analysis, the main interest is in the estimation of the conditional efficiency score, which typically depends on the boundary of the support of the distribution and not on the full conditional distribution. In this paper, we show indeed that the rate for the bandwidths which is optimal for estimating conditional distributions, may not be optimal for the estimation of the efficiency scores. We propose hence a new approach based on the bootstrap which overcomes these difficulties. We analyze and compare, through Monte Carlo simulations, the performances of LSCV techniques with our bootstrap approach in finite samples. As expected, our bootstrap approach shows generally better performances and is more robust to the various Monte Carlo scenarios analyzed. We also illustrate our methodology through an empirical example using an US Aggressive-Growth Mutual Funds data set.

Theoretical properties of bandwidth selectors for kernel density estimation on the circle

We derive the asymptotic properties of the least squares cross-validation (LSCV) selector and the direct plug-in rule (DPI) selector in the kernel density estimation for circular data. The DPI selector has a convergence rate of $$O(n^{-5/14})$$, although the rate of the LSCV selector is $$O(n^{-1/10})$$. Our simulation shows that the DPI selector has more stability than the LSCV selector for small and large sample sizes. In other words, the DPI selector outperforms the LSCV selector in theoretical and practical performance.

Robust Likelihood Cross-Validation for Kernel Density Estimation

Likelihood cross-validation for kernel density estimation is known to be sensitive to extreme observations and heavy-tailed distributions. We propose a robust likelihood-based cross-validation method to select bandwidths in multivariate density estimations. We derive this bandwidth selector within the framework of robust maximum likelihood estimation. This method establishes a smooth transition from likelihood cross-validation for nonextreme observations to least squares cross-validation for extreme observations, thereby combining the efficiency of likelihood cross-validation and the robustness of least-squares cross-validation. We also suggest a simple rule to select the transition threshold. We demonstrate the finite sample performance and practical usefulness of the proposed method via Monte Carlo simulations and a real data application on Chinese air pollution.

Fast bandwidth selection for estimation of the pair correlation function

ABSTRACTKernel estimation is a popular approach to estimation of the pair correlation function which is a fundamental spatial point process characteristic. Least squares cross validation was suggested by Guan [A least-squares cross-validation bandwidth selection approach in pair correlation function estimations. Statist Probab Lett. 2007;77(18):1722–1729] as a data-driven approach to select the kernel bandwidth. The method can, however, be computationally demanding for large point pattern data sets. We suggest a modified least squares cross validation approach that is asymptotically equivalent to the one proposed by Guan but is computationally much faster.

On discrete Epanechnikov kernel functions

Least-squares cross-validation is commonly used for selection of smoothing parameters in the discrete data setting; however, in many applied situations, it tends to select relatively small bandwidths. This tendency to undersmooth is due in part to the geometric weighting scheme that many discrete kernels possess. This problem may be avoided by using alternative kernel functions. Specifically, discrete versions (both unordered and ordered) of the popular Epanechnikov kernel do not have rapidly decaying weights. The analytic properties of these kernels are contrasted with commonly used discrete kernel functions and their relative performance is compared using both simulated and real data. The simulation and empirical results show that these kernel functions generally perform well and in some cases demonstrate substantial gains in terms of mean squared error.

Identifying Uncertainty Distributions and Confidence Regions of Power Plant Parameters

Power system operators, when obtaining a model’s parameter estimates; require additional information to guide their decision on a model’s acceptance. This information has to establish a relationship between the estimates and the chosen model in the parameter space. For this purpose, this paper proposes to extend the usage of the particle filter (PF) as a method for the identification of power plant parameters; and the parameters’ confidence intervals, using measurements. Taking into consideration that the PF is based on the Bayesian filtering concept, the results returned by the filter contain more information about the model and its parameters than usually considered by power system operators. In this paper the samples from the multi-modal posterior distribution of the estimate are used to identify the distribution shape and associated confidence intervals of estimated parameters. Three methods [rule of thumb, least-squares cross validation, plug-in method (HSJM)] for standard deviation (bandwidth) selection of the Gaussian mixture distribution are compared with the uni-modal Gaussian distribution of the parameter estimate. The applicability of the proposed method is demonstrated using field measurements and synthetic data from simulations of a Greek power plant model. The distributions are observed for different system operation conditions that consider different types of noise. The method’s applicability for model validation is also discussed.

A weighted least-squares cross-validation bandwidth selector for kernel density estimation

ABSTRACTSince the late 1980s, several methods have been considered in the literature to reduce the sample variability of the least-squares cross-validation bandwidth selector for kernel density estimation. In this article, a weighted version of this classical method is proposed and its asymptotic and finite-sample behavior is studied. The simulation results attest that the weighted cross-validation bandwidth performs quite well, presenting a better finite-sample performance than the standard cross-validation method for “easy-to-estimate” densities, and retaining the good finite-sample performance of the standard cross-validation method for “hard-to-estimate” ones.

Cross-validation Revisited

Data-based choice of the bandwidth is an important problem in kernel density estimation. The pseudo-likelihood and the least-squares cross-validation bandwidth selectors are well known, but widely criticized in the literature. For heavy-tailed distributions, the L1 distance between the pseudo-likelihood-based estimator and the density does not seem to converge in probability to zero with increasing sample size. Even for normal-tailed densities, the rate of L1 convergence is disappointingly slow. In this article, we report an interesting finding that with minor modifications both the cross-validation methods can be implemented effectively, even for heavy-tailed densities. For both these estimators, the L1 distance (from the density) are shown to converge completely to zero irrespective of the tail of the density. The expected L1 distance also goes to zero. These results hold even in the presence of a strongly mixing-type dependence. Monte Carlo simulations and analysis of the Old Faithful geyser data suggest that if implemented appropriately, contrary to the traditional belief, the cross-validation estimators compare well with the sophisticated plug-in and bootstrap-based estimators.