Generalized Cross-validation Method Research Articles

Nonlinear GCV and quasi-GCV for shrinkage models

The generalized cross-validation (GCV) method has been a popular technique for the selection of tuning parameters for smoothing and penalty, and has been a standard tool to select tuning parameters for shrinkage models in recent works. Its computational ease and robustness compared to the cross-validation method makes it competitive for model selection as well. It is well known that the GCV method performs well for linear estimators, which are linear functions of the response variable, such as ridge estimator. However, it may not perform well for nonlinear estimators since the GCV emphasizes linear characteristics by taking the trace of the projection matrix. This paper aims to explore the GCV for nonlinear estimators and to further extend the results to correlated data in longitudinal studies. We expect that the nonlinear GCV and quasi-GCV developed in this paper will provide similar tools for the selection of tuning parameters in linear penalty models and penalized GEE models.

Period Analysis of Variable Stars by Robust Smoothing

Summary The objective is to estimate the period and the light curve (or periodic function) of a variable star. Previously, several methods have been proposed to estimate the period of a variable star, but they are inaccurate especially when a data set contains outliers. We use a smoothing spline regression to estimate the light curve given a period and then find the period which minimizes the generalized cross-validation (GCV). The GCV method works well, matching an intensive visual examination of a few hundred stars, but the GCV score is still sensitive to outliers. Handling outliers in an automatic way is important when this method is applied in a ‘data mining’ context to a vary large star survey. Therefore, we suggest a robust method which minimizes a robust cross-validation criterion induced by a robust smoothing spline regression. Once the period has been determined, a nonparametric method is used to estimate the light curve. A real example and a simulation study suggest that the robust cross-validation and GCV methods are superior to existing methods.

Hybrid near-field acoustic holography.

Hybrid near-field acoustical holography (NAH) is developed for reconstructing acoustic radiation from an arbitrary object in a cost-effective manner. This hybrid NAH is derived from a modified Helmholtz equation least squares (HELS) formula that expands the acoustic pressure in terms of outgoing and incoming waves. The expansion coefficients are determined by solving an overdetermined linear system of equations obtained by matching the assumed-form solution to measured acoustic pressures through the least squares. Measurements are taken over a conformal surface around a source at close range so that the evanescent waves can be captured. Next, the modified HELS is utilized to regenerate as much acoustic pressures on the conformal surface as necessary and take them as input to the Helmholtz integral formulation implemented numerically by boundary element method (BEM). The acoustic pressures and normal velocities on the source surface are reconstructed by using a modified Tikhnov regularization (TR) with its regularization parameter determined by generalized cross validation (GCV) method. Results demonstrate that this hybrid NAH combines the advantages of HELS and inverse BEM. This is because a majority of the input data are regenerated but not measured, thus the efficiency of reconstruction is greatly enhanced. Meanwhile, the accuracy of reconstruction is ensured by the Helmholtz integral theory and modified TR together with GCV method, provided that HELS converges fast enough on the measurement surface. Numerical examples of reconstructing acoustic quantities on the surface of a simplified engine block are demonstrated. [Work supported by NSF.]

Computation of spherical harmonic coefficients from gravity gradiometry data to be acquired by the GOCE satellite: regularization issues

The issue of optimal regularization is investigated in the context of the processing of satellite gravity gradiometry (SGG) data that will be acquired by the GOCE (Gravity Field and Steady-State Ocean Circulation Explorer) satellite. These data are considered as the input for determination of the Earth’s gravity field in the form of a series of spherical harmonics. Exploitation of a recently developed fast processing algorithm allowed a very realistic setup of the numerical experiments to be specified, in particular: a non-repeat orbit; 1-s sampling rate; half-year duration of data series; and maximum degree and order set to 300. The first goal of the study is to compare different regularization techniques (regularization matrices). The conclusion is that the first-order Tikhonov regularization matrix (the elements are practically proportional to the degree squared) and the Kaula regularization matrix (the elements are proportional to the fourth power of the degree) are somewhat superior to other regularization techniques. The second goal is to assess the generalized cross-validation method for the selection of the regularization parameter. The inference is that the regularization parameter found this way is very reasonable. The time expenditure required by the generalized cross-validation method remains modest even when a half-year set of SGG data is considered. The numerical study also allows conclusions to be drawn regarding the quality of the Earth’s gravity field model that can be obtained from the GOCE SGG data. In particular, it is shown that the cumulative geoid height error between degrees 31 and 200 will not exceed 1 cm.

On Gauss quadrature and partial cross validation

New estimators of expected values Ew( X) of functions of a random variable X are introduced. The new estimators are based on Gauss quadrature, a numerical method frequently used to approximate integrals over finite intervals. The estimators need a small number of numerical evaluations and hence are useful in partial cross validation (PCV) a numerical method for finding optimal smoothing parameters in nonparametric curve estimation. The PCV can considerably reduce the computational cost of the generalized cross validation method typically used to determine the optimal smoothing parameter.

Measurement of emulsion droplet sizes using PFG NMR and regularization methods

The droplet size distributions of emulsions have been measured using pulsed field gradient (PFG) nuclear magnetic resonance (NMR) for many years. This technique finds particular application with emulsions that are concentrated and/or opaque, since such emulsion systems are difficult to characterize by other methods. Most studies employing PFG techniques assume a lognormal form when extracting the droplet size distribution from the experimental data. It is clearly desirable to retrieve a droplet size distribution from the experimental data without assuming such a functional form. This is achieved for the first time using regularization techniques. Regularization based on the distribution area and on its second derivative are compared and assessed along with the following techniques for selecting the optimal regularization parameter: the L-curve method, generalized cross validation (GCV), and the discrepancy principle. Regularization is applied to both simulated data sets and experimental data. It is found that when the experimental error can be estimated accurately, the discrepancy principle with area regularization is the best approach. When the error is not known the GCV method, with second derivative regularization and allowing only nonnegative values, is most effective.

A simple generalised crossvalidation method of span selection for periodogram smoothing

A consistent estimator for the spectral density of a stationary random process can be obtained by smoothing the periodograms across frequency. An important component of smoothing is the choice of the span. Lee ( 1997) proposed a span selector that was erroneously claimed to be unbiased for the mean squared error. The naive use of mean squared error has some important drawbacks in this context because the variance of the periodogram depends on its mean, i.e. the spectrum. We propose a new span selector based on the generalised crossvalidation function derived from the gamma deviance. This criterion, originally developed for use in fitting generalised additive models. utilises the approximate full likelihood of periodograms, which asymptotically behave like independently distributed chi-squared. i.e. gamma. random variables. The proposed span selector is very simple and easily implemented. Simulation results suggest that the proposed span selector generally outperforms those obtained under a mean squared error criterion.

Generalized approach for accelerated maximum likelihood based image restoration applied to three-dimensional fluorescence microscopy.

For deconvolution applications in three-dimensional microscopy we derived and implemented a generic, accelerated maximum likelihood image restoration algorithm. A conjugate gradient iteration scheme was used considering either Gaussian or Poisson noise models. Poisson models are better suited to low intensity fluorescent image data; typically, they show smaller restoration errors and smoother results. For the regularization, we modified the standard Tikhonov method. However, the generic design of the algorithm allows for more regularization approaches. The Hessian matrix of the restoration functional was used to determine the step size. We compared restoration error and convergence behaviour between the classical line-search and the Hessian matrix method. Under typical working conditions, the restoration error did not increase over that of the line-search and the speed of convergence did not significantly decrease allowing for a twofold increase in processing speed. To determine the regularization parameter, we modified the generalized cross-validation method. Tests that were done on both simulated and experimental fluorescence wide-field data show reliable results.

CROSS-REFERENCE MAXIMUM LIKELIHOOD ESTIMATE RECONSTRUCTION FOR POSITRON EMISSION TOMOGRAPHY

Maximum likelihood estimate (MLE) is a widely used approach for PET image reconstruction. However, it has been shown that reconstructing emission tomography based on MLE without regularization would result in noise and edge artifacts. In the attempt to regularize the maximum likelihood estimate, we propose a new and efficient method in this paper to incorporate the correlated but possibly incomplete structure information which may be derived from expertise, PET systems or other imaging modalities. A mean estimate smoothing the MLE locally within each region of interest is derived according to the boundaries provided by the structure information. Since the boundaries may not be correct, a penalized MLE using the mean estimate is sought. The resulting reconstruction is called a cross-reference maximum likelihood estimate (CRMLE). The CRMLE can be obtained through a modified EM algorithm, which is computation and storage efficient. By borrowing the strength from the correct portion of boundary information, the CRMLE is able to extract the useful information to improve reconstruction for different kinds of incomplete and incorrect boundaries in Monte Carlo studies. The proposed CRMLE algorithm not only reduces the estimation errors, but also preserves the correct boundaries. The penalty parameters can be selected through human interactions or automatically data-driven methods, such as the generalized cross validation method.

A computationally efficient superresolution image reconstruction algorithm

Superresolution reconstruction produces a high-resolution image from a set of low-resolution images. Previous iterative methods for superresolution had not adequately addressed the computational and numerical issues for this ill-conditioned and typically underdetermined large scale problem. We propose efficient block circulant preconditioners for solving the Tikhonov-regularized superresolution problem by the conjugate gradient method. We also extend to underdetermined systems the derivation of the generalized cross-validation method for automatic calculation of regularization parameters. The effectiveness of our preconditioners and regularization techniques is demonstrated with superresolution results for a simulated sequence and a forward looking infrared (FLIR) camera image sequence.

Efficient generalized cross-validation with applications to parametric image restoration and resolution enhancement

In many image restoration/resolution enhancement applications, the blurring process, i.e., point spread function (PSF) of the imaging system, is not known or is known only to within a set of parameters. We estimate these PSF parameters for this ill-posed class of inverse problem from raw data, along with the regularization parameters required to stabilize the solution, using the generalized cross-validation method (GCV). We propose efficient approximation techniques based on the Lanczos algorithm and Gauss quadrature theory, reducing the computational complexity of the GCV. Data-driven PSF and regularization parameter estimation experiments with synthetic and real image sequences are presented to demonstrate the effectiveness and robustness of our method.

The influence of the regularization parameter and the first estimate on the performance of tikhonov regularized non-linear image restoration algorithms

This paper reports studies on the influence of the regularization parameter and the first estimate on the performance of iterative image restoration algorithms. We discuss regularization parameter estimation methods that have been developed for the linear Tikhonov-Miller filter to restore images distorted by additive Gaussian noise. We have performed experiments on synthetic data to show that these methods can be used to determine the regularization parameter of non-linear iterative image restoration algorithms, which we use to restore images contaminated by Poisson noise. We conclude that the generalized cross-validation method is an efficient method to determine a value of the regularization parameter close to the optimal value. We have also derived a method to estimate the regularization parameter of a Tikhonov regularized version of the Richardson-Lucy algorithm. These iterative image restoration algorithms need a first estimate to start their iteration. An obvious and frequently used choice for the first estimate is the acquired image. However, the restoration algorithm could be sensitive to the noise present in this image, which may hamper the convergence of the algorithm. We have therefore compared various choices of first estimates and tested the convergence of various iterative restoration algorithms. We found that most algorithms converged for most choices, but that smoothed first estimates resulted in a faster convergence.

Optimal wavelet filter banks for regularized restoration of noisy images

Regularized image restoration methods efficiently handle the ill-posed problem of image restoration. Nevertheless, the issue of selecting the regularization parameter as well as the smoothing filter still constitutes an open research topic. A model of regularized image restoration is introduced and analyzed in this paper. The proposed model assumes that wavelet filter banks replace the smoothing filter of conventional regularized restoration. Filter factorizations for the optimal design of wavelet filter banks using the generalized-cross-validation (GCV) criterion are presented, and novel expressions of the influence matrix, which is used to calculate the GCV error, are derived. The error of the GCV method is expressed in terms of the modulation matrix of the filter bank and the modulation vector of the degradation filter. The expressions are given in general form for optimal wavelet filter bank design upon arbitrary sampling lattices. The numerical examples of image restoration using the proposed method that are presented indicate significant signal-to-noise ratio improvement, ΔSNR, compared to image restoration methods that employ the Laplacian as the smoothing filter.

Kronecker product and SVD approximations in image restoration

Image restoration applications often result in ill-posed least squares problems involving large, structured matrices. One approach used extensively is to restore the image in the frequency domain, thus providing fast algorithms using fast Fourier transforms (FFTs). This is equivalent to using a circulant approximation to a given matrix. Iterative methods may also be used effectively by exploiting the structure of the matrix. While iterative schemes are more expensive than FFT-based methods, it has been demonstrated that they are capable of providing better restorations. As an alternative, we propose an approximate singular value decomposition (SVD), which can be used in a variety of applications. Used as a direct method, the computed restorations are comparable to iterative methods but are computationally less expensive. In addition, the approximate SVD may be used with the generalized cross validation method to choose regularization parameters. It is also demonstrated that the approximate SVD can be an effective preconditioner for iterative methods.

Smoothing Spline Models with Correlated Random Errors

Spline-smoothing techniques are commonly used to estimate the mean function in a nonparametric regression model. Their performances depend greatly on the choice of smoothing parameters. Many methods of selecting smoothing parameters such as generalized maximum likelihood (GML), generalized cross-validation (GCV), and unbiased risk (UBR), have been developed under the assumption of independent observations. They tend to underestimate smoothing parameters when data are correlated. In this article, I assume that observations are correlated and that the correlation matrix depends on a parsimonious set of parameters. I extend the GML, GCV, and UBR methods to estimate the smoothing parameters and the correlation parameters simultaneously. I also relate a smoothing spline model to three mixed-effects models. These relationships show that the smoothing spline estimates evaluated at design points are best linear unbiased prediction (BLUP) estimates and that the GML estimates of the smoothing parameters and the correlation parameters are restricted maximum likelihood (REML) estimates. They also provide a way to fit a spline model with correlated errors using the SAS procedure proc mixed. Simulations are conducted to evaluate and compare the performance of the GML, GCV, UBR methods and the method proposed by Diggle and Hutchinson. The GML method is recommended, because it is stable and works well in all simulations. It performs better than other methods, especially when the sample size is not large. I illustrate my methods with applications to time series data and to spatial data.

Smoothing Spline Models with Correlated Random Errors

Abstract Spline-smoothing techniques are commonly used to estimate the mean function in a nonparametric regression model. Their performances depend greatly on the choice of smoothing parameters. Many methods of selecting smoothing parameters such as generalized maximum likelihood (GML), generalized cross-validation (GCV), and unbiased risk (UBR), have been developed under the assumption of independent observations. They tend to underestimate smoothing parameters when data are correlated. In this article, I assume that observations are correlated and that the correlation matrix depends on a parsimonious set of parameters. I extend the GML, GCV, and UBR methods to estimate the smoothing parameters and the correlation parameters simultaneously. I also relate a smoothing spline model to three mixed-effects models. These relationships show that the smoothing spline estimates evaluated at design points are best linear unbiased prediction (BLUP) estimates and that the GML estimates of the smoothing parameters and the correlation parameters are restricted maximum likelihood (REML) estimates. They also provide a way to fit a spline model with correlated errors using the SAS procedure proc mixed. Simulations are conducted to evaluate and compare the performance of the GML, GCV, UBR methods and the method proposed by Diggle and Hutchinson. The GML method is recommended, because it is stable and works well in all simulations. It performs better than other methods, especially when the sample size is not large. I illustrate my methods with applications to time series data and to spatial data.

Optimal regularization of an inverse dynamics problem

Solving the inverse dynamics problem using Tikhonov regularization and dynamic programming requires the selection of an optimal smoothing parameter. One popular method for choosing the smoothing parameter is the generalized cross-validation method. This method works very well but is computationally expensive for large systems. Another method for selecting an optimal smoothing parameter is the L-curve method. This L-curve is easily computed and may prove very useful for large systems which preclude other methods. In this paper we investigate both of these methods for a simple cantilever beam that is subjected to an unknown force. A simulated strain measurement is used to provide the experimental data.

Generalized cross validation for wavelet thresholding

Noisy data are often fitted using a smoothing parameter, controlling the importance of two objectives that are opposite to a certain extent. One of these two is smoothness and the other is closeness to the input data. The optimal value of this paramater minimizes the error of the result (as compared to the unknown, exact data), usually expressed in the L 2 norm. This optimum cannot be found exactly, simply because the exact data are unknown. In spline theory, the generalized cross validation (GCV) technique has proved to be an effective (though rather slow) statistical way for estimating this optimum. On the other hand, wavelet theory is well suited for signal and image processing. This paper investigates the possibility of using GCV in a noise reduction algorithm, based on wavelet-thresholding, where the threshold can be seen as a kind of smoothing parameter. The GCV method thus allows choosing the (nearly) optimal threshold, without knowing the noise variance. Both an original theoretical argument and practical experiments are used to show this successful combination.

Trade-offs in the identification of surface displacements

This paper deals with the identification of the displacements on a face of an elastic pad, from several types of stress measurements at interior points. The relation between surface displacements and interior stress is modelled via integral relations and approximated by the Boundary Element Method: the (ill-posed) continuous inverse problem is thus reduced to the inversion of input-output data for a discrete model. The ill-posedness is overcome by regularization. Two methods are considered: augmenting the output error by a regularizing function (Tikhonov type regularization), and truncating the Singular Value Decomposition of the model matrix. In both approaches the choice of the regularization parameter is critical. Extensive numerical experiments show that the Tikhonov type regularization, together with the Generalized Cross Validation method for the choice of the regularization parameter, gives the best results. The computational cost of the identification procedure does not depend critically on the dimension of the discrete model; the measurement of all the stress component does not improve the estimate with respect to technologically simpler measurement strategies, provided that high dimensional models are used. Such conclusions may be helpful in the design of tactile sensing systems,

Characterization of the effects of an airborne mixture of chemicals on the respiratory tract and smoothing polynomial spline analysis of the data.

We expanded a previously published (Vijayaraghavan et al. 1994) computerized system to analyze the breathing pattern of unanesthetized mice in order better to recognize and quantify the effects of an airborne mixture of chemicals at three different levels of the respiratory tract. The airborne chemical mixture used was a machining fluid. Such fluids are widely used in industry and a large number of workers are exposed to these airborne mixtures. We found this mixture to be capable of inducing three types of effects on the respiratory tract: sensory irritation of the upper respiratory tract (S), airflow limitation along the conducting airways (A) and pulmonary irritation (P). Depending upon the exposure concentration, mainly S or P effects were obtained but an A effect was also identified. The three types of effects occurred at various times during the exposures and, furthermore, within a group of exposed animals some exhibited one type of effect while others exhibited another type. In order to analyze such complex data sets, two statistical methods for smoothing polynomial splines were utilized: the maximum likelihood (ML) method and generalized cross validation (GCV) method. The results indicated the previous methods used to characterize a single effect of airborne chemicals can now be extended to evaluate mixtures likely to induce multiple types of effects. However, statistical analysis methods, either the ML or GCV methods, or other appropriate methods are needed to evaluate the responses obtained due to the complex effects that a mixture can induce in comparison to single chemicals.