Unbiased Risk Estimate Research Articles

Model averaging for interval-valued data

In recent years, model averaging, by which estimates are obtained based on not one single model but a weighted ensemble of models, has received growing attention as an alternative to model selection. To-date, methods for model averaging have been developed almost exclusively for point-valued data, despite the fact that interval-valued data are commonplace in many applications and the substantial body of literature on estimation and inference methods for interval-valued data. This paper focuses on the special case of interval time series data, and assumes that the mid-point and log-range of the interval values are modelled by a two-equation vector autoregressive with exogenous covariates (VARX) model. We develop a methodology for combining models of varying lag orders based on a weight choice criterion that minimises an unbiased estimator of the squared error risk of the model average estimator. We prove that this method yields predictors of mid-points and ranges with an optimal asymptotic property. In addition, we develop a method for correcting the range forecasts, taking into account the forecast error variance. An extensive simulation experiment examines the performance of the proposed model averaging method in finite samples. We apply the method to an interval-valued data series on crude oil future prices.

Crack Growth Signal Processing Approach Combining Wavelet Threshold Denoising and Variable Amplitude DCPD Technique

The direct current potential drop (DCPD) method is widely used in laboratory environments to monitor the crack initiation and propagation of specimens. In this study, an anti-interference signal processing approach, combining wavelet threshold denoising and a variable current amplitude DCPD signal synthesis technique, was proposed. Adaptive wavelet threshold denoising using Stein’s unbiased risk estimate was applied to the main potential drop signal and the reference potential signal under two different current amplitudes to reduce the interference caused by noise. Thereafter, noise-reduced signals were synthesized to eliminate the time-varying thermal electromotive force. The multiplicative interference signal was eliminated by normalizing the main potential drop signal and the reference potential drop signal. This signal processing approach was applied to the crack growth monitoring data of 316 L stainless steel compact tension specimens in a laboratory environment, and the signal processing results of static cracks and propagation cracks under different load conditions were analyzed. The results showed that the proposed approach can significantly improve the signal-to-noise ratio as well as the accuracy and resolution of the crack growth measurement.

Degrees of Freedom in Functional Principal Components Analysis

This paper develops the analytical form of the degrees of freedom in functional principal components analysis. Under the framework of unbiased risk estimation, we derive an unbiased estimator with a clear analytical formula for the degrees of freedom in the one-way penalized functional principal components analysis paradigm. Specifically, a new analytical formula incorporating binary smoothing parameters is also derived based on the singular value decomposition and half-smoothed method regarding the two-way penalized functional principal components analysis framework. The performance of our procedures is demonstrated by simulation studies.

Finite Sample Improvement of Akaike’s Information Criterion

Considering the selection of frequency histograms, we propose a modification of Akaike's Information Criterion that avoids overfitting, even when the sample size is small. We call this correction an over-penalization procedure. We emphasize that the principle of unbiased risk estimation for model selection can indeed be improved by addressing excess risk deviations in the design of the penalization procedure. On the theoretical side, we prove sharp oracle inequalities for the Kullback-Leibler divergence. These inequalities are valid with positive probability for any sample size and include the estimation of unbounded log-densities. Along the proofs, we derive several analytical lemmas related to the Kullback-Leibler divergence, as well as concentration inequalities, that are of independent interest. In a simulation study, we also demonstrate state-of-the-art performance of our over-penalization criterion for bin size selection, in particular outperforming AICc procedure.

Re-calibrating pure risk integrating individual data from two-phase studies with external summary statistics.

Accurate risk assessment is critical in clinical decision-making. It entails the projected risk based on a risk prediction model agreeing with the observed risk in the target cohort. However, the model often over- or under-estimates the risk. Building a new model for the target cohort would be ideal but costly. It is therefore of great interest to recalibrate an existing model for the target cohort. Existing methods have been proposed to recalibrate the model by leveraging the disease incidence rates from the target cohort. However, they assume the same covariate distribution across cohorts and when the assumption is violated, the recalibrated model can be substantially biased. Further, recalibration is also complicated by the two-phase sampling design that is commonly used for developing risk prediction models. In this paper, we develop a weighted estimating-equation approach accounting for the two-phase design and combine it with a weighted empirical likelihood that leverages the summary information on both disease incidence rates and covariates from the target cohort. We provide a resampling-based inference procedure. Our extensive simulation results show that using the summary information from the target population, the proposed recalibration method yields nearly unbiased risk estimates under a wide range of scenarios. An application to a colorectal cancer study also illustrates that the proposed method yields a well-calibrated model in the targetcohort.

STRETCHING THE NET: MULTIDIMENSIONAL REGULARIZATION

This paper derives asymptotic risk (expected loss) results for shrinkage estimators with multidimensional regularization in high-dimensional settings. We introduce a class of multidimensional shrinkage estimators (MuSEs), which includes the elastic net, and show that—as the number of parameters to estimate grows—the empirical loss converges to the oracle-optimal risk. This result holds when the regularization parameters are estimated empirically via cross-validation or Stein’s unbiased risk estimate. To help guide applied researchers in their choice of estimator, we compare the empirical Bayes risk of the lasso, ridge, and elastic net in a spike and normal setting. Of the three estimators, we find that the elastic net performs best when the data are moderately sparse and the lasso performs best when the data are highly sparse. Our analysis suggests that applied researchers who are unsure about the level of sparsity in their data might benefit from using MuSEs such as the elastic net. We exploit these insights to propose a new estimator, the cubic net, and demonstrate through simulations that it outperforms the three other estimators for any sparsity level.

Estimation under matrix quadratic loss and matrix superharmonicity

Summary We investigate estimation of a normal mean matrix under the matrix quadratic loss. Improved estimation under the matrix quadratic loss implies improved estimation of any linear combination of the columns under the quadratic loss. First, an unbiased estimate of risk is derived and the Efron–Morris estimator is shown to be minimax. Next, a notion of matrix superharmonicity for matrix-variate functions is introduced and shown to have properties analogous to those of the usual superharmonic functions, which may be of independent interest. Then, it is shown that the generalized Bayes estimator with respect to a matrix superharmonic prior is minimax. We also provide a class of matrix superharmonic priors that includes the previously proposed generalization of Stein’s prior. Numerical results demonstrate that matrix superharmonic priors work well for low-rank matrices.

Second-order Stein: SURE for SURE and other applications in high-dimensional inference

Stein’s formula states that a random variable of the form z⊤f(z)−divf(z) is mean-zero for all functions f with integrable gradient. Here, divf is the divergence of the function f and z is a standard normal vector. This paper aims to propose a second-order Stein formula to characterize the variance of such random variables for all functions f(z) with square integrable gradient, and to demonstrate the usefulness of this second-order Stein formula in various applications. In the Gaussian sequence model, a remarkable consequence of Stein’s formula is Stein’s Unbiased Risk Estimate (SURE), an unbiased estimate of the mean squared risk for almost any given estimator μˆ of the unknown mean vector. A first application of the second-order Stein formula is an Unbiased Risk Estimate for SURE itself (SURE for SURE): an unbiased estimate providing information about the squared distance between SURE and the squared estimation error of μˆ. SURE for SURE has a simple form as a function of the data and is applicable to all μˆ with square integrable gradient, for example, the Lasso and the Elastic Net. In addition to SURE for SURE, the following statistical applications are developed: (1) upper bounds on the risk of SURE when the estimation target is the mean squared error; (2) confidence regions based on SURE and using the second-order Stein formula; (3) oracle inequalities satisfied by SURE-tuned estimates under a mild Lipschtiz assumption; (4) an upper bound on the variance of the size of the model selected by the Lasso, and more generally an upper bound on the variance of the empirical degrees-of-freedom of convex penalized estimators; (5) explicit expressions of SURE for SURE for the Lasso and the Elastic Net; (6) in the linear model, a general semiparametric scheme to de-bias a differentiable initial estimator for the statistical inference of a low-dimensional projection of the unknown regression coefficient vector, with a characterization of the variance after debiasing; and (7) an accuracy analysis of a Gaussian Monte Carlo scheme to approximate the divergence of functions f:Rn→Rn.

Honest Confidence Sets for High-Dimensional Regression by Projection and Shrinkage

The issue of honesty in constructing confidence sets arises in nonparametric regression. While optimal rate in nonparametric estimation can be achieved and utilized to construct sharp confidence sets, severe degradation of confidence level often happens after estimating the degree of smoothness. Similarly, for high-dimensional regression, oracle inequalities for sparse estimators could be utilized to construct sharp confidence sets. Yet, the degree of sparsity itself is unknown and needs to be estimated, which causes the honesty problem. To resolve this issue, we develop a novel method to construct honest confidence sets for sparse high-dimensional linear regression. The key idea in our construction is to separate signals into a strong and a weak group, and then construct confidence sets for each group separately. This is achieved by a projection and shrinkage approach, the latter implemented via Stein estimation and the associated Stein unbiased risk estimate. Our confidence set is honest over the full parameter space without any sparsity constraints, while its size adapts to the optimal rate of when the true parameter is indeed sparse. Moreover, under some form of a separation assumption between the strong and weak signals, the diameter of our confidence set can achieve a faster rate than existing methods. Through extensive numerical comparisons on both simulated and real data, we demonstrate that our method outperforms other competitors with big margins for finite samples, including oracle methods built upon the true sparsity of the underlying model.

Improved Generalized Cross-Validation and Unbiased Predictive Risk Estimator Methods Using the RGSVD: Application to Inversion of Potential Field Data

The inversion of potential field data has widely utilized the generalized cross-validation (GCV) and the unbiased predictive risk estimator (UPRE) methods to determine the regularization parameter. However, these two methods are time-consuming and it is difficult for them to determine the optimal linear search range including the optimal regularization. To solve these problems, this article improves the GCV and UPRE methods using the RGSVD (randomized generalized singular value decomposition) algorithm. The improved methods first use the randomized algorithm to compute an approximate generalized singular value decomposition (GSVD) with less computational time. Then, the optimal linear search range is determined based on the generalized singular values. Finally, the GCV and the UPRE functions are efficiently computed on the basis of the results from the RGSVD algorithm. In this way, the GCV and UPRE methods using the RGSVD algorithm are able to determine the optimal regularization parameter fast and effectively. One comparative test shows the effectiveness and efficiency of the GCV and the UPRE methods using the RGSVD algorithm.

AIC for Growth Curve Model with Monotone Missing Data

In this article, we consider an AIC for a one-sample version of the growth curve model when the dataset has a monotone pattern of missing observations. It is well known that the AIC can be regarded as an approximately unbiased estimator of the AIC-type risk defined by the expected -predictive likelihood. Here, the likelihood is based on the observed data. First, when the covariance matrix is known, we derive an AIC, which is an exact unbiased estimator of the AIC-type risk function. Next, when the covariance matrix is unknown, we derive a conventional AIC using the estimators based on the complete data set only. Finally, a numerical example is presented to illustrate our model selection procedure.

Computation of strains from stereo digital image correlation using principal component analysis

Stereo digital image correlation (stereo DIC), a full-field deformation measurement technique, is widely used in the mechanics community due to its accuracy, versatility, and ability to measure 3D displacements over specimen surfaces. There are several strategies reported in the literature to compute strains from DIC displacements. Among them, the one based on principal component analysis (PCA) offers a way to effectively demarcate noise from the data by identifying the number of dominant singular vectors that are further differentiated by fitting an appropriate polynomial function. Though this approach has been used to differentiate 2D-DIC displacements, its implementation in the stereo-DIC workflow has not been reported yet. Moreover, its two important parameters, namely, the number of dominant singular vectors and the order of polynomial function, are still selected based on a heuristic where the user assesses the shape of the singular vectors and values to choose these parameters. In the present work, we address these gaps by presenting our novel PCA-based strain computation approach to stereo DIC, and automate the selection of parameters in our method using existing routines such as Stein’s unbiased risk estimator (SURE) and Bayesian information criterion (BIC). We verify the accuracy of the proposed approach on synthetic and DIC Challenge datasets and validate it on data from two experiments: uniaxial tension test and hydraulic inflation test. We observe an excellent agreement between the computed strains and the ground truth in all the cases. The main benefits of the proposed method are two-fold: (a) it leads to convergence in virtual strain gauge study and (b) it is non-parametric because its key parameters are chosen adaptively based on the data at hand.

Fault detection for rotating machinery using translation-invariant higher-density wavelet packet sliding window block thresholding

The weak incipient fault feature information of mechanical equipment is often submerged in strong background noise, which increases the difficulty of feature extraction. Herein, a novel method called the translation-invariant higher-density wavelet packet sliding window block thresholding is proposed. A higher-density wavelet transform processes one scale function and two wavelet functions, which has an advantage over single wavelet transform in matching various signal feature information. The translation-invariant higher-density wavelet packet transform effectively eliminates the Gibbs phenomenon, and the fine division of frequency bands breaks through the limitations of the higher-density wavelet transform to obtain more comprehensive feature information. At the same time, to accommodate the periodicity of the impact feature signal and the correlation between the decomposition coefficients, a sliding window block thresholding method is proposed. The sliding window is set to independently retain the weak fault features and block thresholding is applied to select the optimal block length and threshold by minimizing the Stein unbiased risk estimator. The results of the simulation and experiment indicate that the proposed method can more effectively reduce the interference of background noise and impurity frequencies, and successfully extract the characteristic frequencies of incipient fault diagnosis.

ENSURE: ENSEMBLE STEIN'S UNBIASED RISK ESTIMATOR FOR UNSUPERVISED LEARNING.

Deep learning algorithms are emerging as powerful alternatives to compressed sensing methods, offering improved image quality and computational efficiency. Unfortunately, fully sampled training images may not be available or are difficult to acquire in several applications, including high-resolution and dynamic imaging. Previous studies in image reconstruction have utilized Stein's Unbiased Risk Estimator (SURE) as a mean square error (MSE) estimate for the image denoising step in an unrolled network. Unfortunately, the end-to-end training of a network using SURE remains challenging since the projected SURE loss is a poor approximation to the MSE, especially in the heavily undersampled setting. We propose an ENsemble SURE (ENSURE) approach to train a deep network only from undersampled measurements. In particular, we show that training a network using an ensemble of images, each acquired with a different sampling pattern, can closely approximate the MSE. Our preliminary experimental results show that the proposed ENSURE approach gives comparable reconstruction quality to supervised learning and a recent unsupervised learning method.

A recursive unbiased risk estimate for the analysis-based $$\ell _1$$ minimization

A recursive unbiased risk estimate for the analysis-based $$\ell _1$$ minimization

Evaluation of MRI Denoising Methods Using Unsupervised Learning.

In this paper we evaluate two unsupervised approaches to denoise Magnetic Resonance Images (MRI) in the complex image space using the raw information that k-space holds. The first method is based on Stein’s Unbiased Risk Estimator, while the second approach is based on a blindspot network, which limits the network’s receptive field. Both methods are tested on two different datasets, one containing real knee MRI and the other consists of synthetic brain MRI. These datasets contain information about the complex image space which will be used for denoising purposes. Both networks are compared against a state-of-the-art algorithm, Non-Local Means (NLM) using quantitative and qualitative measures. For most given metrics and qualitative measures, both networks outperformed NLM, and they prove to be reliable denoising methods.

Automated Data-Driven Selection of the Hyperparameters for Total-Variation-Based Texture Segmentation

Penalized least squares are widely used in signal and image processing. Yet, it suffers from a major limitation since it requires fine-tuning of the regularization parameters. Under assumptions on the noise probability distribution, Stein-based approaches provide unbiased estimator of the quadratic risk. The Generalized Stein Unbiased Risk Estimator is revisited to handle correlated Gaussian noise without requiring to invert the covariance matrix. Then, in order to avoid expansive grid search, it is necessary to design algorithmic scheme minimizing the quadratic risk with respect to regularization parameters. This work extends the Stein’s Unbiased GrAdient estimator of the Risk of Deledalle et al. (SIAM J Imaging Sci 7(4):2448–2487, 2014) to the case of correlated Gaussian noise, deriving a general automatic tuning of regularization parameters. First, the theoretical asymptotic unbiasedness of the gradient estimator is demonstrated in the case of general correlated Gaussian noise. Then, the proposed parameter selection strategy is particularized to fractal texture segmentation, where problem formulation naturally entails inter-scale and spatially correlated noise. Numerical assessment is provided, as well as discussion of the practical issues.

PULNS: Positive-Unlabeled Learning with Effective Negative Sample Selector

Positive-unlabeled learning (PU learning) is an important case of binary classification where the training data only contains positive and unlabeled samples. The current state-of-the-art approach for PU learning is the cost-sensitive approach, which casts PU learning as a cost-sensitive classification problem and relies on unbiased risk estimator for correcting the bias introduced by the unlabeled samples. However, this approach requires the knowledge of class prior and is subject to the potential label noise. In this paper, we propose a novel PU learning approach dubbed PULNS, equipped with an effective negative sample selector, which is optimized by reinforcement learning. Our PULNS approach employs an effective negative sample selector as the agent responsible for selecting negative samples from the unlabeled data. While the selected, likely negative samples can be used to improve the classifier, the performance of classifier is also used as the reward to improve the selector through the REINFORCE algorithm. By alternating the updates of the selector and the classifier, the performance of both is improved. Extensive experimental studies on 7 real-world application benchmarks demonstrate that PULNS consistently outperforms the current state-of-the-art methods in PU learning, and our experimental results also confirm the effectiveness of the negative sample selector underlying PULNS.

An efficient variance estimator for cross-validation under partition sampling

This paper concerns the problem of variance estimation of cross-validation. We consider the unbiased cross-validation risk estimate in the form of a general U-statistic and focus on estimating the variance of the U-statistic risk score. We propose an efficient variance estimator under a half-sampling design where the bias of the estimator can be expressed explicitly. Furthermore, we discuss a practical approach to estimate its bias by a two-layer Monte Carlo method so as to obtain a bias-corrected variance estimator. In the simulation study and real data examples, we evaluate the performance of the proposed variance estimator, in comparison to the commonly used bootstrap and jackknife methods, in the context of model selection under the one-standard-error rule. The numerical results suggest that the proposal yields identical or similar conclusion for model selection compared to its counterparts. Moreover, the developed variance estimator is much more efficient to calculate than its competitors. In the end, we discuss the generalization of the methodology to other partition-sampling scenarios.

Nonlocal total variation regularization with Shape Adaptive Patches for image denoising via Split Bregman method

Non-Local or patch-based approaches are used in most of the state-of-the-art methods for image denoising. Denoising with regular square patches may cause noise halos around edges, particularly high contrasted edges. In order to overcome this drawback, this work presents an extension of the Non-Local TV framework that effectively exploits the potential local geometric features of the image by replacing the simple square patches with different oriented shapes. We first solve the denoised models of ROF-NLTV with different oriented patch shapes by Split Bregman algorithm. Then, use exponentially weighted aggregation method based on Stein’s unbiased risk estimate to combine the estimators obtained in the first step. The numerical results show our method outperforms some previous ones. Moreover, common noise halos around edges usually observed by denoising with Non-Local TV method are reduced thanks our approach.