Soft-thresholding Estimators Research Articles

Bridging between Soft and Hard Thresholding by Scaling

In this article, we developed and analyzed a thresholding method in which soft thresholding estimators are independently expanded by empirical scaling values. The scaling values have a common hyper-parameter that is an order of expansion of an ideal scaling value that achieves hard thresholding. We simply call this estimator a scaled soft thresholding estimator. The scaled soft thresholding is a general method that includes the soft thresholding and non-negative garrote as special cases and gives an another derivation of adaptive LASSO. We then derived the degree of freedom of the scaled soft thresholding by means of the Stein's unbiased risk estimate and found that it is decomposed into the degree of freedom of soft thresholding and the reminder connecting to hard thresholding. In this meaning, the scaled soft thresholding gives a natural bridge between soft and hard thresholding methods. Since the degree of freedom represents the degree of over-fitting, this result implies that there are two sources of over-fitting in the scaled soft thresholding. The first source originated from soft thresholding is determined by the number of un-removed coefficients and is a natural measure of the degree of over-fitting. We analyzed the second source in a particular case of the scaled soft thresholding by referring a known result for hard thresholding. We then found that, in a sparse, large sample and non-parametric setting, the second source is largely determined by coefficient estimates whose true values are zeros and has an influence on over-fitting when threshold levels are around noise levels in those coefficient estimates. In a simple numerical example, these theoretical implications has well explained the behavior of the degree of freedom. Moreover, based on the results here and some known facts, we explained the behaviors of risks of soft, hard and scaled soft thresholding methods.

Computing the proximity operator of the ℓ p norm with 0 < p < 1

Sparse modelling with the l p norm of 0 ≤ p ≤ 1 requires the availability of the proximity operator of the l p norm. The proximity operators of the l0 and l1 norms are the well-known hard- and soft-thresholding estimators, respectively. In this study, the authors give a complete study on the properties of the proximity operator of the l p norm. Based on these properties, explicit formulas of the proximity operators of the l1/2 norm and l2/3 norm are derived with simple proofs; for other values of p, an iterative Newton's method is developed to compute the proximity operator of the l p norm by fully exploring the available proximity operators of the l0, l1/2, l2/3, and l1 norms. As applications, the proximity operator of the l p norm with 0 ≤ p ≤ 1 is applied to the l p -regularisation for compressive sensing and image restoration.

On scaling of soft-thresholding estimator

LASSO is known to have a problem of excessive shrinkage at a sparse representation. To analyze this problem in detail, in this paper, we consider a positive scaling for soft-thresholding estimators that are LASSO estimators in an orthogonal regression problem. We especially consider a non-parametric orthogonal regression problem which includes wavelet denosing. We first gave a risk (generalization error) of LARS (least angle regression) based soft-thresholding with a single scaling parameter. We then showed that an optimal scaling value that minimizes the risk under a sparseness condition is 1+O(logn/n), where n is the number of samples. The important point is that the optimal value of scaling is larger than one. This implies that expanding soft-thresholding estimator shows a better generalization performance compared to a naive soft-thresholding. This also implies that a risk of LARS-based soft-thresholding with the optimal scaling is smaller than without scaling. We then showed their difference is O(logn/n). This also shows an effectiveness of the introduction of scaling. Through simple numerical experiments, we found that LARS-based soft-thresholding with scaling can improve both of sparsity and generalization performance compared to a naive soft-thresholding.

High-dimensional mean estimation viaℓ1penalized normal likelihood

A new method is proposed for estimating the difference between the high-dimensional mean vectors of two multivariate normal populations with equal covariance matrix based on an ℓ 1 penalized normal likelihood. It is well known that the normal likelihood involves the covariance matrix which is usually unknown. We substitute the adaptive thresholding estimator given by Cai and Liu (2011) of the covariance matrix, and then estimate the difference between the mean vectors by maximizing the ℓ 1 penalized normal likelihood. Under the high-dimensional framework where both the sample size and the dimension tend to infinity, we show that the proposed estimator has sign recovery and also derive its mean squared error. We also compare the proposed estimator with the soft-thresholding and the adaptive soft-thresholding estimators which give simple thresholdings for the sample mean vector.

Distributional results for thresholding estimators in high-dimensional Gaussian regression models

We study the distribution of hard-, soft-, and adaptive soft-thresholding estimators within a linear regression model where the number of parameters k can depend on sample size n and may diverge with n. In addition to the case of known error-variance, we define and study versions of the estimators when the error-variance is unknown. We derive the finite-sample distribution of each estimator and study its behavior in the large-sample limit, also investigating the effects of having to estimate the variance when the degrees of freedom n-k does not tend to infinity or tends to infinity very slowly. Our analysis encompasses both the case where the estimators are tuned to perform consistent model selection and the case where the estimators are tuned to perform conservative model selection. Furthermore, we discuss consistency, uniform consistency and derive the uniform convergence rate under either type of tuning.

Penalized partially linear models using sparse representations with an application to fMRI time series

In this paper, we consider modeling the nonparametric component in partially linear models (PLMs) using linear sparse representations, e.g., wavelet expansions. Two types of representations are investigated, namely, orthogonal bases (complete) and redundant overcomplete expansions. For bases, we introduce a regularized estimator of the nonparametric part. The important contribution here is that the nonparametric part can be parsimoniously estimated by choosing an appropriate penalty function for which the hard and soft thresholding estimators are special cases. This allows us to represent in an effective manner a broad class of signals, including stationary and/or nonstationary signals and avoids excessive bias in estimating the parametric component. We also give a fast estimation algorithm. The method is then generalized to handle the case of overcomplete representations. A large-scale simulation study is conducted to illustrate the finite sample properties of the estimator. The estimator is finally applied to real neurophysiological functional magnetic resonance imaging (MRI) data sets that are suspected to contain both smooth and transient drift features.

Hybrid shrinkage estimators using penalty bases for the ordinal one-way layout

This paper constructs improved estimators of the means in the Gaussian saturated one-way layout with an ordinal factor. The least squares estimator for the mean vector in this saturated model is usually inadmissible. The hybrid shrinkage estimators of this paper exploit the possibility of slow variation in the dependence of the means on the ordered factor levels but do not assume it and respond well to faster variation if present. To motivate the development, candidate penalized least squares (PLS) estimators for the mean vector of a one-way layout are represented as shrinkage estimators relative to the penalty basis for the regression space. This canonical representation suggests further classes of candidate estimators for the unknown means: monotone shrinkage (MS) estimators or soft-thresholding (ST) estimators or, most generally, hybrid shrinkage (HS) estimators that combine the preceding two strategies. Adaptation selects the estimator within a candidate class that minimizes estimated risk. Under the Gaussian saturated one-way layout model, such adaptive estimators minimize risk asymptotically over the class of candidate estimators as the number of factor levels tends to infinity. Thereby, adaptive HS estimators asymptotically dominate adaptive MS and adaptive ST estimators as well as the least squares estimator. Local annihilators of polynomials, among them difference operators, generate penalty bases suitable for a range of numerical examples. In case studies, adaptive HS estimators recover high frequency details in the mean vector more reliably than PLS or MS estimators and low frequency details more reliably than ST estimators.

Regularization of Wavelet Approximations

In this paper, we introduce nonlinear regularized wavelet estimators for estimating nonparametric regression functions when sampling points are not uniformly spaced. The approach can apply readily to many other statistical contexts. Various new penalty functions are proposed. The hard-thresholding and soft-thresholding estimators of Donoho and Johnstone are specific members of nonlinear regularized wavelet estimators. They correspond to the lower and upper envelopes of a class of the penalized least squares estimators. Necessary conditions for penalty functions are given for regularized estimators to possess thresholding properties. Oracle inequalities and universal thresholding parameters are obtained for a large class of penalty functions. The sampling properties of nonlinear regularized wavelet estimators are established and are shown to be adaptively minimax. To efficiently solve penalized least squares problems, nonlinear regularized Sobolev interpolators (NRSI) are proposed as initial estimators, which are shown to have good sampling properties. The NRSI is further ameliorated by regularized one-step estimators, which are the one-step estimators of the penalized least squares problems using the NRSI as initial estimators. The graduated nonconvexity algorithm is also introduced to handle penalized least squares problems. The newly introduced approaches are illustrated by a few numerical examples.

Bayesian wavelet denoising: Besov priors and non-Gaussian noises

There has recently been a great research interest in thresholding methods for nonlinear wavelet regression over spaces of smooth functions. Near-minimax convergence rates were, in particular, established for simple hard and soft thresholding rules over Besov and Triebel bodies. In this paper, we propose a Bayesian approach where the functional properties of the underlying signal in noise are directly modeled using Besov norm priors on its wavelet decomposition coefficients. In the context of maximum a posteriori estimation, we first prove that general thresholding rules are obtained in (generalized) dual spaces. In this Tikhonov-type regularization framework, we show that nonstandard soft thresholding estimators are in particular obtained in possibly non-Gaussian noise situations. In the case of the minimum mean square error criterion, a Gibbs sampler is finally presented to estimate the model parameters and the posterior mean estimate of the underlying signal of interest.