Shrinkage Rules Research Articles

Shrinkage rules for variational minimization problems and applications to analytical ultracentrifugation

Abstract Finding a sparse representation of a noisy signal can be modeled as a variational minimization with -sparsity constraints for q less than one. Especially for real-time, online, or iterative applications, in which problems of this type have to be solved multiple times, one needs fast algorithms to compute these minimizers. However, identifying the exact minimizers is computationally expensive. We consider minimization up to a constant factor to circumvent this limitation. We verify that q-dependent modifications of shrinkage rules provide closed formulas for such minimizers. Therefore, their computation is extremely fast. We also introduce a new shrinkage rule which is adapted to q. To support the theoretical results, the proposed method is applied to Landweber iteration with shrinkage used at each iteration step. This approach is utilized to solve the ill-posed problem of analytic ultracentrifugation, a method to determine the size distribution of macromolecules. For relatively pure solutes, our proposed scheme leads to sparser solutions with sharper peaks, higher resolution, and smaller residuals than standard regularization for this problem.

The Curvelet Approach for Denoising in various Imaging Modalities using Different Shrinkage Rules

The images usually bring different kinds of noise in the process of receiving, coding and transmission. In this paper the Curvelet transform is used for de-noising of image. Two digital implementations of the Curvelet transform (a multiscale transform) viz the Unequally Spaced Fast Fourier Transform (USFFT) and the Wrapping Algorithm are used to de-noise images degraded by different types of noises such as Random, Gaussian, Salt and Pepper, Speckle and Poisson noise. This paper aims at the effect the Curvelet transform has in Curvelet shrinkage assuming different types of noise models. A signal to noise ratio as a measure of the quality of de-noising was preferred. The experimental results show that the conventional Curvelet shrinkage approach fails to remove Poisson noise in medical images.

Bayesian estimation of intensity surfaces on the sphere via needlet shrinkage and selection

This paper describes an approach for Bayesian modeling in spherical data sets. Our method is based upon a recent construction called the needlet, which is a particular form of spherical wavelet with many favorable statistical and computational properties. We perform shrinkage and selection of needlet coefficients, focusing on two main alternatives: empirical-Bayes thresholding, and Bayesian local shrinkage rules. We study the performance of the proposed methodology both on simulated data and on two real data sets: one involving the cosmic microwave background radiation, and one involving the reconstruction of a global news intensity surface inferred from published Reuters articles in August, 1996. The fully Bayesian approach based on robust, sparse shrinkage priors seems to outperform other alternatives.

Visual enhancement of digital ultrasound images using multiscale wavelet domain

Ultrasonography has been considered as one of the most powerful techniques for imaging organs and soft tissue structures in the human body. The main disadvantage of medical ultrasonography is the poor quality of images, which are affected by multiplicative speckle noise. In this paper, we present a novel method for despeckling medical ultrasound images. The primary goal of speckle reduction is to remove the speckle without losing much detail contained in an image. To achieve this goal, we make use of the wavelet transform and apply multi-resolution analysis to localize an image into different frequency components or useful subbands and then effectively reduce the speckle in the subbands according to the local statistics within the bands. The main advantage of the wavelet transform is that the image fidelity after reconstruction is visually lossless. The objective of the paper is to investigate the proper selection of wavelet filters and thresholding schemes which yields optimal visual enhancement of ultrasound images, in particular. We employ the wavelet shrinkage denoising techniques with different wavelet bases and decomposition levels on the individual subbands to achieve the best acceptable speckle reduction while maintaining the fidelity of the image and also examine the effects of different thresholding techniques as well as shrinkage rules for denoising ultrasound images. The proposed method consists of the log transformed original ultrasound image being subjected to wavelet transform, which is then denoised by a thresholding technique using a shrinkage rule. Experimental results show that the subband decomposition of ultrasound images, using Bior6.8 and level 3 with soft thresholding based on Bayes shrinkage rule, performs better than other techniques. The performance is measured in terms of Variance, Mean Square Error (MSE), Signal-to-Noise Ratio (SNR), Peak SNR (PSNR) and Correlation Coefficient (CC). The results of wavelet shrinkage techniques are compared with common speckle filters. We observe that the proposed method achieves better visual enhancement of ultrasound images which would lead to more accurate image analysis by the medical experts.

The SURE-LET Approach for MR Brain Image Denoising Using Different Shrinkage Rules

SURE-LET Approach is used for reducing or removing noise in brain Magnetic Resonance Images (MRI). Removing or reducing noise is an active research area in image processing. Rician noise is the dominant noise in MRIs. Due to this type of noise, the abnormal tissue (cancerous tissue) may be misclassified as normal tissue and introduces bias into MRI measurements that can have signi?cant impact on the shapes and orientations of tensors in di?usion tensor MRIs. SURE is a new approach to Orthonormal wavelet image denoising. It is an image-domain minimization of an estimate of the mean squared error—Stein’s unbiased risk estimates (SURE). Here, the denoising process can be expressed as a linear combination of elementary denoising processes-linear expansion of thresholds (LET). Different Shrinkage functions such as Soft and Hard and Shrinkage rules and Universal and BayesShrink are used to remove noise and the performance of these results are compared. The algorithm is applied on brain MRIs with different noisy conditions by varying standard deviation of noise. The performance of this approach is compared with performance of the Curvelet transform.

On testing for serial correlation of unknown form using wavelet thresholding

Omnibus procedures for testing serial correlation are developed, using spectral density estimation and wavelet shrinkage. The asymptotic distributions of the wavelet coefficients under the null hypothesis of no serial correlation are derived. Under some general conditions on the wavelet basis, the wavelet coefficients asymptotically follow a normal distribution. Furthermore, they are asymptotically uncorrelated. Adopting a spectral approach and using results on wavelet shrinkage, new one-sided test statistics are proposed. As a spatially adaptive estimation method, wavelets can effectively detect fine features in the spectral density, such as sharp peaks and high frequency alternations. Using an appropriate thresholding parameter, shrinkage rules are applied to the empirical wavelet coefficients, resulting in a non-linear wavelet-based spectral density estimator. Consequently, the advocated approach avoids the need to select the finest scale J , since the noise in the wavelet coefficients is naturally suppressed. Simple data-dependent threshold parameters are also considered. In general, the convergence of the spectral test statistics toward their respective asymptotic distributions appears to be relatively slow. In view of that, Monte Carlo methods are investigated. In a small simulation study, several spectral test statistics are compared, with respect to level and power, including versions of these test statistics using Monte Carlo simulations.

Bayesian Estimation of Bessel K Form Random Vectors in AWGN

We present new Bayesian estimators for spherically-contoured Bessel K form (BKF) random vectors in additive white Gaussian noise (AWGN). The derivations are an extension of existing results for the scalar BKF and multivariate Laplace (MLAP) densities. MAP and MMSE estimators are derived. We show that the MMSE estimator can be written in exact form in terms of the generalized incomplete Gamma function. Computationally efficient approximations are given. We compare the proposed exact and approximate MMSE estimators with recent results using the BKF density, both in terms of the shrinkage rules and the associated mean-square error.

Locally analytic schemes: A link between diffusion filtering and wavelet shrinkage

We study a class of numerical schemes for nonlinear diffusion filtering that offers insights on the design of novel wavelet shrinkage rules for isotropic and anisotropic image enhancement. These schemes utilise analytical or semi-analytical solutions to dynamical systems that result from space-discrete nonlinear diffusion filtering on minimalistic images with 2 × 2 pixels. We call them locally analytic schemes (LAS) and locally semi-analytic schemes (LSAS), respectively. They can be motivated from discrete energy functionals, offer sharp edges due to their locality, are very simple to implement because of their explicit nature, and enjoy unconditional absolute stability. They are applicable to singular nonlinear diffusion filters such as TV flow, to bounded nonlinear diffusion filters of Perona–Malik type, and to tensor-driven anisotropic methods such as edge-enhancing or coherence-enhancing diffusion filtering. The fact that these schemes use processes within 2 × 2 -pixel blocks allows to connect them to shift-invariant Haar wavelet shrinkage on a single scale. This interpretation leads to novel shrinkage rules for two- and higher-dimensional images that are scalar-, vector- or tensor-valued. Unlike classical shrinkage strategies they employ a diffusion-inspired coupling of the wavelet channels that guarantees an approximation with an excellent degree of rotation invariance. By extending these schemes from a single scale to a multi-scale setting, we end up at hybrid methods that demonstrate the possibility to realise the effects of the most sophisticated diffusion filters within a fairly simplistic wavelet setting that requires only Haar wavelets in conjunction with coupled shrinkage rules.

Improving the Estimation of Eigenvectors Under Quadratic Loss

Improved estimation of eigenvectors of a covariance matrix is considered under uncertain prior information (UP!) regarding the parameter vector. Like statistical models underlying the statistical inferences to be made, the prior information will be susceptible to uncertainty and the practitioners may be reluctant to impose the additional information regarding parameters in the estimation process. A very large gain in precision may be achieved by judiciously exploiting the information about the parameters which in practice will be available in any realistic problem. Several estimators based on preliminary test and the Stein-type shrinkage rules are constructed. The expressions for the bias and risk of the proposed estimators are derived and compared with the usual estimators. We demonstrate how the classical large sample theory of the conventional estimator can be extended to shrinkage and preliminary test estimators for the eigenvector of a covariance matrix. It is established that shrinkage estimators are asymptotically superior to the usual sample estimators. For illustration purposes, the method is applied to three data sets.

Frequentist optimality of Bayesian wavelet shrinkage rules for Gaussian and non-Gaussian noise

The present paper investigates theoretical performance of various Bayesian wavelet shrinkage rules in a nonparametric regression model with i.i.d. errors which are not necessarily normally distributed. The main purpose is comparison of various Bayesian models in terms of their frequentist asymptotic optimality in Sobolev and Besov spaces. We establish a relationship between hyperparameters, verify that the majority of Bayesian models studied so far achieve theoretical optimality, state which Bayesian models cannot achieve optimal convergence rate and explain why it happens.

Wavelet Bayesian Block Shrinkage via Mixtures of Normal-Inverse Gamma Priors

This article proposes a nonlinear block shrinkage method in the wavelet domain for estimating an unknown function in the presence of Gaussian noise. This shrinkage uses an empirical Bayesian blocking approach that accounts for the sparseness of the representation of the unknown function. The modeling is accomplished by using a mixture of two normalinverse gamma (N I G ) distributions as a joint prior on wavelet coefficients and noise variance in each block at a particular resolution level. This method results in an explicit and readily implementable weighted sum of shrinkage rules. An automatic, level-dependent choice for the model hyperparameters, that leads to amplitude-scale invariant solutions, is also suggested. Finally, the performance of the proposed method, BBS (Bayesian block shrinkage), is illustrated on the battery of standard test functions and compared to some existing block-wavelet denoising methods.

Bivariate shrinkage functions for wavelet-based denoising exploiting interscale dependency

Most simple nonlinear thresholding rules for wavelet-based denoising assume that the wavelet coefficients are independent. However, wavelet coefficients of natural images have significant dependencies. We only consider the dependencies between the coefficients and their parents in detail. For this purpose, new non-Gaussian bivariate distributions are proposed, and corresponding nonlinear threshold functions (shrinkage functions) are derived from the models using Bayesian estimation theory. The new shrinkage functions do not assume the independence of wavelet coefficients. We show three image denoising examples in order to show the performance of these new bivariate shrinkage rules. In the second example, a simple subband-dependent data-driven image denoising system is described and compared with effective data-driven techniques in the literature, namely VisuShrink, SureShrink, BayesShrink, and hidden Markov models. In the third example, the same idea is applied to the dual-tree complex wavelet coefficients.

Simultaneous Estimation of Several Intraclass Correlation Coefficients

Based on shrinkage and preliminary test rules, various estimators are proposed for estimation of several intraclass correlation coefficients when independent samples are drawn from multivariate normal populations. It is demonstrated that the James-Stein type estimators are asymptotically superior to the usual estimators. Furthermore, it is also indicated through asymptotic results that none of the preliminary test and shrinkage estimators dominate each other, though they perform relatively well as compared to the classical estimator. The relative dominance picture of the estimators is presented. A Monte Carlo study is performed to appraise the properties of the proposed estimators for small samples.

Almost Everywhere Behavior of General Wavelet Shrinkage Operators

Wavelet shrinkage estimators are obtained by applying a shrinkage rule on the empirical wavelet coefficients. Such simple estimators are now well explored and widely used in wavelet-based nonparametrics. Results of Tao (1996, Appl. Comput. Harmon. Anal.3, 384–387) demonstrated that hard and soft thresholding shrinkage estimators absolutely converge almost everywhere to the original function when the threshold value goes to zero. Such natural and intuitive behavior of threshold estimators is expected, yet this result does not translate to the Fourier expansions. In this paper we show that almost everywhere convergence of shrinkage estimators holds for a range of shrinkage rules, not necessarily thresholding, subject to some mild technical conditions. Comments about the norm convergence are provided as well.

Construction of improved estimators of multinomial proportions

The improved large sample estimation theory for the probabilities of multi¬nomial distribution is developed under uncertain prior information (UPI) that the true proportion is a known quantity. Several estimators based on pretest and the Stein-type shrinkage rules are constructed. The expressions for the bias and risk of the proposed estimators are derived and compared with the maximum likelihood (ml) estimators. It is demonstrated that the shrinkage estimators are superior to the ml estimators. It is also shown that none of the preliminary test and shrinkage estimators dominate each other, though they perform y/ell relative to the ml estimators. The relative dominance picture of the estimators is presented. A simulation study is carried out to assess the performance of the estimators numerically in small samples.

Analysis of multiresolution image denoising schemes using generalized Gaussian and complexity priors

Research on universal and minimax wavelet shrinkage and thresholding methods has demonstrated near-ideal estimation performance in various asymptotic frameworks. However, image processing practice has shown that universal thresholding methods are outperformed by simple Bayesian estimators assuming independent wavelet coefficients and heavy-tailed priors such as generalized Gaussian distributions (GGDs). In this paper, we investigate various connections between shrinkage methods and maximum a posteriori (MAP) estimation using such priors. In particular, we state a simple condition under which MAP estimates are sparse. We also introduce a new family of complexity priors based upon Rissanen's universal prior on integers. One particular estimator in this class outperforms conventional estimators based on earlier applications of the minimum description length (MDL) principle. We develop analytical expressions for the shrinkage rules implied by GGD and complexity priors. This allows us to show the equivalence between universal hard thresholding, MAP estimation using a very heavy-tailed GGD, and MDL estimation using one of the new complexity priors. Theoretical analysis supported by numerous practical experiments shows the robustness of some of these estimates against mis-specifications of the prior-a basic concern in image processing applications.

A Monte Carlo Study of Robustness of Pretest and Shrinkage Estimators in Pooling Coefficients of Variation

Pooling data, when justified, is advantageous for estimating the true parameter. In this paper the problem of estimating the coefficient of variation is considered when it is a priori suspected that two coefficients of variation are the same. Various estimators based on pretest and shrinkage rules are considered. A comparison through the Simulated Mean Squared Error (SMSE) criterion is carried out among various proposed estimators of the target coefficient of variation. The relative simulated efficiencies of the restricted, shrinkage restricted and shrinkage pretest estimators are studied. It is found that the proposed estimators are quite robust when the sample sizes are not too large. The result of Monte Carlo study indicates that the proposed shrinkage pretest estimator is efficient than the usual estimator in a wider range.

Multiple shrinkage and subset selection in wavelets

This paper discusses Bayesian methods for multiple shrinkage estimation in wavelets. Wavelets are used in applications for data denoising, via shrinkage of the coefficients towards zero, and for data compression, by shrinkage and setting small coefficients to zero. We approach wavelet shrinkage by using Bayesian hierarchical models, assigning a positive prior probability to the wavelet coefficients being zero. The resulting estimator for the wavelet coefficients is a multiple shrinkage estimator that exhibits a wide variety of nonlinear patterns. We discuss fast computational implementations, with a focus on easy-to-compute analytic approximations as well as importance sampling and Markov chain Monte Carlo methods. Multiple shrinkage estimators prove to have excellent mean squared error performance in reconstructing standard test functions. We demonstrate this in simulated test examples, comparing various implementations of multiple shrinkage to commonly-used shrinkage rules. Finally, we illustrate our approach with an application to the so-called 'glint' data.

Shrinkage Positive Rule Estimators for Spherically Symmetrical Distributions

Tn the normal case it is well known that, although the James-Stein rule is minimax, it is not admissible and the associated positive rule is one way to improve on it. We extend this result to the class of the spherically symmetric distributions and to a large class of shrinkage rules. Moreover we propose a family of generalized positive rules. We compare our results to those of Berger and Bock ( Statistical Decision Theory and Related Topics, II, Academic Press, New York. 1976). In particular our conditions on the shrinkage estimator are weaker.

The Use of Shrinkage Estimators in Linear Discriminant Analysis

Probably the most common single discriminant algorithm in use today is the linear algorithm. Unfortunately, this algorithm has been shown to frequently behave poorly in high dimensions relative to other algorithms, even on suitable Gaussian data. This is because the algorithm uses sample estimates of the means and covariance matrix which are of poor quality in high dimensions. It seems reasonable that if these unbiased estimates were replaced by estimates which are more stable in high dimensions, then the resultant modified linear algorithm should be an improvement. This paper studies using a shrinkage estimate for the covariance matrix in the linear algorithm. We chose the linear algorithm, not because we particularly advocate its use, but because its simple structure allows one to more easily ascertain the effects of the use of shrinkage estimates. A simulation study assuming two underlying Gaussian populations with common covariance matrix found the shrinkage algorithm to significantly outperform the standard linear algorithm in most cases. Several different means, covariance matrices, and shrinkage rules were studied. A nonparametric algorithm, which previously had been shown to usually outperform the linear algorithm in high dimensions, was included in the simulation study for comparison.