Stein's Unbiased Risk Estimate Research Articles

Low and high dimensional wavelet thresholds for matrix-variate normal distribution

The matrix-variate normal distribution is a probability distribution that is a generalization of the multivariate normal distribution to matrix-valued random variables. In this paper, we introduce a wavelet shrinkage estimator based on Stein’s unbiased risk estimate (SURE) threshold for matrix-variate normal distribution. We find a new SURE threshold for soft thresholding wavelet shrinkage estimator under the reflected normal balanced loss function in low and high dimensional cases. Also, we obtain the restricted wavelet shrinkage estimator based on non-negative sub matrix of the mean matrix. Finally, we present a simulation study to test the validity of the wavelet shrinkage estimator and two real examples for low and high dimensional data sets.

Supervise-Assisted Self-Supervised Deep-Learning Method for Hyperspectral Image Restoration.

Hyperspectral image (HSI) restoration is a challenging research area, covering a variety of inverse problems. Previous works have shown the great success of deep learning in HSI restoration. However, facing the problem of distribution gaps between training HSIs and target HSI, those data-driven methods falter in delivering satisfactory outcomes for the target HSIs. In addition, the degradation process of HSIs is usually disturbed by noise, which is not well taken into account in existing restoration methods. The existence of noise further exacerbates the dissimilarities within the data, rendering it challenging to attain desirable results without an appropriate learning approach. To track these issues, in this article, we propose a supervise-assisted self-supervised deep-learning method to restore noisy degraded HSIs. Initially, we facilitate the restoration network to acquire a generalized prior through supervised learning from extensive training datasets. Then, the self-supervised learning stage is employed and utilizes the specific prior of the target HSI. Particularly, to restore clean HSIs during the self-supervised learning stage from noisy degraded HSIs, we introduce a noise-adaptive loss function that leverages inner statistics of noisy degraded HSIs for restoration. The proposed noise-adaptive loss consists of Stein's unbiased risk estimator (SURE) and total variation (TV) regularizer and fine-tunes the network with the presence of noise. We demonstrate through experiments on different HSI tasks, including denoising, compressive sensing, super-resolution, and inpainting, that our method outperforms state-of-the-art methods on benchmarks under quantitative metrics and visual quality. The code is available at https://github.com/ying-fu/SSDL-HSI.

Self-supervised MRI denoising: leveraging Stein’s unbiased risk estimator and spatially resolved noise maps

Thermal noise caused by the imaged object is an intrinsic limitation in magnetic resonance imaging (MRI), resulting in an impaired clinical value of the acquisitions. Recently, deep learning (DL)-based denoising methods achieved promising results by extracting complex feature representations from large data sets. Most approaches are trained in a supervised manner by directly mapping noisy to noise-free ground-truth data and, therefore, require extensive paired data sets, which can be expensive or infeasible to obtain for medical imaging applications. In this work, a DL-based denoising approach is investigated which operates on complex-valued reconstructed magnetic resonance (MR) images without noise-free target data. An extension of Stein’s unbiased risk estimator (SURE) and spatially resolved noise maps quantifying the noise level with pixel accuracy were employed during the training process. Competitive denoising performance was achieved compared to supervised training with mean squared error (MSE) despite optimizing the model without noise-free target images. The proposed DL-based method can be applied for MR image enhancement without requiring noise-free target data for training. Integrating the noise maps as an additional input channel further enables the regulation of the desired level of denoising to adjust to the preference of the radiologist.

A machine learning approach combined with wavelet analysis for automatic detection of Pc5 geomagnetic pulsations observed at geostationary orbits

Pc5 geomagnetic pulsations are ultra-low frequency (ULF) waves whose period lies within 150–600 s. Their detection is crucial for accurate space weather modelling, monitoring, and analysis. Identification of these pulsations using manual approaches is difficult and time-consuming because of the smaller magnitudes and the limited durations within which they occur. To overcome this challenge, we propose a robust Cascade Forward Neural Network (CFNN) model combined with the wavelet technique for automatically detecting Pc5 events from satellite datasets observed at geostationary orbits. The dataset used in this study is the magnetic field vector measurements retrieved from the Geostationary Operational Environmental Satellite-10 (GOES-10) from 2000 to 2009. Pc5 geomagnetic pulsations were extracted from the toroidal component of the field perturbation using a bandpass Butterworth filter. Continuous wavelet transform (CWT) analysis using the mother Morlet wavelet was utilized to validate the integrity of the extracted signal in the time–frequency domain. The extracted Pc5 signal was decomposed into details and approximations using Daubechies wavelet transform to separate the intelligible signal from the incoherent noise that left their trace on the magnetic field time series. The detail signal was subjected to denoising using the heuristic Stein Unbiased Risk Estimate (SURE) approach with soft thresholding to obtain the denoised Pc5 events. The denoised Pc5 events were utilized as the target in the machine learning whereas the inputs were obtained from 5-dimensional space transformation of the toroidal component of the time series. The developed algorithm performed well and demonstrated a detection accuracy of 80 % and a Root Mean Square Error (RMSE) of 0.13 nT indicating a high performance in detecting Pc5 events. For effective validation of the model, Pc5 events detected by our model were correlated with the Kp index and the amplitude of the Pc5 events observed at geostationary orbit. A good correlation was obtained in both cases, making our model a practical and preferred choice for Pc5 pulsation detection in contrast to conventional frequency analysis tools which take much time for signal processing on large data sets.

Sparse additive models in high dimensions with wavelets

AbstractIn multiple regression, when covariates are numerous, it is often reasonable to assume that only a small number of them has predictive information. In some medical applications for instance, it is believed that only a few genes out of thousands are responsible for cancer. In that case, the aim is not only to propose a good fit, but also to select the relevant covariates (genes). We propose to perform model selection with additive models in high dimensions (sample size and number of covariates). Our approach is computationally efficient thanks to fast wavelet transforms, it does not rely on cross validation, and it solves a convex optimization problem for a prescribed penalty parameter, called the quantile universal threshold. We also propose a second rule based on Stein unbiased risk estimation geared toward prediction. We use Monte Carlo simulations and real data to compare various methods based on false discovery rate (FDR), true positive rate (TPR) and mean squared error. Our approach is the only one to handle high dimensions, and has a good FDR–TPR trade‐off.

Two new Bayesian-wavelet thresholds estimations of elliptical distribution parameters under non-linear exponential balanced loss

The estimation of mean vector parameters is very important in elliptical and spherically models. Among different methods, the Bayesian and shrinkage estimation are interesting. In this paper, the estimation of p-dimensional location parameter for p-variate elliptical and spherical distributions under an asymmetric loss function is investigated. We find generalized Bayes estimator of location parameters for elliptical and spherical distributions. Also we show the minimaxity and admissibility of generalized Bayes estimator in class of We introduce two new shrinkage soft-wavelet threshold estimators based on Huang shrinkage wavelet estimator (empirical) and Stein’s unbiased risk estimator (SURE) for elliptical and spherical distributions under non-linear exponential-balanced loss function. At the end, we present a simulation study to test the validity of the class of proposed estimators and physicochemical properties of the tertiary structure data set that is given to test the efficiency of this estimators in denoising.

ENSURE: A General Approach for Unsupervised Training of Deep Image Reconstruction Algorithms.

Image reconstruction using deep learning algorithms offers improved reconstruction quality and lower reconstruction time than classical compressed sensing and model-based algorithms. Unfortunately, clean and fully sampled ground-truth data to train the deep networks is often unavailable in several applications, restricting the applicability of the above methods. We introduce a novel metric termed the ENsemble Stein's Unbiased Risk Estimate (ENSURE) framework, which can be used to train deep image reconstruction algorithms without fully sampled and noise-free images. The proposed framework is the generalization of the classical SURE and GSURE formulation to the setting where the images are sampled by different measurement operators, chosen randomly from a set. We evaluate the expectation of the GSURE loss functions over the sampling patterns to obtain the ENSURE loss function. We show that this loss is an unbiased estimate for the true mean-square error, which offers a better alternative to GSURE, which only offers an unbiased estimate for the projected error. Our experiments show that the networks trained with this loss function can offer reconstructions comparable to the supervised setting. While we demonstrate this framework in the context of MR image recovery, the ENSURE framework is generally applicable to arbitrary inverse problems.

Robust compressed sensing MRI based on combined nonconvex regularization

Compressed sensing (CS) is well known method for solving fast reconstruction of magnetic resonance (MR) images. However, the quality of the reconstructed images is sensitive to the noise. The reconstructed results often suffer from blurry or jaggy artifacts, which seriously affects the analysis of the medical image. To achieve robustness reconstruction result, in this paper, the combined nonconvex regularization model is proposed to enhance the details recovery and reduce jaggy artifacts. In the proposed framework, the bias-based sharpness enhancement prior is integrated into smoothed L0 (SL0) model, which can make SL0 reconstruction method more robust to noise. To solve the nonconvex optimization problem, a two-cycle iterative algorithm is developed to solve it. In addition, the improved Stein Unbiased Risk Estimator (SURE) is developed to automatically tune the shape parameter that to be effective for a variety of noise levels. The experimental results on MR images consistently demonstrate the excellent performance of the proposed method in both the objective quality metrics and visual inspections.

Deep external and internal learning for noisy compressive sensing

Reconstructing natural image from its corresponding compressive sensing (CS) measurements is an ill-posed problem. Learning accurate prior of desirable image is essential to solve this inverse problem in high quality, especially for CS reconstruction from noisy measurements. The existing learning-based methods cannot effectively simulate whole potential noise in the testing data during external learning, and cannot effectively exploit the information from internal testing data. In this paper, we present an effective convolutional neural network (CNN) based method for CS reconstruction from noisy measurements, which learns the deep prior from an external dataset and internal noisy testing data with Stein’s unbiased risk estimator (SURE). Specifically, we first pre-train an arbitrary CNN for CS reconstruction with an external dataset to learn a common prior. Then, we utilize meta learning to find a generic initial parameter that is suitable for fast internal learning and adaption to various noisy measurements. Finally, we customize the learned network to learn a specific prior for each internal testing data under Gaussian noise or more general mixed Poisson-Gaussian noise. Experimental results show that the proposed method outperforms the state-of-the-art methods under both comprehensively quantitative metrics and perceptive quality.

A Self-Supervised Denoising Method Based on Deep Noise Estimation

Seismic random noise attenuation is an essential procedure when processing seismic data. Due to various acquisition environments and complex geological conditions, random noise in seismic field data exhibits spatiotemporal levels, significantly increasing the difficulty of extracting seismic signals. The deep learning (DL) methods have shown excellent performances for seismic denoising. However, most existing discriminative DL methods cannot fit field data with noise levels and signal structures that differ from the training set. To tackle the unmatching challenge, we propose a self-supervised deep denoising model named noise estimation-based convolution neural network (NE-CNN), which contains a multiscale denoising module as a pretrained model and a dual-path noise estimation module that estimates the noise level of each data patch with a generalized Gaussian distribution and gray-level co-occurrence matrix (GLCM). With Stein’s unbiased risk estimate (SURE), we can fine-tune the pretrained denoising module according to the estimated noise levels in a self-supervised style solely on data to be processed, leading to a boost of robustness to nonstationary seismic noise. In synthetic and field data tests, NE-CNN performed satisfactorily compared with discriminative DL methods in denoising effects on seismic data with nonstationary noise.

Bridging between Soft and Hard Thresholding by Scaling

In this article, we developed and analyzed a thresholding method in which soft thresholding estimators are independently expanded by empirical scaling values. The scaling values have a common hyper-parameter that is an order of expansion of an ideal scaling value that achieves hard thresholding. We simply call this estimator a scaled soft thresholding estimator. The scaled soft thresholding is a general method that includes the soft thresholding and non-negative garrote as special cases and gives an another derivation of adaptive LASSO. We then derived the degree of freedom of the scaled soft thresholding by means of the Stein's unbiased risk estimate and found that it is decomposed into the degree of freedom of soft thresholding and the reminder connecting to hard thresholding. In this meaning, the scaled soft thresholding gives a natural bridge between soft and hard thresholding methods. Since the degree of freedom represents the degree of over-fitting, this result implies that there are two sources of over-fitting in the scaled soft thresholding. The first source originated from soft thresholding is determined by the number of un-removed coefficients and is a natural measure of the degree of over-fitting. We analyzed the second source in a particular case of the scaled soft thresholding by referring a known result for hard thresholding. We then found that, in a sparse, large sample and non-parametric setting, the second source is largely determined by coefficient estimates whose true values are zeros and has an influence on over-fitting when threshold levels are around noise levels in those coefficient estimates. In a simple numerical example, these theoretical implications has well explained the behavior of the degree of freedom. Moreover, based on the results here and some known facts, we explained the behaviors of risks of soft, hard and scaled soft thresholding methods.

Selecting Regularization Parameters for Nuclear Norm--Type Minimization Problems

The reconstruction of a low-rank matrix from its noisy observation finds its usage in many applications. It can be reformulated into a constrained nuclear norm minimization problem, where the bound $\eta$ of the constraint is explicitly given or can be estimated by the probability distribution of the noise. When the Lagrangian method is applied to find the minimizer, the solution can be obtained by the singular value thresholding operator, where the thresholding parameter $\lambda$ is related to the Lagrangian multiplier. In this paper, we first show that the Frobenius norm of the discrepancy between the minimizer and the observed matrix is a strictly monotonically increasing function of $\lambda$. From that we derive a closed form solution for $\lambda$ in terms of $\eta$. Since $\lambda$ is the same as the regularization parameter for the unconstrained regularized problem, our results can be applied to automatically choosing a suitable regularization parameter for the nuclear norm--type regularized minimization problems using the discrepancy principle. The regularization parameters obtained are comparable to (and sometimes better than) those obtained by Stein's unbiased risk estimator approach, while the cost of solving the minimization problem can be reduced by 11--18 times. Numerical experiments with both synthetic data and real MRI data are performed to validate the proposed approach.

Degrees of freedom for off-the-grid sparse estimation

The degrees of freedom quantify the number of parameters of an estimator and is central to various risk minimization procedures. Its computation and properties are well understood when dealing with finite dimensional linear models, possibly regularized using sparsity through the celebrated Lasso estimator. However, for some applications (such as super-resolution in imaging or training multi-layer perceptrons with a single hidden layer) it makes sense to rather consider “continuous” methods. For these applications, these methods avoid the discretization of the parameter space and lead to more efficient numerical solvers. Training these continuous models with a sparsity inducing prior can be achieved by solving a convex optimization problem over the infinite dimensional space of measures, which is often called the Beurling Lasso (Blasso), and is the continuous counterpart of the Lasso. Previous works (Ann. Statist. 35 (2007) 2173–2192; Statist. Sinica 23 (2013) 809–828) show that the size of the smallest solution support is an unbiased estimator of the degrees of freedom of the Lasso. Our main contribution is a proof of a continuous counterpart to this result for the Blasso. In contrast to the Lasso, our new formula shows that the empirical degrees of freedom of the Blasso is not proportional to the size of the support. Our second contribution is a detailed study of the case of sampling Fourier coefficients in 1D, which corresponds to a super-resolution problem. We show that our formula for the unbiased estimation of the degrees of freedom is valid outside of a set of measure zero of observations, which in turn justifies its use to compute an unbiased estimator of the prediction risk using the Stein Unbiased Risk Estimator (SURE). We also report numerical results for both the case of Fourier sampling and the learning of a multilayers perceptron with a single hidden layer.

Progressive Denoising of Monte Carlo Rendered Images

AbstractImage denoising based on deep learning has become a powerful tool to accelerate Monte Carlo rendering. Deep learning techniques can produce smooth images using a low sample count. Unfortunately, existing deep learning methods are biased and do not converge to the correct solution as the number of samples increase. In this paper, we propose a progressive denoising technique that aims to use denoising only when it is beneficial and to reduce its impact at high sample counts. We use Stein's unbiased risk estimate (SURE) to estimate the error in the denoised image, and we combine this with a neural network to infer a per‐pixel mixing parameter. We further augment this network with confidence intervals based on classical statistics to ensure consistency and convergence of the final denoised image. Our results demonstrate that our method is consistent and that it improves existing denoising techniques. Furthermore, it can be used in combination with existing high quality denoisers to ensure consistency. In addition to being asymptotically unbiased, progressive denoising is particularly good at preserving fine details that would otherwise be lost with existing denoisers.

Adaptive singular value shrinkage estimate for low rank tensor denoising

Recently, tensors are widely used to represent higher-order data with internal spatial or temporal relations, e.g. images, videos, hyperspectral images (HSIs). While the true signals are usually corrupted by noises, it is of interest to study tensor recovery problems. To this end, many models have been established based on tensor decompositions. Traditional tensor decomposition models, such as the CP and Tucker factorization, treat every mode of tensors equally. However, in many real applications, some modes of the data act differently from the other modes, e.g. channel mode of images, time mode of videos, band mode of HSIs. The recently proposed model called t-SVD aims to tackle such problems. In this paper, we focus on tensor denoising problems. Specifically, in order to obtain low-rank estimators of true signals, we propose to use different shrinkage functions to shrink the tensor singular values based on the t-SVD. We derive Stein’s unbiased risk estimate (SURE) of the proposed model and develop adaptive SURE-based tuning parameter selection procedure, which is totally data-driven and simultaneous with the estimation process. The whole modeling procedure requires only one round of t-SVD. To demonstrate our model, we conduct experiments on simulation data, images, videos and HSIs. The results show that the proposed SURE approximates the true risk function accurately. Moreover, the proposed model selection procedure picks good tuning parameters out. We show the superiority of our model by comparing with state-of-the-art denoising models. The experiments manifest that our model outperforms in both quantitative metrics (e.g. RSE, PSNR) and visualizing results.

Automated Parameter Selection for Accelerated MRI Reconstruction via Low-Rank Modeling of Local k-Space Neighborhoods

PurposeImage quality in accelerated MRI rests on careful selection of various reconstruction parameters. A common yet tedious and error-prone practice is to hand-tune each parameter to attain visually appealing reconstructions. Here, we propose a parameter tuning strategy to automate hybrid parallel imaging (PI) – compressed sensing (CS) reconstructions via low-rank modeling of local k-space neighborhoods (LORAKS) supplemented with sparsity regularization in wavelet and total variation (TV) domains. MethodsFor low-rank regularization, we leverage a soft-thresholding operation based on singular values for matrix rank selection in LORAKS. For sparsity regularization, we employ Stein's unbiased risk estimate criterion to select the wavelet regularization parameter and local standard deviation of reconstructions to select the TV regularization parameter. Comprehensive demonstrations are presented on a numerical brain phantom and in vivo brain and knee acquisitions. Quantitative assessments are performed via PSNR, SSIM and NMSE metrics. ResultsThe proposed hybrid PI-CS method improves reconstruction quality compared to PI-only techniques, and it achieves on par image quality to reconstructions with brute-force optimization of reconstruction parameters. These results are prominent across several different datasets and the range of examined acceleration rates. ConclusionA data-driven parameter tuning strategy to automate hybrid PI-CS reconstructions is presented. The proposed method achieves reliable reconstructions of accelerated multi-coil MRI datasets without the need for exhaustive hand-tuning of reconstruction parameters.

Low-rank matrix denoising for count data using unbiased Kullback-Leibler risk estimation

Many statistical studies are concerned with the analysis of observations organized in a matrix form whose elements are count data. When these observations are assumed to follow a Poisson or a multinomial distribution, it is of interest to focus on the estimation of either the intensity matrix (Poisson case) or the compositional matrix (multinomial case) when it is assumed to have a low rank structure. In this setting, it is proposed to construct an estimator minimizing the regularized negative log-likelihood by a nuclear norm penalty. Such an approach easily yields a low-rank matrix-valued estimator with positive entries which belongs to the set of row-stochastic matrices in the multinomial case. Then, as a main contribution, a data-driven procedure is constructed to select the regularization parameter in the construction of such estimators by minimizing (approximately) unbiased estimates of the Kullback-Leibler (KL) risk in such models, which generalize Stein's unbiased risk estimation originally proposed for Gaussian data. The evaluation of these quantities is a delicate problem, and novel methods are introduced to obtain accurate numerical approximation of such unbiased estimates. Simulated data are used to validate this way of selecting regularizing parameters for low-rank matrix estimation from count data. For data following a multinomial distribution, the performances of this approach are also compared to K-fold cross-validation. Examples from a survey study and metagenomics also illustrate the benefits of this methodology for real data analysis.

Deep SURE for Unsupervised Remote Sensing Image Fusion

Image fusion is utilized in remote sensing due to the limitation of the imaging sensor and the high cost of simultaneously acquiring high spatial and spectral resolution images. Optical remote sensing imaging systems usually provide images of high spatial resolution but low spectral resolution and vice versa. Therefore, fusing those images to obtain a fused image having both high spectral and spatial resolution is desirable in many applications. This paper proposes a fusion framework using an unsupervised convolutional neural network (CNN) and Stein’s unbiased risk estimate (SURE). We derive a new loss function for a CNN that incorporates back-projection mean-squared error with SURE to estimate the projected mean-square-error (MSE) between the fused image and the ground truth. The main motivation is that training a CNN with this SURE loss function is unsupervised and avoids overfitting. Experimental results for two fusion examples, multispectral and hyperspectral (MS-HS) image fusion, and multispectral and multispectral (MS-MS) image fusion, show that the proposed method yields high quality fused images and outperforms the competitive methods. Codes are be available at https://github.com/hvn2/Deep-SURE-Fusion.

Ground Truth-Free 3-D Seismic Random Noise Attenuation via Deep Tensor Convolutional Neural Networks in the Time-Frequency Domain

The inherent challenge of 3-D seismic noise attenuation is determining how to uncover high-dimensional concise structures that only exist in true signals to eliminate random noise. The prevailing deep learning (DL) denoising methods have achieved promising performance in revealing the compact structures underlying contaminated seismic data. However, as clean ground-truth seismic data are generally unavailable in real-world settings, most existing matrix-based DL denoising schemes fail to automatically describe this type of high-dimensional structure in an unsupervised manner, potentially rendering them unable to effectively perform 3-D seismic data denoising tasks. To tackle this challenge, this article presents a tensor convolutional neural network (TCNN)-based data denoising scheme using Stein’s unbiased risk estimate (SURE) (called SURE-TCNN) to learn intrinsic high-dimensional structures without ground-truth seismic data. Considering that SURE provides an almost unbiased estimate of the mean squared error (MSE), SURE-TCNN has the potential to provide similar results to those of the supervised MSE-based TCNN with ground-truth data. For ease of implementation, the properties of a transform-based tensor-tensor product (t-product) are followed to establish a solid theoretical connection between the SURE-TCNN tensor and matrix. Derived from this connection, the SURE-TCNN weight parameters are determined by implementing matrix-based SURE-convolutional neural networks (CNNs) on each frontal slice in the time-frequency domain (e.g., the wavelet domain). Synthetic and field data examples demonstrate the superior performances of the proposed model against three state-of-the-art (SOTA) methods.

Large Graph Signal Denoising With Application to Differential Privacy

Over the last decade, signal processing on graphs has become a very active area of research. Specifically, the number of applications, for instance in statistical or deep learning, using frames built from graphs, such as wavelets on graphs, has increased significantly. We consider in particular the case of signal denoising on graphs via a data-driven wavelet tight frame methodology. This adaptive approach is based on a threshold calibrated using Stein's unbiased risk estimate adapted to a tight-frame representation. We make it scalable to large graphs using Chebyshev-Jackson polynomial approximations, which allow fast computation of the wavelet coefficients, without the need to compute the Laplacian eigendecomposition. However, the overcomplete nature of the tight-frame, transforms a white noise into a correlated one. As a result, the covariance of the transformed noise appears in the divergence term of the SURE, thus requiring the computation and storage of the frame, which leads to an impractical calculation for large graphs. To estimate such covariance, we develop and analyze a Monte-Carlo strategy, based on the fast transformation of zero mean and unit variance random variables. This new data-driven denoising methodology finds a natural application in differential privacy. A comprehensive performance analysis is carried out on graphs of varying size, from real and simulated data.