Natural Image Denoising Research Articles

Speckle noise removal via learned variational models

In this paper, we address the image denoising problem in presence of speckle degradation typically arising in ultra-sound images. Variational methods and Convolutional Neural Networks (CNNs) are considered well-established methods for specific noise types, such as Gaussian and Poisson noise. Nonetheless, the advances achieved by these two classes of strategies are limited when tackling the de-speckle problem. In fact, variational methods for speckle removal typically amounts to solve a non-convex functional with the related issues from the convergence viewpoint; on the other hand, the lack of large datasets of noise-free ultra-sound images has not allowed the extension of the state-of-the-art CNN denoiser methods to the case of speckle degradation. Here, we aim at combining the classical variational methods with the predictive properties of CNNs by considering a weighted total variation regularized model; the local weights are obtained as the output of a statistically inspired neural network that is trained on a small and composite dataset of natural and synthetic images. The resulting non-convex variational model, which is minimized by means of the Alternating Direction Method of Multipliers (ADMM) is proven to converge to a stationary point. Numerical tests show the effectiveness of our approach for the denoising of natural and satellite images.

CAIR: Combining integrated attention with iterative optimization learning for sparse-view CT reconstruction

Sparse-view CT is an efficient way for low dose scanning but degrades image quality. Inspired by the successful use of non-local attention in natural image denoising and compression artifact removal, we proposed a network combining integrated attention and iterative optimization learning for sparse-view CT reconstruction (CAIR). Specifically, we first unrolled the proximal gradient descent into a deep network and added an enhanced initializer between the gradient term and the approximation term. It can enhance the information flow between different layers, fully preserve the image details, and improve the network convergence speed. Secondly, the integrated attention module was introduced into the reconstruction process as a regularization term. It adaptively fuses the local and non-local features of the image which are used to reconstruct the complex texture and repetitive details of the image, respectively. Note that we innovatively designed a one-shot iteration strategy to simplify the network structure and reduce the reconstruction time while maintaining image quality. Experiments showed that the proposed method is very robust and outperforms state-of-the-art methods in terms of both quantitative and qualitative, greatly improving the preservation of structures and the removal of artifacts.

Improved Ultrafast Power Doppler Imaging by Using Spatiotemporal Non-Local Means Filtering.

The change of microvasculature is associated with the occurrence and development of many diseases. Ultrafast power Doppler imaging (uPDI) is an emerging technology for the visualization of microvessels due to the development of ultrafast plane wave (PW) imaging and advanced clutter filters. However, the low signal-to-noise ratio (SNR) caused by unfocused transmit of PW imaging deteriorates the subsequent imaging of microvasculature. Nonlocal means (NLM) filtering has been demonstrated to be effective in the denoising of both natural and medical images, including ultrasound power Doppler images. However, the feasibility and performance of applying an NLM filter on the ultrasound radio frequency (RF) data have not been investigated so far. In this study, we propose to apply an NLM filter on the spatiotemporal domain of clutter filtered blood flow RF data (St-NLM) to improve the quality of uPDI. Experiments were conducted to compare the proposed method with three different methods (under various similarity window sizes), including conventional uPDI without NLM filtering (Non-NLM), NLM filtering on the obtained power Doppler images (PD-NLM), and NLM filtering on the spatial domain of clutter filtered blood flow RF data (S-NLM). Phantom experiments, in vivo contrast-enhanced human spinal cord tumor experiments, and in vivo contrast-free human liver experiments were performed to demonstrate the superiority of the proposed St-NLM method over the other three methods. Qualitative and quantitative results show that the proposed St-NLM method can effectively suppress the background noise, improve the contrast between vessels and background, and preserve the details of small vessels at the same time. In the human liver study, the proposed St-NLM method achieves 31.05-, 24.49-, and 11.15-dB higher contrast-to-noise ratios (CNRs) and 36.86-, 36.86-, and 15.22-dB lower noise powers than Non-NLM, PD-NLM, and S-NLM, respectively. In the human spinal cord tumor, the full-width at half-maximums (FWHMs) of vessel cross Section are 76, 201, and [Formula: see text] for St-NLM, Non-NLM, and S-NLM, respectively. The proposed St-NLM method can enhance the microvascular visualization in uPDI and has the potential for the diagnosis of many microvessel-change-related diseases.

Contextual Transformation Network for Lightweight Remote-Sensing Image Super-Resolution

Current super-resolution networks typically reduce network parameters and multiadds operations by designing lightweight structures, but lightening the convolution layer is often ignored. In this work, we observe that <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$3 \times 3$ </tex-math></inline-formula> convolutions occupy a high percentage of network parameters in most lightweight super-resolution networks. This motivates us to consider lightening super-resolution networks by replacing <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$3 \times 3$ </tex-math></inline-formula> convolutions with lightweight convolutions, while maintaining the performance. To achieve this, we propose a lightweight convolution layer named contextual transformation layer (CTL). It can yield efficient contextual features through a context feature extraction module and enrich extracted contextual features through a context feature transformation module. Based on CTLs, we build a lightweight super-resolution network called contextual transformation network (CTN) for remote-sensing image super-resolution. Specifically, we use two CTLs to construct a contextual transformation block (CTB) for hierarchical feature learning. Interleaved with a CTB, a context enhancement module (CEM) is employed to enhance the extracted feature representations. All extracted features are processed by a contextual feature aggregation module for final remote-sensing image super-resolution. Extensive experiments are performed on a remote-sensing image super-resolution benchmark named UC Merced. Our method achieves superior results to the other state-of-the-art methods. To demonstrate the generalization ability of our CTL, we extend our CTN to two relevant tasks: natural image super-resolution and natural image denoising. Experimental results on natural image super-resolution benchmarks (i.e., Set5, Set14, B100, Urban100, and Manga109) and natural image denoising benchmarks (i.e., SIDD and DND) further prove the superiority of our method. Our code is publicly available at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/BITszwang/CTNet</uri> .

Hyperspectral Denoising Using Unsupervised Disentangled Spatiospectral Deep Priors

Image denoising is often empowered by accurate prior information. In recent years, data-driven neural network priors have shown promising performance for RGB natural image denoising. Compared to classic handcrafted priors (e.g., sparsity and total variation), the "deep priors" are learned using a large number of training samples -- which can accurately model the complex image generating process. However, data-driven priors are hard to acquire for hyperspectral images (HSIs) due to the lack of training data. A remedy is to use the so-called unsupervised deep image prior (DIP). Under the unsupervised DIP framework, it is hypothesized and empirically demonstrated that proper neural network structures are reasonable priors of certain types of images, and the network weights can be learned without training data. Nonetheless, the most effective unsupervised DIP structures were proposed for natural images instead of HSIs. The performance of unsupervised DIP-based HSI denoising is limited by a couple of serious challenges, namely, network structure design and network complexity. This work puts forth an unsupervised DIP framework that is based on the classic spatio-spectral decomposition of HSIs. Utilizing the so-called linear mixture model of HSIs, two types of unsupervised DIPs, i.e., U-Net-like network and fully-connected networks, are employed to model the abundance maps and endmembers contained in the HSIs, respectively. This way, empirically validated unsupervised DIP structures for natural images can be easily incorporated for HSI denoising. Besides, the decomposition also substantially reduces network complexity. An efficient alternating optimization algorithm is proposed to handle the formulated denoising problem. Semi-real and real data experiments are employed to showcase the effectiveness of the proposed approach.

Removal of Gaussian White Noises from the image by probability map prediction based Deep learning approach

Deep learning methods have gained an increasing research interest, especially in the field of image denoising. Although there are significant differences between the different types of deep learning techniques used for natural image denoising, it includes significant process and procedure differences between them. To be specific, discriminative learning based on deep learning convolutional neural network (CNN) may effectively solve the problem of Gaussian noise. Deep learning based optimization models are useful in predicting the true noise level. However, no relevant research has attempted to summarize the different deep learning approaches for performing image denoising in one location. It has been suggested to build the proposed framework in parallel with the previously trained CNN to enhance the training speed and accuracy in denoising the Gaussian White Noise (GWN). In the proposed architecture, ground truth maps are created by combining the additional patches of input with original pictures to create ground truth maps. Furthermore, by changing kernel weights for forecasting probability maps, the loss function may be reduced to its smallest value. Besides, it is efficient in terms of processing time with less sparsity while enlarging the objects present in the images. As well as in conventional methods, various performance measures such as PSNR, MSE, and SSIM are computed and compared with one another.

Quantitative evaluation of deep convolutional neural network-based image denoising for low-dose computed tomography

To minimize radiation risk, dose reduction is important in the diagnostic and therapeutic applications of computed tomography (CT). However, image noise degrades image quality owing to the reduced X-ray dose and a possible unacceptably reduced diagnostic performance. Deep learning approaches with convolutional neural networks (CNNs) have been proposed for natural image denoising; however, these approaches might introduce image blurring or loss of original gradients. The aim of this study was to compare the dose-dependent properties of a CNN-based denoising method for low-dose CT with those of other noise-reduction methods on unique CT noise-simulation images. To simulate a low-dose CT image, a Poisson noise distribution was introduced to normal-dose images while convoluting the CT unit-specific modulation transfer function. An abdominal CT of 100 images obtained from a public database was adopted, and simulated dose-reduction images were created from the original dose at equal 10-step dose-reduction intervals with a final dose of 1/100. These images were denoised using the denoising network structure of CNN (DnCNN) as the general CNN model and for transfer learning. To evaluate the image quality, image similarities determined by the structural similarity index (SSIM) and peak signal-to-noise ratio (PSNR) were calculated for the denoised images. Significantly better denoising, in terms of SSIM and PSNR, was achieved by the DnCNN than by other image denoising methods, especially at the ultra-low-dose levels used to generate the 10% and 5% dose-equivalent images. Moreover, the developed CNN model can eliminate noise and maintain image sharpness at these dose levels and improve SSIM by approximately 10% from that of the original method. In contrast, under small dose-reduction conditions, this model also led to excessive smoothing of the images. In quantitative evaluations, the CNN denoising method improved the low-dose CT and prevented over-smoothing by tailoring the CNN model.

Denoising convolutional neural network inspired via multi-layer convolutional sparse coding

Sparse prior to image denoising is a classical research field with a long history in computer vision. We propose an end-to-end supervised neural network, named DnMLCSC-net, which is inspired via multi-layer convolutional sparse coding model embedded with symbiotic analysis–synthesis priors for natural image denoising. Unfolding a multi-layer, learned iterative soft thresholding algorithm (ML-LISTA) and developing into a convolutional recurrent neural network, all parameters in the model are updated adaptively to minimize mixed loss via gradient descent using backpropagation. In addition, a combined ReLU function is taken as the activation function. Inconsistent dilated convolution and batch normalization were empirically introduced into the encoding layers corresponding to the first iteration of ML-LISTA. Experimental results show that our network achieves a competitive denoising effect in comparison with several state-of-the-art denoising methods.

Learning Spatially Variant Linear Representation Models for Joint Filtering.

Joint filtering mainly uses an additional guidance image as a prior and transfers its structures to the target image in the filtering process. Different from existing approaches that rely on local linear models or hand-designed objective functions to extract the structural information from the guidance image, we propose a new joint filtering method based on a spatially variant linear representation model (SVLRM), where the target image is linearly represented by the guidance image. However, learning SVLRMs for vision tasks is a highly ill-posed problem. To estimate the spatially variant linear representation coefficients, we develop an effective approach based on a deep convolutional neural network (CNN). As such, the proposed deep CNN (constrained by the SVLRM) is able to model the structural information of both the guidance and input images. We show that the proposed approach can be effectively applied to a variety of applications, including depth/RGB image upsampling and restoration, flash deblurring, natural image denoising, and scale-aware filtering. In addition, we show that the linear representation model can be extended to high-order representation models (e.g., quadratic and cubic polynomial representations). Extensive experimental results demonstrate that the proposed method performs favorably against the state-of-the-art methods that have been specifically designed for each task.

Non-local mean denoising using multiple PET reconstructions

ObjectivesNon-local mean (NLM) filtering has been broadly used for denoising of natural and medical images. The NLM filter relies on the redundant information, in the form of repeated patterns/textures, in the target image to discriminate the underlying structures/signals from noise. In PET (or SPECT) imaging, the raw data could be reconstructed using different parameters and settings, leading to different representations of the target image, which contain highly similar structures/signals to the target image contaminated with different noise levels (or properties). In this light, multiple-reconstruction NLM filtering (MR-NLM) is proposed, which relies on the redundant information provided by the different reconstructions of the same PET data (referred to as auxiliary images) to conduct the denoising process.MethodsImplementation of the MR-NLM approach involved the use of twelve auxiliary PET images (in addition to the target image) reconstructed using the same iterative reconstruction algorithm with different numbers of iterations and subsets. For each target voxel, the patches of voxels at the same location are extracted from the auxiliary PET images based on which the NLM denoising process is conducted. Through this, the exhaustive search scheme performed in the conventional NLM method to find similar patches of voxels is bypassed. The performance evaluation of the MR-NLM filter was carried out against the conventional NLM, Gaussian and bilateral post-reconstruction approaches using the experimental Jaszczak phantom and 25 whole-body PET/CT clinical studies.ResultsThe signal-to-noise ratio (SNR) in the experimental Jaszczak phantom study improved from 25.1 when using Gaussian filtering to 27.9 and 28.8 when the conventional NLM and MR-NLM methods were applied (p value < 0.05), respectively. Conversely, the Gaussian filter led to quantification bias of 35.4%, while NLM and MR-NLM approaches resulted in a bias of 32.0% and 31.1% (p value < 0.05), respectively. The clinical studies further confirm the superior performance of the MR-NLM method, wherein the quantitative bias measured in malignant lesions (hot spots) decreased from − 12.3 ± 2.3% when using the Gaussian filter to − 3.5 ± 1.3% and − 2.2 ± 1.2% when using the NLM and MR-NLM approaches (p value < 0.05), respectively.ConclusionThe MR-NLM approach exhibited promising performance in terms of noise suppression and signal preservation for PET images, thus translating into higher SNR compared to the conventional NLM approach. Despite the promising performance of the MR-NLM approach, the additional computational burden owing to the requirement of multiple PET reconstruction still needs to be addressed.

Ultrasound image de-speckling by a hybrid deep network with transferred filtering and structural prior

Deep neural-network has been widely used in natural image denoising. However, due to the lack of label of real ultrasound (US) B-mode image for de-speckling, the deep neural network is greatly restricted in US image de-speckling. In this paper, we propose to use transfer learning and two types of prior knowledge to construct a hybrid neural network structure for de-speckling. Firstly, based on a given US image model, the speckle noise is similar to Gaussian distribution in the logarithmic transformation domain, called Gaussian prior knowledge. The distribution parameters are estimated in the logarithmic transformation domain based on four typical traditional US image de-speckling methods with maximum likelihood estimation. Secondly, depending on the prior parameters, a transferable denoising network is trained with clean natural image dataset. Finally, a VGGNet is used to extract the structure boundaries before and after US image de-speckling based on the transfer network, and we call it structural prior knowledge. The structural boundaries of a US image should be unchanged after the de-speckling, and hence we use this constraint to fine-tune the transfer network. The proposed de-speckling framework is verified on artificially generated phantom (AGP) images and real US images, and the results demonstrate its effectiveness.

Image denoising in undecimated dual-tree complex wavelet domain using multivariate t-distribution

Denoising of natural images is a basic problem in image processing. The present paper proposes a new algorithm for image denoising based on the maximum a-posteriori (MAP) estimator in undecimated dual-tree complex wavelet transform. The undecimated dual-tree complex wavelet transform (UDT-CWT), along with the directional selectivity of the dual-tree complex wavelet transform (DT-CWT), offers exact translational invariance property through removing the down-sampling of filter outputs together with the up-sampling of the complex filter pairs of DT-CWT. These properties are very important in image denoising. The performance of the MAP estimator depends strongly on the probability of noise-free wavelet coefficients. In our proposed denoising method, multivariate t-distribution is applied as the prior probability of noise-free coefficients. The t-distribution can accurately model the statistics of wavelet coefficients, which have peaky and heavy-tailed characteristics. On the other hand, the multivariate model makes it possible to take into account the dependencies of wavelet coefficients and their neighbors. Also, in our work, the necessary parameters of the multivariate distribution will be estimated in a locally-adaptive way to improve the denoising results via using the correlations among the amplitudes of neighbor coefficients. Simulation results delineate that the proposed algorithm outperforms state-of-the-art denoising algorithms in the literature.

DEMC: A Deep Dual-Encoder Network for Denoising Monte Carlo Rendering

In this paper, we present DEMC, a deep Dual-Encoder network to remove Monte Carlo noise efficiently while preserving details. Denoising Monte Carlo rendering is different from natural image denoising since inexpensive by-products (feature buffers) can be extracted in the rendering stage. Most of them are noise-free and can provide sufficient details for image reconstruction. However, these feature buffers also contain redundant information, which makes Monte Carlo denoising different from natural image denoising. Hence, the main challenge of this topic is how to extract useful information and reconstruct clean images. To address this problem, we propose a novel network structure, Dual-Encoder network with a feature fusion sub-network, to fuse feature buffers firstly, then encode the fused feature buffers and a noisy image simultaneously, and finally reconstruct a clean image by a decoder network. Compared with the state-of-the-art methods, our model is more robust on a wide range of scenes and is able to generate satisfactory results in a significantly faster way.

Low-Dose CT Image Denoising Model Based on Sparse Representation by Stationarily Classified Sub-Dictionaries

Low-dose computed tomography (LDCT) technique is an important imaging modality, but LDCT images are always severely degraded by mottle noise and streak artifacts. The recently proposed nonlocally centralized sparse representation (NCSR) algorithm has good performance in natural image denoising, but it suffers from residual streak artifacts and can't preserve edges structure information well when implemented in LDCT image denoising. In addition, it has high computational complexity. To address this problem, in this paper, we propose an improved model, i.e. SNCSR model, based on the stationary PCA sub-dictionaries, nonlocally centralized sparse representation and relative total variation. In the SNCSR model, in order to learn more accurate sub-dictionaries, the LDCT image is preprocessed by the improved total variation (ITV) model in which the weighted coefficient of the regularization term is constructed depending on a clipped and normalized local activity. In addition, the maximum eigenvalue of the gradient covariance matrix of the image patch is used to distinguish edge structure information from background region so that the restored image can be represented more sparsely. Moreover, unlike the NCSR model that needs to learn sub-dictionaries in each outer loop, the proposed model learns stationary sub-dictionaries only once before iteration starts, which shorten the computation time significantly. At last, the relative total variation (RTV) algorithm is applied to further reduce the residual artifacts in the recovered image more thoroughly. The experiments are performed on the simulated pelvis phantom, the actual thoracic phantom and the clinical abdominal data. Compared with several other competitive denoising algorithms, both subjective visual effect and objective evaluation criteria show that the proposed SNCSR model has lower computational complexity and can improve LDCT images quality more effectively.

Natural Image Denoising and Enhancement Based On Iterative Bilateral Filter

Natural Image Denoising and Enhancement Based On Iterative Bilateral Filter

A patch-based low-rank tensor approximation model for multiframe image denoising

Compared with matrix, tensor is a more natural representation for multiframe image, such as hyperspectral image and MRI image. Low-rankness of tensor is essential to describe the intrinsic geometrical structure of these data. Patch-based low-rank models have shown their ability to exploit spatial redundancy of computer vision data especially for natural image denoising. However, most of the existed patch-based matrix models are based on two dimensional low-rankness, which cannot fully reveal the correlation of every direction in high-order multiframe images; the existed patch-based tensor models either need additional assumptions or need SVD in every loop of iteration which is computationally expensive. In this paper, we propose a novel patch-based model to recover a low-rank tensor by simultaneously performing low-rank matrix factorizations to the all-mode matricizations of the underlying low-rank tensor. An augmented Lagrangian alternating minimization algorithm is implemented to solve the model along with two adaptive rank-adjusting strategies when the exact rank is unknown. We apply the proposed algorithm to multiframe image denoising by exploiting the nonlocal self-similarity. Experimental results show that our algorithm can better preserve the sharpness of important image structures and outperforms several state-of-the-art denoising methods.

Denoising of natural images through robust wavelet thresholding and genetic programming

Digital images play an essential role in analysis tasks that can be applied in various knowledge domains, including medicine, meteorology, geology, and biology. Such images can be degraded by noise during the process of acquisition, transmission, storage, or compression. The use of local filters in image restoration may generate artifacts when these filters are not well adapted to the image content as a result of the heuristic optimization of local filters. Denoising methods based on learning procedure are more capable than parametric filters for addressing the conflicts between noise suppression and artifact reduction. In this study, we present a nonlinear filtering method based on a two-step switching scheme to remove both salt-and-pepper and additive white Gaussian noises. In the switching scheme, two cascaded detectors are used to detect noise, and two corresponding estimators are employed to effectively and efficiently filter the noise in an image. In the process of training, a method according to patch clustering is utilized, and genetic programming (GP) is subsequently applied to determine the optimum filter (wavelet-domain filter) for each individual cluster, while in testing part, the optimum filter trained beforehand by GP is recovered and used on the inputted corrupted patch. This adaptive structure is employed to cope with several noise types. Experimental and comparative analysis results show that the denoising performance of the proposed method is superior to that of existing denoising methods as per both quantitative and qualitative assessments.

Con-Patch: When a Patch Meets Its Context

Measuring the similarity between the patches in images is a fundamental building block in various tasks. Naturally, the patch size has a major impact on the matching quality and on the consequent application performance. Under the assumption that our patch database is sufficiently sampled, using large patches (e.g., 21 × 21 ) should be preferred over small ones (e.g., 7 × 7 ). However, this dense-sampling assumption is rarely true; in most cases, large patches cannot find relevant nearby examples. This phenomenon is a consequence of the curse of dimensionality, stating that the database size should grow exponentially with the patch size to ensure proper matches. This explains the favored choice of small patch size in most applications. Is there a way to keep the simplicity and work with small patches while getting some of the benefits that large patches provide? In this paper, we offer such an approach. We propose to concatenate the regular content of a conventional (small) patch with a compact representation of its (large) surroundings-its context. Therefore, with a minor increase of the dimensions (e.g., with additional ten values to the patch representation), we implicitly/softly describe the information of a large patch. The additional descriptors are computed based on a self-similarity behavior of the patch surrounding. We show that this approach achieves better matches, compared with the use of conventional-size patches, without the need to increase the database-size. Also, the effectiveness of the proposed method is tested on three distinct problems: 1) external natural image denoising; 2) depth image super-resolution; and 3) motion-compensated frame-rate up conversion.

Statistics of Natural Stochastic Textures and Their Application in Image Denoising.

Natural stochastic textures (NSTs), characterized by their fine details, are prone to corruption by artifacts, introduced during the image acquisition process by the combined effect of blur and noise. While many successful algorithms exist for image restoration and enhancement, the restoration of natural textures and textured images based on suitable statistical models has yet to be further improved. We examine the statistical properties of NST using three image databases. We show that the Gaussian distribution is suitable for many NST, while other natural textures can be properly represented by a model that separates the image into two layers; one of these layers contains the structural elements of smooth areas and edges, while the other contains the statistically Gaussian textural details. Based on these statistical properties, an algorithm for the denoising of natural images containing NST is proposed, using patch-based fractional Brownian motion model and regularization by means of anisotropic diffusion. It is illustrated that this algorithm successfully recovers both missing textural details and structural attributes that characterize natural images. The algorithm is compared with classical as well as the state-of-the-art denoising algorithms.

Denoising by low-rank and sparse representations

Due to the ill-posed nature of image denoising problem, good image priors are of great importance for an effective restoration. Nonlocal self-similarity and sparsity are two popular and widely used image priors which have led to several state-of-the-art methods in natural image denoising. In this paper, we take advantage of these priors and propose a new denoising algorithm based on sparse and low-rank representation of image patches under a nonlocal framework. This framework consists of two complementary steps. In the first step, noise removal from groups of matched image patches is formulated as recovery of low-rank matrices from noisy data. This problem is then efficiently solved under asymptotic matrix reconstruction model based on recent results from random matrix theory which leads to a parameter-free optimal estimator. Nonlocal learned sparse representation is adopted in the second step to suppress artifacts introduced in the previous estimate. Experimental results, demonstrate the superior denoising performance of the proposed algorithm as compared with the state-of-the-art methods.