Image Denoisers Research Articles

Pseudo-ISP: Learning pseudo in-camera signal processing pipeline from a color image denoiser

Existing deep denoisers usually ignore the problem of noise discrepancy between training and test images caused by the different noise modeling of sensor and in-camera signal processing (ISP) pipeline, which inevitably degrades the denoising performance. In this paper, we present an unpaired learning scheme to adapt the color image denoisers for handling test images with noise discrepancy. To this end, we consider a practical training setting equipped with a pre-trained color denoiser, a set of test noisy sRGB images, and an unpaired set of clean sRGB images. Then, we propose to alternate between two modules, i.e., Pseudo-ISP training for learning noise modeling of sensor and in-camera signal processing (ISP) pipeline and denoiser adaption. Instead of modeling the complex noise in sRGB images, we assume the signal-dependent and spatially independent noise in rawRGB space, and specifically design the Pseudo-ISP to learn the pseudo ISP pipeline and pseudo rawRGB noise model jointly. Applying the learned Pseudo-ISP to the unpaired set of clean sRGB images, the corresponding realistic noisy sRGB observations can be easily obtained. Conversely, using the generated paired data for model fine-tuning, the color denoiser can be well adapted for test images. Through iterating between Pseudo-ISP training and denoiser adaption, denoising performance of deep denoisers can be further improved. Experiments show that our Pseudo-ISP not only can boost simple Gaussian blurring-based denoiser to achieve competitive performance against CBDNet, but also is effective in improving state-of-the-art deep denoisers, e.g., CBDNet and RIDNet.

Target-Aware Image Denoising for Inverse Monte Carlo Rendering

Physically based differentiable rendering allows an accurate light transport simulation to be differentiated with respect to the rendering input, i.e., scene parameters, and it enables inferring scene parameters from target images, e.g., photos or synthetic images, via an iterative optimization. However, this inverse Monte Carlo rendering inherits the fundamental problem of the Monte Carlo integration, i.e., noise, resulting in a slow optimization convergence. An appealing approach to addressing such noise is exploiting an image denoiser to improve optimization convergence. Unfortunately, the direct adoption of existing image denoisers designed for ordinary rendering scenarios can drive the optimization into undesirable local minima due to denoising bias. It motivates us to reformulate a new image denoiser specialized for inverse rendering. Unlike existing image denoisers, we conduct our denoising by considering the target images, i.e., specific information in inverse rendering. For our target-aware denoising, we determine our denoising weights via a linear regression technique using the target. We demonstrate that our denoiser enables inverse rendering optimization to infer scene parameters robustly through a diverse set of tests.

Convergent Regularization in Inverse Problems and Linear Plug-and-Play Denoisers

AbstractRegularization is necessary when solving inverse problems to ensure the well-posedness of the solution map. Additionally, it is desired that the chosen regularization strategy is convergent in the sense that the solution map converges to a solution of the noise-free operator equation. This provides an important guarantee that stable solutions can be computed for all noise levels and that solutions satisfy the operator equation in the limit of vanishing noise. In recent years, reconstructions in inverse problems are increasingly approached from a data-driven perspective. Despite empirical success, the majority of data-driven approaches do not provide a convergent regularization strategy. One such popular example is given by iterative plug-and-play (PnP) denoising using off-the-shelf image denoisers. These usually provide only convergence of the PnP iterates to a fixed point, under suitable regularity assumptions on the denoiser, rather than convergence of the method as a regularization technique, thatis under vanishing noise and regularization strength. This paper serves two purposes: first, we provide an overview of the classical regularization theory in inverse problems and survey a few notable recent data-driven methods that are provably convergent regularization schemes. We then continue to discuss PnP algorithms and their established convergence guarantees. Subsequently, we consider PnP algorithms with learned linear denoisers and propose a novel spectral filtering technique of the denoiser to control the strength of regularization. Further, by relating the implicit regularization of the denoiser to an explicit regularization functional, we are the first to rigorously show that PnP with a learned linear denoiser leads to a convergent regularization scheme. The theoretical analysis is corroborated by numerical experiments for the classical inverse problem of tomographic image reconstruction.

Image Denoisers in Tomography: Enhancing Quality of Bimodal Imaging for Corrosion Studies

Image Denoisers in Tomography: Enhancing Quality of Bimodal Imaging for Corrosion Studies

Plug-and-Play Image Reconstruction Is a Convergent Regularization Method.

Non-uniqueness and instability are characteristic features of image reconstruction methods. As a result, it is necessary to develop regularization methods that can be used to compute reliable approximate solutions. A regularization method provides a family of stable reconstructions that converge to a specific solution of the noise-free problem as the noise level tends to zero. The standard regularization technique is defined by a variational image reconstruction that minimizes a data discrepancy augmented by a regularizer. The actual numerical implementation makes use of iterative methods, often involving proximal mappings of the regularizer. In recent years, Plug-and-Play (PnP) image reconstruction has been developed as a new powerful generalization of variational methods based on replacing proximal mappings by more general image denoisers. While PnP iterations yield excellent results, neither stability nor convergence in the sense of regularization have been studied so far. In this work, we extend the idea of PnP by considering families of PnP iterations, each accompanied by its own denoiser. As our main theoretical result, we show that such PnP reconstructions lead to stable and convergent regularization methods. This shows for the first time that PnP is as mathematically justified for robust image reconstruction as variational methods.

Wavelet analysis model inspired convolutional neural networks for image denoising

In recent years, many image denoising methods have been proposed based on convolutional neural networks (CNNs). While these methods have shown continuous performance improvement by introducing various mechanisms and structures, their computational cost tends to become increasingly expensive, owing to the resulting complex network architectures. This paper aims at winning the trade-off between computational efficiency and denoising performance for CNN-based image denoisers. Towards this end, we draw inspirations from traditional variational models with wavelet analysis operators for CNN architecture design. A model-inspired CNN is proposed with four key modules: iterative encoding-decoding units inspired by the iterative denoising process, directional convolutions inspired by the separable wavelet filters, inception modules inspired by the multi-scale analysis of wavelets, and stage-wise connections inspired by the adding noise back operation. In experiments, our CNN shows high computational efficiency in both training and test, with competitive results to state-of-the-art approaches.

An unsupervised fusion network for boosting denoising performance

While many efforts have been devoted to addressing image denoising and achieve continuously improving results during the past few decades, it is fair to say that no a stand-alone method is consistently better than others. Nonetheless, many existing denoising methods, each having a different denoising capability, can yield various but complementary denoised images with respect to specific local areas. To effectively exploit the complementarity and diversity among the denoised images obtained with different denoisers, in this work we fuse them to produce an overall better result, which is fundamental to achieve robust and competitive denoising performance especially for complex scenes. A framework called deep fusion network (DFNet) is proposed to generate a consistent estimation about the final denoised image, taking advantage of the complementarity of denoisers and suppressing the bias. Specifically, given a noisy image, we first exploit a set of representative image denoisers to denoise it respectively, and obtain the corresponding initial denoised images. Then these initial denoised images are concatenated and fed into the proposed DFNet, and the proposed DFNet seeks to adjust its network parameters to produce the fused image (as the final denoised image) with an unsupervised training strategy through minimizing the carefully designed loss function. The experimental results show that our approach outperforms the stand-alone methods as well as the ones using combination strategy by large margin both in objective and subjective evaluations. Compared to the those methods that are relatively close to our strategy, the proposed DFNet is extensible and parameter free, which means it can cope with a variable number of different denoisers and avoid the manual intervention during the fusion process. The proposed DFNet has greater flexibility and better practicality.

Image Denoising with Control over Deep Network Hallucination

Deep image denoisers achieve state-of-the-art results but with a hidden cost. As witnessed in recent literature, these deep networks are capable of overfitting their training distributions, causing inaccurate hallucinations to be added to the output and generalizing poorly to varying data. For better control and interpretability over a deep denoiser, we propose a novel framework exploiting a denoising network. We call it controllable confidence-based image denoising (CCID). In this framework, we exploit the outputs of a deep denoising network alongside an image convolved with a reliable filter. Such a filter can be a simple convolution kernel which does not risk adding hallucinated information. We propose to fuse the two components with a frequency-domain approach that takes into account the reliability of the deep network outputs. With our framework, the user can control the fusion of the two components in the frequency domain. We also provide a user-friendly map estimating spatially the confidence in the output that potentially contains network hallucination. Results show that our CCID not only provides more interpretability and control, but can even outperform both the quantitative performance of the deep denoiser and that of the reliable filter, especially when the test data diverge from the training data.

A novel fusion paradigm for multi-channel image denoising

Multi-channel and single-channel image denoising are on two important development fronts. Integrating multi-channel and single-channel image denoisers for further improvement is a valuable research direction. A natural assumption is that using more useful information is helpful to the output results. In this paper, a novel multi-channel and single-channel fusion paradigm (MSF) is proposed. The proposed MSF works by fusing the estimates of a multi-channel image denoiser and a single-channel image denoiser. The performance of recent multi-channel image denoising methods involved in the proposed MSF can be further improved at low additional time-consuming cost. Specifically, the validity principle of the proposed MSF is that the fused single-channel image denoiser can produce auxiliary estimate for the involved multi-channel image denoiser in a designed underdetermined transform domain. Based on the underdetermined transformation, we create a corresponding orthogonal transformation for fusion and better restore the multi-channel images. The quantitative and visual comparison results demonstrate that the proposed MSF can be effectively applied to several state-of-the-art multi-channel image denoising methods.

Revealing stable and unstable modes of denoisers through nonlinear eigenvalue analysis

In this paper, we propose to analyze stable and unstable modes of black-box image denoisers through nonlinear eigenvalue analysis. We aim to find input images for which the denoiser output is proportional to the input. We treat this as a generalized nonlinear eigenproblem. Potential implications are wide, as most image processing algorithms can be viewed as black-box operators. We introduce a generalized nonlinear power-method to solve eigenproblems for such operators. This allows us to reveal stable modes of the denoiser: optimal inputs, achieving superior PSNR in noise removal. Analogously to the linear case, such stable modes show coarse structures and correspond to large eigenvalues. We also provide a method to generate unstable modes, which the denoiser suppresses strongly, which are textural with small eigenvalues. We validate the method using total-variation (TV) and demonstrate it on the EPLL (Zoran–Weiss) and the Non-local means denoisers. Finally, we suggest an encryption–decryption application.

Image De-Noising With Machine Learning: A Review

Images are susceptible to various kinds of noises, which corrupt the pictorial information stored in the images. Image de-noising has become an integral part of the image processing workflow. It is used to attenuate the noises and accentuate the specific image information stored within. Machine learning is an important tool in the image-de-noising workflow in terms of its robustness, accuracy, and time requirement. This paper explores the numerous state-of-the-art machine-learning-based image de-noisers like dictionary learning models, convolutional neural networks and generative adversarial networks for a range of noises like Gaussian, Impulse, Poisson, Mixed and Real-World noises. The motivation, algorithm and framework of different machine learning de-noisers are analyzed. These de-noisers are compared using PSNR as quality assessment metric on some benchmark datasets. The best de-noising results for different noise types are discussed along with future prospects. Among various Gaussian noise de-noisers, GCBD, BRDNet and PReLU network prove to be promising. CNN+LSTM, and MC2RNet are most suitable CNN-based Poisson de-noisers. For impulse noise removal, Blind CNN, and CNN+PSO perform well. For mixed noise removal, WDL, EM-CNN, CNN, SDL, and Mixed CNN are prominent. De-noisers like GRDN and DDFN show accurate results in the domain of real-world de-noising.

Real-Time Denoising of Volumetric Path Tracing for Direct Volume Rendering.

Direct volume rendering (DVR) using volumetric path tracing (VPT) is a scientific visualization technique that simulates light transport with objects' matter using physically-based lighting models. Monte Carlo (MC) path tracing is often used with surface models, yet its application for volumetric models is difficult due to the complexity of integrating MC light-paths in volumetric media with none or smooth material boundaries. Moreover, auxiliary geometry-buffers (G-buffers) produced for volumes are typically very noisy, failing to guide image denoisers relying on that information to preserve image details. This makes existing real-time denoisers, which take noise-free G-buffers as their input, less effective when denoising VPT images. We propose the necessary modifications to an image-based denoiser previously used when rendering surface models, and demonstrate effective denoising of VPT images. In particular, our denoising exploits temporal coherence between frames, without relying on noise-free G-buffers, which has been a common assumption of existing denoisers for surface-models. Our technique preserves high-frequency details through a weighted recursive least squares that handles heterogeneous noise for volumetric models. We show for various real data sets that our method improves the visual fidelity and temporal stability of VPT during classic DVR operations such as camera movements, modifications of the light sources, and editions to the volume transfer function.

Hyperspectral Unmixing via Nonnegative Matrix Factorization With Handcrafted and Learned Priors

Nowadays, nonnegative matrix factorization (NMF)-based methods have been widely applied to blind spectral unmixing. Introducing proper regularizers to NMF is crucial for mathematically constraining the solutions and physically exploiting spectral and spatial properties of images. Generally, properly handcrafted regularizers and solving the associated complex optimization problem are nontrivial tasks. In our work, we propose an NMF-based unmixing framework which jointly uses a learned regularizer from data and a handcrafted regularizer. To be specific, we plug learned priors of abundances where the associated subproblem can be addressed using various image denoisers, and we consider an <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$\ell _{2,1}$ </tex-math></inline-formula> -norm as an example to illustrate the way of integrating handcrafted regularizers. The proposed framework is flexible and extendable. Both synthetic data and real airborne data are conducted to confirm the effectiveness of our method.

When AWGN-Based Denoiser Meets Real Noises

Discriminative learning based image denoisers have achieved promising performance on synthetic noises such as Additive White Gaussian Noise (AWGN). The synthetic noises adopted in most previous work are pixel-independent, but real noises are mostly spatially/channel-correlated and spatially/channel-variant. This domain gap yields unsatisfied performance on images with real noises if the model is only trained with AWGN. In this paper, we propose a novel approach to boost the performance of a real image denoiser which is trained only with synthetic pixel-independent noise data dominated by AWGN. First, we train a deep model that consists of a noise estimator and a denoiser with mixed AWGN and Random Value Impulse Noise (RVIN). We then investigate Pixel-shuffle Down-sampling (PD) strategy to adapt the trained model to real noises. Extensive experiments demonstrate the effectiveness and generalization of the proposed approach. Notably, our method achieves state-of-the-art performance on real sRGB images in the DND benchmark among models trained with synthetic noises. Codes are available at https://github.com/yzhouas/PD-Denoising-pytorch.

Image Denoising via Sequential Ensemble Learning.

Image denoising is about removing measurement noise from input image for better signal-to-noise ratio. In recent years, there has been great progress on the development of data-driven approaches for image denoising, which introduce various techniques and paradigms from machine learning in the design of image denoisers. This paper aims at investigating the application of ensemble learning in image denoising, which combines a set of simple base denoisers to form a more effective image denoiser. Based on different types of image priors, two types of base denoisers in the form of transform-shrinkage are proposed for constructing the ensemble. Then, with an effective re-sampling scheme, several ensemble-learning-based image denoisers are constructed using different sequential combinations of multiple proposed base denoisers. The experiments showed that sequential ensemble learning can effectively boost the performance of image denoising.

An Improved Combination of Image Denoisers Using Spatial Local Fusion Strategy

Image denoising is a well-researched problem in the image processing field. Numerous image denoising algorithms have been proposed in the past. Although researchers have continually focused on improving the denoising algorithm performance regarding denoising effect and outstanding results have been achieved, the improvement amplitude of a single denoising algorithm has decreased over time. Recently, the consensus neural network (CsNet), which combines multiple image denoisers to produce an overall better result compared to single algorithms, was proposed. However, the denoising process of CsNet is time-consuming owing to its MSE optimal weight setting and iterative boosting stages. Therefore, we propose an improved combination of nonlocally centralized sparse representation (NCSR) with a fast and flexible denoising convolutional neural network (FFDNet) using a spatial local fusion strategy (ICID). ICID uses a structural-based patch to decompose their denoised images into the strength, structure, and mean intensity components. Thereafter, an image patch is reconstructed and placed back into the fused image after fusing the three components separately. Experimental results verified that our algorithm is superior to CsNet, and it is faster. The combination of NCSR and FFDNet can harmonize the complementary denoising capabilities of different denoising algorithms. NCSR can preserve as many details as possible in natural images with numerous repeated structures, whereas FFDNet can achieve state-of-the-art results with a sufficiently large training set of images. Moreover, ICID uses the structural-based method, which considers more local details and preserves more textures, resulting in superior performance.

Compressive Computed Tomography Reconstruction through Denoising Approximate Message Passing

X-ray Computed Tomography (CT) reconstruction from a sparse number of views is a useful way to reduce either the radiation dose or the acquisition time, for example in fixed-gantry CT systems, however this results in an ill-posed inverse problem whose solution is typically computationally demanding. Approximate Message Passing (AMP) techniques represent the state of the art for solving under-sampling Compressed Sensing problems with random linear measurements but there are still not clear solutions on how AMP should be modified and how it performs with real world problems. This paper investigates the question of whether we can employ an AMP framework for real sparse view CT imaging? The proposed algorithm for approximate inference in tomographic reconstruction incorporates a number of advances from within the AMP community, resulting in the Denoising Generalized Approximate Message Passing CT algorithm (D-GAMP-CT). Specifically, this exploits the use of sophisticated image denoisers to regularize the reconstruction. While in order to reduce the probability of divergence the (Radon) system and Poisson non-linear noise model are treated separately, exploiting the existence of efficient preconditioners for the former and the generalized noise modelling in GAMP for the latter. Experiments with simulated and real CT baggage scans confirm that the performance of the proposed algorithm outperforms statistical CT optimization solvers.

Image Restoration and Reconstruction using Targeted Plug-and-Play Priors

Leveraging current state-of-the-art denoisers to tackle other inverse problems in imaging is a challenging task, which has recently been the topic of significant research effort. In this paper, we present several contributions to this research front, based on two fundamental building blocks: 1) the recently proposed plug-and-play framework, which allows combining iterative algorithms for imaging inverse problems with state-of-the-art image denoisers, used in black-box fashion; and 2) patch-based denoisers, using Gaussian mixture models (GMM). We exploit the adaptability of GMM to learn class-adapted denoisers, which opens the door to embedding a patch classification step in the algorithmic loop, yielding simultaneous restoration and semantic segmentation. We apply the proposed approach to several standard imaging inverse problems (deblurring, compressive sensing reconstruction, and super-resolution), obtaining results that are competitive with the state of the art.

Line roughness estimation and Poisson denoising in scanning electron microscope images using deep learning

We propose the use of deep supervised learning for the estimation of line edge roughness (LER) and line width roughness (LWR) in low-dose scanning electron microscope (SEM) images. We simulate a supervised learning dataset of 100,800 SEM rough line images constructed by means of the Thorsos method and the ARTIMAGEN library developed by the National Institute of Standards and Technology. We also devise two separate deep convolutional neural networks called SEMNet and EDGENet, each of which has 17 convolutional layers, 16 batch normalization layers, and 16 dropout layers. SEMNet performs the Poisson denoising of SEM images, and it is trained with a dataset of simulated noisy-original SEM image pairs. EDGENet directly estimates the edge geometries from noisy SEM images, and it is trained with a dataset of simulated noisy SEM image-edge array pairs. SEMNet achieved considerable improvements in peak signal-to-noise ratio as well as the best LER/LWR estimation accuracy compared with standard image denoisers. EDGENet offers excellent LER and LWR estimation as well as roughness spectrum estimation.

Optimal Combination of Image Denoisers.

Given a set of image denoisers, each having a different denoising capability, is there a provably optimal way of combining these denoisers to produce an overall better result? An answer to this question is fundamental to designing an ensemble of weak estimators for complex scenes. In this paper, we present an optimal combination scheme by leveraging the deep neural networks and the convex optimization. The proposed framework, called the Consensus Neural Network (CsNet), introduces three new concepts in image denoising: 1) a provably optimal procedure to combine the denoised outputs via convex optimization; 2) a deep neural network to estimate the mean squared error (MSE) of denoised images without needing the ground truths; and 3) an image boosting procedure using a deep neural network to improve the contrast and to recover the lost details of the combined images. Experimental results show that CsNet can consistently improve the denoising performance for both deterministic and neural network denoisers.