Noisy Image Pairs Research Articles

Unsupervised denoising of photoacoustic images based on the Noise2Noise network.

In this study, we implemented an unsupervised deep learning method, the Noise2Noise network, for the improvement of linear-array-based photoacoustic (PA) imaging. Unlike supervised learning, which requires a noise-free ground truth, the Noise2Noise network can learn noise patterns from a pair of noisy images. This is particularly important for in vivo PA imaging, where the ground truth is not available. In this study, we developed a method to generate noise pairs from a single set of PA images and verified our approach through simulation and experimental studies. Our results reveal that the method can effectively remove noise, improve signal-to-noise ratio, and enhance vascular structures at deeper depths. The denoised images show clear and detailed vascular structure at different depths, providing valuable insights for preclinical research and potential clinical applications.

M-Denoiser: Unsupervised image denoising for real-world optical and electron microscopy data

Real-world microscopy data have a large amount of noise due to the limited light/electron that can be used to capture images. The noise of microscopy data is composed of signal-dependent shot noise and signal-independent read noise, and the Poisson–Gaussian noise model is usually used to describe the noise distribution. Meanwhile, the noise is spatially correlated because of the data acquisition process. Due to the lack of clean ground truth, unsupervised and self-supervised denoising algorithms in computer vision shed new light on tackling such tasks by utilizing paired noisy images or one single noisy image. However, they usually make the assumption that the noise is signal-independent or pixel-wise independent, which contradicts with the actual case. Hence, we propose M-Denoiser for denoising real-world microscopy data in an unsupervised manner. Firstly, the shatter module is used to break the dependency and correlation before denoising. Secondly, a novelly designed unsupervised training loss based on a pair of noisy images is proposed for real-world microscopy data. For evaluation, we train our model on optical and electron microscopy datasets. The experimental results show that M-Denoiser achieves the best performance both quantitatively and qualitatively compared with all the baselines.

A self-validation Noise2Noise training framework for image denoising

ABSTRACT Image denoising is a crucial algorithm in image processing that aims to enhance image quality. Deep learning-based image denoising methods can be categorized into supervised and unsupervised approaches. Supervised learning requires pairs of noisy and noise-free training data, which is impractical in real-world scenarios. Unsupervised learning uses pairs of noisy images for training, but it may yield lower accuracy. Additionally, deep learning-based methods often require a large amount of training data. To overcome these challenges, this research proposes a self-validation Noise2Noise (SV-N2N) framework that generates validation sets using only noisy images without requiring noise-free pairs. The proposed SV-N2N method effectively reduces noise, comparable to supervised and unsupervised methods, without requiring a noise-free ground truth, which is efficient for solving real-world scenarios.

Low-dose imaging denoising with one pair of noisy images.

Low-dose imaging techniques have many important applications in diverse fields, from biological engineering to materials science. Samples can be protected from phototoxicity or radiation-induced damage using low-dose illumination. However, imaging under a low-dose condition is dominated by Poisson noise and additive Gaussian noise, which seriously affects the imaging quality, such as signal-to-noise ratio, contrast, and resolution. In this work, we demonstrate a low-dose imaging denoising method that incorporates the noise statistical model into a deep neural network. One pair of noisy images is used instead of clear target labels and the parameters of the network are optimized by the noise statistical model. The proposed method is evaluated using simulation data of the optical microscope, and scanning transmission electron microscope under different low-dose illumination conditions. In order to capture two noisy measurements of the same information in a dynamic process, we built an optical microscope that is capable of capturing a pair of images with independent and identically distributed noises in one shot. A biological dynamic process under low-dose condition imaging is performed and reconstructed with the proposed method. We experimentally demonstrate that the proposed method is effective on an optical microscope, fluorescence microscope, and scanning transmission electron microscope, and show that the reconstructed images are improved in terms of signal-to-noise ratio and spatial resolution. We believe that the proposed method could be applied to a wide range of low-dose imaging systems from biological to material science.

THz imaging enhancement based on Noise2Noise algorithm

AbstractTerahertz (THz) imaging has an outstanding advantage of high resolution due to the high frequency and has promising potential in VideoSAR. However, limited to the THz source power and the air absorption, the THz image usually has a low SNR and is susceptible to noise in remote sensing and imaging. In order to improve the quality of THz images, a THz image enhancement method is proposed based on the noise2noise strategy. The THz images are reconstructed using the AFBP algorithm. They are organized as noisy image pairs and filtered with a mask to remove the influence of moving targets. Then, the Noise2Noise network is constructed based on the CNN network and takes the noisy image pair as input and reference. In the training stage, 1000 noisy image pairs are used as the training set and 100 noisy images are used as the test set to verify the performance of the proposed method. The experimental results of real VideoSAR data have demonstrated that the proposed method is capable of suppressing noise and enhancing the THz image.

A Convolutional Neural Network-Based Conditional Random Field Model for Structured Multi-Focus Image Fusion Robust to Noise

The limited depth of field of optical lenses, makes multi-focus image fusion (MFIF) algorithms of vital importance. Lately, Convolutional Neural Networks (CNN) have been widely adopted in MFIF methods, however their predictions mostly lack structure and are limited by the size of the receptive field. Moreover, since images have noise due to various sources, the development of MFIF methods robust to image noise is required. A novel robust to noise Convolutional Neural Network-based Conditional Random Field (mf-CNNCRF) model is introduced. The model takes advantage of the powerful mapping between input and output of CNN networks and the long range interactions of the CRF models in order to reach structured inference. Rich priors for both unary and smoothness terms are learned by training CNN networks. The α -expansion graph-cut algorithm is used to reach structured inference for MFIF. A new dataset, which includes clean and noisy image pairs, is introduced and is used to train the networks of both CRF terms. A low-light MFIF dataset is also developed to demonstrate real-life noise introduced by the camera sensor. Qualitative and quantitative evaluation prove that mf-CNNCRF outperforms state-of-the-art MFIF methods for clean and noisy input images, while being more robust to different noise types without requiring prior knowledge of noise.

Learning from Multiple Instances: A Two-Stage Unsupervised Image Denoising Framework Based on Deep Image Prior

Supervised image denoising methods based on deep neural networks require a large amount of noisy-clean or noisy image pairs for network training. Thus, their performance drops drastically when the given noisy image is significantly different from the training data. Recently, several unsupervised learning models have been proposed to reduce the dependence on training data. Although unsupervised methods only require noisy images for learning, their denoising effect is relatively weak compared with supervised methods. This paper proposes a two-stage unsupervised deep learning framework based on deep image prior (DIP) to enhance the image denoising performance. First, a two-target DIP learning strategy is proposed to impose a learning restriction on the DIP optimization process. A cleaner preliminary image, together with the given noisy image, was used as the learning target of the two-target DIP learning process. We then demonstrate that adding an extra learning target with better image quality in the DIP learning process is capable of constraining the search space of the optimization process and improving the denoising performance. Furthermore, we observe that given the same network input and the same learning target, the DIP optimization process cannot generate the same denoised images. This indicates that the denoised results are uncertain, although they are similar in image quality and are complemented by local details. To utilize the uncertainty of the DIP, we employ a supervised denoising method to preprocess the given noisy image and propose an up- and down-sampling strategy to produce multiple sampled instances of the preprocessed image. These sampled instances were then fed into multiple two-target DIP learning processes to generate multiple denoised instances with different image details. Finally, we propose an unsupervised fusion network that fuses multiple denoised instances into one denoised image to further improve the denoising effect. We evaluated the proposed method through extensive experiments, including grayscale image denoising, color image denoising, and real-world image denoising. The experimental results demonstrate that the proposed framework outperforms unsupervised methods in all cases, and the denoising performance of the framework is close to or superior to that of supervised denoising methods for synthetic noisy image denoising and significantly outperforms supervised denoising methods for real-world image denoising. In summary, the proposed method is essentially a hybrid method that combines both supervised and unsupervised learning to improve denoising performance. Adopting a supervised method to generate preprocessed denoised images can utilize the external prior and help constrict the search space of the DIP, whereas using an unsupervised method to produce intermediate denoised instances can utilize the internal prior and provide adaptability to various noisy images of a real scene.

LOUD: Local Orthogonalization-Constrained Unsupervised Deep-Learning Denoiser

Random noise attenuation of seismic data is a fundamental problem in seismic data processing. It is not only an important problem itself but also is a crucial step for the subsequent tasks, e.g., migration and inversion. We propose a local orthogonalization constrained unsupervised deep learning denoiser (LOUD) to suppress seismic random noise based on a new loss function that specifically adapts to seismic data. Through unsupervised learning, we eliminate the common need in supervised learning approaches of collecting or generating sizable clean and noisy image pairs, which is challenging and expensive, especially for seismic data. We utilize a deep convolutional autoencoder to reconstruct the clean seismic image and leverage the local signal-and-noise orthogonalization as a constraint to guarantee that the removed noise component is orthogonal to the recovered signal. Experimental results on both synthetic and field datasets exhibit the effectiveness of our proposed method over traditional denoising methods.

Rapid noise removal based dual adversarial network for the Brillouin optical time domain analyzer

We propose a dual adversarial network (DANet) to improve the signal-to-noise ratio (SNR) of the Brillouin optical time domain analyzer. Rather than inferring the conditional posteriori distribution in the conventional maximum a posteriori (MAP) framework, DANet constructs a joint distribution from two different factorizations corresponding to the noise removal and generation tasks. This method utilizes all the information between the clean-noisy image pairs to preserve data completely without requiring traditional image priors and noise distribution assumptions. Additionally, the clean-noisy image pairs produced by the generator can expand the original dataset to retrain and enhance the denoising effect. The performance of DANet is verified using the simulated and experimental data. Without spatial resolution deterioration, an SNR improvement of 35.51 dB is observed in the simulation, and the Brillouin frequency shift (BFS) uncertainty along the fiber is reduced by 3.56 MHz. Experiments yield a maximum SNR improvement of 19.08 dB, with the BFS uncertainty along the fiber reduced by 0.93MHz. Significantly, DANet has a processing time of 1.26 s, which is considerably faster than conventional methods, demonstrating its potential for rapid noise removal tasks.

Noise2Noise Improved by Trainable Wavelet Coefficients for PET Denoising

The significant statistical noise and limited spatial resolution of positron emission tomography (PET) data in sinogram space results in the degradation of the quality and accuracy of reconstructed images. Although high-dose radiotracers and long acquisition times improve the PET image quality, the patients’ radiation exposure increases and the patient is more likely to move during the PET scan. Recently, various data-driven techniques based on supervised deep neural network learning have made remarkable progress in reducing noise in images. However, these conventional techniques require clean target images that are of limited availability for PET denoising. Therefore, in this study, we utilized the Noise2Noise framework, which requires only noisy image pairs for network training, to reduce the noise in the PET images. A trainable wavelet transform was proposed to improve the performance of the network. The proposed network was fed wavelet-decomposed images consisting of low- and high-pass components. The inverse wavelet transforms of the network output produced denoised images. The proposed Noise2Noise filter with wavelet transforms outperforms the original Noise2Noise method in the suppression of artefacts and preservation of abnormal uptakes. The quantitative analysis of the simulated PET uptake confirms the improved performance of the proposed method compared with the original Noise2Noise technique. In the clinical data, 10 s images filtered with Noise2Noise are virtually equivalent to 300 s images filtered with a 6 mm Gaussian filter. The incorporation of wavelet transforms in Noise2Noise network training results in the improvement of the image contrast. In conclusion, the performance of Noise2Noise filtering for PET images was improved by incorporating the trainable wavelet transform in the self-supervised deep learning framework.

Bayesian sparse hierarchical model for image denoising

Sparse models and their variants have been extensively investigated, and have achieved great success in image denoising. Compared with recently proposed deep-learning-based methods, sparse models have several advantages: (1) Sparse models do not require a large number of pairs of noisy images and the corresponding clean images for training. (2) The performance of sparse models is less reliant on the training data, and the learned model can be easily generalized to natural images across different noise domains. In sparse models, ℓ0 norm penalty makes the problem highly non-convex, which is difficult to be solved. Instead, ℓ1 norm penalty is commonly adopted for convex relaxation, which is considered as the Laplacian prior from the Bayesian perspective. However, many previous works have revealed that ℓ1 norm regularization causes a biased estimation for the sparse code, especially for high-dimensional data, e.g., images. In this paper, instead of using the ℓ1 norm penalty, we employ an improper prior in the sparse model and formulate a hierarchical sparse model for image denoising. Compared with other competitive methods, experiment results show that our proposed method achieves a better generalization for images with different characteristics across various domains, and achieves state-of-the-art performance for image denoising on several benchmark datasets.

Goodpoint: unsupervised learning of key point detection and description

Subject of Research. The paper presents the study of algorithms for key point detection and description, widely used in computer vision. Typically, the corner detector acts as a key point detector, including neural key point detectors. For some types of images obtained in medicine, the application of such detectors is problematic due to the small number of detected key points. The paper considers a problem of a neural network key point detector training on unlabeled images. Method. We proposed the definition of key points not depending on specific visual features. A method was considered for training of a neural network model meant for detecting and describing key points on unlabeled data. The application of homographic image transformation was basic to the method. The neural network model was trained to detect the same key points on pairs of noisy images related to a homographic transformation. Only positive examples were used for detector training, just points correctly matched with features produced by the neural network model for key point description. Main Results. The unsupervised learning algorithm is used to train the neural network model. For the ease of comparison, the proposed model has a similar architecture and the same number of parameters as the supervised model. Model evaluation is performed on the three different datasets: natural images, synthetic images, and retinal photographs. The proposed model shows similar results to the supervised model on the natural images and better results on retinal photographs. Improvement of results is demonstrated after additional training of the proposed model on images from the target domain. This is an advantage over a model trained on a labeled dataset. For comparison, the harmonic average of such metrics is used as: the accuracy and the depth of matching by descriptors, reproducibility of key points and image coverage. Practical Relevance. The proposed algorithm makes it possible to train the neural network key point detector together with the feature extraction model on images from the target domain without costly dataset labeling and reduce labor costs for the development of the system that uses the detector.

Noise-Powered Disentangled Representation for Unsupervised Speckle Reduction of Optical Coherence Tomography Images.

Due to its noninvasive character, optical coherence tomography (OCT) has become a popular diagnostic method in clinical settings. However, the low-coherence interferometric imaging procedure is inevitably contaminated by heavy speckle noise, which impairs both visual quality and diagnosis of various ocular diseases. Although deep learning has been applied for image denoising and achieved promising results, the lack of well-registered clean and noisy image pairs makes it impractical for supervised learning-based approaches to achieve satisfactory OCT image denoising results. In this paper, we propose an unsupervised OCT image speckle reduction algorithm that does not rely on well-registered image pairs. Specifically, by employing the ideas of disentangled representation and generative adversarial network, the proposed method first disentangles the noisy image into content and noise spaces by corresponding encoders. Then, the generator is used to predict the denoised OCT image with the extracted content features. In addition, the noise patches cropped from the noisy image are utilized to facilitate more accurate disentanglement. Extensive experiments have been conducted, and the results suggest that our proposed method is superior to the classic methods and demonstrates competitive performance to several recently proposed learning-based approaches in both quantitative and qualitative aspects. Code is available at: https://github.com/tsmotlp/DRGAN-OCT.

Learning synthetic aperture radar image despeckling without clean data

Speckle noise can reduce the image quality of synthetic aperture radar (SAR) and make interpretation more difficult. Existing SAR image despeckling convolutional neural networks require quantities of noisy–clean image pairs. However, obtaining clean SAR images is very difficult. Because continuous convolution and pooling operations result in losing many informational details while extracting the deep features of the SAR image, the quality of recovered clean images becomes worse. Therefore, we propose a despeckling network called multiscale dilated residual U-Net (MDRU-Net). The MDRU-Net can be trained directly using noisy–noisy image pairs without clean data. To protect more SAR image details, we design five multiscale dilated convolution modules that extract and fuse multiscale features. Considering that the deep and shallow features are very distinct in fusion, we design different dilation residual skip connections, which make features at the same level have the same convolution operations. Afterward, we present an effective L_hybrid loss function that can effectively improve the network stability and suppress artifacts in the predicted clean SAR image. Compared with the state-of-the-art despeckling algorithms, the proposed MDRU-Net achieves a significant improvement in several key metrics.

Denoising of Scintillation Camera Images Using a Deep Convolutional Neural Network: A Monte Carlo Simulation Approach.

Scintillation camera images contain a large amount of Poisson noise. We have investigated whether noise can be removed in whole-body bone scans using convolutional neural networks (CNNs) trained with sets of noisy and noiseless images obtained by Monte Carlo simulation. Methods: Three CNNs were generated using 3 different sets of training images: simulated bone scan images, images of a cylindric phantom with hot and cold spots, and a mix of the first two. Each training set consisted of 40,000 noiseless and noisy image pairs. The CNNs were evaluated with simulated images of a cylindric phantom and simulated bone scan images. The mean squared error between filtered and true images was used as difference metric, and the coefficient of variation was used to estimate noise reduction. The CNNs were compared with gaussian and median filters. A clinical evaluation was performed in which the ability to detect metastases for CNN- and gaussian-filtered bone scans with half the number of counts was compared with standard bone scans. Results: The best CNN reduced the coefficient of variation by, on average, 92%, and the best standard filter reduced the coefficient of variation by 88%. The best CNN gave a mean squared error that was on average 68% and 20% better than the best standard filters, for the cylindric and bone scan images, respectively. The best CNNs for the cylindric phantom and bone scans were the dedicated CNNs. No significant differences in the ability to detect metastases were found between standard, CNN-, and gaussian-filtered bone scans. Conclusion: Noise can be removed efficiently regardless of noise level with little or no resolution loss. The CNN filter enables reducing the scanning time by half and still obtaining good accuracy for bone metastasis assessment.

Inertial sensor aided multi-image nonuniform motion blur removal based on motion decomposition

Many factors lead to spatially varying blur kernels in a blurred image, such as camera shake, moving objects, and scene depth variation. The traditional camera shake removal methods either ignore the influence of varying depth values or object motion in dynamic scenes, while the methods not limited to removing camera shake always make simple assumptions about camera motion trajectory. We consider these factors in a unified framework, with the aid of an alternate-exposure capture strategy and simultaneously recorded inertial sensor readings. The inertial measurements relate the long-exposed blurred image to preceding and succeeding short-exposed noisy images. The special exposure arrangement effectively addresses the problem inherent in reconstructing camera motion from inertial measurements. In addition, the noisy image pair bracketing the blurred image is used for motion detection and initial depth map estimation, making the proposed method free of user interaction and additional expensive devices. Contrary to previous methods that individually parametrize the motion blur of the moving foreground layer and the static background layer, we exploit the fact that camera shake has a global influence to decompose the motion of the foreground layer such that a more tight constraint between the motion of layers is established. Given the motion and image data, we propose a single-energy model and minimize it using alternating optimization to estimate the spatially varying motion blur and the latent sharp image. Comparative experimental results demonstrate that our method outperforms conventional camera motion deblurring and object deblurring methods on both synthetic and real scenes.

Blurred and noisy image pairs in parallel optics.

In previous works we have shown that parallel optics (PO) architecture can be used to improve the system matrix condition, which results in improving its immunity to additive noise in the image restoration process. PO is composed of a "main" system and an "auxiliary" system. Previously, we suggested the "trajectories" method to realize PO. In that method, a required auxiliary system is composed from auxiliary optics with a pixel confined response, followed by signal processing. In this paper, we emphasize the important secondary effects of the trajectories method. We show that in such a system, where the postprocessing comes after the detection, the postprocessing acts as a noise filter, hence allowing us to work with noisy data in the auxiliary channel. Roughly speaking, the SNR of an imaging system depends on the numerical aperture (NA). It follows that the main system, which typically has a higher NA, also has a higher SNR. Hence in the PO system, the ratio between the NA values of the main and auxiliary systems is expected to dictate the gap between their SNR values. In this paper, we show that when the system is implemented by the trajectories method, this expectation is too conservative. It is shown that due to the noise filtering, the auxiliary system can be noisier than expected. This claim is proved analytically and verified and exemplified by using experimental measurements.

EXTENDING THE VISUAL CRYPTOGRAPHY ALGORITHM USING IMAGE WATERMARKING TECHNIQUE

Visual cryptography is a secret information sharing technique which shares the information in the form of images. It generates noise-like random pixels on share images to hide secret information which on overlay decrypt the information. This technique is known as conventional visual secret sharing schemes. It suffers a management problem, because of which dealers cannot visually identify each share. This problem is solved by the extended visual cryptography scheme (EVCS), which adds a meaningful cover image in each share. While removing the extra cover image it produces extra noise or degrades the hidden image quality. So we propose a new image watermarking technique in this Visual Cryptography Algorithm that places a small image on the noisy image pair at the bottom right corner. So that the cover images need not be removed and it doesn't degrade  resolution of the secret image.

Performance Evaluation of Stereo Matching Algorithms in the Lack of Visual Features

this paper, we evaluate three different subcategories of image matching algorithms. We consider hierarchical matching, wavelet-based localized correlation and multiresolution subregioning. The importance of this evaluation stems from the fact that these algorithms are all somehow based on a multiresolution scheme, but exhibit different performances when dealing with featureless image pairs, noisy image pairs, or when tuned to different parameters, e.g. the number of resolution levels and the size of the correlation size. We also consider the use of different correlation functions. A data set has been built using random dots stereograms, with a full range of disparities and a controlled amount of noise. The algorithms performances are benchmarked in terms of accuracy and global coherence of the disparity maps.

Bayesian Scale Space Analysis of Differences in Images

This article considers the detection of image features in different spatial scales. The main focus is on capturing the scale-dependent differences in a pair of noisy images, but the technique developed can also be applied to the analysis of single images. The approach proposed uses Bayesian statistical modeling and simulation-based inference, and it can be viewed as a further development of SiZer technology, originally designed for nonparametric curve fitting. Numerical examples include artificial test images and a preliminary analysis of a pair of Landsat images used in satellite-based forest inventory. This article has supplementary material online.