Clean Image Research Articles

Purifying Adversarial Images Using Adversarial Autoencoder With Conditional Normalizing Flows

We present a <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">target-agnostic</i> adversarial autoencoder with conditional normalizing flows specifically designed to, given any <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">unlabeled</i> image dataset, purify adversarial samples into clean images, i.e., remove adversarial noise from the images while preserving their visual quality. In our model interpretation, samples are processed by manifold projection in which the encoder brings the sample back into a posterior data distribution in latent space so that the sample is less likely to be irregular to the learned representation of any target classifier. Normalizing flows conditioned on top of our hybrid network structure and walk-back training are used to deal with common drawbacks of generative model and autoencoder-based approaches: not only the trade-off between compression loss and over-fitting on training data but also the <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">structural model dependency</i> on dataset classes and labels. Experiments demonstrated that our proposed model is preferable to existing target-agnostic adversarial defense methods particularly for large and unlabeled image datasets.

Missing Recovery: Single Image Reflection Removal based on Auxiliary Prior Learning.

Photographs taken through a glass window are susceptible to disturbances due to reflection. Therefore, single image reflection removal is crucial to image quality enhancement. In this paper, a novel learning architecture that can address this ill-posed problem is proposed. First, a novel reflection removal pipeline was designed to reconstruct the missing information caused by the camera imaging process using the proposed missing recovery network. Second, to address the issues in existing reflection removal strategies, we revisit several auxiliary priors and integrate them by defining an energy function. To solve the energy function, a convolutional neural network-based optimization scheme was proposed. Finally, we investigated the dark channel responses of reflection and clean images and found an interesting way to distinguish between these two types of images. We prove this property mathematically and propose a novel loss function called dark channel loss to improve performance. Experiments show that the proposed method outperforms state-of-the-art reflection removal methods both quantitatively and qualitatively.

Speckle Removal Using Dictionary Learning and PnP-Based Fast Iterative Shrinkage Threshold Algorithm

Speckle is a multiplicative granular noise that naturally occurs in the images captured by coherent imaging sensors such as synthetic aperture radar (SAR). It visually degrades the underlying image information and has an impact on subsequent image analysis. This problem is addressed here by developing a sparse representation model and applying an alternating minimization scheme for SAR image despeckling. The proposed method directly deals with the multiplicative noise and the data model is formed by utilizing the speckle statistics. The similar patches are clustered together to adaptively learn the dictionary, and the sparse coefficients are updated using plug-and-play based fast iterative shrinkage threshold algorithm (PnP-FISTA). Finally, the clean image is estimated using Newton’s method. Experiments on simulated and practical SAR images signify that the proposed method performs better compared to the state-of-the-art methods in terms of performance metrics and visual assessment.

Every cell needs a beautiful image: on-the-fly contacting measurements for high-throughput production

The future of the energy transition will lead to a terrawatt-scale photovoltaic market, which can be served cost-effectively primarily by means of high-throughput production of solar cells. In addition to high-throughput production, characterization must be adapted to highest cycle times. Therefore, we present an innovative approach to detect image defects in solar cells using on-the-fly electroluminescence measurements. When a solar cell passes a standard current–voltage (I–V) unit, the cell is stopped, contacted, measured, released, and afterwards again accelerated. In contrast to this, contacting and measuring the sample on-the-fly saves a lot of time. Yet, the resulting images are blurred due to high-speed motion. For the development of such an on-the-fly contact measurement tool, a deblurring method is developed in this work. Our deep-learning-based deblurring model enables to present a clean EL image of the solar cell to the human operator and allows for a proper defect detection, reaching a correlation coefficient of 0.84.

Deep-Based Film Grain Removal and Synthesis.

In this paper, deep learning-based techniques for film grain removal and synthesis that can be applied in video coding are proposed. Film grain is inherent in analog film content because of the physical process of capturing images and video on film. It can also be present in digital content where it is purposely added to reflect the era of analog film and to evoke certain emotions in the viewer or enhance the perceived quality. In the context of video coding, the random nature of film grain makes it both difficult to preserve and very expensive to compress. To better preserve it while compressing the content efficiently, film grain is removed and modeled before video encoding and then restored after video decoding. In this paper, a film grain removal model based on an encoder-decoder architecture and a film grain synthesis model based on a conditional generative adversarial network (cGAN) are proposed. Both models are trained on a large dataset of pairs of clean (grain-free) and grainy images. Quantitative and qualitative evaluations of the developed solutions were conducted and showed that the proposed film grain removal model is effective in filtering film grain at different intensity levels using two configurations: 1) a non-blind configuration where the film grain level of the grainy input is known and provided as input; and 2) a blind configuration where the film grain level is unknown. As for the film grain synthesis task, the experimental results show that the proposed model is able to reproduce realistic film grain with a controllable intensity level specified as input.

A Convolutional Neural Network-Based Conditional Random Field Model for Structured Multi-Focus Image Fusion Robust to Noise

The limited depth of field of optical lenses, makes multi-focus image fusion (MFIF) algorithms of vital importance. Lately, Convolutional Neural Networks (CNN) have been widely adopted in MFIF methods, however their predictions mostly lack structure and are limited by the size of the receptive field. Moreover, since images have noise due to various sources, the development of MFIF methods robust to image noise is required. A novel robust to noise Convolutional Neural Network-based Conditional Random Field (mf-CNNCRF) model is introduced. The model takes advantage of the powerful mapping between input and output of CNN networks and the long range interactions of the CRF models in order to reach structured inference. Rich priors for both unary and smoothness terms are learned by training CNN networks. The α -expansion graph-cut algorithm is used to reach structured inference for MFIF. A new dataset, which includes clean and noisy image pairs, is introduced and is used to train the networks of both CRF terms. A low-light MFIF dataset is also developed to demonstrate real-life noise introduced by the camera sensor. Qualitative and quantitative evaluation prove that mf-CNNCRF outperforms state-of-the-art MFIF methods for clean and noisy input images, while being more robust to different noise types without requiring prior knowledge of noise.

EITGAN: A Transformation-based Network for recovering adversarial examples

&lt;abstract&gt;&lt;p&gt;Adversarial examples have been shown to easily mislead neural networks, and many strategies have been proposed to defend them. To address the problem that most transformation-based defense strategies will degrade the accuracy of clean images, we proposed an Enhanced Image Transformation Generative Adversarial Network (EITGAN). Positive perturbations were employed in the EITGAN to counteract adversarial effects while enhancing the classified performance of the samples. We also used the image super-resolution method to mitigate the effect of adversarial perturbations. The proposed method does not require modification or retraining of the classifier. Extensive experiments demonstrated that the enhanced samples generated by the EITGAN effectively defended against adversarial attacks without compromising human visual recognition, and their classification performance was superior to that of clean images.&lt;/p&gt;&lt;/abstract&gt;

Deep Cleaner—A Few Shot Image Dataset Cleaner Using Supervised Contrastive Learning

Images are increasingly used for AI-based diagnosis and analysis of many diseases like cervical cancer, mouth cancer, glucose analysis from retina etc. In many cases, data collection is done by specialised camera modules which capture images of affected areas. As with any other sources of data, this process is also error-prone and may contain unwanted objects and regions that may require cleaning by removing them. Outliers in these kinds of dataset may adversely affect the performance of machine learning models. Manually cleaning would be a tedious task, especially when the data is collated from different sources. Hence, cleaning the data before training the model is of utmost importance. In this paper, we propose a Few-Shot learning based model pre-trained in supervised contrastive learning settings to automate the process of data cleaning. Our model learns the dataset distribution and distinguishes the accurate data points from noisy data points. We also show that scaling up the model can greatly improve the Few-Shot performance. On the noisy MobileODT cervical data, which was collected from Kaggle, our model obtained 52% accuracy without cleaning data using an EfficientNet architecture for the classification task. Whereas the same architecture with ROI cropping achieved an accuracy of 76.56% after cleaning through the proposed Deep Cleaner approach that requires only 100 clean images. The proposed approach performs 2.74% better than a denoising auto-encoder, which is considered a powerful anomaly detection technique.

Frequency-Domain Deep Guided Image Denoising

Despite the tremendous advances in image denoising techniques, there still remains a challenge to restore a clean image with salient structures based only on one single noisy observation, especially at high noise levels. In this work, to address this issue, we propose a novel frequency-domain guided denoising algorithm to conduct denoising with the help of an additional well-aligned guidance image. Thanks to the structural correlation between the input paired images, the frequency characteristics of the guidance image can indicate whether the frequency coefficients of the noisy target image are contributed by noise or textures. Therefore, the explicit frequency decomposition enables our model to avoid over-smoothing those detailed contents when removing noise. However, as two input images are usually captured in different fields, their structures are not always consistent with each other. Therefore, we model guided denoising with an optimization problem which takes into account both the representation model with respect to the guidance image and the structural fidelity to the noisy target image. Based on this, we further design a convolutional neural network, called as FGDNet, to explore the optimal solution. Due to the visual masking phenomenon, the human visual system is sensitive to noise in the flat areas, but may not perceive noise around edges or textures. Therefore, to handle a denoising task, we expect to remove as much noise as possible to guarantee the spatial smoothness of the flat contents, while preserving the high-frequency structures as well. Through frequency decomposition, our model can process the low-frequency and high-frequency contents separately with different denoising schemes. In the training stage, we also adopt a frequency-relevant loss function to optimize the network. Experimental results show that, compared with the state-of-the-art guided and non-guided denoising algorithms, our FGDNet can achieve higher denoising accuracy and also better visual quality in both flat and texture-rich regions. Source codes are available at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/lustrouselixir/FGDNet</uri> .

Fast Hyperspectral Image Denoising and Destriping Method Based on Graph Laplacian Regularization

Hyperspectral images (HSIs) contain rich spatial and spectral information about the earth, and are widely used in the remote sensing field. However, an HSI is frequently corrupted by various types of noise, such as Gaussian noise, sparse noise, stripe noise, and so on, which severely limits the subsequent application of the HSI. In this paper, we propose a graph Laplacian regularizer (GLR) to exploit the low-rank information across the bands of the HSI. Compared with the traditional low-rank regularization, our graph Laplacian regularization can achieve equivalent or better performance with less time consumption. Besides, a sparse constraint and a low-rank constraint are employed to remove the sparse and stripe noise. In addition, the augmented Lagrangian multiplier is used to solve each component to restore a clean image. Finally, we have carried out experiments on the simulated and real noisy HSI data. The results show the superiority of the proposed method over state-of-the-art methods, in terms of PSNR and SSIM, time cost and visual effect. The MATLAB code is available through: https://github.com/zzhang-99/FGLR.

SAR Despeckling Using a Denoising Diffusion Probabilistic Model

Speckle is a type of multiplicative noise that affects all coherent imaging modalities including Synthetic Aperture Radar (SAR) images. The presence of speckle degrades the image quality and can adversely affect the performance of SAR image applications such as automatic target recognition and change detection. Thus, SAR despeckling is an important problem in remote sensing. In this paper, we introduce SAR-DDPM, a denoising diffusion probabilistic model for SAR despeckling. The proposed method employs a Markov chain that transforms clean images to white Gaussian noise by successively adding random noise. The despeckled image is obtained through a reverse process that predicts the added noise iteratively, using a noise predictor conditioned on the speckled image. Additionally, we propose a new inference strategy based on cycle spinning to improve the despeckling performance. Our experiments on both synthetic and real SAR images demonstrate that the proposed method leads to significant improvements in both quantitative and qualitative results over the state-of-the-art despeckling methods. The code is available at: https://github.com/malshaV/SAR_DDPM.

T-Net: Deep Stacked Scale-Iteration Network for Image Dehazing

Haze reduces the visibility of image content and leads to failure in handling subsequent computer vision tasks. In this paper, we address the problem of single image dehazing by proposing a dehazing network named T-Net, which consists of a backbone network based on the U-Net architecture and a dual attention module. Multi-scale feature fusion can be achieved by using skip connections with a new fusion strategy. Furthermore, by repeatedly unfolding the plain T-Net, Stack T-Net is proposed to take advantage of the dependence of deep features across stages via a recursive strategy. To reduce network parameters, the intra-stage recursive computation of ResNet is adopted in our Stack T-Net. We take both the stage-wise result and the original hazy image as input to each T-Net and finally output the prediction of the clean image. Experimental results on both synthetic and real-world images demonstrate that our plain T-Net and the advanced Stack T-Net perform favorably against state-of-the-art dehazing algorithms and show that our Stack T-Net could further improve the dehazing effect, demonstrating the effectiveness of the recursive strategy.

De-Noising Brain MRI Images by Mixing Concatenation and Residual Learning (MCR)

Brain magnetic resonance images (MRI) are used to diagnose the different diseases of the brain, such as swelling and tumor detection. The quality of the brain MR images is degraded by different noises, usually salt & pepper and Gaussian noises, which are added to the MR images during the acquisition process. In the presence of these noises, medical experts are facing problems in diagnosing diseases from noisy brain MR images. Therefore, we have proposed a de-noising method by mixing concatenation, and residual deep learning techniques called the MCR de-noising method. Our proposed MCR method is to eliminate salt & pepper and gaussian noises as much as possible from the brain MRI images. The MCR method has been trained and tested on the noise quantity levels 2% to 20% for both salt & pepper and gaussian noise. The experiments have been done on publically available brain MRI image datasets, which can easily be accessible in the experiments and result section. The Structure Similarity Index Measure (SSIM) and Peak Signal-to-Noise Ratio (PSNR) calculate the similarity score between the denoised images by the proposed MCR method and the original clean images. Also, the Mean Squared Error (MSE) measures the error or difference between generated denoised and the original images. The proposed MCR de-noising method has a 0.9763 SSIM score, 84.3182 PSNR, and 0.0004 MSE for salt & pepper noise; similarly, 0.7402 SSIM score, 72.7601 PSNR, and 0.0041 MSE for Gaussian noise at the highest level of 20% noise. In the end, we have compared the MCR method with the state-of-the-art de-noising filters such as median and wiener de-noising filters.

Transformer Based Defense GAN Against Palm-Vein Adversarial Attacks

Vein biometrics is a high security and privacy preserving identification technology that has attracted increasing attention over the last decade. Deep neural networks (DNNs), such as convolutional neural networks (CNN), have shown strong capabilities for robust feature representation, and have achieved, as a result, state-of-the-art performance on various vision tasks. Inspired by their success, deep learning models have been widely investigated for vein recognition and have shown significant improvement of identification accuracy compared to handcrafted models. Existing deep learning models, however, are vulnerable to adversarial perturbation attacks, where thoughtfully crafted small perturbations can cause misclassification of legitimate images, degrading, thereby, the efficiency of vein recognition systems. To address this problem, we propose, in this paper, VeinGuard, a novel defense framework to defend deep learning classifiers against adversarial palm-vein image attacks, composed of a local transformer-based GAN and a purifier. VeinGuard comprises two components: a local transformer-based GAN (LTGAN) that learns the distribution of unperturbed vein images and generates high-quality palm-vein images, and a purifier consisting of a trainable residual network and of a pre-trained generator from LTGAN that automatically removes a wide variety of adversarial perturbations. The resulting clean images are fed to vein classifiers for identification, thereby avoiding adversarial attacks. We evaluate VeinGuard on three public vein datasets in terms of white-box attacks, black-box attacks, ablation experiments, and computation time. The experimental results show that VeinGuard allows filtering the perturbations and enables the classifiers to achieve state-of-the-art recognition results for different adversarial attacks.

Deep learning for terahertz image denoising in nondestructive historical document analysis

Historical documents contain essential information about the past, including places, people, or events. Many of these valuable cultural artifacts cannot be further examined due to aging or external influences, as they are too fragile to be opened or turned over, so their rich contents remain hidden. Terahertz (THz) imaging is a nondestructive 3D imaging technique that can be used to reveal the hidden contents without damaging the documents. As noise or imaging artifacts are predominantly present in reconstructed images processed by standard THz reconstruction algorithms, this work intends to improve THz image quality with deep learning. To overcome the data scarcity problem in training a supervised deep learning model, an unsupervised deep learning network (CycleGAN) is first applied to generate paired noisy THz images from clean images (clean images are generated by a handwriting generator). With such synthetic noisy-to-clean paired images, a supervised deep learning model using Pix2pixGAN is trained, which is effective to enhance real noisy THz images. After Pix2pixGAN denoising, 99% characters written on one-side of the Xuan paper can be clearly recognized, while 61% characters written on one-side of the standard paper are sufficiently recognized. The average perceptual indices of Pix2pixGAN processed images are 16.83, which is very close to the average perceptual index 16.19 of clean handwriting images. Our work has important value for THz-imaging-based nondestructive historical document analysis.

NG-GAN: A Robust Noise-Generation Generative Adversarial Network for Generating Old-Image Noise.

Numerous old images and videos were captured and stored under unfavorable conditions. Hence, old images and videos have uncertain and different noise patterns compared with those of modern ones. Denoising old images is an effective technique for reconstructing a clean image containing crucial information. However, obtaining noisy-clean image pairs for denoising old images is difficult and challenging for supervised learning. Preparing such a pair is expensive and burdensome, as existing denoising approaches require a considerable number of noisy-clean image pairs. To address this issue, we propose a robust noise-generation generative adversarial network (NG-GAN) that utilizes unpaired datasets to replicate the noise distribution of degraded old images inspired by the CycleGAN model. In our proposed method, the perception-based image quality evaluator metric is used to control noise generation effectively. An unpaired dataset is generated by selecting clean images with features that match the old images to train the proposed model. Experimental results demonstrate that the dataset generated by our proposed NG-GAN can better train state-of-the-art denoising models by effectively denoising old videos. The denoising models exhibit significantly improved peak signal-to-noise ratios and structural similarity index measures of 0.37 dB and 0.06 on average, respectively, on the dataset generated by our proposed NG-GAN.

Robust Data Augmentation Generative Adversarial Network for Object Detection

Generative adversarial network (GAN)-based data augmentation is used to enhance the performance of object detection models. It comprises two stages: training the GAN generator to learn the distribution of a small target dataset, and sampling data from the trained generator to enhance model performance. In this paper, we propose a pipelined model, called robust data augmentation GAN (RDAGAN), that aims to augment small datasets used for object detection. First, clean images and a small datasets containing images from various domains are input into the RDAGAN, which then generates images that are similar to those in the input dataset. Thereafter, it divides the image generation task into two networks: an object generation network and image translation network. The object generation network generates images of the objects located within the bounding boxes of the input dataset and the image translation network merges these images with clean images. A quantitative experiment confirmed that the generated images improve the YOLOv5 model's fire detection performance. A comparative evaluation showed that RDAGAN can maintain the background information of input images and localize the object generation location. Moreover, ablation studies demonstrated that all components and objects included in the RDAGAN play pivotal roles.

Semisupervised Surveillance Video Character Extraction and Recognition With Attentional Learning Multiframe Fusion

Character extraction in the video is very helpful to the understanding of the video content, especially the artificially superimposed characters such as time and place in the surveillance video. However, the performance of the existing algorithms does not meet the needs of application. Therefore, the authors improve semisupervised surveillance video character extraction and recognition with attentional learning multiframe feature fusion. First, the multiframe fusion strategy based on an attention mechanism is adopted to solve the target missing problem, and the Dense ASPP network is introduced to solve the character multiscale problem. Second, a character image denoising algorithm based on semisupervised fuzzy C-means clustering is proposed to isolate and extract clean binary character images. Finally, for some video characters that may involve privacy, traditional and deep learning-based video restoration algorithms are used for characteristic elimination.

Image-based motion artifact reduction on liver dynamic contrast enhanced MRI

Liver MRI images often suffer from degraded quality due to ghosting or blurring artifacts caused by patient respiratory or bulk motion. In this study, we developed a two-stage deep learning model to reduce motion artifact on dynamic contrast enhanced (DCE) liver MRIs. The stage-I network utilized a deep residual network with a densely connected multi-resolution block (DRN-DCMB) network to remove most motion artifacts. The stage-II network applied the generative adversarial network (GAN) and perceptual loss compensation to preserve image structural features. The stage-I network served as the generator of GAN and its pretrained parameters in stage-I were further updated via backpropagation during stage-II training. The stage-I network was trained using small image patches with simulated motion artifacts including image-space rotational and translational motion, and K-space based centric and interleaved linear motion, sinusoidal, and rotational motion to mimic liver motion patterns. The stage-II network training used full-size images with the same types of simulated motion. The liver DCE-MRI image volumes without obvious motion artifacts in 10 patients were used for the training process, of which 1020 images of 8 patients were used for training and 240 images of 2 patients for validation. Finally, the whole two-stage deep learning model was tested with simulated motion images (312 clean images from 5 test patients) and patient images with real motion artifacts (28 motion images from 12 patients). The resulted images after two-stage processing demonstrated reduced motion artifacts while preserved anatomic details without image blurriness, with SSIM of 0.935 ± 0.092, MSE of 60.7 ± 9.0 × 10-3, and PSNR of 32.054 ± 2.219.

Image and Texture Independent Deep Learning Noise Estimation Using Multiple Frames

In this study, a novel multiple frame based image and texture independent Convolutional Neural Network (CNN) noise estimator is introduced. Noise estimation is a crucial step for denoising algorithms, especially for ones that are called “non-blind”. The estimator works for additive Gaussian noise for varying noise levels. The noise levels studied in this work have a standard deviation equal to 5 to 25 increasing 5 by 5. Since there is no database for noisy multiple images to train and validate the network, two frames of synthetic noisy images with a variety of noise levels are created by adding Additive White Gaussian Noise (AWGN) to each clean image. The proposed method is applied on the most popular gray level images besides the color image databases such as Kodak, McMaster, BSDS500 in order to compare the results with the other works. Image databases comprise indoor and outdoor scenes that have fine details and richer texture. The estimator has an accuracy rate of 99 % for the classification and favourable results for the regression. The proposed method outperforms traditional methods in most cases. And the regression output can be used with any non-blind denoising method.