Clean Image Research Articles

DEMVSNet: Denoising and depth inference for unstructured multi‐view stereo on noised images

Most deep-learning-based multi-view stereo series studies are concerned with improving the depth prediction accuracy of noise-free images. However, it is difficult to obtain off-the-set clean images in practice and 3D convolutional neural networks require a lot of computing resources. To make full use of its computing power, different types of information can be processed simultaneously in the network. For these two issues, this paper proposes a novel multi-stage network architecture to address depth inference and denoising simultaneously. Specifically, 2D feature maps are first converted into 3D cost volumes containing pixel information and depth information through differentiable homography and Gaussian probability mapping. Then, the cost volume is input into the regularisation module in each network stage to obtain the predicted probability volumes. Furthermore, simple static weights lead to training failure, and it is necessary to dynamically adjust the loss function by gradient normalisation. The proposed method can dispose of pixel information and depth information simultaneously and both reach an excellent level. Extensive experimental results show that the authors’ work surpasses the state-of-the-art denoising on the DTU dataset (adding Gaussian–Poisson noise) and is more robust to noise images in depth inference.

Training low dose CT denoising network without high quality reference data

Objective. Currently, the field of low-dose CT (LDCT) denoising is dominated by supervised learning based methods, which need perfectly registered pairs of LDCT and its corresponding clean reference image (normal-dose CT). However, training without clean labels is more practically feasible and significant, since it is clinically impossible to acquire a large amount of these paired samples. In this paper, a self-supervised denoising method is proposed for LDCT imaging. Approach. The proposed method does not require any clean images. In addition, the perceptual loss is used to achieve data consistency in feature domain during the denoising process. Attention blocks used in decoding phase can help further improve the image quality. Main results. In the experiments, we validate the effectiveness of our proposed self-supervised framework and compare our method with several state-of-the-art supervised and unsupervised methods. The results show that our proposed model achieves competitive performance in both qualitative and quantitative aspects to other methods. Significance. Our framework can be directly applied to most denoising scenarios without collecting pairs of training data, which is more flexible for real clinical scenario.

A Hybrid CNN for Image Denoising

Deep convolutional neural networks (CNNs) with strong learning abilities have been used in the field of image super-resolution. However, some CNNs depends on a single deep network to training an image super-resolution model, which will have poor performance in complex screens. To address this problem, we propose a hybrid denoising CNN (HDCNN). HDCNN is composed of a dilated block (DB), RepVGG block (RVB) and feature refinement block (FB), a single convolution. DB combines a dilated convolution, batch normalization (BN), common convolutions, activation function of ReLU to obtain more context information. RVB uses parallel combination of convolution and BN, ReLU to extract complementary width features. FB is used to obtain more accurate information via refining obtained feature from the RVB. A single convolution collaborates a residual learning operation to construct a clean image. These key components make the HDCNN have good performance in image denoising. Experiment shows that the proposed HDCNN enjoys good denoising effect in public datasets.

Synthesized rain images for deraining algorithms

Since most of the rainy scene datasets used for training single image rain removal (SIRR) algorithms are constructed by blending artificial rain streaks with source images, it is difficult for a machine trained with such datasets to understand the patterns of real or realistic rain streaks. So, several studies have been attempted to build a real rainy scene dataset. However, since collecting real rainy scenes itself requires significant costs, the real rainy scene datasets provided by some studies cover only very limited rainy environment(s). This paper presents a new approach to synthesize realistic rainy scenes using GAN, which is a world-first attempt as far as we know. The proposed method builds a representation space to which rain streaks of multiple styles are smoothly mapped by learning the distributions of various rain datasets. The representation space allows control over the generated rain streaks. Also, the proposed method can synthesize multiple rainy scenes per clean (source) scene simultaneously, thereby a synthesized rain image dataset (SyRa) (Dataset can be found here: https://github.com/jaewoong1/SyRa-Synthesized_Rain_dataset) consisting of 11 K clean images and 55 K rainy images was constructed. Finally, this paper provides benchmarking results of several SIRR methods trained with SyRa. This result will be very useful for developing SIRR algorithms that can cope well with the actual rain environment.

Face Identification Using Conditional Generative Adversarial Network

Abstract Most of research studies that have dealt with face corrupted images to the level of being unrecognizable by a human are based on full image reconstruction. In some real scenarios, a single corrupted image might need to be recognized among a limited number of available clean images. This study deals with face identification from artificially corrupted images with various kinds of noises. The work proposes a face identification conditional generative adversarial network (FI-CGAN) model to identify faces based on the CGAN. The proposed models reconstruct the corrupted image based on available clean images to map the corrupted image to a specific label. The classification is made using the nearest neighbor method with peak signal-to-noise ratio, mean squared error and structural similarity index as metrics. The study used the Specs on Faces dataset and achieved a satisfactory performance for face identification.

Hyperspectral Image Denoising via Adversarial Learning

Due to sensor instability and atmospheric interference, hyperspectral images (HSIs) often suffer from different kinds of noise which degrade the performance of downstream tasks. Therefore, HSI denoising has become an essential part of HSI preprocessing. Traditional methods tend to tackle one specific type of noise and remove it iteratively, resulting in drawbacks including inefficiency when dealing with mixed noise. Most recently, deep neural network-based models, especially generative adversarial networks, have demonstrated promising performance in generic image denoising. However, in contrast to generic RGB images, HSIs often possess abundant spectral information; thus, it is non-trivial to design a denoising network to effectively explore both spatial and spectral characteristics simultaneously. To address the above issues, in this paper, we propose an end-to-end HSI denoising model via adversarial learning. More specifically, to capture the subtle noise distribution from both spatial and spectral dimensions, we designed a Residual Spatial-Spectral Module (RSSM) and embed it in an UNet-like structure as the generator to obtain clean images. To distinguish the real image from the generated one, we designed a discriminator based on the Multiscale Feature Fusion Module (MFFM) to further improve the quality of the denoising results. The generator was trained with joint loss functions, including reconstruction loss, structural loss and adversarial loss. Moreover, considering the lack of publicly available training data for the HSI denoising task, we collected an additional benchmark dataset denoted as the Shandong Feicheng Denoising (SFD) dataset. We evaluated five types of mixed noise across several datasets in comparative experiments, and comprehensive experimental results on both simulated and real data demonstrate that the proposed model achieves competitive results against state-of-the-art methods. For ablation studies, we investigated the structure of the generator as well as the training process with joint losses and different amounts of training data, further validating the rationality and effectiveness of the proposed method.

An effective color image segmentation approach using superpixel-neutrosophic C-means clustering and gradient-structural similarity

Due to the insufficient expression of uncertain information in fuzzy sets, the fuzzy C-means clustering algorithm is difficult to deal with clustered boundary and outliers. Therefore, this paper proposes a color image segmentation method based on neutrosophic C-means clustering. First, we improved the simple linear clustering algorithm to obtain accurate and natural adaptive local spatial neighborhoods. Secondly, local neighborhood information is added to the objective function of neutrosophic C-means clustering to obtain a more accurate membership. According to the membership, superpixels are divided into certainty group and uncertainty group. Finally, the certainty group is classified by the maximum membership, and the superpixels of the uncertain group are classified by the structural similarity. Experimental results show that the proposed method has better performance on clean and noise images.

A model-driven network for guided image denoising

Guided image denoising recovers clean target images by fusing guidance images and noisy target images. Several deep neural networks have been designed for this task, but they are black-box methods lacking interpretability. To overcome the issue, this paper builds a more interpretable network. To start with, an observation model is proposed to account for modality gap between target and guidance images. Then, this paper formulates a deep prior regularized optimization problem, and solves it by alternating direction method of multipliers (ADMM) algorithm. The update rules are generalized to design the network architecture. Extensive experiments conducted on FAIP and RNS datasets manifest that the novel network outperforms several state-of-the-art and benchmark methods regarding both evaluation metrics and visual inspection.

FUS-Net: U-Net-Based FUS Interference Filtering.

Imaging applications tailored towards ultrasound-based treatment, such as high intensity focused ultrasound (FUS), where higher power ultrasound generates a radiation force for ultrasound elasticity imaging or therapeutics/theranostics, are affected by interference from FUS. The artifact becomes more pronounced with intensity and power. To overcome this limitation, we propose FUS-net, a method that incorporates a CNN-based U-net autoencoder trained end-to-end on 'clean' and 'corrupted' RF data in Tensorflow 2.3 for FUS artifact removal. The network learns the representation of RF data and FUS artifacts in latent space so that the output of corrupted RF input is clean RF data. We find that FUS-net perform 15% better than stacked autoencoders (SAE) on evaluated test datasets. B-mode images beamformed from FUS-net RF shows superior speckle quality and better contrast-to-noise (CNR) than both notch-filtered and adaptive least means squares filtered RF data. Furthermore, FUS-net filtered images had lower errors and higher similarity to clean images collected from unseen scans at all pressure levels. Lastly, FUS-net RF can be used with existing cross-correlation speckle-tracking algorithms to generate displacement maps. FUS-net currently outperforms conventional filtering and SAEs for removing high pressure FUS interference from RF data, and hence may be applicable to all FUS-based imaging and therapeutic methods.

Effect of number and area of view-fields on the measurement accuracy of hair follicle density in goats (Capra hircus)

Improving the hair follicle density (HFD) has long been a breeding goal, as it can directly affect the fiber yield of animals, such as cashmere goats, sheep, rabbits and alpacas. Due to sample size and time limitations, the HFD cannot be easily obtained from a large histological area. If the area is too small, the experimental sampling error cannot be ignored. The number of view-fields (fieldN), the area of view-field (fieldS), and the total statistical area (total SA, fieldN×fieldS) needed in the HFD examination have not been determined. Thus, we aimed to develop a strategy to address this unclear topic. Goatskin was obtained as the sample, and paraffin sectioning and hematoxylin-eosin staining were done. Images of each skin sample were captured, stitched, and clipped. All hair follicles (HFs) were marked manually as color dots, and the background noise was removed. We acquired 52 clean images (50 mm2 for each) and considered 10 types of fieldN, fieldN∈[1,10] and 14 types of fieldS, from 32.0 mm2 to 0.10 mm2. A Matlab-based algorithm/program, MinSA, was developed to calculate the mean relative error (MRE) under different fieldN and fieldS. In fieldN = 1, the field area was 5.04 mm2 where MRE reached 5%, indicating that if the field area were less than 5.04 mm2 (one field) for the HFD examination, the MRE would be higher than 5%. As the fieldN increased from 2 to 10, the fieldS at 5% MRE gradually reduced, but the total SA increased from 4.45 to 5.49 mm2. This result demonstrates that using multiple view-field was not a good choice. In goats, 1–3 fields are recommended, and the total statistical area should be over 5 mm2 of stained skin section. Furthermore, this algorithm could apply to other economic fur-bearing animals.

Blind denoising using dense hybrid convolutional network

The performance of existing deep convolutional networks is limited when encountering images with different noise levels. In this study, a denoising method with state-of-the-art performance that combines a deep convolutional network with the traditional nonlocal mean denoising method is proposed. The noisy image is first denoised using the nonlocal mean method. Then, the denoised image is input into the proposed dense hybrid convolutional network to be trained, producing a clean image with clear details. The dense hybrid convolutional network comprises three parts: a feature-extracting noise-suppressing module that extracts abstract features from denoised images and suppresses the residual noise by interval convolution; a feature-learning module used for training blurred edges and textures; and a magnifying module that uses deconvolution to restore the feature maps to the original size and reduce the noise again. In contrast to existing denoising algorithms, the method has two desirable properties: 1) it can restore edges and textures clearly while removing the noise; 2) it effectively deals with noise of unknown levels (i.e. blind denoising) with a single network model. The conducted experiments show that the proposed method achieves superior performance compared to those of state-of-the-art denoising methods.

High-Resolution Representations Network for Single Image Dehazing.

Deep learning-based image dehazing methods have made great progress, but there are still many problems such as inaccurate model parameter estimation and preserving spatial information in the U-Net-based architecture. To address these problems, we propose an image dehazing network based on the high-resolution network, called DeHRNet. The high-resolution network originally used for human pose estimation. In this paper, we make a simple yet effective modification to the network and apply it to image dehazing. We add a new stage to the original network to make it better for image dehazing. The newly added stage collects the feature map representations of all branches of the network by up-sampling to enhance the high-resolution representations instead of only taking the feature maps of the high-resolution branches, which makes the restored clean images more natural. The final experimental results show that DeHRNet achieves superior performance over existing dehazing methods in synthesized and natural hazy images.

Adversarial attacks on YOLACT instance segmentation

Adversarial attacks have stimulated research interests in the field of deep learning security. In terms of autonomous driving technology, instance segmentation can help autonomous vehicles identify a drivable area in a driving environment and a traffic accident will happen once instance segmentation is attacked by adversarial examples. However, only few studies have been done on adversarial attacks about instance segmentation. It would be more meaningful and more practical to research adversarial attacks against instance segmentation. In this paper, we propose the improved Projected Gradient Descent (PGD) to produce adversarial examples on the total loss of You Only Look At CoefficienT (YOLACT) instance segmentation. Firstly, we design the loss function and calculate the adversarial gradient. Then we propose the improved PGD. Finally, we obtain adversarial examples of YOLACT instance segmentation. Our attack method is efficient and powerful in both white-box and black-box attack settings, and is applicable in a variety of neural network architectures. On COCO 2017, under white-box attacks, our method achieves 1.14% box mean average precision (mAP) and 1.50% mask mAP on YOLACT with ResNet101 backbone. We also compare the similarity between clean images and adversarial examples among three different backbones and compare the operation time among three different backbones. We also find that box regression loss, classification loss and mask loss are also effective separately for generating adversarial examples. Our research will provide inspiration for further efforts in efficient and effective defense methods on instance segmentation.

Open-Set Adversarial Defense with Clean-Adversarial Mutual Learning

Open-set recognition and adversarial defense study two key aspects of deep learning that are vital for real-world deployment. The objective of open-set recognition is to identify samples from open-set classes during testing, while adversarial defense aims to robustify the network against images perturbed by imperceptible adversarial noise. This paper demonstrates that open-set recognition systems are vulnerable to adversarial samples. Furthermore, this paper shows that adversarial defense mechanisms trained on known classes are unable to generalize well to open-set samples. Motivated by these observations, we emphasize the necessity of an Open-Set Adversarial Defense (OSAD) mechanism. This paper proposes an Open-Set Defense Network with Clean-Adversarial Mutual Learning (OSDN-CAML) as a solution to the OSAD problem. The proposed network designs an encoder with dual-attentive feature-denoising layers coupled with a classifier to learn a noise-free latent feature representation, which adaptively removes adversarial noise guided by channel and spatial-wise attentive filters. Several techniques are exploited to learn a noise-free and informative latent feature space with the aim of improving the performance of adversarial defense and open-set recognition. First, we incorporate a decoder to ensure that clean images can be well reconstructed from the obtained latent features. Then, self-supervision is used to ensure that the latent features are informative enough to carry out an auxiliary task. Finally, to exploit more complementary knowledge from clean image classification to facilitate feature denoising and search for a more generalized local minimum for open-set recognition, we further propose clean-adversarial mutual learning, where a peer network (classifying clean images) is further introduced to mutually learn with the classifier (classifying adversarial images). We propose a testing protocol to evaluate OSAD performance and show the effectiveness of the proposed method on white-box attacks, black-box attacks, as well as the rectangular occlusion attack in multiple object classification datasets.

Semi-supervised learning for raindrop removal on a single image

This article propose s a network that is mainly used to deal with a single image polluted by raindrops in rainy weather to get a clean image without raindrops. In the existing solutions, most of the methods rely on paired images, that is, the rain image and the real image without rain in the same scene. However, in many cases, the paired images are difficult to obtain, which makes it impossible to apply the raindrop removal network in many scenarios. Therefore this article proposes a semi-supervised rain-removing network apply to unpaired images. The model contains two parts: a supervised network and an unsupervised network. After the model is trained, the unsupervised network does not require paired images and it can get a clean image without raindrops. In particular, our network can perform training on paired and unpaired samples. The experimental results show that the best results are achieved not only on the supervised rain-removing network, but also on the unsupervised rain-removing network.

Multiframe blind restoration with image quality prior

In dynamic environments, such as ground-based optic observations for space targets, anisotropic turbulence and random noise tend to produce different discriminate image degradations at every moment. These degradations severely reduce the quality of the target images and make restoration of these images very difficult. However, an inconsistent degradation implies that there is complementary information in such images. In this paper, a ranking network (Ranknet) is first proposed to ensure that input sequences have a consistent distribution upon degradation and squeeze the spatial distribution of the sample set. Then, an extraction-refinement neural network (ERnet) is proposed to extract complementary features and blindly reconstruct a clean image of the observation target. In ERnet, an extraction subnetwork (EN) uses 3D convolutions to extract discriminate features from multiframe input sequences, and a refinement subnetwork (RN) based on 2D convolution restores clean images by refining the effective features. In addition, a spatial–temporal attention module (STAM) is devoted to enhancing features through utilization of the high-quality features. Experimental results on the restorations of space target images and motion blur images confirm the superior performance of ERnet as compared with other state-of-the-art methods.

Learning stochastic object models from medical imaging measurements by use of advanced ambient generative adversarial networks.

Purpose: To objectively assess new medical imaging technologies via computer-simulations, it is important to account for the variability in the ensemble of objects to be imaged. This source of variability can be described by stochastic object models (SOMs). It is generally desirable to establish SOMs from experimental imaging measurements acquired by use of a well-characterized imaging system, but this task has remained challenging. Approach: A generative adversarial network (GAN)-based method that employs AmbientGANs with modern progressive or multiresolution training approaches is proposed. AmbientGANs established using the proposed training procedure are systematically validated in a controlled way using computer-simulated magnetic resonance imaging (MRI) data corresponding to a stylized imaging system. Emulated single-coil experimental MRI data are also employed to demonstrate the methods under less stylized conditions. Results: The proposed AmbientGAN method can generate clean images when the imaging measurements are contaminated by measurement noise. When the imaging measurement data are incomplete, the proposed AmbientGAN can reliably learn the distribution of the measurement components of the objects. Conclusions: Both visual examinations and quantitative analyses, including task-specific validations using the Hotelling observer, demonstrated that the proposed AmbientGAN method holds promise to establish realistic SOMs from imaging measurements.

Unsupervised single image dehazing with generative adversarial network

Most recent learning algorithms for single image dehazing are designed to train with paired hazy and corresponding ground truth images, typically synthesized images. Real paired datasets can help to improve performance, but are tough to acquire. This paper proposes an unsupervised dehazing algorithm based on GAN to alleviate this issue. An end-to-end network based on GAN architecture is established and fed with unpaired clean and hazy images, signifying that the estimation of atmospheric light and transmission is not required. The proposed network consists of three parts: a generator, a global test discriminator, and a local context discriminator. Moreover, a dark channel prior based attention mechanism is applied to handle inconsistency haze. We conduct experiments on RESIDE datasets. Extensive experiments demonstrated the effectiveness of the proposed approach which outperformed previous state-of-the-art unsupervised methods by a large margin.

Reduction of Compton Background Noise for X-ray Fluorescence Computed Tomography with Deep Learning

For bench-top X-ray fluorescence computed tomography (XFCT), the X-ray tube source will bring extreme Compton background noise, resulting in a low signal-to-noise ratio and low contrast detection limit. In this paper, a noise2noise denoising algorithm based on the UNet deep learning network is proposed. The network can use noise image learning to convert the noise image into a clean image. Two sets of phantoms (high concentration Gd phantom and low concentration Bi phantom) are used for scanning to simulate the imaging process under different noise levels and generate the required data set. Additionally, the data set is generated by Geant4 simulation. In the training process, the L1 loss function is used for its good convergence. The image quality is evaluated according to CNR and pixel profile, which shows that our algorithm is better than BM3D, both visually and quantitatively.

Upcycling adversarial attacks for infrared object detection

Recently, infrared object detection (IOD) has been extensively studied due to the rapid growth of deep neural networks (DNNs). An adversarial attack using imperceptible perturbation can dramatically deteriorate the performance of DNNs. Most of the existing adversarial attacks are focused on visible image recognition (VIR), but there are few attacks for IOD. Moreover, the existing attacks are challenging to exploit for state-of-the-art detectors (e.g., EfficientDet) due to low compatibility. To solve the problem, we propose a novel upcycling adversarial attack for IOD by expanding the highly compatible adversarial attacks for the VIR task. We also propose a novel evaluation metric, attack efficiency (AE), to compare the effectiveness of different adversarial attacks. Since the AE value increases with the small perturbation size and the significant performance drop, we can concurrently compare the similarity and performance degradation between adversarial and clean images for various attacks. We validate our approaches through comprehensive experiments on two challenging datasets (e.g., FLIR and MSOD) for the infrared domain.