Single Image Rain Removal Research Articles

DWTN: deep wavelet transform network for lightweight single image deraining

On rainy days the uncertainty of the shape and distribution of rain streaks can cause the images captured by RGB image-based measurement tools to be blurred and distorted. Thanks to the wavelet transform ability to provide spatial and frequency domain information about an image and its multidirectional and multiscale nature, it is widely used in traditional image enhancement methods. In image deraining the distribution of rain streaks is not only related to spatial domain features but is also closely related to frequency domain spatial features. However, deep learning-based rain removal models mainly rely on the spatial features of the image, and RGB data can hardly distinguish rain marks from image details, which leads to the loss of crucial image information during rain removal. We have developed a lightweight single-image rain removal model called the deep wavelet transform network (DWTN) to address this limitation. This method separates image details from rain images and can more effectively remove rain marks. The proposed DWTN has three significant contributions. First, DWTN uses the feature components after the wavelet transform as the input to the model and assigns a separate frequency-aware enhancement block (FAEB) to each element. These blocks focus on specific frequency features that benefit the rain removal task. Second, we introduce a frequency feature fusion block (FFFB) that fuses different wavelet components to reduce noise and enhance the image background through a channel attention mechanism while attenuating rain streaks. Finally, we design a spatial feature enhancement block (SFEB), which uses a spatial attention mechanism to calibrate the spatial position of features to improve the rain removal performance. We evaluate the performance of DWTN using PSNR and SSIM on four synthetic datasets and NIQE and BRISQUE on two real datasets. The results of the evaluation of the six datasets and at least four performance metrics show that the proposed DWTN is superior to existing methods.

AIMHNet: An Attribute-Insensitive Multiscale Hourglass Network for Rain Streak and Raindrop Removal

CNN-based methods have made great progress in single-image rain removal. Most recent methods improve performance by increasing the depth of the network. To fully extract local and global features while reducing inference time, we propose a top-to-down attribute-insensitive multiscale hourglass network for rain streak and raindrop removal. For the rain removal task, we expect that the constructed network can accurately identify the various attributes of the rain information characteristics of the small target. Considering the difference in the size, shape, direction and density of rain streak and raindrop, inspired by the performance of hourglass architecture to capture multiscale features in human pose estimation, we introduce an attribute-insensitive hourglass module to recognize the attributes of rain streak and raindrop in a unified framework. This feature extraction module could capture the characteristics of rain streak and raindrop with different attributes. This stacked hourglass blocks down-sample features and then up-samples them back to the original resolution based on discrete wavelet transform and inverse discrete wavelet transform. We perform extensive experiments on five synthetic and real-world de-raining datasets to validate the effectiveness of our proposed network on rain streak and raindrop removal. The qualitative and quantitative results show that our method is suitable for removing rain streak and raindrop in a unified framework. We present the results of generalization and ablation study for key components, we also report the accuracy of semantic segmentation after preprocessing with all rain removal methods. Our source code will be available on the GitHub: https://github.com/Ruini94/AIMHNet .

Dual-attention U-Net and multi-convolution network for single-image rain removal

Dual-attention U-Net and multi-convolution network for single-image rain removal

Dual parallel multi-scale residual overlay network for single-image rain removal

Dual parallel multi-scale residual overlay network for single-image rain removal

A Survey of Single Image Rain Removal Based on Deep Learning

The rain removal task is to restore a clean image from the contaminated image by separating the background. Since the rise of deep learning in 2016, the task of image deraining has also stepped into the era of deep learning. Numerous researchers have devoted themselves to the field of computer vision and pattern recognition. However, there is still a lack of comprehensive review papers focused on using deep learning to perform rain removal tasks. In this paper, we present a comprehensive review of single image deraining based on deep learning over the past ten years. Two categories of deraining methods are discussed: the data-driven approach and the data-model-based approach. For the first type, we compare the existing network structures and loss functions. For the second type, we analyze the combination of different deraining models with deep learning, and each branch method is introduced in detail. Additionally, we quantitatively investigate the performances of the existing state-of-the-art methods on both publicly synthetic and real datasets. The trend of image deraining is also discussed.

Single‐image deraining algorithm based on multi‐stage recurrent network

AbstractThe damage caused by the rain streak attached to the rainy image to the background seriously affects the analysis of image information and subsequent research. Aiming at the problem that the current single image rain removal methods cannot extract the deep rain streak features, a single image deraining algorithm based on multi‐stage recurrent network is proposed, which is committed to separating the rain streak layer and the background layer. The network consists of three stages, each stage is mainly composed of upper and lower scale dilation blocks, the upper scale dilation block includes multi‐scale dilated convolution and scale fusion, the lower scale dilation block is set the same, and the stages are connected using improved gated recurrent units to facilitate parameter sharing between them. The experimental results show that the proposed algorithm outperforms the comparative algorithm in both quantitative indicators and qualitative analysis on synthetic and real datasets, and the restored image retains more detailed information.

Single Image Rain Removal Algorithm Based on U-Net and Vision Transformer

Rain will significantly lower image quality, which will undoubtedly have an impact on how well outdoor computer vision systems operate, such as autonomous driving. This paper proposes a two-branch deep neural network consisting of an attention guided U-Net and Vision Transformer, which could capture intra-field details and cross-patch relationships to obtain good global and local deraining effects. In order to ensure both branches pose positive effects on the target derain task, we specifically design a patch boot image module to achieve complemented feature fusion, which adaptively selects informative intra-field regions guided by the patch-wise importance map. Moreover, Wasserstein distance between the prediction and reference image is applied as the objective function to pursue better image quality measurement. In comparison to existing algorithms, the suggested method may more effectively remove rain and restore the background details, according to qualitative and quantitative results on the public datasets Rain200L and Rain200H. On the aforementioned datasets, the peaks of the structural similarity (SSIM) and signal-to-noise ratio (PSNR) were 32.55/0.9476 and 26.12/0.8826, respectively.

Memory-efficient multi-scale residual dense network for single image rain removal

Outdoor photography is greatly affected by bad rainfall, which degrades the quality of captured images and hinders practical application. In response to the above problems, we propose a memory-efficient multi-scale residual dense network (MMRD-Net) for single image rain removal. Firstly, we use a combination or fusion of extended convolution blocks of different sizes to accurately extract multi-scale features of rain streaks and raindrops while retaining rich local detail. Secondly, we add attention modules to the residual block, which performs adaptive operations on rainfall and non-rainfall areas to extract more and more accurate image features and details of rain streaks and raindrops. During this period, we opened up a space for shared use by all the concatenation layers in the network, which reduces the memory consumption of the network. It allows for better aggregation of stormwater features at multiple levels and also allows for faster convergence of the network. Finally, we tested on both synthetic and real datasets to obtain subjective results and objective evaluations. The experimental results show that the performance of our algorithm is good and robust for joint rain removal tasks of different densities. Our algorithm outperforms twelve other different classical algorithms in terms of its ability to remove rain streaks. It can produce visually sharper deraining images and the enhanced visibility of the images is effective for computer vision applications (Google Vision API).

Single image rain removal via tetrolet transform and morphological decomposition

Single image rain removal via tetrolet transform and morphological decomposition

Incremental Image De-raining via Associative Memory

While deep learning models have achieved the state-of-the-art performance on single-image rain removal, most methods only consider learning fixed mapping rules on the single synthetic dataset for lifetime. This limits the real-life application as iterative optimization may change mapping rules and training samples. However, when models learn a sequence of datasets in multiple incremental steps, they are susceptible to catastrophic forgetting that adapts to new incremental episodes while failing to preserve previously acquired mapping rules. In this paper, we argue the importance of sample diversity in the episodes on the iterative optimization, and propose a novel memory management method, Associative Memory, to achieve incremental image de-raining. It bridges connections between current and past episodes for feature reconstruction by sampling domain mappings of past learning steps, and guides the learning to trace the current pathway back to the historical environment without storing extra data. Experiments demonstrate that our method can achieve better performance than existing approaches on both inhomogeneous and incremental datasets within the spectrum of highly compact systems.

TA-DNN—two stage attention-based deep neural network for single image rain removal

TA-DNN—two stage attention-based deep neural network for single image rain removal

Adaptive weighted rain streaks model-driven deep network for single image deraining

In the computer vision field, single image rain removal is an extremely important pre-processing task since rain streaks can degrade the performance of subsequent high-level tasks and outdoor image quality. In recent years, massive deep learning-based deraining methods have been proposed to remove rain streaks from rainy images, and they have achieved promising performance. However, most deep learning-based methods lack interpretability and have limited performance in image details restoration and rain streaks removal. In this paper, we propose a novel rain streaks model-driven deep network, MSANet, to alleviate these issues. First of all, instead of simply stacking those available convolutions, pooling, and attention blocks, the structure of MSANet is derived from a prior rain removal model and the structure is thus fully interpretable. Compared to prior models available in the literature, we introduce a weighted convolutional dictionary model to capture the shapes, sizes, directions, regions, and multi-scale information of rain streaks. And to reduce the bias associated with the L2-norm, we utilize an adaptive approach to constrain the fidelity term of the model. Finally, we employ the alternating direction method of multipliers (ADMM) algorithm to optimize this model and unfold the optimization procedure into a new neural network architecture by modeling each operation in the algorithm using network blocks. All the parameters can be automatically learned by end-to-end training. Extensive experiments on several benchmark datasets demonstrate that the proposed network architecture is more efficient than state-of-the-art methods based on both subjective and objective evaluations.

Lightweight Semi-supervised Network for Single Image Rain Removal

Deep learning technologies have shown their advantages in Single Image Rain Removal (SIRR) tasks. However, the derained results of most methods are limited to some challenges. First, due to the lack of real-world rainy/clean image pairs, many methods seriously rely on the labeled synthetic training images and will not effectively remove complex rain streaks in real-world scenarios. Second, most existing SIRR models require high computing power, which considerably limits their real-world applications. To address these issues, we propose a Lightweight Semi-supervised Network (LSNet) for SIRR. Our LSNet utilizes a compact semi-supervised framework to improve generalization ability in real-world rainy images removal. Meanwhile, in our semi-supervised framework, we also design a cascaded sub-network, which progressively removes complex rain streaks via a multi-stage manner. Specially, the multi-stage manner is based on a series of cascaded blocks, where we conduct recursive learning strategy to reduce model parameters. Extensive experimental results demonstrate that our method achieves comparable performance to the state-of-the-arts while has fewer parameters.

Develop Then Rival: A Human Vision-Inspired Framework for Superimposed Image Decomposition

A single superimposed image containing two image views causes visual confusion for both human vision and computer vision. Human vision needs a "develop-then-rival" process to decompose the superimposed image into two individual images, which effectively suppresses visual confusion. However, separating individual image views from a single superimposed image has been an important but challenging task in computer vision area for a long time. In this paper, we propose a human vision-inspired framework for single superimposed image decomposition. We first propose a network to simulate the development stage, which tries to understand and distinguish the semantic information of the two layers of a single superimposed image. To further simulate the rivalry activation/suppression process in human brains, we carefully design a rivalry stage, which incorporates the original mixed input (superimposed image), the activated visual information (outputs of the development stage) together, and then rivals to get images without ambiguity. Experimental results show that our novel framework effectively separates the superimposed images and significantly improves the performance with better output quality compared with state-of-the-art methods. The proposed method also achieves state-of-the-art results on related applications including single image reflection removal, single image rain removal, single image shadow removal, and illumination correction, etc., which validates the generalization of the framework.

PRAGAN: Progressive Recurrent Attention GAN with Pretrained ViT Discriminator for Single-Image Deraining

Images captured in bad weather are not conducive to visual tasks. Rain streaks in rainy images will significantly affect the regular operation of imaging equipment; to solve this problem, using multiple neural networks is a trend. The ingenious integration of network structures allows for full use of the powerful representation and fitting abilities of deep learning to complete low-level visual tasks. In this study, we propose a generative adversarial network (GAN) with multiple attention mechanisms for image rain removal tasks. Firstly, to the best of our knowledge, we propose a pretrained vision transformer (ViT) as the discriminator in GAN for single-image rain removal for the first time. Secondly, we propose a neural network training method that can use a small amount of data for training while maintaining promising results and reliable visual quality. A large number of experiments prove the correctness and effectiveness of our method. Our proposed method achieves better results on synthetic and real image datasets than multiple state-of-the-art methods, even when using less training data.

Recurrent Network Knowledge Distillation for Image Rain Removal

Single-image rain removal (SIRR) based on deep learning has long been a problem of great interest in low-level vision systems. However, traditional convolutional neural network (CNN)-based approaches fail to capture long-range location dependencies effectively and may cause the image background blurred. In this article, we propose a knowledge distilling deraining network (KDRN) to address the SIRR problem. In the proposed network, the teacher regards rain streaks as a linear combination of many residual networks. It is used for image reconstruction at different resolutions. With the aid of a teacher network, the proposed deraining network performs better. A spatial channel aggregation residual attention block (SCARAB) is designed to remove the rain streaks. The block not only concentrates on the rain streak features but also captures the spatial-channel information of the image. For the network structure, we used an end-to-end approach to design the teacher and student networks separately. The proposed KDRN obtains the predicted residual image by a combination of the stage-wise results and the original input image. Extensive experiments show that the proposed KDRN obtains better subjective quality than most of the compared methods, on both heavy and light rain data sets.

Single image deraining via deep residual attention and encoder-decoder network

Single image rain removal remains a crucial and challenging low-level image processing task while significantly for outdoor-based high-level computer vision tasks. Recently, deep convolutional neural networks (CNNs) have become the mainstream structure of removing rain streaks and obtained remarkable performance. However, most of the existing CNN-based methods mainly pay attention to completely removing rain streaks while neglecting the restoration of details after deraining, which suffer from poor visual performance. In this paper, we propose a deep residual attention and encoder-decoder network to overcome the above shortcoming. Specifically, we develop an excellent basic block that contains dual parallel paths which are called rain removal network and detail restore network, respectively, to perform entirely and in-depth mapping relationships from rain to no-rain. The upper rain removal network is composed of dilated convolution and channel attention. This combination can explore the correlation between features from the dimensions of spatial and channel. Meanwhile, for the lower detail restore network, we construct a simple yet effective symmetrical encoder-decoder structure to prevent the loss of global structures information and encourage the details back. Furthermore, our network is end-to-end trainable, easy to implement and without giant parameter quantity. Extensive experiments on synthetic and real-world datasets have shown that our DRAEN achieves better accuracy and visual improvements against recent state-of-the-art methods.

Lightweight single image deraining algorithm incorporating visual saliency

There are still some challenges in the task of single image rain removal, such as artefact remnant, background over-smooth, and increasingly complex and heavy-weight network architecture. Especially too heavy-weight network to fit outdoor detection devices or mobile devices. To address the above challenges, we propose a lightweight single image Deraining algorithm incorporating visual attention saliency mechanisms (LDVS). The proposed network consists of five blocks and two convolution operations, where each block consists of a dilation convolution module and a convolutional block attention module (CBAM). Specifically, visual saliency module CBAM is used for accurate localization of rain streak, and further the combinations of dilated convolution with CBAM is used to extract feature maps of rain streaks faithfully, which is able to remove artefact remnant while maintaining background details. A good tradeoff is presented between the network's weight size and effect of rain removal. Specifically, with only 48,268 parameters, the proposed model can achieve a guaranteed performance. Extensive experiments on a few typical rainy scenarios on synthetic and real-world datasets have demonstrated that to achieve the same level of performance, the proposed method has far smaller size than most of the baselines under both qualitative and quantitative analyses.

Multi-Scale Attention Generative Adversarial Network for Single Image Rain Removal

Multi-Scale Attention Generative Adversarial Network for Single Image Rain Removal

No rain in the world: A novel residual deep attention network for single image rain removal

Clear images are generally desirable in high-level computer vision algorithms which are mostly deployed outdoors. However, affected by the changeable weather in the real world, images are inevitably contaminated by rain streaks. Deep convolutional neural networks (CNNs) have shown significant potential in rain streaks removal. The performance of most existing CNN-based deraining methods is often enhanced by stacking vanilla convolutional layers and some other methods use dilated convolution which can only model local pixel relations to provide the necessary but limited receptive field. Therefore, long-range contextual information is rarely considered for this specific task, thus, deraining a single image remains challenging problem. To address the above problem, an effective residual deep attention network (RDANet) for single image rain removal is proposed. Specifically, we design a strong basic unit that contains dilated convolution, spatial and channel attention module (SCAM) simultaneously. As contextual information is very important for rain removal, the proposed basic unit can capture global long-distance dependencies among pixels in feature maps and model feature relations across channels. Compared with a single dilated convolution, the spatial and channel attention enhance the feature expression ability of the network. Moreover, some previous works have proven that the no-rain information in a rain image will be missing during deraining. To enrich the detailed information in the clean images, we present a residual feature processing group (RFPG) that contains several source skip connections to inject rainy shallow source information into each basic unit. In summary, our model can effectively handle complicated long rain streaks in spatial and the outputs of the network can retain most of the details of the original rain images. Experiments demonstrate the superiority of our RDANet over state-of-the-art methods in terms of both quantitative metrics and visual quality on both synthetic and real rainy images.