Remove Rain Streaks Research Articles

A Lightweight Network for Real-Time Rain Streaks and Rain Accumulation Removal from Single Images Captured by AVs

In autonomous driving, object detection is considered a base step to many subsequent processes. However, object detection is challenged by loss in visibility caused by rain. Rainfall occurs in two main forms, which are streaks and streaks accumulations. Each degradation type imposes different effect on the captured videos; therefore, they cannot be mitigated in the same way. We propose a lightweight network which mitigates both types of rain degradation in real-time, without negatively affecting the object-detection task. The proposed network consists of two different modules which are used progressively. The first one is a progressive ResNet for rain streaks removal, while the second one is a transmission-guided lightweight network for rain streak accumulation removal. The network has been tested on synthetic and real rainy datasets and has been compared with state-of-the-art (SOTA) networks. Additionally, time performance evaluation has been performed to ensure real-time performance. Finally, the effect of the developed deraining network has been tested on YOLO object-detection network. The proposed network exceeded SOTA by 1.12 dB in PSNR on the average result of multiple synthetic datasets with 2.29× speedup. Finally, it can be observed that the inclusion of different lightweight stages works favorably for real-time applications and could be updated to mitigate different degradation factors such as snow and sun blare.

A new image decomposition approach using pixel-wise analysis sparsity model

Decomposing an image into two ‘simpler’ layers has been widely used in low-level vision tasks, such as image recovery and enhancement. It is an ill-posed problem since the number of unknowns are larger than the input. In this paper, a two-step strategy is introduced, including task-aware priors estimate and a decomposition model. A pixel-wise analysis sparsity model is proposed to regularize the separation layers, which supposes the transformed image generated with analysis operator is sparse. Unlike regularizing all pixels with one penalty weight, we try to estimate each pixel’s sparsity level with task-aware priors and to achieve pixel-wise sparse penalty. Additionally, one separation layer is regularized with both synthesis sparsity model and pixel-wise analysis sparsity model to exploit their complementary mechanisms. Unlike the analysis one utilizing image local features, the synthesis one exploits an over-complete dictionary and non-local similarity cues to provide flexible prior for regularizing the decomposition results. The proposed model is solved by an alternating optimization algorithm. We evaluate it with two applications, Retinex model and rain streaks removal. Extensive experiments on multiple enhancement datasets, many synthetic and real rainy images demonstrate that our method can remove imaging noise during Retinex decomposition, and can produce high fidelity deraining results. It achieves competing performance in terms of quantitative metrics and visual quality compared with the state-of-the-art methods.

Recurrent context-aware multi-stage network for single image deraining

Single image rain streak removal is extremely necessary since rainy images can seriously affect many computer vision systems. In this paper, we propose a novel recurrent context-aware multi-stage network (ReCMN) for image rain removal that gradually predicts clean derained results. Specifically, the ReCMN introduces a multi-stage strategy to perform contextual relationship modeling. Firstly, we use the densely residual extraction block (DREB) to guide feature extraction. Then, a multi-scale context aggregation block (MCAB) is designed to utilize the long-distance dependencies and multiple scale features, which can fuse features of different levels to fully exploit contextual information. Finally, we develop a parallel attention block (PAB) to capture the channel and spatial information and only pass effective feature representation. Experimental results demonstrate that our method outperforms several state-of-the-art methods, based on both synthetic datasets and real-world rainy images.

Semi‐MoreGAN: Semi‐supervised Generative Adversarial Network for Mixture of Rain Removal

AbstractReal‐world rain is a mixture of rain streaks and rainy haze. However, current efforts formulate image rain streaks removal and rainy haze removal as separated models, worsening the loss of image details. This paper attempts to solve the mixture of rain removal problem in a single model by estimating the scene depths of images. To this end, we propose a novel SEMI‐supervised Mixture Of rain REmoval Generative Adversarial Network (Semi‐MoreGAN). Unlike most of existing methods, Semi‐MoreGAN is a joint learning paradigm of mixture of rain removal and depth estimation; and it effectively integrates the image features with the depth information for better rain removal. Furthermore, it leverages unpaired real‐world rainy and clean images to bridge the gap between synthetic and real‐world rain. Extensive experiments show clear improvements of our approach over twenty representative state‐of‐the‐arts on both synthetic and real‐world rainy images. Source code is available at https://github.com/syy-whu/Semi-MoreGAN.

Cross-domain collaborative learning for single image deraining

Deep Convolutional Neural Networks (DCNN) have achieved outstanding performance in image deraining tasks. However, current most methods regard rain streak removal as a one-to-one problem, and intra domain shift of different synthetic datasets is usually ignored. Therefore, the deraining models trained on one synthetic dataset cannot effectively remove the rain streak of other synthetic datasets. Also, image deraining models which are trained on the labeled synthetic datasets mostly suffer from performance degradation when tested on the unlabeled real datasets due to the inter domain gap. To address this issue, this paper proposes a Cross-Domain Collaborative Learning (CDCL) framework to minimize the intra domain shift and inter domain gap. Firstly, a dual branch deraining network with collaborative learning is proposed to eliminate the distribution shift of rain streaks of images within synthetic domains. Then, a Cross-Domain Pseudo Label Generation (CDPLG) method is designed to obtain more accurate and robust pseudo labels for real-world rainy images, and the online generated pseudo labels are utilized to train the dual branch deraining network for reducing the domain gap between synthetic domain and real domain. Extensive experiments are conducted on the public benchmark datasets including synthetic datasets and real datasets in image deraining, and experimental results demonstrate that our proposed framework performs favorably compared with the state-of-the-art models.

Single image deraining via deep residual attention and encoder-decoder network

Single image rain removal remains a crucial and challenging low-level image processing task while significantly for outdoor-based high-level computer vision tasks. Recently, deep convolutional neural networks (CNNs) have become the mainstream structure of removing rain streaks and obtained remarkable performance. However, most of the existing CNN-based methods mainly pay attention to completely removing rain streaks while neglecting the restoration of details after deraining, which suffer from poor visual performance. In this paper, we propose a deep residual attention and encoder-decoder network to overcome the above shortcoming. Specifically, we develop an excellent basic block that contains dual parallel paths which are called rain removal network and detail restore network, respectively, to perform entirely and in-depth mapping relationships from rain to no-rain. The upper rain removal network is composed of dilated convolution and channel attention. This combination can explore the correlation between features from the dimensions of spatial and channel. Meanwhile, for the lower detail restore network, we construct a simple yet effective symmetrical encoder-decoder structure to prevent the loss of global structures information and encourage the details back. Furthermore, our network is end-to-end trainable, easy to implement and without giant parameter quantity. Extensive experiments on synthetic and real-world datasets have shown that our DRAEN achieves better accuracy and visual improvements against recent state-of-the-art methods.

Online-Updated High-Order Collaborative Networks for Single Image Deraining

Single image deraining is an important and challenging task for some downstream artificial intelligence applications such as video surveillance and self-driving systems. Most of the existing deep-learning-based methods constrain the network to generate derained images but few of them explore features from intermediate layers, different levels, and different modules which are beneficial for rain streaks removal. In this paper, we propose a high-order collaborative network with multi-scale compact constraints and a bidirectional scale-content similarity mining module to exploit features from deep networks externally and internally for rain streaks removal. Externally, we design a deraining framework with three sub-networks trained in a collaborative manner, where the bottom network transmits intermediate features to the middle network which also receives shallower rainy features from the top network and sends back features to the bottom network. Internally, we enforce multi-scale compact constraints on the intermediate layers of deep networks to learn useful features via a Laplacian pyramid. Further, we develop a bidirectional scale-content similarity mining module to explore features at different scales in a down-to-up and up-to-down manner. To improve the model performance on real-world images, we propose an online-update learning approach, which uses real-world rainy images to fine-tune the network and update the deraining results in a self-supervised manner. Extensive experiments demonstrate that our proposed method performs favorably against eleven state-of-the-art methods on five public synthetic datasets and one real-world dataset.

No rain in the world: A novel residual deep attention network for single image rain removal

Clear images are generally desirable in high-level computer vision algorithms which are mostly deployed outdoors. However, affected by the changeable weather in the real world, images are inevitably contaminated by rain streaks. Deep convolutional neural networks (CNNs) have shown significant potential in rain streaks removal. The performance of most existing CNN-based deraining methods is often enhanced by stacking vanilla convolutional layers and some other methods use dilated convolution which can only model local pixel relations to provide the necessary but limited receptive field. Therefore, long-range contextual information is rarely considered for this specific task, thus, deraining a single image remains challenging problem. To address the above problem, an effective residual deep attention network (RDANet) for single image rain removal is proposed. Specifically, we design a strong basic unit that contains dilated convolution, spatial and channel attention module (SCAM) simultaneously. As contextual information is very important for rain removal, the proposed basic unit can capture global long-distance dependencies among pixels in feature maps and model feature relations across channels. Compared with a single dilated convolution, the spatial and channel attention enhance the feature expression ability of the network. Moreover, some previous works have proven that the no-rain information in a rain image will be missing during deraining. To enrich the detailed information in the clean images, we present a residual feature processing group (RFPG) that contains several source skip connections to inject rainy shallow source information into each basic unit. In summary, our model can effectively handle complicated long rain streaks in spatial and the outputs of the network can retain most of the details of the original rain images. Experiments demonstrate the superiority of our RDANet over state-of-the-art methods in terms of both quantitative metrics and visual quality on both synthetic and real rainy images.

Beyond Monocular Deraining: Parallel Stereo Deraining Network Via Semantic Prior

Rain is a common natural phenomenon. Taking images in the rain however often results in degraded quality of images, thus compromises the performance of many computer vision systems. Most existing de-rain algorithms use only one single input image and aim to recover a clean image. Few work has exploited stereo images. Moreover, even for single image based monocular deraining, many current methods fail to complete the task satisfactorily because they mostly rely on per pixel loss functions and ignore semantic information. In this paper, we present a Paired Rain Removal Network (PRRNet), which exploits both stereo images and semantic information. Specifically, we develop a Semantic-Aware Deraining Module (SADM) which solves both tasks of semantic segmentation and deraining of scenes, and a Semantic-Fusion Network (SFNet) and a View-Fusion Network (VFNet) which fuse semantic information and multi-view information respectively. In addition, we also introduce an Enhanced Paired Rain Removal Network (EPRRNet) which exploits semantic prior to remove rain streaks from stereo images. We first use a coarse deraining network to reduce the rain streaks on the input images, and then adopt a pre-trained semantic segmentation network to extract semantic features from the coarse derained image. Finally, a parallel stereo deraining network fuses semantic and multi-view information to restore finer results. We also propose new stereo based rainy datasets for benchmarking. Experiments on both monocular and the newly proposed stereo rainy datasets demonstrate that the proposed method achieves the state-of-the-art performance. https://github.com/HDCVLab/Stereo-Image-Deraining .

A two‐stage method for single image de‐raining based on attention smoothed dilated network

Rain can severely hamper the visibility of scene objects. Although existing deep learning methods have reported promising performance, they often fail to obtain satisfactory results in many practical situations, especially when the input image contains both rain streaks and haze-like degradation. In this paper, a new two-stage method based on attention smoothed dilated network (SDN) is proposed. Unlike most fully-supervised methods, the mixture of rain streaks and haze-like effects is considered in the model. The proposed method consists of two stages. First, a generative adversarial network guided by the rain-streak attention map is proposed to remove rain streaks, where a multi-stage attention module is used to accurately locate rain streaks in the generator. Second, haze-like effects are further removed through SDN with the same structure as the generator. Extensive experiments on multiple datasets show that the method outperforms the state-of-the-art in both objective evaluation and visual quality.

LRP-net: A lightweight recursive pyramid network for single image deraining

Single image deraining, as a low-level computer vision task, has been drawn extensive attention in recent years. Rain streaks can degrade subjective visibility quality, meanwhile, bring significant difficulties to subsequent high-level computer vision tasks such as object detection. Nowadays, deep-learning based methods, specifically Convolutional Neural Networks (CNN) based ones are adopted to remove the rain streaks and become the state-of-the-art. However, existing popular deep-networks have complicated branches and numerous layers, which strengthen the ability of removing rain streaks and result in high memory and computational cost inevitably. This restricts many applications in real-time and limited computation resource scene, especially on mobile or edge devices. To handle this issue, this paper proposes a novel Lightweight Recursive Pyramid network (LRP-Net) with a small number of parameters for single image deraining. To begin with, we propose a novel Lightweight Pyramid Deraining (LPD) block which consists of a multi-scale pyramid convolution for sufficient feature extraction and a pointwise convolution for feature fusion. Meanwhile, we also design a novel group convolution strategy in LPD for the sake of remarkable parameter reduction. Secondly, we combine a recursive deraining mechanism, a critical component that serves as a feature fusion iterator to construct our LRPNet a powerful and lightweight multi-stage model. In the benefit of the combination between the LPD block and recursive mechanism, the total number of parameters in LRP-Net is only 130 k, which is nearly a 40- reduction compared with the latest state-of-the-art models. The extensive experiments demonstrate the superiority of LRP-Net in both quantitative assessments and visual quality.

Multi-directional rain streak removal based on infimal convolution of oscillation TGV

Image deraining is essential in both image processing and computer vision. However it is still challenging to remove rain streaks, due to the fact that they are complex and usually have multiple directions. In this paper, we propose a novel infimal convolution type functional based model to remove multi-directional rain streaks. Specifically, we employ total generalized variation (TGV) to represent the rain-free background and m-fold infimal convolution of oscillation TGV (ICTGVmosci) to capture the rain streaks with m different directions. The existence of solutions to the proposed model is proved. We also design an algorithm using the classical first-order Primal–Dual method for solving the proposed model. Numerical experiments on both synthetic and real images demonstrate that the proposed model outperforms the state-of-the-art methods quantitatively and qualitatively.

An Optimum Deraining Scheme using Sparse Coding

Rain streak removal is a challenging and interesting task of image processing where the rain streaks will be removed from an image with rain streaks. In the literature, a large number of proposals are made where rain streak removal is considered as image enhancement or denoising. In this paper, a model of rain streak removal was proposed using sparse coding. First the regularization terms of the rain streak removal are defined. Then, a suitable dictionary of sub- dictionary with respect to specific patch of input rainy image are prepared. Finally, the Sparse code is applied on the patches of input image individually. The simulation results prove that the proposed technique performs very well even if the raindrop size is above a certain threshold.

Video Deraining Using the Visual Properties of Rain Streaks

In computer vision applications, the visibility of the video content is crucial to perform analysis for better accuracy. The visibility can be affected by several atmospheric interferences in challenging weather &#x2013; one such interference is the appearance of rain streaks. Recently, rain streak removal has achieved plenty of interest among researchers, as it has some exciting applications such as autonomous cars, intelligent traffic monitoring systems, multimedia, etc. In this paper, we propose a novel and simple method of rain streak removal by combining three novel extracted visual features focusing on the temporal appearance, wide shape and relative location of the rain streak. We called it the TAWL (<i>Temporal Appearance, Width, and Location</i>) method. The proposed TAWL method adaptively uses features from different resolutions and frame rates. Moreover, it progressively processes features from the upcoming frames so that it can remove rain in real-time. Experiments have been conducted using video sequences with both real rain and synthetic rain to compare the performance of the proposed method against the relevant state-of-the-art methods. The experimental results demonstrate that the proposed method outperforms the state-of-the-art methods by removing more rain streaks while keeping other moving regions.

Research and Implementation of Image Rain Removal Based on Deep Learning

Abstract The traditional rain removal algorithm needs to optimize a large number of parameters, and it is only effective for rain of a specific shape, and the model generalization ability is poor. In recent years, the performance of rain removal methods based on deep learning is better than many traditional methods, but there are problems such as incomplete or excessive rain removal, and incomplete texture reconstruction of background details. This paper proposes a rain removal network based on generative confrontation, which connects the high and low frequency parts and integrates them into the model. At the same time, the attention mechanism cyclic neural network is organically combined, which can better preserve the background texture while removing rain. Theoretical can produce better rain streak removal with better color distortion.

Deep Feature Interactive Aggregation Network for Single Image Deraining

Single image deraining task aims at removing rain streaks from a degraded input and reconstructing the high-quality image. In recent years, image processing tasks mostly apply the U-shaped architecture to capture rich contextual information. However, it is difficult to achieve long-range pixel dependencies due to the local receptive field of the convolution operation. In this paper, we propose a deep feature interactive aggregation network for single image deraining to enhance the long-range dependencies among features and realize the interaction of information. To fully utilize the high-level semantic features, we design a long-range dependency feature aggregation module to greatly improve the representational ability of the original U-shaped architecture. It aggregates multi-scale features and calculates the interactive attention of non-overlapping patches among feature maps. In addition, we adopt group normalization to retain the independence of each given image. It interacts with the information among features in an individual image and normalizes the channels of each group to weaken the correlation between batch data processing. Experimental results on widely acknowledged datasets also demonstrate the superiority of our proposed network over previous state-of-the-art methods.

Progressive Subtractive Recurrent Lightweight Network for Video Deraining

Presence of rainy artifacts severely degrade the overall visual quality of a video and tend to overlap with the useful information present in the video frames. This degraded video affects the effectiveness of many automated applications like traffic monitoring, surveillance, <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">etc.</i> As video deraining is a pre-processing step for automated applications, it is highly demanded to have a lightweight deraining module. Therefore, in this paper, a <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">“Progressive Subtractive Recurrent Lightweight Network”</i> is proposed for video deraining. Initially, the Multi-Kernel feature Sharing Residual Block (MKSRB) is designed to learn different sizes of rain streaks which facilitates the complete removal of rain streaks through progressive subtractions. These MKSRB features are merged with previous frame output recurrently to maintain the temporal consistency. Further, multi-receptive feature subtraction is performed through Multi-scale Multi-Receptive Difference Block (MMRDB) to avoid loss of details and extract high-frequency information. Finally, progressively learned features through MKSRB and recurrent feature merging are aggregated with fused MMRDB features which outputs the rain-free frame. Substantial experiments on prevailing synthetic datasets and real-world videos verify the superior performance of the proposed method over the existing state-of-the-art methods for video deraining.

Low-rank appearance-preserving rain streak removal from single images

Low-rank appearance-preserving rain streak removal from single images

Performance analysis of a dual stage deep rain streak removal convolution neural network module with a modified deep residual dense network

Performance analysis of a dual stage deep rain streak removal convolution neural network module with a modified deep residual dense network

PFDN: Pyramid Feature Decoupling Network for Single Image Deraining.

Restoring images degraded by rain has attracted more academic attention since rain streaks could reduce the visibility of outdoor scenes. However, most existing deraining methods attempt to remove rain while recovering details in a unified framework, which is an ideal and contradictory target in the image deraining task. Moreover, the relative independence of rain streak features and background features is usually ignored in the feature domain. To tackle these challenges above, we propose an effective Pyramid Feature Decoupling Network (i.e., PFDN) for single image deraining, which could accomplish image deraining and details recovery with the corresponding features. Specifically, the input rainy image features are extracted via a recurrent pyramid module, where the features for the rainy image are divided into two parts, i.e., rain-relevant and rain-irrelevant features. Afterwards, we introduce a novel rain streak removal network for rain-relevant features and remove the rain streak from the rainy image by estimating the rain streak information. Benefiting from lateral outputs, we propose an attention module to enhance the rain-irrelevant features, which could generate spatially accurate and contextually reliable details for image recovery. For better disentanglement, we also enforce multiple causality losses at the pyramid features to encourage the decoupling of rain-relevant and rain-irrelevant features from the high to shallow layers. Extensive experiments demonstrate that our module can well model the rain-relevant information over the domain of the feature. Our framework empowered by PFDN modules significantly outperforms the state-of-the-art methods on single image deraining with multiple widely-used benchmarks, and also shows superiority in the fully-supervised domain.