Remove Rain Streaks Research Articles

Semi-RainGAN: A Semisupervised Coarse-to-Fine Guided Generative Adversarial Network for Mixture of Rain Removal

Images taken in various real-world scenarios meet the symmetrical goal of simultaneously removing foreground rain-induced occlusions and restoring the background details. This inspires us to remember the principle of symmetry; real-world rain is a mixture of rain streaks and rainy haze and degrades the visual quality of the background. Current efforts formulate image rain streak removal and rainy haze removal as separate models, which disrupts the symmetrical characteristics of real-world rain and background, leading to significant performance degradation. To achieve this symmetrical balance, we propose a novel semisupervised coarse-to-fine guided generative adversarial network (Semi-RainGAN) for the mixture of rain removal. Beyond existing wisdom, Semi-RainGAN is a joint learning paradigm of the mixture of rain removal and attention and depth estimation. Additionally, it introduces a coarse-to-fine guidance mechanism that effectively fuses estimated image, attention, and depth features. This mechanism enables us to achieve symmetrically high-quality rain removal while preserving fine-grained details. To bridge the gap between synthetic and real-world rain, Semi-RainGAN makes full use of unpaired real-world rainy and clean images, enhancing its generalization to real-world scenarios. Extensive experiments on both synthetic and real-world rain datasets demonstrate clear visual and numerical improvements of Semi-RainGAN over sixteen state-of-the-art models.

Background-detail restoration image deraining network based on convolutional dictionary network

The heavy rain streaks can seriously fade the color of images and destroy the background texture information. Although the existing deraining methods perform well in rain removal, we find that most of them tend to favor rain removal or restoration. In this paper, we propose a background-detail restoration image deraining network based on convolutional dictionary network to achieve effective deraining results with better-restored background details. Our framework includes two sub-networks, namely the rain-removal network and the background-restoration network. The former sub-network is constructed as the multi-stage architecture. In order to accurately maintain as much background detail as possible while removing rain streaks, we break down the overall rain removal process into more manageable steps. We introduce convolutional dictionary network and Multi-Scale Detail Restoration Block (MSDRB) to achieve the balance between the effectiveness of deraining and background restoration. The second sub-network is Residual-Unet with Squeeze-and-Excitation (RUSE) operation, which can utilize the output of every stage generated from the former sub-network to fine-tune the background details in rain-free images. Experiments on benchmark public synthetic datasets and real-world datasets demonstrate the effectiveness and generalization capability of the proposed method in rain streak removal and background restoration with different rain patterns.

Progressive dilation dense residual fusion network for single‐image deraining

AbstractRain removal is very important for many applications in computer vision, and it is a challenging problem due to its ill‐posed nature, especially for single‐image deraining. In order to remove rain streaks more thoroughly, as well as to retain more details, a progressive dilation dense residual fusion network is proposed. The entire network is designed in a cascade manner with multiple fusion blocks. The fusion block consists of a dilation dense residual block (DDRB) and a dense residual feature fusion block (DRFFB), where DDRB is created for feature extraction and DRFFB is mainly designed for feature fusion operation. Meanwhile, detail compensation memory mechanism (DCMM) is leveraged between each of two cascade modules to retain more background details. Compared with previous state‐of‐the‐art methods, extensive experiments show that the proposed method can achieve better results, in terms of rain streaks removal and background details preservation. Furthermore, the authors’ network also shows its superiority for image noise removal.

Subband Differentiated Learning Network for Rain Streak Removal

Due to the adverse effects of far more complex rain streaks on the performance of outdoor vision systems, it is critical to accurately eliminate rain streaks from a single rainy image. Recently, many algorithms have made significant progress in rain streak removal. However, these approaches do not give enough consideration to the established fact that rain streaks reflected in different wavelet subbands may have distinct spatial and intensity distributions, resulting in residual degraded components in subbands if not addressed properly. To deal with this problem, we propose a subband differentiated learning network (SDLNet) for rain streak removal. First, we put forward a wavelet subband mask module (WSMM) to produce a wavelet mask, in which different concentrations of rain streak features are provided for different subbands. Second, we advance a WSMM-driven repetitive learning resblock (WSMM-driven RLR) with a novel repetitive learning mechanism (RLM). This RLM enables the WSMM-driven RLR to further probe rain streak features. Third, we propose an adaptive wavelet subband loss (AWSL) function, with the same loss structure but a different loss weight for each subband, exploiting unique error priors in subbands. Finally, the proposed SDLNet employs both a UNet-like architecture with the novel WSMM-driven RLR and the AWSL function, generating a derained image. Compared with state-of-the-art methods, experimental results show that our proposed SDLNet achieves better rain streak elimination performance, where the average PSNR of derained images is improved up to 1.86 dB, while recovering more texture details of clean images with better subjective quality in recovered images.

Symmetric Enhancement of Visual Clarity through a Multi-Scale Dilated Residual Recurrent Network Approach for Image Deraining

Images captured during rainy days present the challenge of maintaining a symmetrical balance between foreground elements (like rain streaks) and the background scenery. The interplay between these rain-obscured images is reminiscent of the principle of symmetry, where one element, the rain streak, overshadows or disrupts the visual quality of the entire image. The challenge lies not just in eradicating the rain streaks but in ensuring the background is symmetrically restored to its original clarity. Recently, numerous deraining algorithms that employ deep learning techniques have been proposed, demonstrating promising results. Yet, achieving a perfect symmetrical balance by effectively removing rain streaks from a diverse set of images, while also symmetrically restoring the background details, is a monumental task. To address this issue, we introduce an image-deraining algorithm that leverages multi-scale dilated residual recurrent networks. The algorithm begins by utilizing convolutional activation layers to symmetrically process both the foreground and background features. Then, to ensure the symmetrical dissemination of the characteristics of rain streaks and the background, it employs long short-term memory networks in conjunction with gated recurrent units across various stages. The algorithm then incorporates dilated residual blocks (DRB), composed of dilated convolutions with three distinct dilation factors. This integration expands the receptive field, facilitating the extraction of deep, multi-scale features of both the rain streaks and background information. Furthermore, considering the complex and diverse nature of rain streaks, a channel attention (CA) mechanism is incorporated to capture richer image features and enhance the model’s performance. Ultimately, convolutional layers are employed to fuse the image features, resulting in a derained image. An evaluation encompassing seven benchmark datasets, assessed using five quality metrics against various conventional and modern algorithms, confirms the robustness and flexibility of our approach.

Image deraining based on dual-channel component decomposition

Image deraining aims to remove rain streaks from images and reduce information loss in outdoor images caused by rain. As a fundamental task in image processing, image deraining not only enhances the visibility of images but also provides necessary image restoration for advanced vision tasks. Existing image deraining models mostly train end-to-end models by minimizing the similarity between the output image of the model and the rain-free ground truth. Although these methods have achieved significant results, they often perform poorly in the face of dense and changing rain streak scenes. In this paper, we propose a novel method, called Dual-Channel Component Decomposition Network (DCD-Net). The basic idea of DCD-Net is to leverage the separability prior of rainy images, treats the rain-free background layer and the rain streak mask layer as two parallel component extraction tasks. To this end, it builds a dual-branch parallel networks that extract the rain-free background image and decouple the reconstruction information of the rain streak mask, respectively. It finally applies a composite multi-level contrastive supervision to the output of the above dual-branch parallel network, thereby achieving rain streak removal. Extensive experiments on various datasets demonstrate that the proposed model outperforms existing methods in deraining dense rain streak images.

Context and detail interaction network for stereo rain streak and raindrop removal

Recently stereo image deraining has attracted lots of attention due to its superiority of abundant information from cross views. Exploring interaction information across stereo views is the key to improving the performance of stereo image deraining. In this paper, we design a general coarse-to-fine deraining framework for stereo rain streak and raindrop removal, called CDINet, comprising a stereo rain removal subnet and a stereo detail recovery subnet to restore images progressively. Two types of interaction modules are devised to explore interaction information for rain removal and detail recovery, respectively. Specifically, a global context interaction module is proposed to learn long-range dependencies of stereo images and remove rain by utilizing stereo structural information. A local detail interaction module is designed to model local contextual correlation, which aims at restoring the detail information by using neighborhood information from cross views. Extensive experiments are conducted on the two datasets including a synthetic rain streak removal dataset (RainKITTI) and a real raindrop removal dataset (Stereo Waterdrop), which demonstrates that our method sets new state-of-the-art deraining performance in terms of both quantitative and qualitative metrics with faster speed.

An investigation of rain streak removal models based on expert experience and deep learning

Computer vision technology has a wide range of applications in today's society, and image rain removal is of great importance in outdoor vision capture. Today's image de-rain techniques are divided into video de-rain, and image de-rain, with the image de-rain task being more difficult than the video de-rain task due to the lack of a time factor. Current image rain removal methods are divided into three main types: filter-based methods, a priori knowledge-based methods and deep learning methods. Although these methods can achieve the image rain removal requirements to a certain extent, there is still no highly generalized method that can better solve the image rain removal problem in all cases. This paper first considers a filter-based approach, which takes less time to run but is difficult to remove cleanly for complex rain streaks. Secondly, this paper examines an a priori knowledge-based approach, which requires the study of how rain images are constructed as a priori knowledge and then uses the existing prior knowledge to remove rain from the images. This approach has a high reliance on prior knowledge and poor generalization. Finally, this paper investigates a deep learning-based method that requires a large number of supervised samples for training and has a better rain removal effect, but ignores the prior knowledge of rain streaks and is prone to overfitting. Based on these three methods we collected some data and experimental results in this field and summarized and analyzed them, giving reasons for the strengths and weaknesses of these three models and presenting new perspectives for future improvements in image de-rain methods.

Memory-efficient multi-scale residual dense network for single image rain removal

Outdoor photography is greatly affected by bad rainfall, which degrades the quality of captured images and hinders practical application. In response to the above problems, we propose a memory-efficient multi-scale residual dense network (MMRD-Net) for single image rain removal. Firstly, we use a combination or fusion of extended convolution blocks of different sizes to accurately extract multi-scale features of rain streaks and raindrops while retaining rich local detail. Secondly, we add attention modules to the residual block, which performs adaptive operations on rainfall and non-rainfall areas to extract more and more accurate image features and details of rain streaks and raindrops. During this period, we opened up a space for shared use by all the concatenation layers in the network, which reduces the memory consumption of the network. It allows for better aggregation of stormwater features at multiple levels and also allows for faster convergence of the network. Finally, we tested on both synthetic and real datasets to obtain subjective results and objective evaluations. The experimental results show that the performance of our algorithm is good and robust for joint rain removal tasks of different densities. Our algorithm outperforms twelve other different classical algorithms in terms of its ability to remove rain streaks. It can produce visually sharper deraining images and the enhanced visibility of the images is effective for computer vision applications (Google Vision API).

Unsupervised video rain streaks removal with deep foreground–background modeling

Outdoor video rain streaks removal is an important inverse problem in video processing that benefits subsequent applications. Traditional methods utilize prior information with interpretable domain knowledge while they are not tenable to capture complex structures of real-world videos. Deep learning methods learn a deraining mapping with a large model capacity brought by deep neural networks and their performances highly depend on the volume and diversity of training data. To address the challenging video deraining problem, we suggest an unsupervised video rain streaks removal method by solely using the observed rainy video. For the complex clean video, inspired by the classical foreground–background decomposition, we employ a deep convolutional neural network to capture the moving foreground and a disentangled deep spatial–temporal network with an affine operator to capture the underlying low-rank structure of the dynamic background. The foreground and background components are well balanced by a learnable probability mask. For structured rain streaks, we introduce a learnable total variation regularization whose parameters (i.e., rain directions) can be unsupervisedly learned. The deep modeling of the complex clean video and the simple yet effective modeling of structured rain streaks under the physical interpretable decomposition framework, which benefit each other in nature, are organically integrated to boost the deraining performance. Extensive experiments on synthetic and real-world rainy videos demonstrate the superiority of our method over state-of-the-art traditional and deep learning-based video deraining methods.

Removing Rain Streaks from Visual Image Using a Combination of Bilateral Filter and Generative Adversarial Network

Images acquired using vision sensors are easily affected by environmental limitations, especially rain streaks. These streaks will seriously reduce image quality, which, in turn, reduces the accuracy of the algorithms that use the resulting images in vision sensor systems. In this paper, we proposed a method that combined the bilateral filter with the generative adversarial network to eliminate the interference of rain streaks. Unlike other methods that use all the information in an image as the input to the generative adversarial network, we used a bilateral filter to preprocess and separate the high frequency part of the original image. The generator for the high-frequency layer of the image was designed to generate an image with no rain streaks. The high-frequency information of the image was used in a high-frequency global discriminator designed to measure the authenticity of the generated image from multiple perspectives. We also designed a loss function based on the structural similarity index to further improve the effect of removal of the rain streaks. An ablation experiment proved the validity of the method. We also compared images in synthetic and real-world datasets. Our method could retain more image information, and the generated image was clearer.

P‐4.4: A Real‐Time Video Rain Removal Algorithm Using Layer Priors

Vision system is widely used in fields such as target detection and security monitoring, but since its heavy dependence on the qualities of input images is in conflict with the fact that rain and other bad weather will cause the quality of input image to decline, the video deraining algorithm is of great importance. Based on the rain streaks removal algorithm using layer priors, this paper proposes a real-time video rain removal algorithm using layer priors, which regards the rain image as the linear superimposition of background layer and raindrops layer, and decomposes them via MAP. During the procession, temporal direction gradient and vertical direction gradient are adopted as explicit priors, while CNN is adopted as implicit prior. After analyzing the experiment results, it could be observed that the proposed algorithm can not only output rain-free images of higher quality, but also greatly reduce the operation time, which meet the real-time requirement to some extent.

Adaptive weighted rain streaks model-driven deep network for single image deraining

In the computer vision field, single image rain removal is an extremely important pre-processing task since rain streaks can degrade the performance of subsequent high-level tasks and outdoor image quality. In recent years, massive deep learning-based deraining methods have been proposed to remove rain streaks from rainy images, and they have achieved promising performance. However, most deep learning-based methods lack interpretability and have limited performance in image details restoration and rain streaks removal. In this paper, we propose a novel rain streaks model-driven deep network, MSANet, to alleviate these issues. First of all, instead of simply stacking those available convolutions, pooling, and attention blocks, the structure of MSANet is derived from a prior rain removal model and the structure is thus fully interpretable. Compared to prior models available in the literature, we introduce a weighted convolutional dictionary model to capture the shapes, sizes, directions, regions, and multi-scale information of rain streaks. And to reduce the bias associated with the L2-norm, we utilize an adaptive approach to constrain the fidelity term of the model. Finally, we employ the alternating direction method of multipliers (ADMM) algorithm to optimize this model and unfold the optimization procedure into a new neural network architecture by modeling each operation in the algorithm using network blocks. All the parameters can be automatically learned by end-to-end training. Extensive experiments on several benchmark datasets demonstrate that the proposed network architecture is more efficient than state-of-the-art methods based on both subjective and objective evaluations.

DSDNet: Toward single image deraining with self-paced curricular dual stimulations

A crucial challenge regarding the single image deraining task is to completely remove rain streaks while still preserving explicit image details. Due to the inherent overlapping between rain streaks and background scenes, the texture details could be inevitably lost when clearing rain away from the degraded image, making the two purposes contradictory. Existing deep learning based approaches endeavor to resolve the two issues successively in a cascaded framework or to treat them as independent tasks in a parallel structure. However, none of the models explores a proper interaction between rain distributions and hidden feature responses, which intuitively would provide more clues to facilitate the procedures of rain streak removal as well as detail restoration. In this paper, we investigate the impact of rain streak detection for single image deraining and propose a novel deep network with dual stimulations, namely, DSDNet. The proposed DSDNet utilizes a dual-stream pipeline to separately estimate rain streaks and a loss of details, and more importantly, an additional mask that indicates both location and intensity of rains is jointly predicted. In particular, the rain mask is involved in a tailored stimulation strategy that is deployed into each stream of the proposed model, serving as guidance for allowing the network to focus on rain removal and detail recovery in rain regions rather than non-rain areas. Moreover, we incorporate a self-paced semi-curriculum learning design to alleviate the learning ambiguity brought by the prediction of the rain mask and thus accelerate the training process. Extensive experiments demonstrate the proposed method outperforms the state-of-the-art methods on several benchmarks, including in both synthetic and real-world scenarios. The effectiveness of the proposed method is also validated via joint single image deraining, detection, and segmentation tasks.

Recurrent wavelet structure-preserving residual network for single image deraining

The combination of deep learning and image prior has been widely used in single image deraining since 2017. Recent studies have demonstrated an excellent deraining effect on the high-frequency part of rain images, but less attention was paid to the low-frequency part of rain images. The rain streaks remain in the low-frequency part of rain images, thus limiting the deraining effect. Since the rain streaks in rain images are often mixed with object edges and background scenes, it is challenging to separate rain from them by directly learning the deraining function in the image domain. To solve these problems, we propose a novel Recurrent Wavelet Structure-preserving Residual Network (RWSRNet), which mainly preserves and introduces the low-frequency sub-images of each level into the low-frequency rain removal sub-networks that are greatly different from the state-of-the-art approaches introducing wavelet transform. In addition, we also share the low-frequency structure information to the high-frequency sub-networks through block connection, which further enriches the detailed information, facilitates convergence, and strengthens the ability of our network to remove rain streaks in high frequency. Finally, we fuse the derained low-frequency sub-images of each level through the proposed image weighted blending module and finally reconstruct the low- and high-frequency sub-images into clean images through inverse wavelet transform recursively. The experimental results indicate that the proposed method achieves an excellent deraining effect on both low- and high-frequency parts of rain images and has better performance in low-frequency preservation and high-frequency enhancement in comparison with the state-of-the-art approaches on synthetic and real image datasets.

Rain Streaks Removal in Images using Extended Generative Adversarial-based Deraining Framework

Rain Streaks Removal in Images using Extended Generative Adversarial-based Deraining Framework

Rain Removal from Light Field Images with 4D Convolution and Multi-scale Gaussian Process.

Existing deraining methods mainly focus on a single input image. However, with just a single input image, it is extremely difficult to accurately detect and remove rain streaks, in order to restore a rain-free image. In contrast, a light field image (LFI) embeds abundant 3D structure and texture information of the target scene by recording the direction and position of each incident ray via a plenoptic camera, which has emerged as a popular device in the computer vision and graphics research communities. However, making full use of the abundant information available from LFIs, such as 2D array of sub-views and the disparity map of each sub-view, for effective rain removal is still a challenging problem. In this paper, we propose a novel network, 4D-MGP-SRRNet, for rain streak removal from LFIs. Our method takes as input all sub-views of a rainy LFI. In order to make full use of the LFI, we adopt 4D convolutional layers to build the proposed rain steak removal network to simultaneously process all sub-views of the LFI. In the proposed network, the rain detection model, MGPDNet, with a novel Multi-scale Self-guided Gaussian Process (MSGP) module is proposed to detect high-resolution rain streaks from all sub-views of the input LFI at multi-scales. Semi-supervised learning is introduced for MSGP to accurately detect rain streaks by training on both virtual-world rainy LFIs and real-world rainy LFIs at multi-scales via calculating pseudo ground truths for real-world rain streaks. We then feed all sub-views subtracting the predicted rain streaks into a 4D convolution-based Depth Estimation Residual Network (DERNet) to estimate the depth maps, which are later converted into fog maps. Finally, all sub-views concatenated with the corresponding rain streaks and fog maps are fed into a powerful rainy LFI restoring model based on the adversarial recurrent neural network to progressively eliminate rain streaks and recover the rain-free LFI. Extensive quantitative and qualitative evaluations conducted on both synthetic LFIs and real-world LFIs demonstrate the effectiveness of our proposed method.

Spatial Non-Stationary Near-Field Channel Modeling and Validation for Massive MIMO Systems

Existing deraining methods focus mainly on a single input image. However, with just a single input image, it is extremely difficult to accurately detect and remove rain streaks, in order to restore a rain-free image. In contrast, a light field image (LFI) embeds abundant 3D structure and texture information of the target scene by recording the direction and position of each incident ray via a plenoptic camera. LFIs are becoming popular in the computer vision and graphics communities. However, making full use of the abundant information available from LFIs, such as 2D array of sub-views and the disparity map of each sub-view, for effective rain removal is still a challenging problem. In this paper, we propose a novel method, 4D-MGP-SRRNet, for rain streak removal from LFIs. Our method takes as input all sub-views of a rainy LFI. To make full use of the LFI, it adopts 4D convolutional layers to simultaneously process all sub-views of the LFI. In the pipeline, the rain detection network, MGPDNet, with a novel Multi-scale Self-guided Gaussian Process (MSGP) module is proposed to detect high-resolution rain streaks from all sub-views of the input LFI at multi-scales. Semi-supervised learning is introduced for MSGP to accurately detect rain streaks by training on both virtual-world rainy LFIs and real-world rainy LFIs at multi-scales via computing pseudo ground truths for real-world rain streaks. We then feed all sub-views subtracting the predicted rain streaks into a 4D convolution-based Depth Estimation Residual Network (DERNet) to estimate the depth maps, which are later converted into fog maps. Finally, all sub-views concatenated with the corresponding rain streaks and fog maps are fed into a powerful rainy LFI restoring model based on the adversarial recurrent neural network to progressively eliminate rain streaks and recover the rain-free LFI.

Towards Robust Rain Removal with Unet++

Image deraining has become a hot topic in the field of computer vision. It is the process of removing rain streaks from an image to reconstruct a high-quality background. This study aims at improving the performance of image rain streak removal and reducing the disruptive effects caused by rain. To better fit the rain removal task, an innovative image deraining method is proposed, where a kernel prediction network with Unet++ is designed and used to filter rainy images, and rainy-day images are used to estimate the pixel-level kernel for rain removal. To minimize the gap between synthetic and real data and improve the performance in real rainy image handling, a loss function and an effective data optimization method are suggested. In contrast with other methods, the loss function consists of Structural Similarity Index loss, edge loss, and L1 loss, and it is adopted to improve performance. The proposed algorithm can improve the Peak Signal-to-Noise ratio by 1.3% when compared to conventional approaches. Experimental results indicate that the proposed method can achieve a better efficiency and preserve more image structure than several classical methods.

Event-Aware Video Deraining via Multi-Patch Progressive Learning.

In this paper, we address the problem of video-based rain streak removal by developing an event-aware multi-patch progressive neural network. Rain streaks in video exhibit correlations in both temporal and spatial dimensions. Existing methods have difficulties in modeling the characteristics. Based on the observation, we propose to develop a module encoding events from neuromorphic cameras to facilitate deraining. Events are captured asynchronously at pixel-level only when intensity changes by a margin exceeding a certain threshold. Due to this property, events contain considerable information about moving objects including rain streaks passing though the camera across adjacent frames. Thus we suggest that utilizing it properly facilitates deraining performance non-trivially. In addition, we develop a multi-patch progressive neural network. The multi-patch manner enables various receptive fields by partitioning patches and the progressive learning in different patch levels makes the model emphasize each patch level to a different extent. Extensive experiments show that our method guided by events outperforms the state-of-the-art methods by a large margin in synthetic and real-world datasets.