Rain Image Research Articles

Single-image rain removal using deep residual network

Rain not only reduces visibility, but also prevents many computer vision algorithms from functioning properly. Removing rain from a single image is an important issue in the field of computer vision. Rain usually produces a phenomenon similar to fog, which is more pronounced when the rain is dense. We constructed a rain model, including rain streaks and fog generated by rainfall. Based on this model, we constructed a multi-task learning network to learn the rain streaks and fog in the image as well as the final clean background. To take advantage of contextual information, we use an expanded convolution network to handle different levels of rain and fog. We learn the residual images of fog and rain based on the deep residual network to avoid the influence of inaccurate parameter estimation on the results. Experiments show that the method in this paper can effectively remove rain and fog in the image.

Rain streaks removal from single image based on texture constraint of background scene

Removing rain streaks from single image is mathematically ill-defined and the known strategies tend to over-smooth the background and remove image details. To tackle this problem, this paper proposes a novel two-stage system. Firstly, in view of the frequency and direction of rain streaks, Gabor filter is first used to suppress rain streaks. Based on the bases extracted from natural images, several Gabor filtered images extracted from the input rain image are fused by a new proposed method, and the fused result provides a texture constraint for the non-rain part. Secondly, a low-rank model is adopted to describe rain streaks as they take on similar and repeated patterns in an imaging scene. By combining the texture constraint and the low-rank model, a global cost function is constructed. Finally, an optimization strategy is proposed based on the split Bregman technique, which solves the function with high performance. A large number of experimental results show the competitive performance and efficiency of the proposed method in comparison with the state-of-the-art methods, making it more applicable in real practices.

Rain Detection and Removal via Shrinkage-based Sparse Coding and Learned Rain Dictionary

Abstract Rain removal is essential for achieving autonomous driving because it preserves the details of objects that are useful for feature extraction and removes the rain structures that hinder feature extraction. Based on a linear superposition model in which the observed rain image is decomposed into two layers, a rain layer and a non-rain layer, conventional rain removal methods estimate these two layers alternatively from an observed single image based on prior modeling. However, the prior knowledge used for the rain structures is not always correct because various types of rain structures can be observed in the rain images, which results in inaccurate rain removal. Therefore, in this article, a novel rain removal method based on the use of a scribbled rain image set and a new shrinkage-based sparse coding model is proposed. The scribbled rain images have information about which pixels have rain structures. Thus, various types of rain structures can be modeled, owing to the abundance of rain structures in the rain image set. To detect the rain regions, two types of approaches, one based on reconstruction error comparison (REC) via a learned rain dictionary and the other based on a deep convolutional neural network (DCNN), are presented. With the rain regions, the proposed shrinkage-based sparse coding model determines how much to reduce the sparse codes of the rain dictionary and maintain the sparse codes of the non-rain dictionary for accurate rain removal. Experimental results verified that the proposed shrinkage-based sparse coding model could remove rain structures and preserve objects’ details due to the REC- or DCNN-based rain detection using the scribbled rain image set. Moreover, it was confirmed that the proposed method is more effective at removing rain structures from similar objects’ structures than conventional methods.

Rain-removing Algorithm for Image Based on Deep Learning

Aiming at the problem that the decrease of driver’s visual definition is affected by rainy weather, a new deep learning image rain removal algorithm is proposed. The network consists of a rainwater detection subnetwork and a rainwater removal subnetwork in series. The rainwater detection subnetwork adopts residual learning network, which can accurately learn the difference between rain images and non-rain images. The rain-removing subnetwork uses a U-shaped network with dense connections. On the one hand, it uses the U-shaped network to retain the details of the background, and on the other hand, the DenseNet is used to multiplex the lower layer features to higher layer features to improve the accuracy of rain removal, by combining them, it alleviates the contradiction between the loss of background details caused by excessive rain removal and the incomplete rain-removing. Experimental results show that this kind of deep rain-removing network can detect and remove the rainwater in the image well.

Remoción de lluvia en imágenes por medio de una arquitectura de autoencoder

Objetivo: Usar técnicas computacionales para eliminar lluvia en imágenes. La motivación viene dada por el hecho de que, para muchos sistemas de visión por computadora, capturar correctamente la escena es clave, y si estos sistemas reciben imágenes degradadas por lluvia como entrada, su funcionamiento puede verse comprometido. Metodología: Se creó un conjunto de datos compuesto por 11000 imágenes sintéticas de lluvia. Estas fueron redimensionadas y normalizadas, para luego utilizar 9000 de ellas como conjunto de entrenamiento en la arquitectura autoencoder. El autoencoder genera una versión sin lluvia de la imagen, la cual es pasada a una etapa de corrección de iluminación para producir la imagen final sin lluvia. Resultados: Se encontró que el mejor desempeño lo tenía el autoencoder de 6 capas. Se evaluó con las 2000 imágenes restantes, lo que resultó en un error cuadrático medio de 0,61 y un índice de similitud estructural de 0,8493, lo que significa que el modelo fue capaz de recuperar una gran cantidad de información original de las imágenes degradadas por la lluvia. Conclusiones: Los resultados obtenidos son superiores a aquellos de la literatura que se basan en el dominio espacial / frecuencial. Se determinó, sin embargo, que es posible obtener mejores resultados si se considera el dominio de la frecuencia como parte de la arquitectura, debido a las propiedades de esta. Por lo tanto, se propone como trabajo futuro combinar enfoques basados en el aprendizaje de máquina con el procesamiento de imágenes basado en el dominio de la frecuencia.

Dilated Residual Spatial Attentive Generative Adversarial Network for Single Image De-raining

This paper propose a dilated residual spatial attentive generative adversarial network for single image de-raining. The improved spatial attentive mechanism is combined with the dilated convolution residual module to optimize the Condition Generative Adversarial Network(CGAN) structure, to solve two problems still exist in the task of removing rain from a single image: First, the rain streaks contained in the dataset we can use are limited, and in the case of real rainy days, the rain streak density is diverse, it is impossible to simulate them completely and accurately. Then, the existing rain removal models cannot remove rain streaks properly for images with different rain streak density which attend to over or under rain removal. In our methods, Firstly, the dilated convolution module is used to enhance the feature extraction of rain streaks, and then an attention map is generated by the spatial attention mechanism module to accurately locate the position of rain streaks, and the rain streaks are extracted as the foreground information by combining the two parts of the network to guide the subsequent rain removal operation; Then the rain streaks is removed through the Contextual auto-encoder, combining with PatchGAN discriminator. The experimental results on two synthetic dataset and real-world rain image show that the network model proposed in this paper has high generalization ability under different rainfall conditions. it improves the rain removal effect of the model on the most realistic public dataset, and improves the recovery ability of image details while effectively removing rain on other public datasets, which verifies the effectiveness of the model in improving the generalization ability and improving the image quality after rain removal. At the same time, the effect of rain removal is better than other methods on the real rain images, which proves that the method proposed in this paper has high practical application value.

Double Recurrent Dense Network for Single Image Deraining

Rain streaks can affect visual visibility, and hence disable many visual algorithms. So we present a double recurrent dense network for removing rain streaks from single image. Assume the rain image is the superposition of the clean image and the rain streaks, we directly learn the rain streaks from the rainy image. In contrast to other models, we introduce a double recurrent scheme to promote better information reuse of rain streaks and relative clean image. For rain streaks, the LSTM cascaded by DenseNet blocks is used as the basic model. The relative clean image predicted by subtracting the rain streaks from the rainy image is then input to the basic model in an iterative way. Benefiting from double recurrent schemes, our model makes full use of rain streaks and image detail information and thoroughly removes rain streaks. Furthermore, we adopt a mix of $L_{1}$ loss, $L_{2}$ loss and SSIM loss to guarantee good rain removing performance. We conduct a plenty of experiments on synthetic and real rainy images, even on similar denoise task, the results not only show our model significantly outperforms the state-of-art methods for removing rain streaks, but also exhibit our model has a high effectiveness for similar task, i.e. image denoising.

Cumulative Rain Density Sensing Network for Single Image Derain

This paper focuses on single image derain, which aims to restore clear image from single rain image. Through full consideration of different frequency information preservation and the complicated interactions between rain-streaks and background, a novel end-to-end cumulative rain-density sensing network (CRDNet) is proposed for adaptive rain-streaks removal. An effective W-Net with powerful learning ability is proposed as a key component to recover rain-invariant low-frequency signals. A cumulative rain-density classifier with a novel cost-sensitive label encoding strategy is proposed as an auxiliary network to improve discriminative power of extracted high-frequency rain-streaks through multi-task training. The proposed CRDNet has been compared with state-of-the-art methods on two public datasets. The quantitative and visual experimental results demonstrate that it can achieve excellent performance with great improvement. Related source code and models are available on github https://github.com/peylnog/CRDNet .

Investigating Collaborative Layer Projection for Robust Rain Scene Modeling

Rain is a common weather phenomenon that severely degrades outdoor image quality, affecting information extraction. The existing methods can be roughly classified as model-based and data-driven approaches. Model-based methods tend to utilize complex but limiting priors, resulting in optimization difficulty. Data-driven techniques emphasize the establishment of a network architecture and strongly depend on training pairs; hence, they become invalid in practical scenarios. To mitigate these problems, we develop a flexible collaborative layer projection framework for efficient and effective single-image rain streak removal task. We introduce a solution space to standardize fidelity without loss of generality to build a general deraining model. Then, a Collaborative Layer Projection Framework (CLPF) is presented for solving this model. Using the projection framework, various types of techniques (e.g., learnable architectures and optimization models) can be easily integrated to realize the desired performance. Extensive evaluations of synthetic and real rain images demonstrate that the proposed method outperforms state-of-the-art methods. In addition, we extend the method to other visual areas, such as video deraining and image dehazing. Our framework also performs relatively well on these issues.

Removing Arbitrary-Scale Rain Streaks via Fractal Band Learning With Self-Supervision

Data-driven rain streak removal methods, most of which rely on synthesized paired data, usually come across the generalization problem when being applied in real scenarios. In this paper, we propose a novel deep-learning based rain streak removal method injected with self-supervision to obtain the capacity of removing more varied-scale rain streaks in practical applications. To this end, in this work, efforts are made from two perspectives. First, considering that rain streak removal is highly correlated with texture characteristics, we create a fractal band learning (FBL) network based on frequency band recovery. It integrates commonly seen band feature operations as neural forms and effectively improves the capacity to capture discriminative features for deraining. Second, to further improve the generalization ability of FBL to remove rain streaks of varied scales, we incorporate scale-robust self-supervision to regularize the network training. The constraint forces the extracted features of an input rain image at different scales to be equivalent after rescaling operations. Therefore, our method can offer similar responses based on solely image content without the interference of scale change and is capable to remove varied-scale rain streaks. Extensive experiments in quantitative and qualitative evaluations demonstrate the superiority of our method for rain streak removal, especially for the real cases where very large rain streaks exist, and prove the effectiveness of each component.

Single-image de-raining using low-rank matrix approximation

Most existing image de-raining methods are based on the assumption that rain has certain shape or chromatic priors, which may be inaccurate for some rain cases and may lead to unsatisfactory de-rained results. In this paper, the proposed image de-raining method is built on the priors of clear images that many self-similar patches exist in natural images. And similar patches satisfy the low-rank property when grouped together. This property has led to many powerful image/video denoising schemes with impressive performance but has never been used for single-image de-raining. We formulate the problem of removing rain as a low-rank matrix approximation problem by adopting this self-similarity image prior with a designed patch matching strategy applicable for rain images. The resulting nuclear norm minimization problem can be efficiently solved by many recently developed methods. Experiments on both real rain images and synthetic rain images show that the proposed method is competitive against other state-of-the-art methods.

Single image rain removal via a deep decomposition–composition network

Rain effect in images typically is annoying for many multimedia and computer vision tasks. For removing rain effect from a single image, deep leaning techniques have been attracting considerable attentions. This paper designs a novel multi-task leaning architecture in an end-to-end manner to reduce the mapping range from input to output and boost the performance. Concretely, a decomposition net is built to split rain images into clean background and rain layers. Different from previous architectures, our model consists of, besides a component representing the desired clean image, an extra component for the rain layer. During the training phase, we further employ a composition structure to reproduce the input by the separated clean image and rain information for improving the quality of decomposition. Experimental results on both synthetic and real images are conducted to reveal the high-quality recovery by our design, and show its superiority over other state-of-the-art methods. Furthermore, our design is also applicable to other layer decomposition tasks like dust removal. More importantly, our method only requires about 50ms to process a testing image in VGA resolution on a GTX 1080 GPU with promising rain removal quality, making it attractive for practical use. The synthesized dataset and code are publicly available at https://sites.google.com/view/xjguo/rain.

Double fidelity terms unidirectional variation model for single-image rain removal

Videos and images captured in poor weather conditions not only have seriously degraded visual quality, but they can also restrict the performance of many computer vision algorithms. One such severe weather condition is rain, which causes complex local intensity fluctuations and fuzzy background images. We study the single-image rain removal to improve the visual quality of images captured in rain. We propose a rain streak removal method with a double fidelity terms unidirectional variational model based on wavelet transform, which is utilized to separate the rain streaks. The rain streaks component is only included in the approximate component and the corresponding high-frequency components. The double fidelity terms unidirectional variational model is then used for removing the rain streaks. The regularization term mainly introduces image-prior information by unidirectional variation. The numerical algorithm is realized by the augmented Lagrange algorithm. Experimental results on a variety of simulated and real rain images show that the proposed method can efficiently remove rain streaks. Experimental results demonstrate the good performance of the proposed model, removing rain streaks while reducing information loss from the background. The proposed method also requires much less computation.

A Tensor-Based Low-Rank Model for Single-Image Rain Streaks Removal

Images were taken in rainy weather always contain unexpected rain streaks, which severely affect the subsequent image processing procedures in outdoor vision systems. Removing rain streaks from a single image is a challenging task and recently has been investigated extensively. In this paper, we propose a novel tensor-based low-rank model for removing rain streaks from a single rainy image. Our method fully exploits the similar repeated patterns and directional smoothness of rain streaks. Different from the existing matrix-based methods, we stack the rain patches to construct a three-order tensor and characterize the similarity and repeated patterns by considering the low-rankness in tensor form. We further regularize the low-rankness by the efficient tensor nuclear norm (TNN) so that the intrinsic spatial structures of rain streaks can be preserved. Moreover, two unidirectional total variation terms are employed to depict the directional smoothness of rain streaks and the rain-free image. The sparsity of rain streaks is also enhanced by an l 1 norm. We develop an efficient alternating direction method of multipliers (ADMM) to tackle the proposed model. The experimental results on synthetic and real-world rain images show that our method outperforms the state-of-the-art methods quantitatively and visually.

Multi‐stage filtering for single rainy image enhancement

Rain image enhancement is important for outdoor computer vision applications. In this study, the authors propose a multi-stage filtering method for single rainy image enhancement. It is based on their new rainy image model, and consists of two main operations: rain streaks removal and rain fog removal. For rain streaks removal, based on one key observation that the low-pass version of a rainy image and that of a non-rainy image of the same scene are almost the same after appropriate low-pass filtering, they remove rain streaks from rainy images by decomposing an input rainy image (or a rainy component image) into the low-frequency (LF) part and the high-frequency (HF) part via an LF smooth filter, i.e. the traditional Gaussian filter with a simple subtraction operation in multiple different stages. After rain streaks removal, dark channel prior-based method was employed for rain fog removal. Experimental results show that the proposed algorithm generated comparable outputs with most of the state-of-the-art algorithms with low computation cost.

Single image deraining using deep convolutional networks

A deep learning-based single image deraining algorithm is proposed in this work. Instead of modeling a rain layer as a linear function between the rain image and its clear version as previous works do, we directly formulate the clear image as the result of a non-linear mapping of thrain image. We construct a coarse deraining convolutional network and a refinement convolutional network to learn this non-linear mapping function. The coarse deraining network is trained to detect the rain streaks with different directions, and restore a raw derained result. The refinement network aims at refining the result according to the raw derained image and the original rain image. By combining the two networks, we are able to well-restore the rain-free image. Experimental results demonstrate that the proposed deraining method can produce high-quality clear images from both synthetic and real-world rain images, outperforming the state-of-the-art methods qualitatively and quantitatively.

A directional global sparse model for single image rain removal

• Formulate a simple but efficient directional model for single image rain removal. • Present a novel rain rotation strategy for the final model. • The designed algorithm is fast and holds the complexity of O(n.logn)). • This method obtains the state-of-the-art results and the corresponding code is available. Rain removal from a single image is an important issue in the fields of outdoor vision. Rain, a kind of bad weather that is often seen, usually causes complex local intensity changes in images and has negative impact on vision performance. Many existing rain removal approaches have been proposed recently, such as some dictionary learning-based methods and layer decomposition-based methods. Although these methods can improve the visibility of rain images, they fail to consider the intrinsic directional and structural information of rain streaks, thus usually leave undesired rain streaks or change the background intensity of rain-free region significantly. In the paper, we propose a simple but efficient method to remove rain streaks from a single rainy image. The proposed method formulates a global sparse model that involves three sparse terms by considering the intrinsic directional and structural knowledge of rain streaks, as well as the property of image background information. We employ alternating direction method of multipliers (ADMM) to solve the proposed convex model which guarantees the global optimal solution. Results on a variety of synthetic and real rainy images demonstrate that the proposed method outperforms two recent state-of-the-art rain removal methods. Moreover, the proposed method needs no training and requires much less computation significantly.

Single image deraining via decorrelating the rain streaks and background scene in gradient domain

Single image based rain removal is very challenging due to the lack of temporal and context information, and the existing techniques are usually unpractical in real-time applications as they are time-consuming, and make images blurred in varying degrees. To tackle this issue, this paper proposes a novel framework, based on a new observation that the background has a reasonably low correlation with rain streaks in gradient domain. The framework mainly contains three steps: 1) a rain-free direction with respect to a rain image or a block therein is proposed, describing the fact that there exists a direction along which the image is least-affected in gradient domain; 2) by combing total variation, low-rank constraint and a de-correlation term, a novel decomposition model is proposed to explicitly extract the rain and rain-free gradient components along the direction perpendicular to the just calculated rain-free direction; 3) the rain-free image is reconstructed using Poisson equation, which effectively resists the sparse noise contained in gradients. The favorable performance of the proposed framework has been confirmed by many experimental results, and especially the computational complexity is low.

Rain streak removal based on non-negative matrix factorization

A rain streak in an image can degrade visual quality of that image to the human eye. Unfortunately, removing the rain streak from a single image represents a very challenging task. In this paper, a single image rain removal process based on non-negative matrix factorization is proposed. First, the rain image is broken down into a low-frequency and high-frequency part by a Gaussian filter. Therefore, the rain component, which lies mostly in the middle frequency range, can be discarded in high and low frequency domains. Next, non-negative matrix factorization (NMF) method is applied to deal with the rain streak in the low frequency domain. Finally, Canny edge detection and block copy strategy are performed separately to remove the rain component in the high frequency domain to improve image quality. In comparison with state-of-the-art approaches, the proposed method achieves competitive results without the need for an extra image database to train the dictionary.

Coronal rain in magnetic bipolar weak fields

We intend to investigate the underlying physics for the coronal rain phenomenon in a representative bipolar magnetic field, including the formation and the dynamics of coronal rain blobs. With the MPI-AMRVAC code, we performed three dimensional radiative magnetohydrodynamic (MHD) simulation with strong heating localized on footpoints of magnetic loops after a relaxation to quiet solar atmosphere. Progressive cooling and in-situ condensation starts at the loop top due to radiative thermal instability. The first large-scale condensation on the loop top suffers Rayleigh-Taylor instability and becomes fragmented into smaller blobs. The blobs fall vertically dragging magnetic loops until they reach low beta regions and start to fall along the loops from loop top to loop footpoints. A statistic study of the coronal rain blobs finds that small blobs with masses of less than 10^10 g dominate the population. When blobs fall to lower regions along the magnetic loops, they are stretched and develop a non-uniform velocity pattern with an anti-parallel shearing pattern seen to develop along the central axis of the blobs. Synthetic images of simulated coronal rain with Solar Dynamics Observatory Atmospheric Imaging Assembly well resemble real observations presenting dark falling clumps in hot channels and bright rain blobs in a cool channel. We also find density inhomogeneities during a coronal rain "shower", which reflects the observed multi-stranded nature of coronal rain.