Raindrop Removal Research Articles

AIMHNet: An Attribute-Insensitive Multiscale Hourglass Network for Rain Streak and Raindrop Removal

CNN-based methods have made great progress in single-image rain removal. Most recent methods improve performance by increasing the depth of the network. To fully extract local and global features while reducing inference time, we propose a top-to-down attribute-insensitive multiscale hourglass network for rain streak and raindrop removal. For the rain removal task, we expect that the constructed network can accurately identify the various attributes of the rain information characteristics of the small target. Considering the difference in the size, shape, direction and density of rain streak and raindrop, inspired by the performance of hourglass architecture to capture multiscale features in human pose estimation, we introduce an attribute-insensitive hourglass module to recognize the attributes of rain streak and raindrop in a unified framework. This feature extraction module could capture the characteristics of rain streak and raindrop with different attributes. This stacked hourglass blocks down-sample features and then up-samples them back to the original resolution based on discrete wavelet transform and inverse discrete wavelet transform. We perform extensive experiments on five synthetic and real-world de-raining datasets to validate the effectiveness of our proposed network on rain streak and raindrop removal. The qualitative and quantitative results show that our method is suitable for removing rain streak and raindrop in a unified framework. We present the results of generalization and ablation study for key components, we also report the accuracy of semantic segmentation after preprocessing with all rain removal methods. Our source code will be available on the GitHub: https://github.com/Ruini94/AIMHNet .

Single image deraindrop leveraging luminance priors and context aggregation

Many deep learning-based raindrop removal methods have been proposed recently and demonstrated good performance. However, most deep learning based raindrop removal methods struggle to restore distorted color and texture resulting from the refraction of raindrops emitted from different background points. To overcome this challenge, we propose a novel recurrent single-image deraindrop approach that utilizes luminance priors and contextual feature aggregation. Our method decomposes the deraindrop process into a raindrop detection network and a raindrop removal network, leveraging the luminance differences between the raindrop and non-raindrop regions for accurate detection. Contextual feature aggregation, achieved by dilated convolutions with different rates, helps recover raindrop attachment areas and maintain consistency in color and texture. We have extensively evaluated our method on synthetic and real raindrop datasets, demonstrating its effectiveness, generalization capability, and superior performance compared to state-of-the-art methods.

Frequency-oriented hierarchical fusion network for single image raindrop removal.

Single image raindrop removal aims at recovering high-resolution images from degraded ones. However, existing methods primarily employ pixel-level supervision between image pairs to learn spatial features, thus ignoring the more discriminative frequency information. This drawback results in the loss of high-frequency structures and the generation of diverse artifacts in the restored image. To ameliorate this deficiency, we propose a novel frequency-oriented Hierarchical Fusion Network (HFNet) for raindrop image restoration. Specifically, to compensate for spatial representation deficiencies, we design a dynamic adaptive frequency loss (DAFL), which allows the model to adaptively handle the high-frequency components that are difficult to recover. To handle spatially diverse raindrops, we propose a hierarchical fusion network to efficiently learn both contextual information and spatial features. Meanwhile, a calibrated attention mechanism is proposed to facilitate the transfer of valuable information. Comparative experiments with existing methods indicate the advantages of the proposed algorithm.

MARR-GAN: Memristive Attention Recurrent Residual Generative Adversarial Network for Raindrop Removal.

Since machine learning techniques for raindrop removal have not been capable of completely removing raindrops and have failed to take into account the constraints of edge devices with limited resources, a novel software-hardware co-designed method with a memristor for raindrop removal, named memristive attention recurrent residual generative adversarial network (MARR-GAN), is introduced in this research. A novel raindrop-removal network is specifically designed based on attention gate connections and recurrent residual convolutional blocks. By replacing the basic convolution unit with recurrent residual convolution unit, improved capturing of the changes in raindrop appearance over time is achieved, while preserving the position and shape information in the image. Additionally, an attention gate is utilized instead of the original skip connection to enhance the overall structural understanding and local detail preservation, facilitating a more comprehensive removal of raindrops across various areas of the image. Furthermore, a hardware implementation scheme for MARR-GAN is presented in this paper, where deep learning algorithms are seamlessly integrated with neuro inspired computing chips, utilizing memristor crossbar arrays for accelerated real-time image-data processing. Compelling evidence of the efficacy and superiority of MARR-GAN in raindrop removal and image restoration is provided by the results of the empirical study.

Dual-Pixel Raindrop Removal.

Removing raindrops in images has been addressed as a significant task for various computer vision applications. In this paper, we propose the first method using a dual-pixel(DP) sensor to better address raindrop removal. Our key observation is that raindrops attached to a glass window yield noticeable disparities in DP's left-half and right-half images, while almost no disparity exists for in-focus backgrounds. Therefore, the DP disparities can be utilized for robust raindrop detection. The DP disparities also bring the advantage that the occluded background regions by raindrops are slightly shifted between the left-half and the right-half images. Therefore, fusing the information from the left-half and the right-half images can lead to more accurate background texture recovery. Based on the above motivation, we propose a DP Raindrop Removal Network(DPRRN) consisting of DP raindrop detection and DP fused raindrop removal. To efficiently generate a large amount of training data, we also propose a novel pipeline to add synthetic raindrops to real-world background DP images. Experimental results on constructed synthetic and real-world datasets demonstrate that our DPRRN outperforms existing state-of-the-art methods, especially showing better robustness to real-world situations. Our source codes and datasets will be available at http://www.ok.sc.e.titech.ac.jp/res/SIR/dprrn/dprrn.html.

Attention-Based Real Image Restoration.

Deep convolutional neural networks perform better on images containing spatially invariant degradations, also known as synthetic degradations; however, their performance is limited on real-degraded photographs and requires multiple-stage network modeling. To advance the practicability of restoration algorithms, this article proposes a novel single-stage blind real image restoration network (R²Net) by employing a modular architecture. We use a residual on the residual structure to ease low-frequency information flow and apply feature attention to exploit the channel dependencies. Furthermore, the evaluation in terms of quantitative metrics and visual quality for four restoration tasks, i.e., denoising, super-resolution, raindrop removal, and JPEG compression on 11 real degraded datasets against more than 30 state-of-the-art algorithms, demonstrates the superiority of our R²Net. We also present the comparison on three synthetically generated degraded datasets for denoising to showcase our method's capability on synthetics denoising. The codes, trained models, and results are available on https://github.com/saeed-anwar/R2Net.

Restoring Vision in Adverse Weather Conditions With Patch-Based Denoising Diffusion Models.

Image restoration under adverse weather conditions has been of significant interest for various computer vision applications. Recent successful methods rely on the current progress in deep neural network architectural designs (e.g., with vision transformers). Motivated by the recent progress achieved with state-of-the-art conditional generative models, we present a novel patch-based image restoration algorithm based on denoising diffusion probabilistic models. Our patch-based diffusion modeling approach enables size-agnostic image restoration by using a guided denoising process with smoothed noise estimates across overlapping patches during inference. We empirically evaluate our model on benchmark datasets for image desnowing, combined deraining and dehazing, and raindrop removal. We demonstrate our approach to achieve state-of-the-art performances on both weather-specific and multi-weather image restoration, and experimentally show strong generalization to real-world test images.

Context and detail interaction network for stereo rain streak and raindrop removal

Recently stereo image deraining has attracted lots of attention due to its superiority of abundant information from cross views. Exploring interaction information across stereo views is the key to improving the performance of stereo image deraining. In this paper, we design a general coarse-to-fine deraining framework for stereo rain streak and raindrop removal, called CDINet, comprising a stereo rain removal subnet and a stereo detail recovery subnet to restore images progressively. Two types of interaction modules are devised to explore interaction information for rain removal and detail recovery, respectively. Specifically, a global context interaction module is proposed to learn long-range dependencies of stereo images and remove rain by utilizing stereo structural information. A local detail interaction module is designed to model local contextual correlation, which aims at restoring the detail information by using neighborhood information from cross views. Extensive experiments are conducted on the two datasets including a synthetic rain streak removal dataset (RainKITTI) and a real raindrop removal dataset (Stereo Waterdrop), which demonstrates that our method sets new state-of-the-art deraining performance in terms of both quantitative and qualitative metrics with faster speed.

Mutual channel prior guided dual-domain interaction network for single image raindrop removal

Removing raindrops adhering to lenses or glass is challenging since raindrops have more complex internal structures and optical effects than rain streaks. Existing raindrop removal methods typically employ single-channel masks for raindrop representation. However, these methods ignore the complementarity of priors between different channels, thereby losing the color and transparency information of raindrops. Moreover, simply applying the same processing across different pixels is insufficient to deal with spatially-varying degradation. To address these limitation, we propose a novel Mutual channel prior guided Dual-domain Interaction Network (MDINet) for single image raindrop removal. Specifically, to effectively guide the restoration of the raindrop region, we propose a mutual channel prior by modeling the multi-channel raindrop soft mask, in which the enriched color and transparency information of raindrops are represented and learned by our model. To tackle the spatially-varying degradation, we design the dual-domain interaction block to capture global low-frequency content and local high-frequency details. Simultaneously, a spatially adaptive modulation block is devised to handle complicated and diverse raindrops. Moreover, we propose a contrastive discrimination strategy to motivate the restored image to approach the clean image and stay away from the raindrop image in the representation space. Extensive experiments on benchmark datasets and real-world images indicate that the proposed method achieves qualitative and quantitative superiority against state-of-the-art methods.

UnfairGAN: An enhanced generative adversarial network for raindrop removal from a single image

Image deraining is a new challenging problem in real-world applications, such as autonomous vehicles. In a bad weather condition of heavy rainfall, raindrops, mainly hitting glasses or windshields, can significantly reduce observation ability. Moreover, raindrops spreading over the glass can yield refraction’s physical effect, which seriously impedes the sightline or undermine machine learning systems. In this paper, we propose an enhanced generative adversarial network to deal with the challenging problems of raindrops. UnfairGAN is an enhanced generative adversarial network that can utilize prior high-level information, such as edges and rain estimation, to boost deraining performance. UnfairGAN can effectively conserve the essential details caused by heavy raindrops and remove artifacts caused by the instability of training the discriminator. This method is based on three main primary advantages. First, UnfairGAN consists of an advanced loss function of the discriminator that can improve the instabilities of traditional GAN. Second, UnfairGAN comprises a new advanced activation function that is able to increase the learning effectiveness of image classification and reconstruction. Finally, UnfairGAN is basically built on a new end-to-end cascade network, namely DRD-UNet, used to probe hierarchical features for image restoration effectively. When evaluating competing methods on the well-known Raindrop dataset, we achieve a peak signal-to-noise ratio value of 31.56 while retaining the essential details in the image. Besides, we introduce a new large image dataset (DeepRaindrops) for training deep learning networks of removing raindrops. In this dataset, our proposed method is superior to other state-of-the-art approaches of deraining raindrops regarding quantitative metrics and visual quality. Our source codes for UnfairGAN are available at https://github.com/ZeroZero19/UnfairGAN.git.

Adherent mist and raindrop removal from a single image using attentive convolutional network

Temperature-difference-induced mist and environmentally-induced raindrops adhering to glass products such as windshield and camera lens, can often block vision and severely degrade the image seen through the lens. Despite posing a challenge to various vision systems, including autonomous driving and security surveillance, this problem has not received sufficient attention from researchers. In this work, we discuss the image degradation caused by adherent mist and raindrops. An attentive convolutional network is designed to visually remove the adherent mist and raindrops from a single image. Considering the variations in coexistence and reginal characteristics of adherent mist and raindrops, we propose interpolation-based pyramid attention blocks to perceive spatial information at different scales without rigid priors. Experiments show that the proposed method can improve the visibility of severely degraded images in real-world scenarios, both qualitatively and quantitatively. Further application experiments demonstrate that this practical problem is critical to high-level vision situations.

Laplacian encoder-decoder network for raindrop removal

• Rain droplets removed by generating the Laplacian pyramid levels of rain-free images. • The Laplacian pyramid decomposition improves with respect to a simple encoder-decoder. • The training benefits from comparing each generated level with the ground truth. • Reconstructing levels before comparison, balances their impact and provides context. • Better high frequencies will improve structural similarity and signal-to-noise ratio. Digital raindrop removal is a branch of image restoration that aims at identifying adherent droplets on a glass surface and replacing them with plausible content. When successfully performed, raindrop removal was proven in the past to positively affect both the perceived appearance of the scene, and the performance of computer vision tasks such as semantic segmentation and object detection. In this paper, we design and implement a new encoder-decoder neural network for supervised raindrop removal. Our network, given a rainy input image, produces as output the Laplacian pyramid of a rain-free version of the input, making it possible to handle the variety of appearances of rain droplets by processing different frequency bands independently. To this end, we define and experimentally prove the effectiveness of a custom loss function that combines the errors of the different Laplacian frequency bands. We test our model for raindrop removal on a standard dataset, using multiple objective metrics to provide a detailed analysis of its performance. We confirm the superiority of our proposal in a comparison with other methods from the state of the art.

Semi-supervised learning for raindrop removal on a single image

This article propose s a network that is mainly used to deal with a single image polluted by raindrops in rainy weather to get a clean image without raindrops. In the existing solutions, most of the methods rely on paired images, that is, the rain image and the real image without rain in the same scene. However, in many cases, the paired images are difficult to obtain, which makes it impossible to apply the raindrop removal network in many scenarios. Therefore this article proposes a semi-supervised rain-removing network apply to unpaired images. The model contains two parts: a supervised network and an unsupervised network. After the model is trained, the unsupervised network does not require paired images and it can get a clean image without raindrops. In particular, our network can perform training on paired and unpaired samples. The experimental results show that the best results are achieved not only on the supervised rain-removing network, but also on the unsupervised rain-removing network.

Feature-Aligned Video Raindrop Removal With Temporal Constraints.

Existing adherent raindrop removal methods focus on the detection of the raindrop locations, and then use inpainting techniques or generative networks to recover the background behind raindrops. Yet, as adherent raindrops are diverse in sizes and appearances, the detection is challenging for both single image and video. Moreover, unlike rain streaks, adherent raindrops tend to cover the same area in several frames. Addressing these problems, our method employs a two-stage video-based raindrop removal method. The first stage is the single image module, which generates initial clean results. The second stage is the multiple frame module, which further refines the initial results using temporal constraints, namely, by utilizing multiple input frames in our process and applying temporal consistency between adjacent output frames. Our single image module employs a raindrop removal network to generate initial raindrop removal results, and create a mask representing the differences between the input and initial output. Once the masks and initial results for consecutive frames are obtained, our multiple-frame module aligns the frames in both the image and feature levels and then obtains the clean background. Our method initially employs optical flow to align the frames, and then utilizes deformable convolution layers further to achieve feature-level frame alignment. To remove small raindrops and recover correct backgrounds, a target frame is predicted from adjacent frames. A series of unsupervised losses are proposed so that our second stage, which is the video raindrop removal module, can self-learn from video data without ground truths. Experimental results on real videos demonstrate the state-of-art performance of our method both quantitatively and qualitatively.

Improved Sea-Ice Identification Using Semantic Segmentation With Raindrop Removal

Sea-ice identification is an essential process for safety critical navigation support of surface vessels in polar waters. Semantic segmentation has drawn much attention as an enabling technique for fast detection of objects in a scene including sea-ice conditions. Identifying sea-ice is a challenging problem, especially in the presence of raindrops. The raindrop alters the boundaries of the objects in the scene, and thus, degrades the identification performance. In this work, a raindrop removing framework is developed to enhance the classification performance. Three deep-learning semantic segmentation networks are trained to classify the scene of sea-ice images into ice, water, ship, and sky. The deep-learning networks are VGG-16, fully convolutional network, and pyramid scene parsing network. Transfer learning along with data augmentation operations have been implemented to improve the training process. Results illustrate that data augmentation operations enhance the performance of the three models. Moreover, the raindrop removing framework improves the models’ performance, e.g. the average intersection over union of the VGG-16 model is improved from 85.91% to 91.70%.

Single Image Raindrop Removal Using a Non-Local Operator and Feature Maps in the Frequency Domain

Taking a photo on a rainy day may result in a photo with raindrops. Images containing raindrops have a significant impact on the visual impression and accuracy when applied to image recognition systems. Thus, an automatic high-quality raindrop removal method is desired for outdoor image processing systems as well as for acquiring good-looking images. Several existing methods have been proposed to tackle this problem, but they often fail to keep global consistency and generate unnatural patterns. In this paper, we tackle this problem by introducing a non-local operator. The non-local operator combines features in distant locations with matrix multiplication and enables consistency in distant locations. In addition, high-frequency components such as edges are more affected in images with raindrops. Inspired by the nature that high-frequency components can be separated from other components in the frequency domain, we also propose to process feature maps in the frequency domain, which are obtained by the fast Fourier transform operation and processed by several convolution layers. Experimental results show that our method effectively removes raindrops and achieves state-of-the-art performance.

Mutual Channel Prior Guided Dual-Domain Transformer for Single Image Raindrop Removal

Mutual Channel Prior Guided Dual-Domain Transformer for Single Image Raindrop Removal

Raindrop Removal On A Single Image Using The Generative Adversarial Network

The presence of raindrops attached to the window glass or vehicle glass reduces visibility of the actual scene. The area covered by raindrops forms a round image and reflects light from the actual scene, this image is called Raindrop. In some cases, the camera's focus is blocked by Raindrop will result in a blurry image. The problem faced is that there is no actual landscape image, so to overcome this this research tries to adapt the research from Rui Qian and MaybeShewill-CV which uses the Generative Adversarial Network architecture, by adding the Raindrop and Groundtruh datasets from observations at Darussalam Gontor University. The purpose of this study is to remove raindrops from a single image. This is important to research because it provides updates and optimizes the results of previous research. This study shows the accuracy of PSNR 21.37 and SSIM 0.7561. The model managed to remove Raindrops from the image, but still couldn't match the Groundtruth image. Inability to handle raindrops due to lack of time to run a large number of epochs to produce PSNR values ​​above 40 db and SSIM above 0.9. PSNR and SSIM values ​​can continue to increase along with the addition of the dataset as well as the number of epoch training models carried out.

Enhanced context encoding for single image raindrop removal

Enhanced context encoding for single image raindrop removal

Raindrop removal from a single image using a two-step generative adversarial network

Raindrop removal from a single image using a two-step generative adversarial network