Multi-scale Generative Adversarial Networks Research Articles

Lightweight multi-scale generative adversarial network with attention for image denoising

Lightweight multi-scale generative adversarial network with attention for image denoising

GR-GAN: A unified adversarial framework for single image glare removal and denoising

In this paper, we introduce the single image glare removal (SIGR) task. SIGR aims to eliminate glare or light caused by external environment in lighting scene. To efficiently improve natural image quality and usability, many research tasks, such as image deraining and shadow removal, have been investigated a lot. However, SIGR is still underexplored. Therefore, we propose to construct a dataset and explore deep learning-based models for SIGR. Our contributions can be summarized as follows: (1) We establish a new benchmark dataset for SIGR, termed De-Glare, aiming to propel research of SIGR. This dataset comprises pairs of {glare, glare-free} images sourced from both real-world and synthetic data, utilized for training and evaluating models. (2) We conduct a comprehensive benchmarking of extensive state-of-the-art (SoTA) methods on the constructed De-Glare dataset and provide insightful analyses based on the results. (3) An innovative approach for single image glare removal employing a multi-scale generative adversarial network (GR-GAN) model is proposed. Glare images typically exhibit irregular glare shapes and cluttered backgrounds. To address these irregular glare patterns, we introduce a deformable convolution-based glare attention detector (GAD) designed to generate an attention map which specifics glare spots or rays in the input image. In pursuit of enhancing the perceptual quality of output image, GR-GAN adaptively filters out irrelevant noises and enhances salient features through a generator with cascaded pyramid neck (CPN) network. This work can provide useful insights for developing better SIGR models. Without specific tuning, our method achieves the SoTA results on multiple computer vision tasks, including the image deraining and image shadow removal.

SIGAN: A Multi-Scale Generative Adversarial Network for Underwater Sonar Image Super-Resolution

Super-resolution (SR) is a technique that restores image details based on existing information, enhancing the resolution of images to prevent quality degradation. Despite significant achievements in deep-learning-based SR models, their application in underwater sonar scenarios is limited due to the lack of underwater sonar datasets and the difficulty in recovering texture details. To address these challenges, we propose a multi-scale generative adversarial network (SIGAN) for super-resolution reconstruction of underwater sonar images. The generator is built on a residual dense network (RDN), which extracts rich local features through densely connected convolutional layers. Additionally, a Convolutional Block Attention Module (CBAM) is incorporated to capture detailed texture information by focusing on different scales and channels. The discriminator employs a multi-scale discriminative structure, enhancing the detail perception of both generated and high-resolution (HR) images. Considering the increased noise in super-resolved sonar images, our loss function emphasizes the PSNR metric and incorporates the L2 loss function to improve the quality of the output images. Meanwhile, we constructed a dataset for side-scan sonar experiments (DNASI-I). We compared our method with the current state-of-the-art super-resolution image reconstruction methods on the public dataset KLSG-II and our self-built dataset DNASI-I. The experimental results show that at a scale factor of 4, the average PSNR value of our method was 3.5 higher than that of other methods, and the accuracy of target detection using the super-resolution reconstructed images can be improved to 91.4%. Through subjective qualitative comparison and objective quantitative analysis, we demonstrated the effectiveness and superiority of the proposed SIGAN in the super-resolution reconstruction of side-scan sonar images.

End-to-end latent fingerprint enhancement using multi-scale Generative Adversarial Network

Latent fingerprint enhancement is paramount as it dramatically influences matching accuracy. This process is often challenging due to varying structured noise and background patterns. The prints may be of arbitrary sizes and scales with a high degree of occlusion. There is a need for creating an end-to-end system that handles different conditions reliably to streamline this often lengthy and tricky process. In this work, we propose a Generative Adversarial Network (GAN) based architecture that effectively captures multi-scale context using Atrous Spatial Pyramid Pooling (ASPP). We have trained the network on a synthetically generated dataset, carefully designed to represent real-world latent prints. By avoiding the reconstruction of spurious ridges and only enhancing valid ridges, we avoid the generation of false minutiae, leading to better matching performance. We obtained state-of-the-art results in Sensor to Latent matching using the IIITD MOLF and Latent to Latent Matching using IIITD Latent datasets.

Multi-Scale Generative Adversarial Network With Multi-Head External Attention for Image Inpainting

Multi-Scale Generative Adversarial Network With Multi-Head External Attention for Image Inpainting

Advancing generalizations of multi-scale GAN via adversarial perturbation augmentations

The multi-scale generative adversarial networks (MS-GANs) typically introduce a series of GANs to process hierarchical image patches from a single image, which suites image manipulation scenarios where it takes one image as input to produce manipulated results. However, during MS-GAN training, leveraging only a single image in hierarchical patches brings data insufficiency for model generalization, which limits MS-GAN to produce fine-grained texture details. To address this problem, we theoretically formulate MS-GAN generalization from the PAC-Bayes bound perspective in this work. Under this formulation, we obtain a non-vacuous upper bound of the generalization error, and propose an adversarial perturbation augmentation method named AP-Aug. Given an input image, our AP-Aug produces adversarial perturbations to augment the original image. The augmented training images benefit MS-GAN to approach the generalization upper bound. To this end, we advance the generalizations of MS-GAN to improve image generation performance. In the experiments, our AP-Aug benefits MS-GAN to achieve high quality image generation under several image manipulation scenarios including paint2image, image style transfer, and image super-resolution. The improvement shows that our AP-Aug advances the MS-GAN generalizations for high quality image generations, which perform favorably against state-of-the-art approaches.

Underwater Image Translation via Multi-Scale Generative Adversarial Network

The role that underwater image translation plays assists in generating rare images for marine applications. However, such translation tasks are still challenging due to data lacking, insufficient feature extraction ability, and the loss of content details. To address these issues, we propose a novel multi-scale image translation model based on style-independent discriminators and attention modules (SID-AM-MSITM), which learns the mapping relationship between two unpaired images for translation. We introduce Convolution Block Attention Modules (CBAM) to the generators and discriminators of SID-AM-MSITM to improve its feature extraction ability. Moreover, we construct style-independent discriminators that enable the discriminant results of SID-AM-MSITM to be not affected by the style of images and retain content details. Through ablation experiments and comparative experiments, we demonstrate that attention modules and style-independent discriminators are introduced reasonably and SID-AM-MSITM performs better than multiple baseline methods.

A regional solar forecasting approach using generative adversarial networks with solar irradiance maps

The intermittent and stochastic nature of solar resource hinders the integration of solar energy into modern power system. Solar forecasting has become an important tool for better photovoltaic (PV) power integration, effective market design, and reliable grid operation. Nevertheless, most existing solar forecasting methods are dedicated to improving forecasting accuracy at site-level (e.g. for individual PV power plants) regardless of the impacts caused by the accumulated penetration of distributed PV systems. To tackle with this issue, this article proposes a novel generative approach for regional solar forecasting considering an entire geographical region of a flexible spatial scale. Specifically, we create solar irradiance maps (SIMs) for solar forecasting for the first time by using spatial Kriging interpolation with satellite-derived solar irradiance data. The sequential SIMs provide a comprehensive view of how solar intensity varies over time and are further used as the inputs for a multi-scale generative adversarial network (GAN) to predict the next-step SIMs. The generated SIM frames can be further transformed into PV power output through a irradiance-to-power model. A case study is conducted in a 24 × 24 km area of Brisbane to validate the proposed method by predicting of both solar irradiance and the output of behind-the-meter (BTM) PV systems at unobserved locations. The approach demonstrates comparable accuracy in terms of solar irradiance forecasting and better predictions in PV power generation compared to the conventional forecasting models with a highest average forecasting skill of 10.93±2.35% for all BTM PV systems. Thus, it can be potentially used to assist solar energy assessment and power system control in a highly-penetrated region.

Blank Strip Filling for Logging Electrical Imaging Based on Multiscale Generative Adversarial Network

The Fullbore Formation Micro Imager (FMI) represents a proficient method for examining subterranean oil and gas deposits. Despite its effectiveness, due to the inherent configuration of the borehole and the logging apparatus, the micro-resistivity imaging tool cannot achieve complete coverage. This limitation manifests as blank regions on the resulting micro-resistivity logging images, thus posing a challenge to obtaining a comprehensive analysis. In order to ensure the accuracy of subsequent interpretation, it is necessary to fill these blank strips. Traditional inpainting methods can only capture surface features of an image, and can only repair simple structures effectively. However, they often fail to produce satisfactory results when it comes to filling in complex images, such as carbonate formations. In order to address the aforementioned issues, we propose a multiscale generative adversarial network-based image inpainting method using U-Net. Firstly, in order to better fill the local texture details of complex well logging images, two discriminators (global and local) are introduced to ensure the global and local consistency of the image; the local discriminator can better focus on the texture features of the image to provide better texture details. Secondly, in response to the problem of feature loss caused by max pooling in U-Net during down-sampling, the convolution, with a stride of two, is used to reduce dimensionality while also enhancing the descriptive ability of the network. Dilated convolution is also used to replace ordinary convolution, and multiscale contextual information is captured by setting different dilation rates. Finally, we introduce residual blocks on the U-Net network in order to address the degradation problem caused by the increase in network depth, thus improving the quality of the filled logging images. The experiment demonstrates that, in contrast to the majority of existing filling algorithms, the proposed method attains superior outcomes when dealing with the images of intricate lithology.

Reconstruction of three-dimensional porous media using multi-scale generative adversarial networks

Numerical simulation methods have been widely used in the reconstruction of porous media, but their applicability usually suffers from the huge hardware burden and computational time. Recently, with the rapid development of deep learning, its powerful ability in feature extraction has been applied to reconstruct porous media. As an important branch of deep learning, generative adversarial network (GAN) has been considered as an effective method to extract features from training images (TIs). Based on GAN, this paper proposes a method for the reconstruction of porous media using multi-scale GAN (multi-GAN), which learns the features of the TI at different scales separately by using a convolutional pyramid structure. The proposed method uses only a single three-dimensional (3D) TI of porous media for reconstruction and the reconstructed images contain structural information of the TI at different scales. Compared with some typical methods, this method has certain advantages in terms of reconstruction quality and efficiency.

Noise suppression of distributed fiber-optical acoustic sensing seismic data by attention-guided multiscale generative adversarial network

Distributed fiber-optical acoustic sensing (DAS) is an emerging technology, which uses optical fiber as a sensor for signal acquisition and recently has been applied to seismic exploration. Seismic records collected with DAS equipment often are contaminated with multicomponent noise induced by complex causes which consequently affect the subsequent imaging, inversion, and even the interpretation work. Therefore, research on effective noise suppression algorithm in DAS seismic data has become a hot topic in geophysical prospecting. In this study, we develop an attention-guided multiscale generative adversarial network (AMGAN) based on the traditional GAN architecture and discuss its feasibility in multicomponent DAS noise suppression. In AMGAN, multiscale ideas are introduced to extract features of raw DAS data in different scales. In addition, attention mechanism is imported and trained at each hierarchy to help extract and fuse the features from different scales. Overall, AMGAN, attributed to the inversely fused multiscale features, can reveal more detailed reflected signal information in DAS data denoising tasks. The synthetic and field DAS data experiments indicate that AMGAN can effectively remove the multicomponent seismic noise in DAS data and recover the weak seismic events with advantages in clarity and continuity.

P‐4.11: Image Deraining using Multi‐scale Generative Adversarial Network

The images obtained for automatic driving and security monitoring are easily affected by rain, which will bring serious difficulties to subsequent image‐processing tasks. To tackle the problem, deraining algorithms are widely adopted. However, most of the traditional methods fail to restore rain images when raindrops come in different sizes and densities. In this paper, a single‐image deraining algorithm based on multi‐scale generative adversarial network (MS‐GAN) is proposed. This algorithm uses three‐level Laplace pyramid to decompose the input rain image into different scales, inputs them into three modified residual networks and uses adversarial network to improve the effect of learning. Experimental results show that the proposed algorithm can remove raindrops more efficiently and reduce loss of detail, even when raindrops is of different sizes.

Adaptive Visual Field Multi-scale Generative Adversarial Networks Image Inpainting Base on Coordinate-Attention

Adaptive Visual Field Multi-scale Generative Adversarial Networks Image Inpainting Base on Coordinate-Attention

Infrared and Visible Image Homography Estimation Using Multiscale Generative Adversarial Network

In computer vision, the homography estimation of infrared and visible multi-source images based on deep learning is a current research hotspot. Existing homography estimation methods ignore the feature differences of multi-source images, which leads to poor homography performance in infrared and visible image scenes. To address this issue, we designed an infrared and visible image homography estimation method using a Multi-scale Generative Adversarial Network, called HomoMGAN. First, we designed two shallow feature extraction networks to extract fine features of infrared and visible images, respectively, which extract important features in source images from two dimensions: color channel and imaging space. Second, we proposed an unsupervised generative adversarial network to predict the homography matrix directly. In our adversarial network, the generator captures meaningful features for homography estimation at different scales by using an encoder–decoder structure and further predicts the homography matrix. The discriminator recognizes the feature difference between the warped and target image. Through the adversarial game between the generator and the discriminator, the fine features of the warped image in the homography estimation process are closer to the fine features of the target image. Finally, we conduct extensive experiments in the synthetic benchmark dataset to verify the effectiveness of HomoMGAN and its components. We conduct extensive experiments and the results show that HomoMGAN outperforms existing state-of-the-art methods in the synthetic benchmark datasets both qualitatively and quantitatively.

Reconstruction of Partially Broken Vascular Structures in X-Ray Images via Vesselness-Loss-Based Multi-Scale Generative Adversarial Networks

Reconstruction of Partially Broken Vascular Structures in X-Ray Images via Vesselness-Loss-Based Multi-Scale Generative Adversarial Networks

An inverse halftoning method for various types of halftone images based on multi-scale generative adversarial network

The inverse halftoning method based on deep neural network has been widely studied in last few years. However, the existing methods only apply single-size convolution kernel to extract image features and train the proposed model with only one-scale loss function, which leads to blurring details and residual halftone noise patterns in the restored continuous-tone image. To alleviate these problems, this paper proposes an inverse halftone method for various types of halftone images based on multi-scale generative adversarial network. In the generative model, an elaborated multi-scale feature extraction model is firstly designed to extract halftone image features in parallel, so as to obtain more comprehensive image information from the input halftone image and promote the effective fusion of the extracted features. Moreover, a detail enhancement subnetwork is employed to fine-tune the image details. In adversarial model, a multi-scale discriminator is used to further enhance the image details by learning the distribution characteristics of image information. The experiments on different types of halftone images show that the proposed method significantly outperforms the existing state-of-art methods.

Unsupervised Learning of Depth Estimation and Camera Pose With Multi-Scale GANs

Unsupervised learning methods have achieved remarkable performance in monocular depth estimation and camera pose, which mostly solve the multi-task learning problem by using their inner geometry consistency as the self-supervision signal. While most existing approaches mostly adopt the generative model to obtain the depth map prediction, so in the resolution of depth map there is room for improvement. To this end, we present our unsupervised learning architecture based on adversarial learning model, which is used for unsupervised learning of high-resolution single view depth and camera pose. Specifically, we present a multi-scale deep convolutional Generative Adversarial Network (GAN) based learning system, which consists of three networks (pose estimation network PCNN, Generator-D and Discriminator-D for depth map prediction). Furthermore, in order to generate high-resolution depth map, we propose a multi-scale GAN model (MSGAN) to decompose the hard high-quality image generation problem into more manageable sub-problems through a coarse-to-fine process. Then, we modify the overall generation architecture of GAN model by changing the down-sampling and up-sampling components to improve the quality and accuracy of the depth map prediction. Finally, in order to improve the rate of convergence, we use the Least Square Error to increase the penalty for outliers. Detailed quantitative and qualitative evaluations of the proposed framework on the KITTI dataset show that the proposed method provides better results for both pose estimation and depth recovery.

Improving Image Super-Resolution Based on Multiscale Generative Adversarial Networks.

Convolutional neural networks have greatly improved the performance of image super-resolution. However, perceptual networks have problems such as blurred line structures and a lack of high-frequency information when reconstructing image textures. To mitigate these issues, a generative adversarial network based on multiscale asynchronous learning is proposed in this paper, whereby a pyramid structure is employed in the network model to integrate high-frequency information at different scales. Our scheme employs a U-net as a discriminator to focus on the consistency of adjacent pixels in the input image and uses the LPIPS loss for perceptual extreme super-resolution with stronger supervision. Experiments on benchmark datasets and independent datasets Set5, Set14, BSD100, and SunHays80 show that our approach is effective in restoring detailed texture information from low-resolution images.

A Fast Multi-Scale Generative Adversarial Network for Image Compressed Sensing.

Recently, deep neural network-based image compressed sensing methods have achieved impressive success in reconstruction quality. However, these methods (1) have limitations in sampling pattern and (2) usually have the disadvantage of high computational complexity. To this end, a fast multi-scale generative adversarial network (FMSGAN) is implemented in this paper. Specifically, (1) an effective multi-scale sampling structure is proposed. It contains four different kernels with varying sizes so that decompose, and sample images effectively, which is capable of capturing different levels of spatial features at multiple scales. (2) An efficient lightweight multi-scale residual structure for deep image reconstruction is proposed to balance receptive field size and computational complexity. The key idea is to apply smaller convolution kernel sizes in the multi-scale residual structure to reduce the number of operations while maintaining the receptive field. Meanwhile, the channel attention structure is employed for enriching useful information. Moreover, perceptual loss is combined with MSE loss and adversarial loss as the optimization function to recover a finer image. Numerous experiments show that our FMSGAN achieves state-of-the-art image reconstruction quality with low computational complexity.

A low-cost pathological image digitalization method based on 5 times magnification scanning.

Digital pathology has aroused widespread interest in modern pathology. The key to digitalization is to scan the whole slide image (WSI) at high magnification. The file size of each WSI at 40 times magnification (40×) may range from 1 gigabyte (GB) to 5 GB depending on the size of the specimen, which leads to huge storage capacity, very slow scanning and network exchange, seriously increasing time and storage costs for digital pathology. We design a strategy to scan slides with low resolution (LR) (5×), and a superresolution (SR) method is proposed to restore the image details during diagnosis. The method is based on a multiscale generative adversarial network, which can sequentially generate three high-resolution (HR) images: 10×, 20×, and 40×. A dataset consisting of 100,000 pathological images from 10 types of human body systems is used for training and testing. The differences between the generated images and the real images have been extensively evaluated using quantitative evaluation, visual inspection, medical scoring, and diagnosis. The file size of each 5× WSI is approximately 15 Megabytes. The peak-signal-to-noise ratios (PSNRs) of 10× to 40× generated images are 24.167±3.734 dB, 22.272±4.272 dB, and 20.436±3.845 dB, and the structural similarity (SSIM) index values are 0.845±0.089, 0.680±0.150, and 0.559±0.179, which are better than those of other SR networks and conventional digital zoom methods. Visual inspections show that the generated images have details similar to the real images. Visual scoring average with 0.95 confidence interval from three pathologists are 3.630±1.024, 3.700±1.126, and 3.740±1.095, respectively, and the P value of analysis of variance is 0.367, indicating the pathologists confirm that generated images include sufficient information for diagnosis. The average value of the Kappa test of the diagnoses of paired generated and real images is 0.990, meaning the diagnosis of generated images is highly consistent with that of the real images. The proposed method can generate high-quality 10×, 20×, 40× images from 5× images, which can effectively reduce the time and storage costs of digitalization up to 1/64 of the previous costs, which shows the potential for clinical applications and is expected to be an alternative digitalization method after large-scale evaluation.