Multi-scale Generative Research Articles

End-to-end latent fingerprint enhancement using multi-scale Generative Adversarial Network

Latent fingerprint enhancement is paramount as it dramatically influences matching accuracy. This process is often challenging due to varying structured noise and background patterns. The prints may be of arbitrary sizes and scales with a high degree of occlusion. There is a need for creating an end-to-end system that handles different conditions reliably to streamline this often lengthy and tricky process. In this work, we propose a Generative Adversarial Network (GAN) based architecture that effectively captures multi-scale context using Atrous Spatial Pyramid Pooling (ASPP). We have trained the network on a synthetically generated dataset, carefully designed to represent real-world latent prints. By avoiding the reconstruction of spurious ridges and only enhancing valid ridges, we avoid the generation of false minutiae, leading to better matching performance. We obtained state-of-the-art results in Sensor to Latent matching using the IIITD MOLF and Latent to Latent Matching using IIITD Latent datasets.

Chemical fault diagnosis network based on single domain generalization

Recent developments in fault diagnosis have leveraged domain generalization to address the issue of domain shift. Most existing methods focus on learning domain-invariant representations from multiple source domains. However, collecting valuable fault samples from varying operational conditions is challenging, and it is common for available data to originate from a single operational condition. Thus, this paper introduces a Multi-scale generative and adversarial Metric networks (MGAMN) for Chemical Process Fault Diagnosis. To enhance model generalization, a domain generation module was developed to create augmented domains with significant distributional differences from the source domain. The diagnostic task module then extracts domain-invariant features from both the source and augmented domains. A multi-scale generation strategy is established, utilizing multi-scale deep separable convolutions (Dsc) to ensure that the generated samples contain rich state information. Additionally, an adversarial training and metric learning strategy is designed to learn generalized features capable of resisting unknown domain shifts. Extensive diagnostic experiments on the non-isothermal continuous stirred tank reactor (CSTR) and the Tennessee Eastman Process (TEP) chemical datasets validate the effectiveness of the proposed method. Moreover, ablation studies confirm the effectiveness of the proposed modular strategy, demonstrating significant potential for practical applications.

Multi-Scale Generative Adversarial Network With Multi-Head External Attention for Image Inpainting

Multi-Scale Generative Adversarial Network With Multi-Head External Attention for Image Inpainting

Underwater Image Translation via Multi-Scale Generative Adversarial Network

The role that underwater image translation plays assists in generating rare images for marine applications. However, such translation tasks are still challenging due to data lacking, insufficient feature extraction ability, and the loss of content details. To address these issues, we propose a novel multi-scale image translation model based on style-independent discriminators and attention modules (SID-AM-MSITM), which learns the mapping relationship between two unpaired images for translation. We introduce Convolution Block Attention Modules (CBAM) to the generators and discriminators of SID-AM-MSITM to improve its feature extraction ability. Moreover, we construct style-independent discriminators that enable the discriminant results of SID-AM-MSITM to be not affected by the style of images and retain content details. Through ablation experiments and comparative experiments, we demonstrate that attention modules and style-independent discriminators are introduced reasonably and SID-AM-MSITM performs better than multiple baseline methods.

Blank Strip Filling for Logging Electrical Imaging Based on Multiscale Generative Adversarial Network

The Fullbore Formation Micro Imager (FMI) represents a proficient method for examining subterranean oil and gas deposits. Despite its effectiveness, due to the inherent configuration of the borehole and the logging apparatus, the micro-resistivity imaging tool cannot achieve complete coverage. This limitation manifests as blank regions on the resulting micro-resistivity logging images, thus posing a challenge to obtaining a comprehensive analysis. In order to ensure the accuracy of subsequent interpretation, it is necessary to fill these blank strips. Traditional inpainting methods can only capture surface features of an image, and can only repair simple structures effectively. However, they often fail to produce satisfactory results when it comes to filling in complex images, such as carbonate formations. In order to address the aforementioned issues, we propose a multiscale generative adversarial network-based image inpainting method using U-Net. Firstly, in order to better fill the local texture details of complex well logging images, two discriminators (global and local) are introduced to ensure the global and local consistency of the image; the local discriminator can better focus on the texture features of the image to provide better texture details. Secondly, in response to the problem of feature loss caused by max pooling in U-Net during down-sampling, the convolution, with a stride of two, is used to reduce dimensionality while also enhancing the descriptive ability of the network. Dilated convolution is also used to replace ordinary convolution, and multiscale contextual information is captured by setting different dilation rates. Finally, we introduce residual blocks on the U-Net network in order to address the degradation problem caused by the increase in network depth, thus improving the quality of the filled logging images. The experiment demonstrates that, in contrast to the majority of existing filling algorithms, the proposed method attains superior outcomes when dealing with the images of intricate lithology.

Adaptive Visual Field Multi-scale Generative Adversarial Networks Image Inpainting Base on Coordinate-Attention

Adaptive Visual Field Multi-scale Generative Adversarial Networks Image Inpainting Base on Coordinate-Attention

Multi-scale Generative Adversarial Deblurring Network with Gradient Guidance

<>With regards to the lack of crisp edges and a poor recovery of high frequency information such as details in deblurred motion pictures, this research proposes a multi-scale adversarial deblurring network with gradient guidance (MADN). The algorithm uses the classical generative adversarial network (GAN) framework, consisting of a generator and a discriminator. The generator includes a multi-scale convolutional network and a gradient feature extraction network. The multi-scale convolutional network extracts image features at different scales with a nested connection residual codec structure to improve the image edge structure recovery and to increase the perceptual field. This gradient network incorporates with intermediate scale features to extract the gradient features of blurred images to obtain their high frequency information. The generator combines the gradient and multiscale features to recover the remaining high-frequency information in a deblurred image. The loss function of MADN is formed in this research combining adversarial loss, pixel L2-norm loss and mean absolute error. Compared to those experimental results obtained from current deblurring algorithms, our experimental results indicate visually clearer images retaining more information such as edges and details. This MADN algorithm enhances the peak signal-to-noise ratio by an average of 3.32dB and the structural similarity by an average of 0.053.<>

Infrared and Visible Image Homography Estimation Using Multiscale Generative Adversarial Network

In computer vision, the homography estimation of infrared and visible multi-source images based on deep learning is a current research hotspot. Existing homography estimation methods ignore the feature differences of multi-source images, which leads to poor homography performance in infrared and visible image scenes. To address this issue, we designed an infrared and visible image homography estimation method using a Multi-scale Generative Adversarial Network, called HomoMGAN. First, we designed two shallow feature extraction networks to extract fine features of infrared and visible images, respectively, which extract important features in source images from two dimensions: color channel and imaging space. Second, we proposed an unsupervised generative adversarial network to predict the homography matrix directly. In our adversarial network, the generator captures meaningful features for homography estimation at different scales by using an encoder–decoder structure and further predicts the homography matrix. The discriminator recognizes the feature difference between the warped and target image. Through the adversarial game between the generator and the discriminator, the fine features of the warped image in the homography estimation process are closer to the fine features of the target image. Finally, we conduct extensive experiments in the synthetic benchmark dataset to verify the effectiveness of HomoMGAN and its components. We conduct extensive experiments and the results show that HomoMGAN outperforms existing state-of-the-art methods in the synthetic benchmark datasets both qualitatively and quantitatively.

Adaptive Multi-Scale Generative Models for Complex Data Synthesis

Adaptive Multi-Scale Generative Models for Complex Data Synthesis

Improving Image Super-Resolution Based on Multiscale Generative Adversarial Networks.

Convolutional neural networks have greatly improved the performance of image super-resolution. However, perceptual networks have problems such as blurred line structures and a lack of high-frequency information when reconstructing image textures. To mitigate these issues, a generative adversarial network based on multiscale asynchronous learning is proposed in this paper, whereby a pyramid structure is employed in the network model to integrate high-frequency information at different scales. Our scheme employs a U-net as a discriminator to focus on the consistency of adjacent pixels in the input image and uses the LPIPS loss for perceptual extreme super-resolution with stronger supervision. Experiments on benchmark datasets and independent datasets Set5, Set14, BSD100, and SunHays80 show that our approach is effective in restoring detailed texture information from low-resolution images.

A Fast Multi-Scale Generative Adversarial Network for Image Compressed Sensing.

Recently, deep neural network-based image compressed sensing methods have achieved impressive success in reconstruction quality. However, these methods (1) have limitations in sampling pattern and (2) usually have the disadvantage of high computational complexity. To this end, a fast multi-scale generative adversarial network (FMSGAN) is implemented in this paper. Specifically, (1) an effective multi-scale sampling structure is proposed. It contains four different kernels with varying sizes so that decompose, and sample images effectively, which is capable of capturing different levels of spatial features at multiple scales. (2) An efficient lightweight multi-scale residual structure for deep image reconstruction is proposed to balance receptive field size and computational complexity. The key idea is to apply smaller convolution kernel sizes in the multi-scale residual structure to reduce the number of operations while maintaining the receptive field. Meanwhile, the channel attention structure is employed for enriching useful information. Moreover, perceptual loss is combined with MSE loss and adversarial loss as the optimization function to recover a finer image. Numerous experiments show that our FMSGAN achieves state-of-the-art image reconstruction quality with low computational complexity.

TMS-GAN: A Twofold Multi-Scale Generative Adversarial Network for Single Image Dehazing

In recent years, learning-based single image dehazing networks have been comprehensively developed. However, performance improvement is limited due to domain shift between trained synthetic hazy images and untrained real-world hazy images. To alleviate this issue, this paper proposes a real-world dehazing targeted training scheme which nearly realizes paired real-world data training. As a result, a Twofold Multi-scale Generative Adversarial Network (TMS-GAN) consisting of a Haze-generation GAN (HgGAN) and a Haze-removal GAN (HrGAN) is designed. HgGAN attributes real haze properties to synthetic images and HrGAN removes haze from both synthetic and generated fake realistic data under supervision. Thus, the proposed method can better adapt to real-world image dehazing using this cooperative training scheme. Meanwhile, several structural advances of TMS-GAN also improve dehazing performance. Specifically, a haze residual map based on atmospheric scattering model is deduced in HgGAN for fake realistic data generation. The dual-branch generator in HrGAN draws attention to detail restoration by one branch along with another color-branch. A plug-and-play Multi-attention Progressive Fusion Module (MAPFM) is proposed and inserted in both HgGAN and HrGAN. MAPFM incorporates multi-attention mechanism to guide multi-scale feature fusion in a progressive manner, in which Adjacency-attention Block (AAB) can capture contributing features of each level and Self-attention Block (SAB) can establish non-local dependency of feature fusion. Experiments on mainstream benchmarks show that the proposed framework is superior especially on real-world hazy images among single image dehazing methods.

Multi-Focus Image Fusion Based on Multi-Scale Generative Adversarial Network.

The methods based on the convolutional neural network have demonstrated its powerful information integration ability in image fusion. However, most of the existing methods based on neural networks are only applied to a part of the fusion process. In this paper, an end-to-end multi-focus image fusion method based on a multi-scale generative adversarial network (MsGAN) is proposed that makes full use of image features by a combination of multi-scale decomposition with a convolutional neural network. Extensive qualitative and quantitative experiments on the synthetic and Lytro datasets demonstrated the effectiveness and superiority of the proposed MsGAN compared to the state-of-the-art multi-focus image fusion methods.

Multiscale Generative Adversarial Network Based on Wavelet Feature Learning for SAR-to-Optical Image Translation

Multiscale Generative Adversarial Network Based on Wavelet Feature Learning for SAR-to-Optical Image Translation

Attention-Based Multi-Scale Generative Adversarial Network for synthesizing contrast-enhanced MRI.

In clinical practice, about 35% of MRI scans are enhanced with Gadolinium - based contrast agents (GBCAs) worldwide currently. Injecting GBCAs can make the lesions much more visible on contrast-enhanced scans. However, the injection of GBCAs is high-risk, time-consuming, and expensive. Utilizing a generative model such as an adversarial network (GAN) to synthesize the contrast-enhanced MRI without injection of GBCAs becomes a very promising alternative method. Due to the different features of the lesions in contrast-enhanced images while the single-scale feature extraction capabilities of the traditional GAN, we propose a new generative model that a multi-scale strategy is used in the GAN to extract different scale features of the lesions. Moreover, an attention mechanism is also added in our model to learn important features automatically from all scales for better feature aggregation. We name our proposed network with an attention-based multi-scale contrasted-enhanced-image generative adversarial network (AMCGAN). We examine our proposed AMCGAN on a private dataset from 382 ankylosing spondylitis subjects. The result shows our proposed network can achieve state-of-the-art in both visual evaluations and quantitative evaluations than traditional adversarial training.Clinical Relevance-This study provides a safe, convenient, and inexpensive tool for the clinical practices to get contrast-enhanced MRI without injection of GBCAs.

Extraction of urban multi-class from high-resolution images using pyramid generative adversarial networks

The extraction of man-made structures from high-resolution images plays a vital role in various urban applications. This task is regularly complicated due to the heterogeneous appearance of the objects in the satellite images. In this work, we propose a multi-scale Generative Adversarial network to classify high-resolution images into urban classes (surface, building, tree, low-vegetation, car). We use uNet with EfficientNet B3 architecture as a generator and we use ResNet 18 architecture as a discriminator. We use a conditional generative loss based on the Dice coefficient and softmax functions. Experiments on the Vaihingen and Potsdam datasets were conducted to demonstrate the performance and we compare the results with other architectures. The results demonstrate the validity and higher performance of the proposed multi-scale network for extracting classes in urban areas with average F1-score 89.0% and 88.8%, and average accuracy 89.9% and 91.8% for Vaihingen and Potsdam datasets, respectively.

Semisupervised Multiscale Generative Adversarial Network for Semantic Segmentation of Remote Sensing Image

Semantic segmentation of remote sensing images based on deep neural networks has gained wide attention recently. Although many methods have achieved amazing performance, they need large amounts of labeled images to distinguish the differences in angle, color, size, and other aspects for small targets in remote sensing data sets. However, with a few labeled images, it is difficult to extract the key features of small targets. We propose a semisupervised multiscale generative adversarial network (GAN), which not only utilizes the multipath input and atrous spatial pyramid pooling (ASPP) module but leverages unlabeled images and semisupervised learning strategy to improve the performance of small target segmentation in semantic segmentation when labeled data amount is small. Experimental results show that our model outperforms state-of-the-art methods with insufficient labeled data.

Multi-scale generative adversarial network for improved evaluation of cell–cell interactions observed in organ-on-chip experiments

Organs On a Chip (OOCs) represent a sophisticated approach for exploring biological mechanisms and developing therapeutic agents. In conjunction with high-quality time-lapse microscopy (TLM), OOCs allow for the visualization of reconstituted complex biological processes, such as multi-cell-type migration and cell–cell interactions. In this context, increasing the frame rate is desirable to reconstruct accurately cell-interaction dynamics. However, a trade-off between high resolution and carried information content is required to reduce the overall data volume. Moreover, high frame rates increase photobleaching and phototoxicity. As a possible solution for these problems, we report a new hybrid-imaging paradigm based on the integration of OOC/TLMs with a Multi-scale Generative Adversarial Network (GAN) predicting interleaved video frames with the aim to provide high-throughput videos. We tested the performance of the predictive capability of GAN on synthetic videos, as well as on real OOC experiments dealing with tumor–immune cell interactions. The proposed approach offers the possibility to acquire a reduced number of high-quality TLM images without any major loss of information on the phenomena under investigation.

Multi-scale generative adversarial inpainting network based on cross-layer attention transfer mechanism

Deep learning-based methods have recently shown promising results in image inpainting. These methods generate patches with visually plausible image structures and textures, which are semantically coherent with the context of surrounding regions. However, existing methods tend to generate artifacts which are inconsistent with surrounding regions, especially when dealing with complex images. Aiming at the limitations current in deep learning-based methods, this paper proposes a multi-scale generative adversarial network model based on cross-layer attention transfer mechanism. Cross-Layer Attention Transfer Module (CL-ATM) is presented to guide the filling of the corresponding low-level semantic feature map by using the high-level semantic feature map, so as to ensure visual and semantic consistency of inpainting. Meanwhile, a multi-scale generator and the multi-scale discriminators are added into the network structure. Different scales of discriminators have different receptive fields, which enable the generator to produce images with better global consistency and more details. Qualitative and quantitative experiments show that our method has superior performance against state-of-art inpainting models.

Adversarial Learning for Multiscale Crowd Counting Under Complex Scenes.

In this article, a multiscale generative adversarial network (MS-GAN) is proposed for generating high-quality crowd density maps of arbitrary crowd density scenes. The task of crowd counting has many challenges, such as severe occlusions in extremely dense crowd scenes, perspective distortion, and high visual similarity between the pedestrians and background elements. To address these problems, the proposed MS-GAN combines a multiscale convolutional neural network (generator) and an adversarial network (discriminator) to generate a high-quality density map and accurately estimate the crowd count in complex crowd scenes. The multiscale generator utilizes the fusion features from multiple hierarchical layers to detect people with large-scale variation. The resulting density map produced by the multiscale generator is processed by a discriminator network trained to solve a binary classification task between a poor quality density map and real ground-truth ones. The additional adversarial loss can improve the quality of the density map, which is critical to accurately estimate the crowd counts. The experiments were conducted on multiple datasets with different crowd scenes and densities. The results showed that the proposed method provided better performance compared to current state-of-the-art methods.