Adversarial Generative Network Research Articles

Bubble detection in photoresist with small samples based on GAN augmentations and modified YOLO

Photoresist coating is a key procedure in semiconductor wafer surface processing, and bubbles in a photoresist dropper usually affect the uniformity of photoresist film, which seriously reduces the etching quality of semiconductor wafer surface. However, the problems of blurred edges of photoresist bubbles and complex corrugations in photoresist itself make the automatic detection of photoresist bubbles very challenging. To overcome this problem, a high-precision photoresist bubble detection method based on deep learning is proposed in this paper. First, aiming at the problem that the photoresist bubble samples are extremely lacking and cannot meet the requirement of deep learning model training, this paper proposes a sample automatic generator bubbleGAN based on the adversarial generative network, so as to effectively increase the number of photoresist bubble samples with better diversity of the target to be detected, thus improving the performance of the detection model; Second, aiming at the challenges of target detection caused by the blurred edges of photoresist bubbles and the complex ripples of photoresist itself, this paper proposes a novel bubble detection method mYOLO. Based on the recently developed high-performance target detection method YOLOX, we optimize its structure and activation function to reduce the difficulty of model training, solve the dead zone problem of activation function, and improve the accuracy of model detection. The experimental results on the dataset collected by the actual rotary coating machine prove the effectiveness of the proposed algorithm. In addition, we further compare the impact of different manual data augmentation methods on the detection model.

Illegal Domain Name Generation Algorithm Based on Character Similarity of Domain Name Structure

Detecting and controlling illegal websites (gambling and pornography sites) through illegal domain names has been an unsolved problem. Therefore, how to mine and discover potential illegal domain names in advance has become a current research hotspot. This paper studies a method of generating illegal domain names based on the character similarity of domain name structure. Firstly, the K-means algorithm classified illegal domain names with similar structures. Then, put the classified clusters into the adversarial generative network for training. Finally, through a specific result verification method, the experiment shows that the average concentration of the generation algorithm is 23.82%, the effective concentration is 63.54%, and the expansion rate is 7.5. By comparing the results with the enumeration algorithm, the generation algorithm has greatly improved in terms of generation efficiency and accuracy.

주성분 분석을 잠재공간에 적용한 최소제곱 오차 이미지 생성 네트워크

Image generation is an important area of artificial intelligence that involves creating new images from existing dataset. It involves learning the distribution of target images from randomly generated vectors. Like other deep learning models, the image generation model requires a vast refined data set to produce high-quality results. When there is little data, there is a problem that the diversity and quality of generated images are compromised. In this paper, we propose a new generative model that applies PCA to the generator of the least square error adversarial generative network that, in turn, generates high-quality images even with a small data set. Unlike the existing models that generate target data from randomly generated noise, in the proposed method the direction of the image to be generated is guided by extracting the features of the target data through PCA. The results section shows the superior performance of the proposed model against a different number of images in dataset.

Robust visual tracking using very deep generative model

Deep learning algorithms provide visual tracking robustness at an unprecedented level, but realizing an acceptable performance is still challenging because of the natural continuous changes in the features of foreground and background objects over videos. One of the factors that most affects the robustness of tracking algorithms is the choice of network architecture parameters, especially the depth. A robust visual tracking model using a very deep generator (RTDG) was proposed in this study. We constructed our model on an ordinary convolutional neural network (CNN), which consists of feature extraction and binary classifier networks. We integrated a generative adversarial network (GAN) into the CNN to enhance the tracking results through an adversarial learning process performed during the training phase. We used the discriminator as a classifier and the generator as a store that produces unlabeled feature-level data with different appearances by applying masks to the extracted features. In this study, we investigated the role of increasing the number of fully connected (FC) layers in adversarial generative networks and their impact on robustness. We used a very deep FC network with 22 layers as a high-performance generator for the first time. This generator is used via adversarial learning to augment the positive samples to reduce the gap between the hungry deep learning algorithm and the available training data to achieve robust visual tracking. The experiments showed that the proposed framework performed well against state-of-the-art trackers on OTB-100, VOT2019, LaSOT and UAVDT benchmark datasets.

Data driven recurrent generative adversarial network for generalized zero shot image classification

Traditional Generative Adversarial Network (GAN) based Generalized Zero Shot Learning (GZSL) methods usually suffer from a problem that these methods ignore the differences between classes when using the standard normal distribution to fit the true distribution of each category, and the incompleteness of a single adversarial training makes the model unable to capture all the characteristics of the samples. To address this problem, a data-driven recurrent adversarial generative network is proposed in this paper. We first synthesize visual prototypes for unseen classes using the transformation from semantic attributes to visual prototypes learned on seen classes. Then, some noise is generated from these prototypes to synthesize the unseen samples according to the corresponding semantic attributes. During the sample generation process, a recurrent generative adversarial network is designed to facilitate the generated visual features to be more representative. Extensive experiments on five popular datasets as well as detailed ablation studies demonstrate the effectiveness and superiority of the proposed method.

Text-Guided Image Manipulation via Generative Adversarial Network With Referring Image Segmentation-Based Guidance

This study proposed a novel text-guided image manipulation method that introduces referring image segmentation into a generative adversarial network. The proposed text-guided image manipulation method aims to manipulate images containing multiple objects while preserving text-unrelated regions. The proposed method assigns the task of distinguishing between text-related and unrelated regions in an image to segmentation guidance based on referring image segmentation. With this architecture, the adversarial generative network can focus on generating new attributes according to the text description and reconstructing text-unrelated regions. For the challenging input images with multiple objects, the experimental results demonstrate that the proposed method outperforms conventional methods in terms of image manipulation precision.

CRVC-GAN: combining cross-scale fusion and recursion for video colorization adversarial generative networks

We proposed an innovative end-to-end generative adversarial network structure (CRVC-GAN) for video colorization, which is dedicated to solving the temporal consistency and coloring quality problems in the video coloring domain. For the temporal consistency problem, we utilize the network structure of recurrent neural network (RNN) because the model shape of RNN does not change with the input length and the calculation takes into account the history information to ensure the degree of correlation between successive frames. This better ensures the continuity between frames, so the consecutive frames get good temporal consistency in coloring. For the coloring quality problem, we utilize the UNet++ coding method to decode the video frames, integrating different levels of features to improve accuracy, and then enhance the object boundary information in the video frames, thus improving the coloring quality of grayscale video frames. In addition, we innovatively utilize the style transformation loss function in terms of images for model training to ensure temporal consistency and coloring quality by exploiting the slight difference in style variation from frame to frame. The experimental results show that, compared with the existing methods and through its own designed balance index, the colorization and temporary consistency balance index, CRVC-GAN obtained certain advantages on the DAVIS and Videvo public datasets with the results of 0.56176 and 0.74286, respectively, and achieved a better balance between time consistency and color quality.

A scenario generation method of integrated energy systems based on deep convolution generative adversarial nets

The problems of insufficient sensors and limited collection samples of energy networks in integrated energy systems can lead to problems of missing data and data anomalies, which poses greater challenges for scheduling, planning, and operation of integrated energy systems. Therefore, accurate load scenarios and dimensioning of existing data are required. Therefore, this paper proposes a method for generating load scenes of integrated energy systems based on adversarial generative networks, which is based on deep convolutional generative adversarial network architecture, with alternate training of discriminators and generators, providing sufficient amount of data for the subsequent construction of real-time simulation models of integrated energy systems. The experimental results show that the proposed method can generate the load scenarios of the integrated energy system well and realize the dimensionality increase of the data.

Research on Sustainable Reuse of Urban Ruins Based on Artificial Intelligence Technology: A Study of Guangzhou

In recent years, with the continuous deepening of the urbanization process, the problem of urban ruins (URs) has become prominent. This significantly affects the happiness of residents around the URs, the overall image of the city, and the environment, and it has become an important issue in urban construction. At present, the types of urban ruins mainly include industrial ruins, abandoned urban buildings, and war sites. Generally, methods such as demolition and reconstruction of original buildings or upgrading and transformation are used to reuse URs, and some of them have achieved fruitful results. However, the current renovation of URs is based on fragmented renovation strategies for different URs without a systematic and universally applicable renovation methodology. With the development of artificial intelligence, technologies such as Generative Adversarial Network (GAN), Easy DL, and Natural Language Processing (NLP) can provide technical support for urban ruin reconstruction, from design to operation. Specifically in the present study, the ten representative URs in Guangzhou are first evaluated by the Analytic Hierarchy Process and then combined with AI methods, such as the adversarial generative networks and big data applications, into the reuse design of URs. Finally, a complete research system is established to implement URs’ projects, which provides a clearer systematic planning strategy for the reuse of URs in the future.

Ultra-Short-Term Load Dynamic Forecasting Method Considering Abnormal Data Reconstruction Based on Model Incremental Training

In order to reduce the influence of abnormal data on load forecasting effects and further improve the training efficiency of forecasting models when adding new samples to historical data set, an ultra-short-term load dynamic forecasting method considering abnormal data reconstruction based on model incremental training is proposed in this paper. Firstly, aiming at the abnormal data in ultra-short-term load forecasting, a load abnormal data processing method based on isolation forests and conditional adversarial generative network (IF-CGAN) is proposed. The isolation forest algorithm is used to accurately eliminate the abnormal data points, and a conditional generative adversarial network (CGAN) is constructed to interpolate the abnormal points. The load-influencing factors are taken as the condition constraints of the CGAN, and the weighted loss function is introduced to improve the reconstruction accuracy of abnormal data. Secondly, aiming at the problem of low model training efficiency caused by the new samples in the historical data set, a model incremental training method based on a bidirectional long short-term memory network (Bi-LSTM) is proposed. The historical data are used to train the Bi-LSTM, and the transfer learning is introduced to process the incremental data set to realize the adaptive and rapid adjustment of the model weight and improve the model training efficiency. Finally, the real power grid load data of a region in eastern China are used for simulation analysis. The calculation results show that the proposed method can reconstruct the abnormal data more accurately and improve the accuracy and efficiency of ultra-short-term load forecasting.

Filter-GAN: Imbalanced Malicious Traffic Classification Based on Generative Adversarial Networks with Filter

In recent years, with the rapid development of Internet services in all walks of life, a large number of malicious acts such as network attacks, data leakage, and information theft have become major challenges for network security. Due to the difficulty of malicious traffic collection and labeling, the distribution of various samples in the existing dataset is seriously imbalanced, resulting in low accuracy of malicious traffic classification based on machine learning and deep learning, and poor model generalization ability. In this paper, a feature image representation method and Adversarial Generative Network with Filter (Filter-GAN) are proposed to solve these problems. First, the feature image representation method divides the original session traffic into three parts. The Markov matrix is extracted from each part to form a three-channel feature image. This method can transform the original session traffic format into a uniform-length matrix and fully characterize the network traffic. Then, Filter-GAN uses the feature images to generate few attack samples. Compared with general methods, Filter-GAN can generate more efficient samples. Experiments were conducted on public datasets. The results show that the feature image representation method can effectively characterize the original session traffic. When the number of samples is sufficient, the classification accuracy can reach 99%. Compared with unbalanced datasets, Filter-GAN has significantly improved the recognition accuracy of small-sample datasets, with a maximum improvement of 6%.

An Adversarial Generative Network Designed for High-Resolution Monocular Depth Estimation from 2D HiRISE Images of Mars

In computer vision, stereoscopy allows the three-dimensional reconstruction of a scene using two 2D images taken from two slightly different points of view, to extract spatial information on the depth of the scene in the form of a map of disparities. In stereophotogrammetry, the disparity map is essential in extracting the digital terrain model (DTM) and thus obtaining a 3D spatial mapping, which is necessary for a better analysis of planetary surfaces. However, the entire reconstruction process performed with the stereo-matching algorithm can be time consuming and can generate many artifacts. Coupled with the lack of adequate stereo coverage, it can pose a significant obstacle to 3D planetary mapping. Recently, many deep learning architectures have been proposed for monocular depth estimation, which aspires to predict the third dimension given a single 2D image, with considerable advantages thanks to the simplification of the reconstruction problem, leading to a significant increase in interest in deep models for the generation of super-resolution images and DTM estimation. In this paper, we combine these last two concepts into a single end-to-end model and introduce a new generative adversarial network solution that estimates the DTM at 4× resolution from a single monocular image, called SRDiNet (super-resolution depth image network). Furthermore, we introduce a sub-network able to apply a refinement using interpolated input images to better enhance the fine details of the final product, and we demonstrate the effectiveness of its benefits through three different versions of the proposal: SRDiNet with GAN approach, SRDiNet without adversarial network, and SRDiNet without the refinement learned network plus GAN approach. The results of Oxia Planum (the landing site of the European Space Agency’s Rosalind Franklin ExoMars rover 2023) are reported, applying the best model along all Oxia Planum tiles and releasing a 3D product enhanced by 4×.

Integrating Adversarial Generative Network with Variational Autoencoders towards Cross-Modal Alignment for Zero-Shot Remote Sensing Image Scene Classification

Remote sensing image scene classification takes image blocks as classification units and predicts their semantic descriptors. Because it is difficult to obtain enough labeled samples for all classes of remote sensing image scenes, zero-shot classification methods which can recognize image scenes that are not seen in the training stage are of great significance. By projecting the image visual features and the class semantic features into the latent space and ensuring their alignment, the variational autoencoder (VAE) generative model has been applied to address remote-sensing image scene classification under a zero-shot setting. However, the VAE model takes the element-wise square error as the reconstruction loss, which may not be suitable for measuring the reconstruction quality of the visual and semantic features. Therefore, this paper proposes to augment the VAE models with the generative adversarial network (GAN) to make use of the GAN’s discriminator in order to learn a suitable reconstruction quality metric for VAE. To promote feature alignment in the latent space, we have also proposed cross-modal feature-matching loss to make sure that the visual features of one class are aligned with the semantic features of the class and not those of other classes. Based on a public dataset, our experiments have shown the effects of the proposed improvements. Moreover, taking the ResNet models of ResNet18, extracting 512-dimensional visual features, and ResNet50 and ResNet101, both extracting 2048-dimensional visual features for testing, the impact of the different visual feature extractors has also been investigated. The experimental results show that better performance is achieved by ResNet18. This indicates that more layers of the extractors and larger dimensions of the extracted features may not contribute to the image scene classification under a zero-shot setting.

Chinese Font Generation from Stroke Semantic and Attention Mechanism

In order to help designers develop computer Chinese fonts efficiently, a Chinese character font generation algorithm is proposed. The algorithm can generate full font character images with a relatively consistent style based on small font datasets. For the sake of solving problems in stroke adhesion and structural errors when using adversarial generative network (GAN), the prior information of stroke semantics is introduced to improve the deep neural network for alleviating wrong stroke generation. Moreover, a skip connection module equipped with attention mechanism is integrated in the network model to reduce incomplete font structure and training period. In this module, features in encoder can be projected to that in de- coder, which is helpful in eliminating the information loss of the decoder so as to avoid generating structural errors. Comparative experiments between the proposed algorithm and other methods including pix2pix, zi2zi and DCFont are conducted on public Chinese font datasets. The results show the proposed algorithm is better that the others from the perspective of SSIM and average in loss function. The industry application and scope of our algorithm are given in finally.

A Rolling Bearing Fault Diagnosis Based on Conditional Depth Convolution Countermeasure Generation Networks under Small Samples.

Aiming at the problems of low fault diagnosis accuracy caused by insufficient samples and unbalanced data sample distribution in bearing fault diagnosis, this paper proposes a fault diagnosis method for rolling bearings referencing conditional deep convolution adversarial generative networks (C−DCGAN) for efficient data augmentation. Firstly, the concept of conditional constraints is used to guide and improve the sample generation process of the original generative adversarial network, and specific constraints are added to the data generation model to perform a balanced expansion of muti-category fault data for small sample data sets. Secondly, aiming at the phenomena of training instability, gradient disappearance and gradient explosion in the imbalanced sample set, it is proposed to optimize the structure of the generative network by using the structure of self-defined skip connections and spectral normalization, while using the Wasserstein distance with penalty term instead of cross entropy. The function is used as the loss function of the generative adversarial network to improve the stable feature extraction ability of the generative network and the effect of the training process; in this way, simulation sample data with only a small variation from the real data distribution can be generated. Finally, the complete fault data set (after mixing the original data with sufficient fault category and sample number) and the generated data are input into the one-dimensional convolution neural network for fault diagnosis of rolling bearing. The experiment’s results show that the diagnosis method in this paper can improve the fault classification effect of rolling bearings by generating balanced and sufficient sample data.

Multi-polarization fusion generative adversarial networks for clear underwater imaging

Polarization imaging has become a promising way for clear underwater vision, which depends on the difference of polarization characteristics between backscattered light and target signal. In this paper, to achieve clear underwater color polarization imaging, we propose a learning-based method that uses multi-polarization fusion adversarial generative networks to learn the relationship between polarization information and object radiance. The proposed method is especially designed for multiple polarimetric images, which can effectively extract the polarization correlations of images with different polarization states. Moreover, to train the proposed network, we build, for the first time to our knowledge, a color polarization image dataset from natural underwater environments through passive polarization imaging. The experimental results in laboratory and natural underwater environments show that it is feasible to introduce polarization information into learning-based image recovery, and deep learning technology is conducive to the extraction of polarization information. Comparing with other methods, the proposed method can effectively remove the backscattered light and recover the object radiance.

Human body structure extraction of anime characters based on deep learning

With the popularity of social networking services and smartphones, photo processing applications have become widely used, and there is a growing interest in more advanced photo processing techniques. In the field of image generation, methods based on adversarial generative networks (GANs) have shown particularly good results. We discussed the anime characters’ body structure extraction methods based on generative adversarial networks (GANs). This research is expected to be applied in the field of drawing that can be used as a tool to significantly reduce learning period or provide a new study method for drawing beginners.

Adversarial and adaptive tone mapping operator: multi-scheme generation and multi-metric evaluation

Tone mapping is one of the main techniques to convert high-dynamic range (HDR) images into low-dynamic range (LDR) images. We propose to use a variant of generative adversarial networks to adaptively tone map images. We designed a conditional adversarial generative network composed of a U-Net generator and patchGAN discriminator to adaptively convert HDR images into LDR images. We extended previous work to include additional metrics such as tone-mapped image quality index (TMQI), structural similarity index measure, Fréchet inception distance, and perceptual path length. In addition, we applied face detection on the Kalantari dataset and showed that our proposed adversarial tone mapping operator generates the best LDR image for the detection of faces. One of our training schemes, trained via 256 × 256 resolution HDR–LDR image pairs, results in a model that can generate high TMQI low-resolution 256 × 256 and high-resolution 1024 × 2048 LDR images. Given 1024 × 2048 resolution HDR images, the TMQI of the generated LDR images reaches a value of 0.90, which outperforms all other contemporary tone mapping operators.

Pseudo Generation of Metallographic Images and Verification of Superiority for Discrimination Problems -Applying Adversarial Generative Networks-

Pseudo Generation of Metallographic Images and Verification of Superiority for Discrimination Problems -Applying Adversarial Generative Networks-

Conditional generation of medical images via disentangled adversarial inference.

Synthetic medical image generation has a huge potential for improving healthcare through many applications, from data augmentation for training machine learning systems to preserving patient privacy. Conditional Adversarial Generative Networks (cGANs) use a conditioning factor to generate images and have shown great success in recent years. Intuitively, the information in an image can be divided into two parts: 1) content which is presented through the conditioning vector and 2) style which is the undiscovered information missing from the conditioning vector. Current practices in using cGANs for medical image generation, only use a single variable for image generation (i.e., content) and therefore, do not provide much flexibility nor control over the generated image. In this work we propose DRAI-a dual adversarial inference framework with augmented disentanglement constraints-to learn from the image itself, disentangled representations of style and content, and use this information to impose control over the generation process. In this framework, style is learned in a fully unsupervised manner, while content is learned through both supervised learning (using the conditioning vector) and unsupervised learning (with the inference mechanism). We undergo two novel regularization steps to ensure content-style disentanglement. First, we minimize the shared information between content and style by introducing a novel application of the gradient reverse layer (GRL); second, we introduce a self-supervised regularization method to further separate information in the content and style variables. For evaluation, we consider two types of baselines: single latent variable models that infer a single variable, and double latent variable models that infer two variables (style and content). We conduct extensive qualitative and quantitative assessments on two publicly available medical imaging datasets (LIDC and HAM10000) and test for conditional image generation, image retrieval and style-content disentanglement. We show that in general, two latent variable models achieve better performance and give more control over the generated image. We also show that our proposed model (DRAI) achieves the best disentanglement score and has the best overall performance.