Recurrent Generative Adversarial Network Research Articles

SM-CycleGAN: crop image data enhancement method based on self-attention mechanism CycleGAN

Crop disease detection and crop baking stage judgement require large image data to improve accuracy. However, the existing crop disease image datasets have high asymmetry, and the poor baking environment leads to image acquisition difficulties and colour distortion. Therefore, we explore the potential of the self-attention mechanism on crop image datasets and propose an innovative crop image data-enhancement method for recurrent generative adversarial networks (GANs) fused with the self-attention mechanism to significantly enhance the perception and information capture capabilities of recurrent GANs. By introducing the self-attention mechanism module, the cycle-consistent GAN (CycleGAN) is more adept at capturing the internal correlations and dependencies of image data, thus more effectively capturing the critical information among image data. Furthermore, we propose a new enhanced loss function for crop image data to optimise the model performance and meet specific task requirements. We further investigate crop image data enhancement in different contexts to validate the performance and stability of the model. The experimental results show that, the peak signal-to-noise ratio of the SM-CycleGAN for tobacco images and tea leaf disease images are improved by 2.13% and 3.55%, and the structural similarity index measure is improved by 1.16% and 2.48% compared to CycleGAN, respectively.

Assessment and Prediction of University English Teaching Effect Using Progressive Recurrent Generative Adversarial Network

Assessment and Prediction of University English Teaching Effect Using Progressive Recurrent Generative Adversarial Network

Short‐term power forecasting method for 5G photovoltaic base stations on non‐sunny days based on SDN‐integrated INGO‐BP and RGAN

AbstractIn response to the suboptimal efficiency observed in the network configuration and administration of 5G photovoltaic base stations (PVBSs), as well as the inherent limitations in accurately forecasting photovoltaic power during inclement weather conditions, this research article introduces a concise and effective method for short‐term power prediction of PVBSs when subjected to non‐sunny weather conditions, leveraging the paradigm of software defined networking. This method ingeniously amalgamates the improved Northern Goshawk optimized back propagation neural network (INGO‐BP) and recurrent generative adversarial networks (RGAN). The investigation commences by delineating a comprehensive PVBS system framework underpinned by SDN principles. Subsequently, it employs the Pearson correlation coefficient technique and principal component analysis for the judicious selection of pertinent features from the PVBS photovoltaic dataset, thereby ameliorating the issue of multicollinearity within the predictive model. To culminate the research endeavour, the INGO algorithm is enlisted to optimize the weightings and bias parameters of the BP neural network, while RGAN is harnessed for the acquisition of distinctive distribution patterns in photovoltaic data across diverse meteorological conditions. This augmentation enhances the model's capacity to discern the intricate long‐term mappings between photovoltaic power and meteorological data. The empirical findings substantiate the elevated accuracy of the INGO‐BP prediction model across sunny, cloudy, and rainy weather conditions, with a discernible enhancement in predictive accuracy for PVBS performance under non‐sunny weather conditions facilitated by data augmentation via RGAN.

Application of Deep Learning in Image Recognition of Citrus Pests

The occurrence of pests at high frequencies has been identified as a major cause of reduced citrus yields, and early detection and prevention are of great significance to pest control. At present, studies related to citrus pest identification using deep learning suffer from unbalanced sample sizes between data set classes, which may cause slow convergence of network models and low identification accuracy. To address the above problems, this study built a dataset including 5182 pest images in 14 categories. Firstly, we expanded the dataset to 21,000 images by using the Attentive Recurrent Generative Adversarial Network (AR-GAN) data augmentation technique, then we built Visual Geometry Group Network (VGG), Residual Neural Network (ResNet) and MobileNet citrus pest recognition models by using transfer learning, and finally, we introduced an appropriate attention mechanism according to the model characteristics to enhance the ability of the three models to operate effectively in complex, real environments with greater emphasis placed on incorporating the deep features of the pests themselves. The results showed that the average recognition accuracy of the three models reached 93.65%, the average precision reached 93.82%, the average recall reached 93.65%, and the average F1-score reached 93.62%. The integrated application of data augmentation, transfer learning and attention mechanisms in the research can significantly enhance the model’s ability to classify citrus pests while saving training cost and time, which can be a reference for researchers in the industry or other fields.

Cross-Correlated Scenario Generation for Renewable-Rich Power Systems Using Implicit Generative Models

Generation of realistic scenarios is an important prerequisite for analyzing the reliability of renewable-rich power systems. This paper satisfies this need by presenting an end-to-end model-free approach for creating representative power system scenarios on a seasonal basis. A conditional recurrent generative adversarial network serves as the main engine for scenario generation. Compared to prior scenario generation models that treated the variables independently or focused on short-term forecasting, the proposed implicit generative model effectively captures the cross-correlations that exist between the variables considering long-term planning. The validity of the scenarios generated using the proposed approach is demonstrated through extensive statistical evaluation and investigation of end-application results. It is shown that analysis of abnormal scenarios, which is more critical for power system resource planning, benefits the most from cross-correlated scenario generation.

Data driven recurrent generative adversarial network for generalized zero shot image classification

Traditional Generative Adversarial Network (GAN) based Generalized Zero Shot Learning (GZSL) methods usually suffer from a problem that these methods ignore the differences between classes when using the standard normal distribution to fit the true distribution of each category, and the incompleteness of a single adversarial training makes the model unable to capture all the characteristics of the samples. To address this problem, a data-driven recurrent adversarial generative network is proposed in this paper. We first synthesize visual prototypes for unseen classes using the transformation from semantic attributes to visual prototypes learned on seen classes. Then, some noise is generated from these prototypes to synthesize the unseen samples according to the corresponding semantic attributes. During the sample generation process, a recurrent generative adversarial network is designed to facilitate the generated visual features to be more representative. Extensive experiments on five popular datasets as well as detailed ablation studies demonstrate the effectiveness and superiority of the proposed method.

TR-GAN: Multi-Session Future MRI Prediction With Temporal Recurrent Generative Adversarial Network

Magnetic Resonance Imaging (MRI) has been proven to be an efficient way to diagnose Alzheimer's disease (AD). Recent dramatic progress on deep learning greatly promotes the MRI analysis based on data-driven CNN methods using a large-scale longitudinal MRI dataset. However, most of the existing MRI datasets are fragmented due to unexpected quits of volunteers. To tackle this problem, we propose a novel Temporal Recurrent Generative Adversarial Network (TR-GAN) to complete missing sessions of MRI datasets. Unlike existing GAN-based methods, which either fail to generate future sessions or only generate fixed-length sessions, TR-GAN takes all past sessions to recurrently and smoothly generate future ones with variant length. Specifically, TR-GAN adopts recurrent connection to deal with variant input sequence length and flexibly generate future variant sessions. Besides, we also design a multiple scale & location (MSL) module and a SWAP module to encourage the model to better focus on detailed information, which helps to generate high-quality MRI data. Compared with other popular GAN architectures, TR-GAN achieved the best performance in all evaluation metrics of two datasets. After expanding the Whole MRI dataset, the balanced accuracy of AD vs. cognitively normal (CN) vs. mild cognitive impairment (MCI) and stable MCI vs. progressive MCI classification can be increased by 3.61% and 4.00%, respectively.

Restored Action Generative Adversarial Imitation Learning from observation for robot manipulator

In this paper, a new imitation learning algorithm is proposed based on the Restored Action Generative Adversarial Imitation Learning (RAGAIL) from observation. An action policy is trained to move a robot manipulator similar to a demonstrator’s behavior by using the restored action from state-only demonstration. To imitate the demonstrator, the trajectory is generated by Recurrent Generative Adversarial Networks (RGAN), and the action is restored from the output of the tracking controller constructed by the state and the generated target trajectory. The proposed imitation learning algorithm is not required to access the demonstrator’s action (internal control signal such as force/torque command) and provides better learning performances. The effectiveness of the proposed method is validated through the experimental results of the robot manipulator.

Generation of Realistic Navigation Paths for Web Site Testing Using RNN and GAN

For applications that have not yet been launched, a reliable way for creating online navigation logs may be crucial, enabling developers to test their products as though they were being used by real users. This might lead to faster and lower-cost program testing and enhancement, especially in terms of usability and interaction. In this work we propose a method for using deep learning approaches such as recurrent neural networks (RNN) and generative adversarial neural networks (GANN) to produce high-quality weblogs. Eventually, we can utilize the created data for automated testing and improvement of Web sites prior to their release with the aid of model-driven development tools such as IFML Editor.

Internet of things‐based deeply proficient monitoring and protection system for crop field

AbstractThe production rate of crops is significantly declining due to natural disasters, animal interventions and plant diseases. Internet of things (IoT) and wireless sensor networks are widely applied in crop field monitoring systems to observe the quality of each plant and the field. This work proposes IoT based crop field protection system (ICFPS) that monitors and protects the crop fields from animal intrusions. This proposed system uses ultrasonic sensors, hyperspectral cameras, voice recorded buzzers and other agriculture sensors to protect the entire crop field. This system uses numerous sensor nodes and cameras for gathering field objects (images and environmental objects). The proposed ICFPS creates deep learning techniques such as recurrent convolutional neural networks (RCNN) and recurrent generative adversarial neural networks (RGAN) for feature extraction, disease detection and field data monitoring practices. This proposed work develops a smart city‐based agriculture system using cognitive learning approaches. This proposed system analyses crop field data and provide automatic alerts regarding animal interferences and crop diseases. Moreover, the cognitive smart crop field system observes various field conditions which support for good production rate. In this system, sensors and camera‐enabled agriculture drones are coordinated with each other to collect the field data regularly. At the same time, the proposed work trains the RCNN and RGAN units using effective crop field datasets to attain realistic decisions within minimal time intervals. The experiment details and results show the proposed ICFPS works with 8%–10% of more classification accuracy than existing systems.

Estimating Effective Connectivity by Recurrent Generative Adversarial Networks.

Estimating effective connectivity from functional magnetic resonance imaging (fMRI) time series data has become a very hot topic in neuroinformatics and brain informatics. However, it is hard for the current methods to accurately estimate the effective connectivity due to the high noise and small sample size of fMRI data. In this paper, we propose a novel framework for estimating effective connectivity based on recurrent generative adversarial networks, called EC-RGAN. The proposed framework employs the generator that consists of a set of effective connectivity generators based on recurrent neural networks to generate the fMRI time series of each brain region, and uses the discriminator to distinguish between the joint distributions of the real and generated fMRI time series. When the model is well-trained and generated fMRI data is similar to real fMRI data, EC-RGAN outputs the effective connectivity by means of the causal parameters of the effective connectivity generators. Experimental results on both simulated and real-world fMRI time series data demonstrate the efficacy of our proposed framework.

Cine Cardiac MRI Motion Artifact Reduction Using a Recurrent Neural Network.

Cine cardiac magnetic resonance imaging (MRI) is widely used for the diagnosis of cardiac diseases thanks to its ability to present cardiovascular features in excellent contrast. As compared to computed tomography (CT), MRI, however, requires a long scan time, which inevitably induces motion artifacts and causes patients' discomfort. Thus, there has been a strong clinical motivation to develop techniques to reduce both the scan time and motion artifacts. Given its successful applications in other medical imaging tasks such as MRI super-resolution and CT metal artifact reduction, deep learning is a promising approach for cardiac MRI motion artifact reduction. In this paper, we propose a novel recurrent generative adversarial network model for cardiac MRI motion artifact reduction. This model utilizes bi-directional convolutional long short-term memory (ConvLSTM) and multi-scale convolutions to improve the performance of the proposed network, in which bi-directional ConvLSTMs handle long-range temporal features while multi-scale convolutions gather both local and global features. We demonstrate a decent generalizability of the proposed method thanks to the novel architecture of our deep network that captures the essential relationship of cardiovascular dynamics. Indeed, our extensive experiments show that our method achieves better image quality for cine cardiac MRI images than existing state-of-the-art methods. In addition, our method can generate reliable missing intermediate frames based on their adjacent frames, improving the temporal resolution of cine cardiac MRI sequences.

Recurrent generative adversarial networks for unsupervised WCE video summarization

Wireless capsule endoscopy (WCE) produces amounts of redundant images in one examination, which is very laborious and time-consuming for a physician to review these. It has been extremely needed for a technique that automatically produces a shortened and informative WCE video summary from its original video. This paper considers unsupervised WCE video summarization, and casts it as a sequence-to-sequence learning problem. Our key idea is to learn a deep summarizer network to minimize information loss between training videos and their summaries, in an unsupervised way. To this end, we propose a hybrid yet effective unsupervised WCE video summarization method using long short-term memory (LSTM), variational autoencoder (VAE), pointer network (Ptr-Net), generative adversarial network (GAN), and de-redundancy mechanism (DM) etc. techniques. The proposed model termed Adv-Ptr-Der-SUM adopts a generative adversarial framework, consisting of a summarizer and a discriminator. The summarizer is the VAE-based LSTM architecture with Ptr-Net and DM that aims to learn the conditional probability of output sequence and provide a compact summary. The discriminator is another LSTM aimed at distinguishing between the original video and reconstructed video from the summarizer. The summarizer and discriminator are adversarially trained to optimize the summarizer and produce optimal WCE video summary. Extensive experiments on our WCE-2019-Video dataset show that our model can outperform other video summarization approaches by a large margin in both supervised and unsupervised settings. Also, the proposed model is applied to two public multimedia benchmark datasets, verifying its effectiveness and generality, and demonstrating that it can achieve a competitive result.

Text to photo-realistic image synthesis via chained deep recurrent generative adversarial network

Despite the promising progress made in recent years, automatically generating high-resolution realistic images from text descriptions remains a challenging task due to semantic gap between human-written descriptions and diversities of visual appearance. Most existing approaches generate the rough images with the given text descriptions, while the relationship between sentence semantics and visual content is not holistically exploited. In this paper, we propose a novel chained deep recurrent generative adversarial network (CDRGAN) for synthesizing images from text descriptions. Our model uses carefully designed chained deep recurrent generators that simultaneously recovers global image structures and local details. Specially, our method not only considers the logic relationships of image pixels, but also removes computational bottlenecks through parameters sharing. We evaluate our method on three public benchmarks: CUB, Oxford-102 and MS COCO datasets. Experimental results show that our method significantly outperforms the state-of-the-art approaches consistently across different evaluation metrics.

EEGFuseNet: Hybrid Unsupervised Deep Feature Characterization and Fusion for High-Dimensional EEG With an Application to Emotion Recognition

How to effectively and efficiently extract valid and reliable features from high-dimensional electroencephalography (EEG), particularly how to fuse the spatial and temporal dynamic brain information into a better feature representation, is a critical issue in brain data analysis. Most current EEG studies work in a task driven manner and explore the valid EEG features with a supervised model, which would be limited by the given labels to a great extent. In this paper, we propose a practical hybrid unsupervised deep convolutional recurrent generative adversarial network based EEG feature characterization and fusion model, which is termed as EEGFuseNet. EEGFuseNet is trained in an unsupervised manner, and deep EEG features covering both spatial and temporal dynamics are automatically characterized. Comparing to the existing features, the characterized deep EEG features could be considered to be more generic and independent of any specific EEG task. The performance of the extracted deep and low-dimensional features by EEGFuseNet is carefully evaluated in an unsupervised emotion recognition application based on three public emotion databases. The results demonstrate the proposed EEGFuseNet is a robust and reliable model, which is easy to train and performs efficiently in the representation and fusion of dynamic EEG features. In particular, EEGFuseNet is established as an optimal unsupervised fusion model with promising cross-subject emotion recognition performance. It proves EEGFuseNet is capable of characterizing and fusing deep features that imply comparative cortical dynamic significance corresponding to the changing of different emotion states, and also demonstrates the possibility of realizing EEG based cross-subject emotion recognition in a pure unsupervised manner.

Waterdrop Removal From Hot-Rolled Steel Strip Surfaces Based on Progressive Recurrent Generative Adversarial Networks

Automated visual inspection (AVI) instrument of surface defects for hot-rolled steel strips is conventionally installed closely before the terminal crimping machine, where the adjacent upstream process is laminar spray cooling. Waterdrops, spreading more or less over the steel strip surface, often trigger false alarms, which is a quite common problem in AVI. Stimulated by the idea of image rain removal in visual enhancement field, this article considers the surface waterdrops, pseudodefects in essence, as a conceptual “rain-like layer.” A targeted method, namely progressive recurrent generative adversarial network (PReGAN), is designed for active waterdrop tracking and fine-grained image inpainting. Meanwhile, a steel surface database (2400 raw images with the resolution of <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$1000\times1000$ </tex-math></inline-formula> ) captured from actual hot-rolling line is constructed for the first time for open evaluation of waterdrop removal. The experimental results indicate that images enhanced by the PReGAN perform the most informative and spotless, with 52.2073 peak signal-to-noise ratio (PSNR) and 0.9502 structural similarity (SSIM) index, when compared with <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">four</i> prestigious networks. Assisted by the PReGAN, the false alarms are proved to be reduced at least a half during the application tests using <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">four</i> traditional simple detection methods.

Remaining useful life prediction of planet bearings based on conditional deep recurrent generative adversarial network and action discovery

Planet bearing vibration signals are complex and non-stationary owing to the time-varying vibration transmission path from fault point to accelerometers. In addition, due to the influence of speed and load variation, planetary gearboxes often operate under varying conditions, which causes the acquired vibration signals to be nonlinear. These seriously degrade the performance of the RUL prediction methods. Moreover, there are only few training samples for the RUL prediction of planet bearings. To address these problems, an RUL prediction method of planet bearings using conditional deep recurrent generative adversarial network (C-DRGAN) and action discovery (AD) is presented. First, gated recurrent unit neural network is integrated with conditional generative adversarial network to construct the C-DRGAN, which extracts fault features from nonlinear and non-stationary signals so as to realize the RUL prediction of planet bearings under small samples and varying conditions. Then, the training algorithm based on AD is employed to train the C-DRGAN to enhance convergence speed and reduce training time. Finally, a multiple linear regression classifier is utilized to predict the RUL of planet bearings. The performance of the presented method is validated through an accelerated fatigue life experiment of planet bearings. Experimental analyses demonstrate that this method possesses strong processing adaptability to nonlinear and non-stationary signals and obtains excellent performance under small samples.

Time-Series Regeneration With Convolutional Recurrent Generative Adversarial Network for Remaining Useful Life Estimation

For health prognostic task, ever-increasing efforts have been focused on machine learning based methods, which are capable of yielding accurate remaining useful life (RUL) estimation for industrial equipment or components without exploring the degradation mechanism. A prerequisite ensuring the success of these methods depends on a wealth of run-to-failure data; however, run-to-failure data may be insufficient in practice. That is, conducting a substantial amount of destructive experiments not only is of high cost but also may cause catastrophic consequences. Out of this consideration, an enhanced RUL framework focusing on data self-generation is put forward for both noncyclic and cyclic degradation patterns for the first time. It is designed to enrich data from a data-driven way, generating realistic-like time-series to enhance current RUL methods. First, high-quality data generation is ensured through the proposed convolutional recurrent generative adversarial network, which adopts a two-channel fusion convolutional recurrent neural network. Next, a hierarchical framework is proposed to combine generated data into current RUL estimation methods. Finally, in this article the efficacy of the proposed method is verified through both noncyclic and cyclic degradation systems. With the enhanced RUL framework, an aero-engine system following noncyclic degradation has been tested using three typical RUL models. State-of-the-art RUL estimation results are achieved by enhancing capsule network with generated time-series. Specifically, estimation errors evaluated by the index score function have been reduced by 21.77$\%$ and 32.67$\%$ for the two employed operating conditions, respectively. Besides, the estimation error is reduced to zero for the lithium-ion battery system, which presents cyclic degradation.

End-to-End Recurrent Generative Adversarial Network for Traffic and Surveillance Applications

In video frame segmentation, many existing deep networks and contemporary approaches give a remarkable performance with the assumption that the only foreground is moving, and the background is stationary. However, in the presence of infrequent motion of foreground objects, sudden illumination changes in the background, bad weather, and dynamic background, the accurate foreground object(s) segmentation is a challenging task. Generative adversarial networks (GAN) based training shows fruitful results in various fields like image-to-image style transfer, image enhancement, semantic segmentation, image super-resolution, etc. The limited results of hand-crafted approaches for moving object segmentation (MOS) and the robustness of adversarial training for a given task inspired us to propose a novel approach for moving object segmentation (MOS). In this context, an end-to-end generative adversarial network (two generators) with recurrent technique is proposed for MOS and is named as RMS-GAN. The proposed RMS-GAN is able to incorporate foreground probability knowledge with residual and weight sharing based recurrent technique for accurate segmentation. The recurrent technique helps us to exhibit the temporal behavior between successive video frames, which is more prominent for any video processing applications. Also, to enhance the spatial coherence of the obtained foreground probability map using the generator-1 network, the cascaded architecture of two generators is proposed. The effectiveness of the proposed approach is evaluated both qualitatively and quantitatively on three benchmark video datasets for MOS. Experimental result analysis shows that the proposed network outperforms the existing state-of-the-art methods on three benchmark datasets for MOS.

Improved recurrent generative adversarial networks with regularization techniques and a controllable framework

Generative Adversarial Network (GAN), a deep learning framework to generate synthetic but realistic samples, has produced astonishing results for image synthesis. However, because GAN is routinely used for image datasets, regularization methods for GAN have been developed for convolutional layers. In this study, to expand these methods for time-series data, which are one of the most common data types in various real datasets, modified regularization methods are proposed for Long Short-Term Memory (LSTM)-based GANs. Specifically, the spectral normalization, hinge loss, orthogonal regularization, and the truncation trick are modified and assessed for LSTM-based GANs. Furthermore, a conditional GAN architecture called Controllable GAN (ControlGAN) is applied to LSTM-based GANs to produce the desired samples. The evaluations are conducted with sine wave data, air pollution datasets, and a medical time-series dataset obtained from intensive care units. As a result, ControlGAN with the spectral normalization on gates and cell states consistently outperforms the others, including the conventional model, called Recurrent Conditional GAN (RCGAN).