Real Data Distribution Research Articles

A novel asexual-reproduction evolutionary neural network for wind power prediction based on generative adversarial networks

Accurate forecasts of wind power generation are essential for the operation of wind farms. But for the newly developed stations, it is difficult to make accurate prediction because there are no sufficient historical data available. It will thus be interesting to explore new data augmentation and prediction modeling approach adaptive to such new-built wind farms. In this regard, a novel asexual-reproduction evolutionary neural network (ARENN) for short-term wind power prediction based on Wasserstein generative adversarial network with gradient penalty (WGANGP) and ensemble empirical mode decomposition (EEMD) is presented in this paper. To solve the dilemma that new-built wind farms lack sufficient wind power data, the WGANGP is first applied to generate realistic data with a similar distribution of real data to augment the training dataset, which is further decomposed into a series of more stable subsequences by the EEMD so as to reduce the prediction difficulty of the machine learning model. In this study, a novel ARENN prediction model is developed to make the short-term wind power prediction, in which an asexual-reproduction evolutionary approach is first proposed to optimize the neural network based on a set of different loss functions that facilitate the population of network parameters approximating to the global optimum along different error surfaces in the evolutionary process. The proposed approach is validated on the data collected from the wind farm located in Spain and the predicted results demonstrate the advantage of our proposed approach over other methods involved in this study.

Distance constraint between features for unsupervised domain adaptive person re-identification

Many superior person re-identification (re-ID) approaches face a challenge: their performance show disastrous when all supervised models are generalized to a new domain. Some researchers have recently proposed domain adaptive person re-ID methods to address this difficulty, whereas they directly leveraged the target-domain data that would generate lots of noisy pseudo labels. Hence, this paper proposes a distance constraint between features (DCF) method, which clusters the feature distribution fitted the real target-domain data. We assemble different parts of one person for multi-scale self-supervised learning. After introducing domain invariance and designing the inter-image and inter-class distance constraint to regulate distances between target samples, the feature distribution extracted from the encoder trained by the source data can fit the real target data distribution, which leads our domain adaptive model to enjoy the more reliable clustering results and thus obtain a great identification performance in target domain. Extensive experiments demonstrate our approach outperforms state-of-the-art methods on three large-scale released datasets.

On the effectiveness of dual discriminator weighted generative adversarial network

Generative adversarial network (GAN) has made great progress in image generation and reconstruction, but it may cause the mode collapse problem in practice. We proposed the dual discriminator weighted generative adversarial network (D2WGAN), whose objective function weights the Kullback–Leibler divergence (KL divergence) and the reverse KL divergence and uses the complementary characteristics of these two divergences to make the generated models more diverse. Moreover, we proved the theoretical conditional optimality of the D2WGAN to show that the generator can learn the real data distribution. Finally, we conduct experiments on a large amount of synthetic data and real-world datasets (e.g., MNIST and CIFAR-10). The results show that, compared with the traditional dual discriminator generative adversarial network and GAN, the proposed D2WGAN can process multiple mode data and generate better sample diversities.

Determining overfitting and underfitting in generative adversarial networks using Fréchet distance

Generative adversarial networks (GANs) can be used in a wide range of applications where drawing samples from a data probability distribution without explicitly representing it is essential. Unlike the deep convolutional neural networks (CNNs) trained for mapping an input to one of the multiple outputs, monitoring the overfitting and underfitting in GANs is not trivial since they are not classifying but generating a data. While training set and validation set accuracy give a direct sense of success in terms of overfitting and underfitting for CNNs during the training process, evaluating the GANs mainly depends on the visual inspection of the generated samples and generator/discriminator costs of the GANs. Unfortunately, visual inspection is far away of being objective and generator/discriminator costs are very nonintuitive. In this paper, a method was proposed for quantitatively determining the overfitting and underfitting in the GANs during the training process by calculating the approximate derivative of the Fréchet distance between generated data distribution and real data distribution unconditionally or conditioned on a specific class. Both of the distributions can be obtained from the distribution of the embedding in the discriminator network of the GAN. The method is independent of the design architecture and the cost function of the GAN and empirical results on MNIST and CIFAR-10 support the effectiveness of the proposed method.

One for More: Selecting Generalizable Samples for Generalizable ReID Model

Current training objectives of existing person Re-IDentification (ReID) models only ensure that the loss of the model decreases on selected training batch, with no regards to the performance on samples outside the batch. It will inevitably cause the model to over-fit the data in the dominant position (e.g., head data in imbalanced class, easy samples or noisy samples). The latest resampling methods address the issue by designing specific criterion to select specific samples that trains the model generalize more on certain type of data (e.g., hard samples, tail data), which is not adaptive to the inconsistent real world ReID data distributions. Therefore, instead of simply presuming on what samples are generalizable, this paper proposes a one-for-more training objective that directly takes the generalization ability of selected samples as a loss function and learn a sampler to automatically select generalizable samples. More importantly, our proposed one-for-more based sampler can be seamlessly integrated into the ReID training framework which is able to simultaneously train ReID models and the sampler in an end-to-end fashion. The experimental results show that our method can effectively improve the ReID model training and boost the performance of ReID models.

Stochastic Predictions of Ore Production in an Underground Limestone Mine Using Different Probability Density Functions: A Comparative Study Using Big Data from ICT System

This study stochastically predicted ore production through discrete event simulation using four different probability density functions for truck travel times. An underground limestone mine was selected as the study area. The truck travel time was measured by analyzing the big data acquired from information and communications technology (ICT) systems in October 2018, and probability density functions (uniform, triangular, normal, and observed probability distribution of real data) were determined using statistical values. A discrete event simulation model for a truck haulage system was designed, and truck travel times were randomly generated using a Monte Carlo simulation. The ore production that stochastically predicted fifty times for each probability density function was analyzed and represented as a value of lower 10% (P10), 50% (P50), and 90% (P90). Ore production was underestimated when a uniform and triangular distribution was used, as the actual ore production was similar to that of P90. Conversely, the predicted ore production of P50 was relatively consistent with the actual ore production when using the normal and observed probability distribution of real data. The root mean squared error (RMSE) for predicting ore production for ten days in October 2018 was the lowest (24.9 ton/day) when using the observed probability distribution.

Convolutional Two-Stream Generative Adversarial Network-Based Hyperspectral Feature Extraction

Hyperspectral image processing is faced with difficulties considering its redundant features and complex information. Studies on hyperspectral feature extraction in the deep learning domain have become increasingly popular. The mainstream techniques fully consider the spatial information in local neighborhoods when extracting spectral features by constructing deep neural networks. Deep generative models simulate the intrinsic structure of samples by adequately training, showing their potential values for signal processing. In this article, a convolutional two-stream network (cs <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sup> GAN-FE) based on the improved Wasserstein generative adversarial network (WGAN) is proposed for unsupervised hyperspectral spatial–spectral feature extraction. The improved WGAN is composed of one generator and one discriminator; the former perceives real data distributions, and the latter determines the attribution of generated data. The designed two-stream strategy is not a simple extension of a one-stream strategy and considers both the static spectral–spatial information and the dynamic spectral reflectance variation in multiple bands. Intrinsic spatial–spectral features are extracted by the trained discriminator considering sample distributions and feature relationships. The loss function is also improved for the unique structure of cs <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sup> GAN-FE. Various state-of-the-art techniques are chosen for comparison. Experimental results show the feasibility and potential of this network. Besides, experiments with the random split and the disjointed split both show that the proposed method can outperform other comparison techniques.

Improving generative adversarial networks with simple latent distributions

Generative Adversarial Networks (GANs) have drawn great attention recently since they are the powerful models to generate high-quality images. Although GANs have achieved great success, they usually suffer from unstable training and consequently may lead to the poor generations in some cases. Such drawback is argued mainly due to the difficulties in measuring the divergence between the highly complicated the real and fake data distributions, which are normally in the high-dimensional space. To tackle this problem, previous researchers attempt to search a proper divergence capable of measuring the departure of the complex distributions. In contrast, we attempt to alleviate this problem from a different perspective: while retaining the information as much as possible of the original high dimensional distributions, we learn and leverage an additional latent space where simple distributions are defined in a low-dimensional space; as a result, we can readily compute the distance between two simple distributions with an available divergence measurement. Concretely, to retain the data information, the mutual information is maximized between the variables for the high dimensional complex distributions and the low dimensional simple distributions. The departure of the resulting simple distributions are then measured in the original way of GANs. Additionally, for simplifying the optimization further, we optimize directly the lower bound for mutual information. Termed as SimpleGAN, we conduct the proposed approach over the several different baseline models, i.e., conventional GANs, DCGAN, WGAN-GP, WGAN-GP-res, and LSWGAN-GP on the benchmark CIFAR-10 and STL-10 datasets. SimpleGAN shows the obvious superiority on these baseline models. Furthermore, in comparison with the existing methods measuring directly the distribution departure in the high-dimensional space, our method clearly demonstrates its superiority. Finally, a series of experiments show the advantages of the proposed SimpleGAN.

EEG data augmentation for emotion recognition with a multiple generator conditional Wasserstein GAN

EEG-based emotion recognition has attracted substantial attention from researchers due to its extensive application prospects, and substantial progress has been made in feature extraction and classification modelling from EEG data. However, insufficient high-quality training data are available for building EEG-based emotion recognition models via machine learning or deep learning methods. The artificial generation of high-quality data is an effective approach for overcoming this problem. In this paper, a multi-generator conditional Wasserstein GAN method is proposed for the generation of high-quality artificial that covers a more comprehensive distribution of real data through the use of various generators. Experimental results demonstrate that the artificial data that are generated by the proposed model can effectively improve the performance of emotion classification models that are based on EEG.

DGSAN: Discrete generative self-adversarial network

Although GAN-based methods have received many achievements in the last few years, they have not been entirely successful in generating discrete data. The most crucial challenge of these methods is the difficulty of passing the gradient from the discriminator to the generator when the generator outputs are discrete. Despite the fact that several attempts have been made to alleviate this problem, none of the existing GAN-based methods have improved the performance of text generation compared with the maximum likelihood approach in terms of both the quality and the diversity. In this paper, we proposed a new framework for generating discrete data by an adversarial approach in which there is no need to pass the gradient to the generator. The proposed method has an iterative manner in which each new generator is defined based on the last discriminator. It leverages the discreteness of data and the last discriminator to model the real data distribution implicitly. Moreover, the method is supported with theoretical guarantees, and experimental results generally show the superiority of the proposed DGSAN method compared to the other popular or recent methods in generating discrete sequential data.

Gaussian quadrature method for GLD parameter estimation

Generalized lambda distribution (GLD) is a very flexible distribution that has many applications in estimating the distribution of real data. Its flexibility is due to its λs parameters that each of them somehow determine the form of distribution. Since the shape of distribution is very sensitive to the changes of each λs , the smallest error in estimating each parameter results in an error in other parameters, and then resulting in a total error in estimating the data distribution. One of the most accurate and widely used methods for estimating GLD parameters is the momentum method, which requires integral calculations. This paper uses Gaussian quadrature to integrate GLD equations. The results show that in terms of both percentile index and momentum comparison, the new method leads to more accurate calculation of the GLD parameters.

TWGAN: Twin Discriminator Generative Adversarial Networks

Generative Adversarial Networks (GAN) has become more and more popular these years. However, it is difficult to train and suffers from the training instability problem. To tackle this difficulty, this paper proposes a novel approach. Our idea is intuitive but proven to be very useful. In essence, it combines saturating loss and non-saturating loss into the loss function. Thus it will exploit the complementary statistical properties from two kinds of loss functions to effectively improve the training stability. We term our method twin discriminator Generative Adversarial Networks (TWGAN) which, unlike GAN, has a generator and a twin discriminator. The twin discriminator consists of two discriminators with identical architecture and both of them aim to distinguish whether the samples are from real data or fake data. We develop theoretical analysis to show that, given the optimal discriminators, optimizing the generator of TWGAN reduces to minimizing the Kullback-Leibler (KL) divergence between the distribution of generated data (Pg) and the distribution of real data (Pdata), hence effectively addressing the training instability problem. Extensive experiments on MNIST, Fashion MNIST, CIFAR-10/100 and STL-10 datasets demonstrate that the competitive performance of our TWGAN in generating good quality and diverse samples over baselines. The obtained highest inception score (IS) and lowest Frechet Inception Distance (FID), compared with other state-of-the-art GANs, show the superiority of our TWGAN.

SELF-COLLISION AVOIDANCE OF ARM ROBOT USING GENERATIVE ADVERSARIAL NETWORK AND PARTICLES SWARM OPTIMIZATION (GAN-PSO)

Collision avoidance of Arm Robot is designed for the robot to collide objects, colliding environment, and colliding its body. Self-collision avoidance was successfully trained using Generative Adversarial Networks (GANs) and Particle Swarm Optimization (PSO). The Inverse Kinematics (IK) with 96K motion data was extracted as the dataset to train data distribution of 3.6K samples and 7.2K samples. The proposed method GANs-PSO can solve the common GAN problem such as Mode Collapse or Helvetica Scenario that occurs when the generator always gets the same output point which mapped to different input values. The discriminator produces the random samples' data distribution in which present the real data distribution (generated by Inverse Kinematic analysis). The PSO was successfully reduced the number of training epochs of the generator only with 5000 iterations. The result of our proposed method (GANs-PSO) with 50 particles was 5000 training epochs executed in 0.028ms per single prediction and 0.027474% Generator Mean Square Error (GMSE).

Towards Sustainable Energy Efficiency With Intelligent Electricity Theft Detection in Smart Grids Emphasising Enhanced Neural Networks

In smart grids, electricity theft is the most significant challenge. It cannot be identified easily since existing methods are dependent on specific devices. Also, the methods lack in extracting meaningful information from high-dimensional electricity consumption data and increase the false positive rate that limit their performance. Moreover, imbalanced data is a hurdle in accurate electricity theft detection (ETD) using data driven methods. To address this problem, sampling techniques are used in the literature. However, the traditional sampling techniques generate insufficient and unrealistic data that degrade the ETD rate. In this work, two novel ETD models are developed. A hybrid sampling approach, i.e., synthetic minority oversampling technique with edited nearest neighbor, is introduced in the first model. Furthermore, AlexNet is used for dimensionality reduction and extracting useful information from electricity consumption data. Finally, a light gradient boosting model is used for classification purpose. In the second model, conditional wasserstein generative adversarial network with gradient penalty is used to capture the real distribution of the electricity consumption data. It is constructed by adding auxiliary provisional information to generate more realistic data for the minority class. Moreover, GoogLeNet architecture is employed to reduce the dataset's dimensionality. Finally, adaptive boosting is used for classification of honest and suspicious consumers. Both models are trained and tested using real power consumption data provided by state grid corporation of China. The proposed models' performance is evaluated using different performance metrics like precision, recall, accuracy, F1-score, etc. The simulation results prove that the proposed models outperform the existing techniques, such as support vector machine, extreme gradient boosting, convolution neural network, etc., in terms of efficient ETD.

Spectral–Spatial Attention Feature Extraction for Hyperspectral Image Classification Based on Generative Adversarial Network

Recent research shows that generative adversarial network (GAN) based deep learning derived frameworks can improve the accuracy of hyperspectral image (HSI) classification on limited labeled samples. However, several studies point out that existing GAN-based methods are heavily affected by the complexity and inefficient description issues of HSIs. The discriminator in GAN always attempts to interpret high-dimensional nonlinear spectral knowledge of HSIs, thus resulting in the Hughes phenomenon. Another critical issue is sample generation. The generator is only used as a regularizer for the discriminator, which seriously restricts the performance for classification. In this article, we propose SSAT-GAN, a semisupervised spectral–spatial attention feature extraction approach based on the GAN that feeds raw data into a deep learning framework, in an end-to-end fashion. First, the unlabeled data is added into the discriminator to alleviate the problems of training samples and supplies a reconstructed real HSI data distribution through adversarial training. Second, to enhance the description of HSIs, we build spectral–spatial attention modules (SSAT) and extend them to the discriminator and the generator to extract discriminative characteristics from abundant spatial contexts and spectral signatures. The SSAT modules learn a three-dimensional filter bank with spectral–spatial attention weights to obtain meaningful feature maps to improve the discrimination of the feature representation. In terms of the mode collapse of GANs, the mean minimization loss is employed for unsupervised learning. Experimental results from three real datasets indicate that SSAT-GAN has certain advantages over the state-of-the-art methods.

Wasserstein GAN: Deep Generation Applied on Financial Time Series

Modeling financial time series is challenging due to their high volatility and unexpected happenings on the market. Most financial models and algorithms trying to fill the lack of historical financial time series struggle to perform and are highly vulnerable to overfitting. As an alternative, we introduce in this paper a deep neural network called the WGAN-GP, a data-driven model that focuses on sample generation. The WGAN-GP consists of a generator and discriminator function which utilize an LSTM architecture. The WGAN-GP is supposed to learn the underlying structure of the input data, which in our case, is the Bitcoin. Bitcoin is unique in its behavior; the prices fluctuate what makes guessing the price trend hardly impossible. Through adversarial training, the WGAN-GP should learn the underlying structure of the bitcoin and generate very similar samples of the bitcoin distribution. The generated synthetic time series are visually indistinguishable from the real data. But the numerical results show that the generated data were close to the real data distribution but distinguishable. The model mainly shows a stable learning behavior. However, the model has space for optimization, which could be achieved by adjusting the hyperparameters.

Manifold-Based Nonlocal Second-Order Regularization for Hyperspectral Image Inpainting

The low-dimensional manifold of image patches has been introduced as regularizer term, and shown effective in hyperspectral image inpainting. However, in this article, we find that using only the low-dimensional property of manifold may not always generate smooth results. In terms of this, we first present a higher order term to the low-dimensional manifold model, namely nonlocal second-order regularization (NSR), which provides better approximation to the real data distribution and manifests both the properties of low dimensionality and smoothness. Moreover, in order to balance the known and unknown sets, we further propose a weighted version of NSR, called WNSR. The generalized minimal residual algorithm is adopted to solve this unsymmetrical model, in which a semi-patch is applied for acceleration of the nearest neighbor search. Finally, we conduct intensive numerical experiments on five well-known datasets to verify the superiority of our method. The inpainting results show that our proposed (W)NSR significantly outperforms the state-of-the-art methods with respect to both visual and numerical quality.

A Framework of Composite Functional Gradient Methods for Generative Adversarial Models.

Generative adversarial networks (GAN) are trained through a minimax game between a generator and a discriminator to generate data that mimics observations. While being widely used, GAN training is known to be empirically unstable. This paper presents a new theory for generative adversarial methods that does not rely on the traditional minimax formulation. Our theory shows that with a strong discriminator, a good generator can be obtained by composite functional gradient learning, so that several distance measures (including the KL divergence and the JS divergence) between the probability distributions of real data and generated data are simultaneously improved after each functional gradient step until converging to zero. This new point of view leads to stable procedures for training generative models. It also gives a new theoretical insight into the original GAN. Empirical results on image generation show the effectiveness of our new method.

Learning-Based Hyperspectral Imagery Compression through Generative Neural Networks

Hyperspectral images (HSIs), which obtain abundant spectral information for narrow spectral bands (no wider than 10 nm), have greatly improved our ability to qualitatively and quantitatively sense the Earth. Since HSIs are collected by high-resolution instruments over a very large number of wavelengths, the data generated by such sensors is enormous, and the amount of data continues to grow, HSI compression technique will play more crucial role in this trend. The classical method for HSI compression is through compression and reconstruction methods such as three-dimensional wavelet-based techniques or the principle component analysis (PCA) transform. In this paper, we provide an alternative approach for HSI compression via a generative neural network (GNN), which learns the probability distribution of the real data from a random latent code. This is achieved by defining a family of densities and finding the one minimizing the distance between this family and the real data distribution. Then, the well-trained neural network is a representation of the HSI, and the compression ratio is determined by the complexity of the GNN. Moreover, the latent code can be encrypted by embedding a digit with a random distribution, which makes the code confidential. Experimental examples are presented to demonstrate the potential of the GNN to solve image compression problems in the field of HSI. Compared with other algorithms, it has better performance at high compression ratio, and there is still much room left for improvements along with the fast development of deep-learning techniques.

The Odd Generalized NH Inverse Exponential Model: Theory and Application

In this paper we introduce a new lifetime distribution derived from odd generalized NH-G (OGNH-G) family technique called OGNH-inverse exponential (OGNHIE) distribution. We establish various mathematical properties. The maximum likelihood (ML) estimates for the OGNHIE parameters are derived. Finally the model is applied to a real dataset. We apply goodness of fit statistics and graphical tools to examine the adequacy of the OGNHIE distribution. The importance of this research lies in deriving a new distribution under the name OGNHIE, which is considered the best distributions in analyzing data of life times at present if compared to many distribution in analysis real data.