Real Data Distribution Research Articles

An Iterative Method for Unsupervised Robust Anomaly Detection Under Data Contamination.

Most deep anomaly detection models are based on learning normality from datasets due to the difficulty of defining abnormality by its diverse and inconsistent nature. Therefore, it has been a common practice to learn normality under the assumption that anomalous data are absent in a training dataset, which we call normality assumption. However, in practice, the normality assumption is often violated due to the nature of real data distributions that includes anomalous tails, i.e., a contaminated dataset. Thereby, the gap between the assumption and actual training data affects detrimentally in learning of an anomaly detection model. In this work, we propose a learning framework to reduce this gap and achieve better normality representation. Our key idea is to identify sample-wise normality and utilize it as an importance weight, which is updated iteratively during the training. Our framework is designed to be model-agnostic and hyperparameter insensitive so that it applies to a wide range of existing methods without careful parameter tuning. We apply our framework to three different representative approaches of deep anomaly detection that are classified into one-class classification-, probabilistic model-, and reconstruction-based approaches. In addition, we address the importance of a termination condition for iterative methods and propose a termination criterion inspired by the anomaly detection objective. We validate that our framework improves the robustness of the anomaly detection models under different levels of contamination ratios on five anomaly detection benchmark datasets and two image datasets. On various contaminated datasets, our framework improves the performance of three representative anomaly detection methods, measured by area under the ROC curve.

Adaptive Learning of the Latent Space of Wasserstein Generative Adversarial Networks

Generative models based on latent variables, such as generative adversarial networks (GANs) and variational auto-encoders (VAEs), have gained lots of interests due to their impressive performance in many fields. However, many data such as natural images usually do not populate the ambient Euclidean space but instead reside in a lower-dimensional manifold. Thus an inappropriate choice of the latent dimension fails to uncover the structure of the data, possibly resulting in mismatch of latent representations and poor generative qualities. Towards addressing these problems, we propose a novel framework called the latent Wasserstein GAN (LWGAN) that fuses the Wasserstein auto-encoder and the Wasserstein GAN so that the intrinsic dimension of the data manifold can be adaptively learned by a modified informative latent distribution. We prove that there exist an encoder network and a generator network in such a way that the intrinsic dimension of the learned encoding distribution is equal to the dimension of the data manifold. We theoretically establish that our estimated intrinsic dimension is a consistent estimate of the true dimension of the data manifold. Meanwhile, we provide an upper bound on the generalization error of LWGAN, implying that we force the synthetic data distribution to be similar to the real data distribution from a population perspective. Comprehensive empirical experiments verify our framework and show that LWGAN is able to identify the correct intrinsic dimension under several scenarios, and simultaneously generate high-quality synthetic data by sampling from the learned latent distribution.

Generative models for synthetic data generation: application to pharmacokinetic/pharmacodynamic data.

The generation of synthetic patient data that reflect the statistical properties of real data plays a fundamental role in today's world because of its potential to (i) be enable proprietary data access for statistical and research purposes and (ii) increase available data (e.g., in low-density regions-i.e., for patients with under-represented characteristics). Generative methods employ a family of solutions for generating synthetic data. The objective of this research is to benchmark numerous state-of-the-art deep-learning generative methods across different scenarios and clinical datasets comprising patient covariates and several pharmacokinetic/pharmacodynamic endpoints. We did this by implementing various probabilistic models aimed at generating synthetic data, such as the Multi-layer Perceptron Conditioning Generative Adversarial Neural Network (MLP cGAN), Time-series Generative Adversarial Networks (TimeGAN), and a more traditional approach like Probabilistic Autoregressive (PAR). We evaluated their performance by calculating discriminative and predictive scores. Furthermore, we conducted comparisons between the distributions of real and synthetic data using Kolmogorov-Smirnov and Chi-square statistical tests, focusing respectively on covariate and output variables of the models. Lastly, we employed pharmacometrics-related metric to enhance interpretation of our results specific to our investigated scenarios. Results indicate that multi-layer perceptron-based conditional generative adversarial networks (MLP cGAN) exhibit the best overall performance for most of the considered metrics. This work highlights the opportunities to employ synthetic data generation in the field of clinical pharmacology for augmentation and sharing of proprietary data across institutions.

Data-Driven Generative Model Aimed to Create Synthetic Data for the Long-Term Forecast of Gas Turbine Operation

Abstract The prediction of gas turbine (GT) future health state plays a strategic role in the current energy sector. However, training an accurate prognostic model is challenging in case of limited historical data (e.g., new installation). Thus, this paper develops a Generative Adversarial Network (GAN) model aimed to generate synthetic data that can be used for data augmentation. The GAN model includes two neural networks, i.e., a generator and a discriminator. The generator aims to generate synthetic data that mimic the real data. The discriminator is a binary classification network. During the training process, the generator is optimized to fool the discriminator in distinguishing between real and synthetic data. The real data employed in this paper were taken from the literature, gathered from three GTs, and refer to two quantities, i.e., corrected power output and compressor efficiency, which are tracked during several years. Three different analyses are presented to validate the reliability of the synthetic dataset. First, a visual comparison of real and synthetic data is performed. Then, two metrics are employed to quantitively evaluate the similarity between real and synthetic data distributions. Finally, a prognostic model is trained by only using synthetic data and then employed to predict real data. The results prove the high reliability of the synthetic data, which can be thus exploited to train a prognostic model. In fact, the prediction error of the prognostic model on the real data is lower than 2.5% even in the case of long-term prediction.

A metaheuristic algorithmic framework for solving the hybrid flow shop scheduling problem with unrelated parallel machines

The hybrid flow shop scheduling problem (HFSP), as a realistic extension of the classical flow shop scheduling problem, widely exists in real-world industrial production systems. In practice, the fact that machines are unrelated is important and cannot be neglected. This study focuses on the HFSP with unrelated parallel machines (HFSP-UPM) to minimize the makespan. To address this problem, a hybrid representation that combines single-sequence coding and full-sequence coding is developed to search for solution space that is not covered by common encoding methods. An initialization block integrating random heuristic strategies is proposed for improving the quality of the initial sparrow swarm. To improve the diversity of a sparrow swarm further, a perturbation block embedded with a set of historical best positions and an enhancement strategy are developed. A critical set based local search block is designed for high-intensity local exploitation of promising regions when necessary. Several benchmark cases from the literature as well as some randomly generated instances characterized by the distribution of real data from factory studies are employed to participate in the test. The test results reveal the effectiveness of the proposed perturbation block and local search block. Compared to state-of-the-art metaheuristic algorithms, the enhanced sparrow search algorithm (ESSA) demonstrates higher convergence accuracy when handling instances. The value of the gap between a solution found by the ESSA and a feasible solution found by the mixed-integer programming (MIP) model can reach 0.6%.

Anomaly prediction of Internet behavior based on generative adversarial networks.

With the popularity of Internet applications, a large amount of Internet behavior log data is generated. Abnormal behaviors of corporate employees may lead to internet security issues and data leakage incidents. To ensure the safety of information systems, it is important to research on anomaly prediction of Internet behaviors. Due to the high cost of labeling big data manually, an unsupervised generative model-Anomaly Prediction of Internet behavior based on Generative Adversarial Networks (APIBGAN), which works only with a small amount of labeled data, is proposed to predict anomalies of Internet behaviors. After the input Internet behavior data is preprocessed by the proposed method, the data-generating generative adversarial network (DGGAN) in APIBGAN learns the distribution of real Internet behavior data by leveraging neural networks' powerful feature extraction from the data to generate Internet behavior data with random noise. The APIBGAN utilizes these labeled generated data as a benchmark to complete the distance-based anomaly prediction. Three categories of Internet behavior sampling data from corporate employees are employed to train APIBGAN: (1) Online behavior data of an individual in a department. (2) Online behavior data of multiple employees in the same department. (3) Online behavior data of multiple employees in different departments. The prediction scores of the three categories of Internet behavior data are 87.23%, 85.13%, and 83.47%, respectively, and are above the highest score of 81.35% which is obtained by the comparison method based on Isolation Forests in the CCF Big Data & Computing Intelligence Contest (CCF-BDCI). The experimental results validate that APIBGAN predicts the outlier of Internet behaviors effectively through the GAN, which is composed of a simple three-layer fully connected neural networks (FNNs). We can use APIBGAN not only for anomaly prediction of Internet behaviors but also for anomaly prediction in many other applications, which have big data infeasible to label manually. Above all, APIBGAN has broad application prospects for anomaly prediction, and our work also provides valuable input for anomaly prediction-based GAN.

Challenges of Using Synthetic Data Generation Methods for Tabular Microdata

The generation of synthetic data holds significant promise for augmenting limited datasets while avoiding privacy issues, facilitating research, and enhancing machine learning models’ robustness. Generative Adversarial Networks (GANs) stand out as promising tools, employing two neural networks—generator and discriminator—to produce synthetic data that mirrors real data distributions. This study evaluates GAN variants (CTGAN, CopulaGAN), a variational autoencoder, and copulas on diverse real datasets of different complexity encompassing numerical and categorical attributes. The results highlight CTGAN’s sensitivity to training parameters and TVAE’s robustness across datasets. Scalability challenges persist, with GANs demanding substantial computational resources. TVAE stands out for its high utility across all datasets, even for high-dimensional data, though it incurs higher privacy risks, which is indicative of the curse of dimensionality. While no single model universally excels, understanding the trade-offs and leveraging model strengths can significantly enhance synthetic data generation (SDG). Future research should focus on adaptive learning mechanisms, scalability enhancements, and standardized evaluation metrics to advance SDG methods effectively. Addressing these challenges will foster broader adoption and application of synthetic data.

Generative adversarial networks for multi-fidelity matrix completion with massive missing entries

Multi-fidelity matrices refer to a pair of data matrices, whose entries are the measurements of a specific physical quantity at different fidelity levels w.r.t. two environment variables arranged as the row and the column of matrices, respectively. Although a high-fidelity (HF) matrix can provide an accurate description of the physical quantity, it usually has a number of missing entries because of high acquisition cost. In contrast, its low-fidelity (LF) counterpart is complete but the accuracy of its entries cannot meet the actual requirement. Multi-fidelity matrix completion (MMC) aims to complete the missing entries of the HF matrix by fusing the LF matrix and the available entries of the HF matrix. In this paper, we develop a generative adversarial network for multi-fidelity matrix completion (GAN-MMC), which is composed of a generator and a discriminator. The generator input is the LF matrix and its output is the approximation of the complete HF matrix. The discriminator identifies whether the HF approximation completed by the generator accords with the real HF data distribution. A matrix data augmentation method is proposed to enrich the samples for training GAN-MMC. Moreover, a correlation coefficient-based term is also imposed into the generative loss in order to make the generator capture the relatedness information between the available HF entries and the corresponding LF entries. The numerical experiments show that the proposed model can effectively complete the HF matrix with massive missing entries.

InfoCAVB-MemoryFormer: Forecasting of wind and photovoltaic power through the interaction of data reconstruction and data augmentation

Rare or missing data pose significant challenges in the prediction of wind power (WP) and photovoltaic power (PV). Many methods address the data scarcity issue solely through augmentation techniques, often neglecting the impact of missing data on the augmentation process. When data augmentation is performed on missing datasets, the prediction accuracy cannot be further improved, and this is called as an extreme data scarcity problem. To solve this problem, we introduce two methods, called Information Maximizing Collaborative Adversarial Variational Bayes (InfoCAVB) and MemoryFormer, to achieve data augmentation based on missing data reconstruction. In this paper, the augmentation and reconstruction process are performed at the same time. When the reconstruction process is performed, InfoCAVB utilizes adversarial training to construct variational bayes that approximates the posterior distribution of real data to recover missing data. Meanwhile, in the augmentation process, InfoCAVB maximizes the mutual information between real data and reconstruction data to establish correspondences between specific dimensions of augmentation data and the features of real and reconstruction data, respectively. The advantage of InfoCAVB lies in incorporating data augmentation into data reconstruction through the information maximizing principle. Finally, we propose a MemoryFormer adapted for InfoCAVB to predict WP and PV, and the benefit of MemoryFormer lies in excavating the potential temporal correlations between reconstructed and augmented data. MemoryFormer embeds the distribution of real data into the generated data through memory units, ensuring consistent distribution during the training process of discrete reconstructed and augmented data. Experimental results indicate that the proposed InfoCAVB-MemoryFormer reduces the RMSE averages by 15.7%, 14.1%, and 5.92% for WP prediction and 28.23%, 22.67%, and 12.21% for PV prediction compared to other state-of-the-art model models, demonstrating the effectiveness of the proposed approach in extreme data scarcity scenarios.

Adaptive denoising autoencoder for robust fault detection

In real chemical processes, the collected data is often subject to interference from ambient environmental noise, resulting in a decline in the detection performance. Although denoising autoencoder can realize robust fault detection to a certain extent, its performance is limited by the addition of artificial noise, which refers to Gaussian noise or the dropout noise. Moreover, the artificial noise added is often far from the distribution of real data, making it difficult to suppress the noise in real data. To address this issue, an adaptive denoising autoencoder (ADAE) based on the diffusion model is proposed. ADAE adopts a chain-like structure, concatenating traditional DAEs by replacing the manually injected noise in each DAE block with noise generated by diffusion model at various diffusion steps. In order to better capture the key features of the original process data, a novel noise prediction network is proposed within the diffusion model, utilizing dilated and gated convolutions to capture the dynamic information of the process data. Ultimately, adaptation is achieved through a chain of DAEs, suppressing noise similar to the distribution of the original data. The superiority of ADAE in noise suppression is then verified through numerical examples and the Tennessee Eastman process (TEP), comparing it with other methods.

Skeleton-aware Graph-based Adversarial Networks for Human Pose Estimation from Sparse IMUs

Recently, sparse-inertial human pose estimation (SI-HPE) with only a few IMUs has shown great potential in various fields. The most advanced work in this area achieved fairish results using only six IMUs. However, there are still two major issues that remain to be addressed. First, existing methods typically treat SI-HPE as a temporal sequential learning problem and often ignore the important spatial prior of skeletal topology. Second, there are far more synthetic data in their training data than real data, and the data distribution of synthetic data and real data is quite different, which makes it difficult for the model to be applied to more diverse real data. To address these issues, we propose “Graph-based Adversarial Inertial Poser (GAIP)”, which tracks body movements using sparse data from six IMUs. To make full use of the spatial prior, we design a multi-stage pose regressor with graph convolution to explicitly learn the skeletal topology. A joint position loss is also introduced to implicitly mine spatial information. To enhance the generalization ability, we propose supervising the pose regression with an adversarial loss from a discriminator, bringing the ability of adversarial networks to learn implicit constraints into full play. Additionally, we construct a real dataset that includes hip support movements and a synthetic dataset containing various motion categories to enrich the diversity of inertial data for SI-HPE. Extensive experiments demonstrate that GAIP produces results with more precise limb movement amplitudes and relative joint positions, accompanied by smaller joint angle and position errors compared to state-of-the-art counterparts. The datasets and codes are publicly available at https://cslinzhang.github.io/GAIP/ .

A Direct Approach for Local Quasi-Geoid Modeling Based on Spherical Radial Basis Functions Using a Noisy Satellite-Only Global Gravity Field Model

A Direct Approach for Local Quasi-Geoid Modeling Based on Spherical Radial Basis Functions Using a Noisy Satellite-Only Global Gravity Field Model

Power quality disturbances classification with imbalanced/insufficient samples based on WGAN-GP-SA and DCNN

The accurate identification of power quality disturbance signals (PQDs) is of great importance to ensure the safe and stable operation of power systems. Considering that there are insufficient samples and imbalance samples of power quality disturbance signals in real systems, the hybrid classification method based on WGAN-GP-SA (Wasserstein generative adversarial networks with gradient penalty and Squeeze-and-Excitation block with atrous spatial pyramid pooling module) and DCNN (deep convolutional neural network) is proposed for recognition of PQDs. Firstly, time-frequency features are extracted by combining improved Kalman and continuous wavelet transforms (CWT); secondly, the original training time-frequency maps are expanded by using WGAN-GP-SA, which can learn the distribution of real data and generate similar samples, and PSNR, SSIM, FID and other indexes are introduced for evaluation of generated data; finally, classification experiments are conducted on two data sets using DCNN, and simulation results show that the accuracy is improved by 2.82 % compared with the original data without augmentation. Furthermore, t-distributed stochastic neighbor embedding algorithm (t-SNE) is introduced for visually analysing of classification performance of various datasets, and the visualization indicates the superiority of the hybrid classification method.

Score mismatching for generative modeling

We propose a new score-based model with one-step sampling. Previously, score-based models were burdened with heavy computations due to iterative sampling. For substituting the iterative process, we train a standalone generator to compress all the time steps with the gradient backpropagated from the score network. In order to produce meaningful gradients for the generator, the score network is trained to simultaneously match the real data distribution and mismatch the fake data distribution. This model has the following advantages: (1) For sampling, it generates a fake image with only one step forward. (2) For training, it only needs 10 diffusion steps. (3) Compared with consistency model, it is free of the ill-posed problem caused by consistency loss. On the popular CIFAR-10 dataset, our model outperforms Consistency Model and Denoising Score Matching, which demonstrates the potential of the framework. We further provide more examples on the MINIST and LSUN datasets. The code is available on GitHub.

Iterative and mixed-spaces image gradient inversion attack in federated learning

As a distributed learning paradigm, federated learning is supposed to protect data privacy without exchanging users’ local data. Even so, the gradient inversion attack, in which the adversary can reconstruct the original data from shared training gradients, has been widely deemed as a severe threat. Nevertheless, most existing researches are confined to impractical assumptions and narrow range of applications. To mitigate these shortcomings, we propose a comprehensive framework for gradient inversion attack, with well-designed algorithms for image and label reconstruction. For image reconstruction, we fully utilize the generative image prior, which derives from wide-used generative models, to improve the reconstructed results, by additional means of iterative optimization on mixed spaces and gradient-free optimizer. For label reconstruction, we design an adaptive recovery algorithm regarding real data distribution, which can adjust previous attacks to more complex scenarios. Moreover, we incorporate a gradient approximation method to efficiently fit our attack for FedAvg scenario. We empirically verify our attack framework using benchmark datasets and ablation studies, considering loose assumptions and complicated circumstances. We hope this work can greatly reveal the necessity of privacy protection in federated learning, while urge more effective and robust defense mechanisms.

A roundtrip probability estimation method for mechanical equipment fault detection under imbalanced samples

Aiming at high misdetection of mechanical faults under imbalanced samples, a roundtrip probability-based method is proposed. By roundtrip mapping between latent variables and real fault data, biased estimation of the probability distribution of real fault data is obtained. Further, virtual fault data are sampled according to such distribution to increase sample amount. For recognition of real and virtual data, loss function based on binary cross-entropy is designed. For reconstruction between fault data and its roundtrip mapped results, objective function based on mean square error is designed. Thus, it preserves boundary data and avoids too many virtual data in central area. Meanwhile, a strategy for eliminating abnormal samples is designed to reduce boundary deviation. For supporting the advantage of roundtrip, in-depth reasons for misdetection are analyzed from empirical risk and structural risk. Experiments on 30 benchmark imbalanced test sets show that fault detection rate increases after amount enhancement. Additionally, it is verified on blade cracking and bearing fault detection. Results show that F1 score increases from 0.485 to 0.51 and 0.725 to 0.775 for such two cases.

A Model-Based Reinforcement Learning Method with Conditional Variational Auto-Encoder

Model-based reinforcement learning can effectively improve the sample efficiency of reinforcement learning, but the environment model in this method has errors. The model errors can mislead the policy optimization, leading to suboptimal policy. To improve the generalization ability of the environment model, existing methods often use ensemble models or Bayesian models to build the environment model. However, these methods are computationally intensive and complex to update. Since the generated model can describe the stochastic nature of the environment, this paper proposes a model-based reinforcement learning method based on a conditional variational auto-encoder. In this paper, we use a conditional variational auto-encoder to learn task-related representations and apply the generative model to predict environmental changes. Considering the problem of multi-step error accumulation, model adaptation is utilized to minimize the difference between simulated and real data distributions. Furthermore, the experiments verified that the proposed method can learn task-relevant representations and accelerate policy learning.

Data Augmentation of a Corrosion Dataset for Defect Growth Prediction of Pipelines Using Conditional Tabular Generative Adversarial Networks.

Due to corrosion characteristics, there are data scarcity and uneven distribution in corrosion datasets, and collecting high-quality data is time-consuming and sometimes difficult. Therefore, this work introduces a novel data augmentation strategy using a conditional tabular generative adversarial network (CTGAN) for enhancing corrosion datasets of pipelines. Firstly, the corrosion dataset is subjected to data cleaning and variable correlation analysis. The CTGAN is then used to generate external environmental factors as input variables for corrosion growth prediction, and a hybrid model based on machine learning is employed to generate corrosion depth as an output variable. The fake data are merged with the original data to form the synthetic dataset. Finally, the proposed data augmentation strategy is verified by analyzing the synthetic dataset using different visualization methods and evaluation indicators. The results show that the synthetic and original datasets have similar distributions, and the data augmentation strategy can learn the distribution of real corrosion data and sample fake data that are highly similar to the real data. Predictive models trained on the synthetic dataset perform better than predictive models trained using only the original dataset. In comparative tests, the proposed strategy outperformed other data generation methods.

Distributed Traffic Synthesis and Classification in Edge Networks: A Federated Self-Supervised Learning Approach

With the rising demand for wireless services and increased awareness of the need for data protection, existing network traffic analysis and management architectures are facing unprecedented challenges in classifying and synthesizing the increasingly diverse services and applications. This paper proposes FS-GAN, a federated self-supervised learning framework to support automatic traffic analysis and synthesis over a large number of heterogeneous datasets. FS-GAN is composed of multiple distributed Generative Adversarial Networks (GANs), with a set of generators, each being designed to generate synthesized data samples following the distribution of an individual service traffic, and each discriminator being trained to differentiate the synthesized data samples and the real data samples of a local dataset. A federated learning-based framework is adopted to coordinate local model training processes of different GANs across different datasets. FS-GAN can classify data of unknown types of service and create synthetic samples that capture the traffic distribution of the unknown types. We prove that FS-GAN can minimize the Jensen-Shannon Divergence (JSD) between the distribution of real data across all the datasets and that of the synthesized data samples. FS-GAN also maximizes the JSD among the distributions of data samples created by different generators, resulting in each generator producing synthetic data samples that follow the same distribution as one particular service type. Extensive simulation results show that the classification accuracy of FS-GAN achieves over 20% improvement in average compared to the state-of-the-art clustering-based traffic analysis algorithms. FS-GAN also has the capability to synthesize highly complex mixtures of traffic types without requiring any human-labeled data samples.

A bearing fault diagnosis method with an improved residual Unet diffusion model under extreme data imbalance

As a vital constituent of rotating machinery, rolling bearings assume a pivotal function in ensuring the stable operation of equipment. Recently, deep learning (DL)-based methods have been able to diagnose bearing faults accurately. However, in practical applications, the severe data imbalance caused by the limited availability of fault data compared to the abundance of healthy data poses challenges to the effective training of DL models, leading to a decrease in diagnostic accuracy. In this paper, a bearing fault diagnosis method with the improved residual Unet diffusion model (IResUnet-DM) based on a data generation strategy is proposed to solve the extreme data imbalance. Initially, a deep feature extraction network named improved residual Unet is built, which effectively enhances the information learning ability from vibration signals of the Unet network by one-dimensional residual block and self-attention block. Furthermore, the IResUnet-DM is constructed, which generates vibration signals under extreme data imbalance based on a probability model. The variational bound on the negative log-likelihood of the distribution of generated data was optimized to make the generated data similar to the real data distribution. Finally, wide deep convolutional neural network and one-dimensional ResNet classification networks were used for fault identification to verify the validity and generalization of the IResUnet-DM. Experiment results at different data imbalance rates on two bearing datasets demonstrate that the proposed method can effectively improve fault diagnosis accuracy under extreme data imbalances and outperform the comparison method.