Wasserstein Autoencoder Research Articles

Adaptive Learning of the Latent Space of Wasserstein Generative Adversarial Networks

Generative models based on latent variables, such as generative adversarial networks (GANs) and variational auto-encoders (VAEs), have gained lots of interests due to their impressive performance in many fields. However, many data such as natural images usually do not populate the ambient Euclidean space but instead reside in a lower-dimensional manifold. Thus an inappropriate choice of the latent dimension fails to uncover the structure of the data, possibly resulting in mismatch of latent representations and poor generative qualities. Towards addressing these problems, we propose a novel framework called the latent Wasserstein GAN (LWGAN) that fuses the Wasserstein auto-encoder and the Wasserstein GAN so that the intrinsic dimension of the data manifold can be adaptively learned by a modified informative latent distribution. We prove that there exist an encoder network and a generator network in such a way that the intrinsic dimension of the learned encoding distribution is equal to the dimension of the data manifold. We theoretically establish that our estimated intrinsic dimension is a consistent estimate of the true dimension of the data manifold. Meanwhile, we provide an upper bound on the generalization error of LWGAN, implying that we force the synthetic data distribution to be similar to the real data distribution from a population perspective. Comprehensive empirical experiments verify our framework and show that LWGAN is able to identify the correct intrinsic dimension under several scenarios, and simultaneously generate high-quality synthetic data by sampling from the learned latent distribution.

Towards a configurable and non-hierarchical search space for NAS

Neural Architecture Search (NAS) outperforms handcrafted Neural Network (NN) design. However, current NAS methods generally use hard-coded search spaces, and predefined hierarchical architectures. As a consequence, adapting them to a new problem can be cumbersome, and it is hard to know which of the NAS algorithm or the predefined hierarchical structure impacts performance the most. To improve flexibility, and be less reliant on expert knowledge, this paper proposes a NAS methodology in which the search space is easily customizable, and allows for full network search. NAS is performed with Gaussian Process (GP)-based Bayesian Optimization (BO) in a continuous architecture embedding space. This embedding is built upon a Wasserstein Autoencoder, regularized by both a Maximum Mean Discrepancy (MMD) penalization and a Fully Input Convex Neural Network (FICNN) latent predictor, trained to infer the parameter count of architectures. This paper first assesses the embedding’s suitability for optimization by solving 2 computationally inexpensive problems: minimizing the number of parameters, and maximizing a zero-shot accuracy proxy. Then, two variants of complexity-aware NAS are performed on CIFAR-10 and STL-10, based on two different search spaces, providing competitive NN architectures with limited model sizes.

Bridging spherical mixture distributions and word semantic knowledge for Neural Topic Modeling

Neural Topic Modeling has attracted significant attention from the Natural Language Processing community due to its black-box inference property and has made some progress. However, existing approaches have three main drawbacks: (1). hard to integrate external semantic knowledge to improve topic quality; (2). the posterior collapse issue of the Variational Auto-Encoder (VAE) harms the topic diversity; (3). cannot capture the semantic scopes of extracted topics. To address these limitations jointly, we model topics with a mixture of von Mises-Fisher distributions to incorporate word semantic knowledge and propose the Spherical Embedded Topic (SET) model based on the Wasserstein Auto-Encoder. Experiments have been conducted on three widely used text corpora, and the results show that SET obtains more coherent and diverse topics, outperforming the state-of-the-art approaches. Moreover, the ability of SET to capture semantic scopes of topics is also validated by our experiments.

Out-of-Domain Generalization From a Single Source: An Uncertainty Quantification Approach.

We are concerned with a worst-case scenario in model generalization, in the sense that a model aims to perform well on many unseen domains while there is only one single domain available for training. We propose Meta-Learning based Adversarial Domain Augmentation to solve this Out-of-Domain generalization problem. The key idea is to leverage adversarial training to create "fictitious" yet "challenging" populations, from which a model can learn to generalize with theoretical guarantees. To facilitate fast and desirable domain augmentation, we cast the model training in a meta-learning scheme and use a Wasserstein Auto-Encoder to relax the widely used worst-case constraint. We further improve our method by integrating uncertainty quantification for efficient domain generalization. Extensive experiments on multiple benchmark datasets indicate its superior performance in tackling single domain generalization.

Uncertainty-Aware Heterogeneous Representation Learning in POI Recommender Systems

Discovering interesting yet unvisited point-of-interests (POIs) is among the most practical applications but challenging problems in location-based social networks (LBSNs). Popular approaches face several issues, such as data sparsity and difficulties in modeling latent nonlinearity between users and POIs. Furthermore, the uncertainty in LBSNs poses additional obstacles to learning good representations of users’ general and current interests. To effectively address these issues, we postulate that fusing multiple sources of information is paramount. Toward that, we propose a novel deep generative recommender system—Wasserstein autoencoder for POI recommendation (WaPOIR). It unifies the information from users’ personal preference, social influence, and geographical data, and captures users’ general interests from historical check-ins, while modeling users’ current interests from recently visited POIs. Unlike previous methods, WaPOIR learns the latent distribution of data in the Wasserstein space as a potential representation for each POI and each user in LBSNs. This enables simultaneous maintenance of social and POI interactions and modeling the uncertainty of their relationships. WaPOIR is a stochastic recommendation approach that allows Bayesian inference and approximation of variational posterior distribution. Extensive experiments conducted on real-world LBSN datasets demonstrate that WaPOIR achieves better performance over the state-of-the-art approaches.

Bottlenecks CLUB: Unifying Information-Theoretic Trade-Offs Among Complexity, Leakage, and Utility

Bottleneck problems are an important class of optimization problems that have recently gained increasing attention in the domain of machine learning and information theory. They are widely used in generative models, fair machine learning algorithms, design of privacy-assuring mechanisms, and appear as information-theoretic performance bounds in various multi-user communication problems. In this work, we propose a general family of optimization problems, termed as <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">complexity-leakage-utility bottleneck (CLUB)</i> model, which (i) provides a unified theoretical framework that generalizes most of the state-of-the-art literature for the information-theoretic privacy models, (ii) establishes a new interpretation of the popular generative and discriminative models, (iii) constructs new insights for the generative compression models, and (iv) can be used to obtain fair generative models. We first formulate the CLUB model as a complexity-constrained privacy-utility optimization problem. We then connect it with the closely related bottleneck problems, namely information bottleneck (IB), privacy funnel (PF), deterministic IB (DIB), conditional entropy bottleneck (CEB), and conditional PF (CPF). We show that the CLUB model generalizes all these problems as well as most other information-theoretic privacy models. Then, we construct the deep variational CLUB (DVCLUB) models by employing neural networks to parameterize variational approximations of the associated information quantities. Building upon these information quantities, we present unified objectives of the <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">supervised</i> and <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">unsupervised</i> DVCLUB models. Leveraging the DVCLUB model in an unsupervised setup, we then connect it with state-of-the-art generative models, such as variational auto-encoders (VAEs), generative adversarial networks (GANs), as well as the Wasserstein GAN (WGAN), Wasserstein auto-encoder (WAE), and adversarial auto-encoder (AAE) models through the optimal transport (OT) problem. We then show that the DVCLUB model can also be used in fair representation learning problems, where the goal is to mitigate the undesired bias during the training phase of a machine learning model. We conduct extensive quantitative experiments on colored-MNIST and CelebA datasets.

Graph Wasserstein Autoencoder-Based Asymptotically Optimal Motion Planning With Kinematic Constraints for Robotic Manipulation

This paper presents a learning based motion planning method for robotic manipulation, aiming to solve the asymptotically-optimal motion planning problem with nonlinear kinematics in a complex environment. The core of the proposed method is based on a novel neural network model, i.e., graph wasserstein autoencoder (GraphWAE) network, which is used to represent the implicit sampling distributions of the configuration space (C-space) for sampling-based planning algorithms. Through learning the implicit distributions, we can guide the planning process to search or extend in the desired region to reduce the collision checks dramatically for fast and high-quality motion planning. The theoretical analysis and proofs are given to demonstrate the probabilistic completeness and asymptotic optimality of the proposed method. Numerical simulations and experiments are conducted to validate the effectiveness of the proposed method through a series of planning problems from 2D, 6D and 12D robot C-spaces in the challenging scenes. Results indicate that the proposed method can achieve better planning performance than the state-of-the-art planning algorithms. Note to Practitioners—The motivation of this work is to develop a fast and high-quality asymptotically optimal motion planning method for practical applications such as autonomous driving, robotic manipulation and others. Due to the time consumption caused by collision detection, current planning algorithms usually take much time to converge to the optimal motion path especially in the complicated environment. In this paper, we present a neural network model based on GraphWAE to learn the biasing sampling distributions as the sample generation source to further reduce or avoid collision checks of sampling-based planning algorithms. The proposed method is general and can be also deployed in other sampling-based planning algorithms for improving planning performance in different robot applications.

A novel conditional weighting transfer Wasserstein auto-encoder for rolling bearing fault diagnosis with multi-source domains

Transfer learning based on a single source domain to a target domain has received a lot of attention in the cross-domain fault diagnosis tasks of rolling bearing. However, the practical issues often contain multiple source domain data, and the information contained in the target domain is quite different from one source domain to another. Therefore, the transfer pattern from multiple source domains to the target domain undoubtedly has brighter application prospects. Based on these discussions, a multi-source domain transfer learning approach called conditional weighting transfer Wasserstein auto-encoder is developed to deal with the challenges of cross-domain fault diagnosis. Different from the traditional distribution alignment idea of directly aligning the source and target domains, the proposed framework adopts an indirect latent alignment idea to achieve better feature alignment, that is, indirectly aligning the feature distribution of source and target in the latent feature space with the help of Gaussian prior distribution. Furthermore, considering the variability of different source domains containing information about the target domain, an ingenious conditional weighting strategy is designed to quantify the similarity of different source domains to target domain, and further help the proposed model to minimize the discrepancy in conditional distribution. The cross-domain fault diagnosis tasks adequately verify that the proposed framework can sufficiently transfer knowledge from all source domains to the target domain, and has extensive application prospects.

Accelerating the discovery of anticancer peptides targeting lung and breast cancers with the Wasserstein autoencoder model and PSO algorithm.

In the development of targeted drugs, anticancer peptides (ACPs) have attracted great attention because of their high selectivity, low toxicity and minimal non-specificity. In this work, we report a framework of ACPs generation, which combines Wasserstein autoencoder (WAE) generative model and Particle Swarm Optimization (PSO) forward search algorithm guided by attribute predictive model to generate ACPs with desired properties. It is well known that generative models based on Variational AutoEncoder (VAE) and Generative Adversarial Networks (GAN) are difficult to be used for de novo design due to the problems of posterior collapse and difficult convergence of training. Our WAE-based generative model trains more successfully (lower perplexity and reconstruction loss) than both VAE and GAN-based generative models, and the semantic connections in the latent space of WAE accelerate the process of forward controlled generation of PSO, while VAE fails to capture this feature. Finally, we validated our pipeline on breast cancer targets (HIF-1) and lung cancer targets (VEGR, ErbB2), respectively. By peptide-protein docking, we found candidate compounds with the same binding sites as the peptides carried in the crystal structure but with higher binding affinity and novel structures, which may be potent antagonists that interfere with these target-mediated signaling.

The short texts classification based on neural network topic model

Aiming at the low effectiveness of short texts feature extraction, this paper proposes a short texts classification model based on the improved Wasserstein-Latent Dirichlet Allocation (W-LDA), which is a neural network topic model based on the Wasserstein Auto-Encoder (WAE) framework. The improvements of W-LDA are as follows: Firstly, the Bag of Words (BOW) input in the W-LDA is preprocessed by Term Frequency–Inverse Document Frequency (TF-IDF); Subsequently, the prior distribution of potential topics in W-LDA is replaced from the Dirichlet distribution to the Gaussian mixture distribution, which is based on the Variational Bayesian inference; And then the sparsemax function layer is introduced after the hidden layer inferred by the encoder network to generate a sparse document-topic distribution with better topic relevance, the improved W-LDA is named the Sparse Wasserstein-Variational Bayesian Gaussian mixture model (SW-VBGMM); Finally, the document-topic distribution generated by SW-VBGMM is input to BiGRU (Bidirectional Gating Recurrent Unit) for the deep feature extraction and the short texts classification. Experiments on three Chinese short texts datasets and one English dataset represent that our model is better than some common topic models and neural network models in the four evaluation indexes (accuracy, precision, recall, F1 value) of text classification.

A new data generation approach with modified Wasserstein auto-encoder for rotating machinery fault diagnosis with limited fault data

Limited fault data restrict deep learning methods in solving fault diagnosis problems in rotating machinery. Using limited fault data to generate massive data with similar distributions is an important premise in applying deep learning methods to solve these problems. In this study, a new data generation approach is designed using a modified Wasserstein auto-encoder (MWAE) to generate data that are highly similar to the known data. The sliced Wasserstein distance is introduced to measure the distribution difference. A squeeze-and-excitation attention mechanism is developed for the encoder to extract more representative features from limited data. The sliced Wasserstein distance with a gradient penalty is designed as the regularization term to minimize the difference between the posterior distribution QZ obtained by the encoder and the predefined Gaussian prior distribution PZ. A dataset of simulated high-speed aeronautical bearings is used to illustrate the data generation ability of the proposed method; a comparison with results obtained using other methods confirms its excellent data generation ability.

Sliced Wasserstein based Canonical Correlation Analysis for Cross-Domain Recommendation

• A cross-domain recommendation model based on Sliced Wasserstein autoencoder is proposed. • An improved cross-domain transformation loss of orthogonal transformation is proposed. • Rigorous experiments were carried out to demonstrate the efficiency of the model. To solve the problem of data sparsity and cold start, the cross-domain recommendation is a promising research direction in the recommender system . The goal of cross-domain recommendation is to transfer learned knowledge from the source domain to the target domain by different means to improve the performance of the recommendation. But most approaches face the distribution misalignment . In this paper, we propose a joint learning cross-domain recommendation model that can extract domain-specific and common features simultaneously, and only use the implicit feedback data of users without additional auxiliary information. To the best of our knowledge, it is the first attempt to combine the sliced Wasserstein distance and canonical correlation analysis for the cross-domain recommendation scenario. Our one intuition is to reduce the reconstruction error caused by the variational inference based autoencoder model by the optimal transportation theory. Another attempt is to improve the correlation between domains by combining the idea of the canonical correlation analysis. With rigorous experiments, we empirically demonstrated that our model can achieve better performance compared with the state-of-the-art methods.

A lightweight propagation path aggregating network with neural topic model for rumor detection

The structure information associated with message propagation has been proved to be effective to distinguish false and true rumors. However, existing methods lack an efficient way to learn the representation of the whole rumors which captures the intrinsic mechanism of rumor propagation structures and semantics. In this study, we propose a lightweight propagation path aggregating (PPA) neural network for rumor embedding and classification. In the network, we first model the propagation structure of each rumor as an independent set of propagation paths in which each path represents the source post in a different talking context. We then aggregate all paths to obtain the representation of the whole propagation structure. Besides, we utilize a neural topic model in the Wasserstein autoencoder (WAE) framework to capture event insensitive stance patterns in response propagation trees where no source post is included. Empirical studies demonstrate that 1) PPA achieves the state-of-the-art performance with much less parameters and training time, 2) PPA can further benefit from the pre-trained neural topic model which enables to fully use unlabeled data, thus improves the performance of PPA especially when labeled samples are limited or rumors are spreading at early stage. Meanwhile, this topic model offers an explicit interpretation of stance patterns in the form of topics, consequently improves interpretability of the PPA network. The source code can be available at https://github.com/zperfet/PathFakeGit.

Data augmentation for rolling bearing fault diagnosis using an enhanced few-shot Wasserstein auto-encoder with meta-learning

Despite the advance of intelligent fault diagnosis for rolling bearings, in industries, data-driven methods still suffer from data acquisition and imbalance. We propose an enhanced few-shot Wasserstein auto-encoder (fs-WAE) to reverse the negative effect of imbalance. Firstly, an enhanced WAE is proposed for data augmentation, in which squeeze-and-excitation blocks are applied to calibrate channel-wise feature responses adaptively, strengthening the representational power of encoder. Secondly, a meta-learning strategy called Reptile is utilized to further enhance the mapping ability of WAE from prior distribution to vibration signals in the face of small dataset. Finally, gradient penalty is introduced as a regularization term to provide a flexible optimization function. The proposed method is applied to the pattern recognition based on experimental and engineering datasets. Moreover, comparative results demonstrate the utility and superiority of fs-WAE over other models in terms of efficiency and the resilience to imbalance degree.

Power Electric Transformer Fault Diagnosis Based on Infrared Thermal Images Using Wasserstein Generative Adversarial Networks and Deep Learning Classifier

The safety of electric power networks depends on the health of the transformer. However, once a variety of transformer failure occurs, it will not only reduce the reliability of the power system but also cause major accidents and huge economic losses. Until now, many diagnosis methods have been proposed to monitor the operation of the transformer. Most of these methods cannot be detected and diagnosed online and are prone to noise interference and high maintenance cost that will cause obstacles to the real-time monitoring system of the transformer. This paper presents a full-time online fault monitoring system for cast-resin transformer and proposes an overheating fault diagnosis method based on infrared thermography (IRT) images. First, the normal and fault IRT images of the cast-resin transformer are collected by the proposed thermal camera monitoring system. Next is the model training for the Wasserstein Autoencoder Reconstruction (WAR) model and the Differential Image Classification (DIC) model. The differential image can be acquired by the calculation of pixel-wise absolute difference between real images and regenerated images. Finally, in the test phase, the well-trained WAR and DIC models are connected in series to form a module for fault diagnosis. Compared with the existing deep learning algorithms, the experimental results demonstrate the great advantages of the proposed model, which can obtain the comprehensive performance with lightweight, small storage size, rapid inference time and adequate diagnostic accuracy.

Enabling rapid COVID-19 small molecule drug design through scalable deep learning of generative models

We improved the quality and reduced the time to produce machine learned models for use in small molecule antiviral design. Our globally asynchronous multi-level parallel training approach strong scales to all of Sierra with up to 97.7% efficiency. We trained a novel, character-based Wasserstein autoencoder that produces a higher quality model trained on 1.613 billion compounds in 23 minutes while the previous state of the art takes a day on 1 million compounds. Reducing training time from a day to minutes shifts the model creation bottleneck from computer job turnaround time to human innovation time. Our implementation achieves 318 PFLOPs for 17.1% of half-precision peak. We will incorporate this model into our molecular design loop enabling the generation of more diverse compounds; searching for novel, candidate antiviral drugs improves and reduces the time to synthesize compounds to be tested in the lab.

Closed‐form expressions for maximum mean discrepancy with applications to Wasserstein auto‐encoders

The maximum mean discrepancy (MMD) has found numerous applications in statistics and machine learning, among which is its use as a penalty in the Wasserstein auto‐encoder (WAE). In this paper, we compute closed‐form expressions for estimating the Gaussian kernel‐based MMD between a given distribution and the standard multivariate normal distribution. This formula reveals a connection to the Baringhaus–Henze–Epps–Pulley (BHEP) statistic of the Henze–Zirkler test and provides further insights about the MMD. We introduce the standardized version of MMD as a penalty for the WAE training objective, allowing for a better interpretability of MMD values and more compatibility across different hyperparameter settings. Next, we propose code normalization–using batch normalization at the code layer–which has the benefits of making the kernel width selection easier, reducing the training effort, and preventing outliers in the aggregate code distribution. Our experiments on synthetic and real data show that the analytic formulation improves over the commonly used stochastic approximation of the MMD and demonstrate that code normalization provides significant benefits when training WAEs.

Wavelet Loss Function for Auto-Encoder

In the field of image generation, especially for auto-encoder models, how to extract better features and obtain better quality reconstruction samples by modifying network structure and training algorithms has always been the focus of attention. For example, Variational Auto-Encoder (VAE), which is a very popular auto-encoder model, is a theoretically rigorously derived image generation model. For commonly used auto-encoders, such as VAE, Regularized Auto-Encoder (RAE), and Wasserstein Auto-Encoder (WAE), the mean square error (MSE) is all used as the loss function of the reconstructed process, which makes the blur problem of the reconstruction samples unavoidable. Especially for larger-sized images, the blur phenomenon is more obvious. To solve this problem, Perceptual Loss Function is used in some cases. Although it can improve the image quality to a certain extent, the amount of calculation is large and the image quality improvement in auto-encoder is also relatively limited. For this reason, we try to propose a new loss function, Wavelet loss function, to better generate and reconstruct images. Wavelet transform is applied to the reconstructed image loss function of the auto-encoder, and the frequency characteristics of the decomposed image are used to constrain it. We conducted comparative experiments on two larger-size image datasets (FaceSrub, COIL20) and a small-size image dataset (Fashion_MNIST), and proved the effectiveness of the wavelet loss function. At the same time, we propose a new image quality index: wavelet high-frequency signal-to-noise ratio (WHF-SNR), which can better measure the quality of the reconstructed image of the auto-encoder.

Pixel-Wise Wasserstein Autoencoder for Highly Generative Dehazing.

We propose a highly generative dehazing method based on pixel-wise Wasserstein autoencoders. In contrast to existing dehazing methods based on generative adversarial networks, our method can produce a variety of dehazed images with different styles. It significantly improves the dehazing accuracy via pixel-wise matching from hazy to dehazed images through 2-dimensional latent tensors of the Wasserstein autoencoder. In addition, we present an advanced feature fusion technique to deliver rich information to the latent space. For style transfer, we introduce a mapping function that transforms existing latent spaces to new ones. Thus, our method can produce highly generative haze-free images with various tones, illuminations, and moods, which induces several interesting applications, including low-light enhancement, daytime dehazing, nighttime dehazing, and underwater image enhancement. Experimental results demonstrate that our method quantitatively outperforms existing state-of-the-art methods for synthetic and real-world datasets, and simultaneously generates highly generative haze-free images, which are qualitatively diverse.

SeGMA: Semi-Supervised Gaussian Mixture Autoencoder.

We propose a semi-supervised generative model, SeGMA, which learns a joint probability distribution of data and their classes and is implemented in a typical Wasserstein autoencoder framework. We choose a mixture of Gaussians as a target distribution in latent space, which provides a natural splitting of data into clusters. To connect Gaussian components with correct classes, we use a small amount of labeled data and a Gaussian classifier induced by the target distribution. SeGMA is optimized efficiently due to the use of the Cramer-Wold distance as a maximum mean discrepancy penalty, which yields a closed-form expression for a mixture of spherical Gaussian components and, thus, obviates the need of sampling. While SeGMA preserves all properties of its semi-supervised predecessors and achieves at least as good generative performance on standard benchmark data sets, it presents additional features: 1) interpolation between any pair of points in the latent space produces realistically looking samples; 2) combining the interpolation property with disentangling of class and style information, SeGMA is able to perform continuous style transfer from one class to another; and 3) it is possible to change the intensity of class characteristics in a data point by moving the latent representation of the data point away from specific Gaussian components.