Variational Autoencoder Framework Research Articles

Learned image compression via neighborhood-based attention optimization and context modeling with multi-scale guiding

In recent years, learned image compression has witnessed significant advancements. However, many existing learned image compression models predominantly rely on Convolutional Neural Networks (CNNs) and predicting the distribution of latent features to use ever more expensive entropy models, which cannot fully capture detail redundancy and lead to prohibitively slow decoding. To address this challenge, this paper introduces novel image compression models, MGIC and RMGIC, which consist of the Residual Neighborhood-based Attention Module (RNAM), Multi-Scale Guide Entropy Model (MGEM) and Refined Multi-Scale Guide Entropy Model (RMGEM). Specifically, Neighborhood Attention (NA) is combined with global feature learning to construct the RNAM, which can accurately capture the correlation of spatial neighborhoods and is more suitable for image compression. The proposed RNAM is flexible and could work as a plug-and-play component to enhance CNN-based models. Additionally, MGEM employs two mask convolutions with two kernel sizes to leverage multi-scale information for context modeling and eliminating the effect of parallel decoding on model performance. Building upon MGEM, RMGEM incorporates a refined two-stage spatial context modeling technique to enhance the utilization of spatial neighborhoods and achieve more precise entropy prediction. Furthermore, MGIC and RMGIC are derived by integrating RNAM, MGEM, and RMGEM into the variational autoencoder framework. The proposed methods can save more bandwidth and improve the quality of information storage. Extensive experimental evaluations have shown that the proposed method is effective and achieves state-of-the-art performance.

Obstacle Avoidance Learning for Robot Motion Planning in Human–Robot Integration Environments

In the human-robot integration environment, it is essential for mobile robots to pass through the crowd and complete the navigation task smoothly. However, the current mainstream robot navigation algorithms only treat pedestrians as dynamic obstacles and passively avoid pedestrians in local planning. When encountering fast-moving pedestrians, local path planning often fails, causing the robot to stagnate, spin or shake in place, which in turn reduces the navigation efficiency and results in unnatural navigation trajectories. To address this problem, it is desirable for the robot to find a safe and convenient temporary target to avoid the collision with fast-moving pedestrians. In this paper, we propose an obstacle avoidance learning method with the temporary target for the robot motion planning in the human-robot integration environment. The temporary target distribution is learned from imitations by using a Conditional Variational Autoencoder (CVAE) framework, whereby the dynamic scenario information including pedestrian information, the environmental information, and the robot information are considered as the generation conditions. With the proposed method, the mobile robot first navigates to the temporary target area, and then plans the path toward the final target point. Experimental studies reveal that the proposed method can achieve satisfactory performance with respect to different scenario conditions.

Similarity-assisted variational autoencoder for nonlinear dimension reduction with application to single-cell RNA sequencing data

BackgroundDeep generative models naturally become nonlinear dimension reduction tools to visualize large-scale datasets such as single-cell RNA sequencing datasets for revealing latent grouping patterns or identifying outliers. The variational autoencoder (VAE) is a popular deep generative method equipped with encoder/decoder structures. The encoder and decoder are useful when a new sample is mapped to the latent space and a data point is generated from a point in a latent space. However, the VAE tends not to show grouping pattern clearly without additional annotation information. On the other hand, similarity-based dimension reduction methods such as t-SNE or UMAP present clear grouping patterns even though these methods do not have encoder/decoder structures.ResultsTo bridge this gap, we propose a new approach that adopts similarity information in the VAE framework. In addition, for biological applications, we extend our approach to a conditional VAE to account for covariate effects in the dimension reduction step. In the simulation study and real single-cell RNA sequencing data analyses, our method shows great performance compared to existing state-of-the-art methods by producing clear grouping structures using an inferred encoder and decoder. Our method also successfully adjusts for covariate effects, resulting in more useful dimension reduction.ConclusionsOur method is able to produce clearer grouping patterns than those of other regularized VAE methods by utilizing similarity information encoded in the data via the highly celebrated UMAP loss function.

StVAE deconvolves cell-type composition in large-scale cellular resolution spatial transcriptomics.

Recent rapid developments in spatial transcriptomic techniques at cellular resolution have gained increasing attention. However, the unique characteristics of large-scale cellular resolution spatial transcriptomic datasets, such as the limited number of transcripts captured per spot and the vast number of spots, pose significant challenges to current cell-type deconvolution methods. In this study, we introduce stVAE, a method based on the variational autoencoder framework to deconvolve the cell-type composition of cellular resolution spatial transcriptomic datasets. To assess the performance of stVAE, we apply it to five datasets across three different biological tissues. In the Stereo-seq and Slide-seqV2 datasets of the mouse brain, stVAE accurately reconstructs the laminar structure of the pyramidal cell layers in the cortex, which are mainly organized by the subtypes of telencephalon projecting excitatory neurons. In the Stereo-seq dataset of the E12.5 mouse embryo, stVAE resolves the complex spatial patterns of osteoblast subtypes, which are supported by their marker genes. In Stereo-seq and Pixel-seq datasets of the mouse olfactory bulb, stVAE accurately delineates the spatial distributions of known cell types. In summary, stVAE can accurately identify spatial patterns of cell types and their relative proportions across spots for cellular resolution spatial transcriptomic data. It is instrumental in understanding the heterogeneity of cell populations and their interactions within tissues. stVAE is available in GitHub (https://github.com/lichen2018/stVAE) and Figshare (https://figshare.com/articles/software/stVAE/23254538).

Dependency-Aware Neural Topic Model

Existing topic models that aim to discover relationships between topics typically focus on two types of relations: undirected correlation and hierarchy with a tree structure. However, these approaches are limited in their ability to capture complicated dependencies between topics, such as causality, where topics may depend on multiple other topics. To address this limitation, we propose a novel Dependency-Aware Neural Topic Model (DepNTM). This model takes the dependency relationships between topics as a generative process, where each topic is generated by other topics based on their dependencies. Specifically, we utilize a Structural Causal Model within a variational autoencoder (VAE) framework to learn the topic dependencies in a Directed Acyclic Graph (DAG). To further map the latent topics with comprehensible semantic information and ensure interpretability, we use labels of documents as the supervision signal. We conduct experiments on two public real-world corpora from the arXiv system and Stack Overflow website. The experimental results show that DepNTM outperforms nine state-of-the-art (SOTA) topic models in terms of perplexity (reduced by 9% on avg. vs. SOTA baselines), topic quality (improved by 58%) in most cases. We further conduct abundant analytical experiments to demonstrate the reliability of DepNTM.

VNAGT: Variational Non-Autoregressive Graph Transformer Network for Multi-Agent Trajectory Prediction

Accurately predicting the trajectory of road agents in complex traffic scenarios is challenging because the movement patterns of agents are complex and stochastic, not only depending on their own past trajectories but also being closely related to the social interaction with other surrounding agents. Besides accuracy, efficient prediction with low inference latency is also a highly desirable feature for the practical application of trajectory prediction. To address these issues, we propose the VNAGT model, a variational non-autoregressive graph transformer that adopts the framework of conditional variational autoencoder and incorporates the non-autoregressive approach such that diverse trajectories and low prediction latency can be achieved simultaneously. In order to capture the social and temporal interaction, we put forward a unified graph attention-based module that is applicable for homogeneous and heterogeneous multi-agents such that the class information can be seamlessly integrated when it is available. Non-autoregressive decoding is combined with variational learning to produce multiple plausible predictions with low latency. We train and validate the model on two real-world homogeneous and heterogeneous trajectory datasets. The experimental results demonstrate the superior performance of our model in comparison with the state-of-the-art methods.

A VAE-Based User Preference Learning and Transfer Framework for Cross-Domain Recommendation

The core idea of cross-domain recommendation is to alleviate the problem of data scarcity. Previous methods have made brilliant successes. However, many of them mainly focus on learning an ideal mapping function across-domains, ignoring the user preferences within a specific domain, which leads to suboptimal results. In this paper, we propose a Cross-Domain Recommendation Variational AutoEncoder framework (CDRVAE), a novel extension of a variational autoencoder on cross-domain recommendations for user behaviour distribution modeling. It applies a new hybrid architecture of VAE as the backbone and simultaneously constructs two information flows, within-domain and cross-domain modeling. For the former, an asymmetric codec structure is designed to reconstruct preference distribution from domain-specific latent factors. To relieve the posterior collapse dilemma, a combined prior is employed to increase the distribution complexity. The equivalent transition by a transformation matrix and the unobserved interaction generation by cross-domain reconstruction contribute to the latter. We combine all the above components for the more accurate and reliable user features. Extensive experiments are conducted on three public benchmark datasets to validate the effectiveness of the proposed CDRVAE. Experimental results demonstrate that CDRVAE is consistently superior to other state-of-the-art alternative baseline models.

Disease-specific resting-state EEG network variations in schizophrenia revealed by the contrastive machine learning

Given a multitude of genetic and environmental factors, when investigating the variability in schizophrenia (SCZ) and the first-degree relatives (R-SCZ), latent disease-specific variation is usually hidden. To reliably investigate the mechanism underlying the brain deficits from the aspect of functional networks, we newly iterated a framework of contrastive variational autoencoders (cVAEs) applied in the contrasts among three groups, to disentangle the latent resting-state network patterns specified for the SCZ and R-SCZ. We demonstrated that the comparison in reconstructed resting-state networks among SCZ, R-SCZ, and healthy controls (HC) revealed network distortions of the inner-frontal hypoconnectivity and frontal-occipital hyperconnectivity, while the original ones illustrated no differences. And only the classification by adopting the reconstructed network metrics achieved satisfying performances, as the highest accuracy of 96.80% ± 2.87%, along with the precision of 95.05% ± 4.28%, recall of 98.18% ± 3.83%, and F1-score of 96.51% ± 2.83%, was obtained. These findings consistently verified the validity of the newly proposed framework for the contrasts among the three groups and provided related resting-state network evidence for illustrating the pathological mechanism underlying the brain deficits in SCZ, as well as facilitating the diagnosis of SCZ.

Diverse Image Captioning via Conditional Variational Autoencoder and Dual Contrastive Learning

Diverse image captioning has achieved substantial progress in recent years. However, the discriminability of generative models and the limitation of cross entropy loss are generally overlooked in the traditional diverse image captioning models, which seriously hurts both the diversity and accuracy of image captioning. In this paper, aiming to improve diversity and accuracy simultaneously, we propose a novel Conditional Variational Autoencoder (DCL-CVAE) framework for diverse image captioning by seamlessly integrating sequential variational autoencoder with contrastive learning. In the encoding stage, we first build conditional variational autoencoders to separately learn the sequential latent spaces for a pair of captions. Then, we introduce contrastive learning in the sequential latent spaces to enhance the discriminability of latent representations for both image-caption pairs and mismatched pairs. In the decoding stage, we leverage the captions sampled from the pre-trained Long Short-Term Memory (LSTM), LSTM decoder as the negative examples and perform contrastive learning with the greedily sampled positive examples, which can restrain the generation of common words and phrases induced by the cross entropy loss. By virtue of dual constrastive learning, DCL-CVAE is capable of encouraging the discriminability and facilitating the diversity, while promoting the accuracy of the generated captions. Extensive experiments are conducted on the challenging MSCOCO dataset, showing that our proposed methods can achieve a better balance between accuracy and diversity compared to the state-of-the-art diverse image captioning models.

Towards inclusive underground public transportation: Gender differences on thermal comfort

As urbanization and societal needs grow, the demand for inclusive, efficient, and sustainable underground public transportation has increased, necessitating a greater focus on enhancing passengers' thermal comfort. However, limited research has addressed the subway station environment, and restricted datasets are challenging to build an accurate nonlinear model. Therefore, this study proposes a thermal comfort data generation algorithm based on 5161 measured data via the Variational Autoencoder (VAE) framework to break data collection constraints and increase thermal comfort model accuracy. By employing a gradient boosting algorithm on the extensively generated dataset, this study investigated factors influencing thermal comfort in subway stations, emphasizing individual gender differences and dynamic environmental variables. The VAE model enhanced the thermal comfort dataset, resulting in a 29% and 35% increase in male and female test set accuracy, respectively. The study suggests that males with a BMI over 27 were more sensitive to temperature. As for the gender-friendly environmental features, people feel most comfortable at temperatures between 24 and 29 °C and relative humidity levels between 20% and 70%. Additionally, the study reveals that a stable environment promotes enhanced thermal comfort for passengers. These insights contribute to developing sustainable, equitable, and user-friendly underground public transportation systems.

C-NTPP: Learning Cluster-Aware Neural Temporal Point Process

Event sequences in continuous time space are ubiquitous across applications and have been intensively studied with both classic temporal point process (TPP) and its recent deep network variants. This work is motivated by an observation that many of event data exhibit inherent clustering patterns in terms of the sparse correlation among events, while such characteristics are seldom explicitly considered in existing neural TPP models whereby the history encoders are often embodied by RNNs or Transformers. In this work, we propose a c-NTPP (Cluster-Aware Neural Temporal Point Process) model, which leverages a sequential variational autoencoder framework to infer the latent cluster each event belongs to in the sequence. Specially, a novel event-clustered attention mechanism is devised to learn each cluster and then aggregate them together to obtain the final representation for each event. Extensive experiments show that c-NTPP achieves superior performance on both real-world and synthetic datasets, and it can also uncover the underlying clustering correlations.

Boosting Variational Inference With Margin Learning for Few-Shot Scene-Adaptive Anomaly Detection

Anomaly detection in surveillance videos aims to identify frames where abnormal events happen. Existing approaches assume that the training and testing videos are from the same scene, exhibiting poor generalization performance when encountering an unseen scene. In this paper, we propose a Variational Anomaly Detection Network (VADNet), which is characterized by its high scene-adaptation - it can identify abnormal events in a new scene only via referring to a few normal samples without fine-tuning. Our model embodies two major innovations. First, a novel Variational Normal Inference (VNI) module is proposed to formulate image reconstruction in a conditional variational auto-encoder (CVAE) framework, which learns a probabilistic decision model instead of a traditional deterministic one. Secondly, a Margin Learning Embedding (MLE) module is leveraged to boost the variational inference and aid in distinguishing normal events. We theoretically demonstrate that minimizing the triplet loss in MLE module facilitates maximizing the evidence lower bound (ELBO) of CVAE, which promotes the convergence of VNI. By incorporating variational inference with margin learning, VADNet becomes much more generative that is able to handle the uncertainty caused by the changed scene and limited reference data. Extensive experiments on several datasets demonstrate that the proposed VADNet can adapt to a new scene effectively without fine-tuning and achieve remarkable performance, which outperforms other methods significantly and establishes new state-of-the-art in the case of few-shot scene-adaptive anomaly detection. We believe our method is closer to real-world application due to its strong generalization ability. All codes are released in https://github.com/huangxx156/VADNet.

Edge-enhanced Global Disentangled Graph Neural Network for Sequential Recommendation

Sequential recommendation has been a widely popular topic of recommender systems. Existing works have contributed to enhancing the prediction ability of sequential recommendation systems based on various methods, such as recurrent networks and self-attention mechanisms. However, they fail to discover and distinguish various relationships between items, which could be underlying factors which motivate user behaviors. In this article, we propose an Edge-Enhanced Global Disentangled Graph Neural Network (EGD-GNN) model to capture the relation information between items for global item representation and local user intention learning. At the global level, we build a global-link graph over all sequences to model item relationships. Then a channel-aware disentangled learning layer is designed to decompose edge information into different channels, which can be aggregated to represent the target item from its neighbors. At the local level, we apply a variational auto-encoder framework to learn user intention over the current sequence. We evaluate our proposed method on three real-world datasets. Experimental results show that our model can get a crucial improvement over state-of-the-art baselines and is able to distinguish item features.

Knowledge Base Embedding for Sampling-Based Prediction

Each link prediction task requires different degrees of answer diversity. While a link prediction task may expect up to a couple of answers, another may expect nearly a hundred answers. Given this fact, the performance of a link prediction model can be estimated more accurately if a flexible number of obtained answers are estimated instead of a predefined number of answers. Inspired by this, in this article, we analyze two evaluation criteria for link prediction tasks, respectively ranking-based protocol and sampling-based protocol. Furthermore, we study two classes of models on link prediction task, direct model and latent-variable model respectively, to demonstrate that latent-variable model performs better under the sampling-based protocol. We then propose a latent-variable model where the framework of Conditional Variational AutoEncoder (CVAE) is applied. Experimental study suggests that the proposed model performs comparably to the current state-of-the-art even under the conventional rank-based protocol. Under the sampling-based protocol, the proposed model is shown to outperform various state-of-the-art models.

Motion Planning of Manipulator by Points-Guided Sampling Network

This paper proposes a network called points-guided sampling net (PGSN) to guide the sampling process in sampling-based motion planner by utilizing the geometric information of obstacles. The geometric information is extracted from the point cloud of obstacles. By analyzing the properties of the point cloud, we propose a VAE feature extraction net that incorporates the variational autoencoder (VAE) framework with unique architectures designed for point clouds. Furthermore, we design a multi-modal sampling net to model the probability distribution of the states based on training trajectories taken from different environments. Based on PGSN, we propose a sampling-based motion planning algorithm called the point-guided rapidly-exploring random tree (PG-RRT). Three experiments are conducted to verify the proposed PGSN: Exp I shows the proposed VAE feature extraction net can successfully extract geometric features from the inputted point cloud; Exp II verifies the multi-modal sampling net successfully chooses corresponding mode with respect to extracted features; Exp III demonstrates the efficacy of our PG-RRT algorithm by showing PG-RRT outperforms other algorithms. Moreover, we provide theoretical analysis and insights towards understanding our model. Note to Practitioners—Obstacles cause lots of the sampling space invalid, thus the traditional sampling-based motion planning (SBMP) algorithm is usually unable to generate a trajectory within a reasonable short period of time. To improve the success rate and efficiency of SBMP, this paper proposes a novel deep neural network called points-guided sampling net (PGSN). PGSN is designed to exploit: (1) environmental point clouds and (2) training trajectories from multiple environments with different obstacles. In the first step, the point clouds include important geometric information. To utilize this information, we adopt a variational autoencoder approach which combines an encoder and a decoder together to extract geometric features more accurately from point clouds. In the second step, trajectories from multiple environments have a multi-modal property which can be represented by a truncated multivariate Gaussian mixture model. We propose a multi-modal sampling net to learn optimal parameters of this model from the training trajectories, and to select corresponding mode based on the extracted features. Experiments demonstrate that the proposed algorithm is feasible and can achieve higher success rate than the state-of-the-art methods. Our method uses a single frame of point cloud to improve efficiency, therefore multiple point clouds from different perspective maybe needed when objects occlude with each other.

Data Augmentation in High Dimensional Low Sample Size Setting Using a Geometry-Based Variational Autoencoder.

In this paper, we propose a new method to perform data augmentation in a reliable way in the High Dimensional Low Sample Size (HDLSS) setting using a geometry-based variational autoencoder (VAE). Our approach combines the proposal of 1) a new VAE model, the latent space of which is modeled as a Riemannian manifold and which combines both Riemannian metric learning and normalizing flows and 2) a new generation scheme which produces more meaningful samples especially in the context of small data sets. The method is tested through a wide experimental study where its robustness to data sets, classifiers and training samples size is stressed. It is also validated on a medical imaging classification task on the challenging ADNI database where a small number of 3D brain magnetic resonance images (MRIs) are considered and augmented using the proposed VAE framework. In each case, the proposed method allows for a significant and reliable gain in the classification metrics. For instance, balanced accuracy jumps from 66.3% to 74.3% for a state-of-the-art convolutional neural network classifier trained with 50 MRIs of cognitively normal (CN) and 50 Alzheimer disease (AD) patients and from 77.7% to 86.3% when trained with 243 CN and 210 AD while improving greatly sensitivity and specificity metrics.

An entangled mixture of variational autoencoders approach to deep clustering

We present a novel deep clustering algorithm that utilizes a variational autoencoder (VAE) framework with an entangled multi encoder-decoder neural architecture. Our model enforces a complementary structure that guides the learned latent representations towards a better space arrangement. It differs from previous VAE-based clustering algorithms by employing a new generative model that uses multiple encoder-decoders that are entangled to provide a joint clustering decision. The optimal clustering is found by optimizing a lower bound of the model likelihood function. Both the reconstruction component and the regularization component of the ELBO objective function are explicitly involved in the clustering procedure. We show that this modeling results in both better clustering capabilities and improved data generation. The proposed method is evaluated on standard datasets and is shown to significantly outperform state-of-the-art deep clustering methods.

Learning Explainable Disentangled Representations of E-Commerce Data by Aligning Their Visual and Textual Attributes

Understanding multimedia content remains a challenging problem in e-commerce search and recommendation applications. It is difficult to obtain item representations that capture the relevant product attributes since these product attributes are fine-grained and scattered across product images with huge visual variations and product descriptions that are noisy and incomplete. In addition, the interpretability and explainability of item representations have become more important in order to make e-commerce applications more intelligible to humans. Multimodal disentangled representation learning, where the independent generative factors of multimodal data are identified and encoded in separate subsets of features in the feature space, is an interesting research area to explore in an e-commerce context given the benefits of the resulting disentangled representations such as generalizability, robustness and interpretability. However, the characteristics of real-word e-commerce data, such as the extensive visual variation, noisy and incomplete product descriptions, and complex cross-modal relations of vision and language, together with the lack of an automatic interpretation method to explain the contents of disentangled representations, means that current approaches for multimodal disentangled representation learning do not suffice for e-commerce data. Therefore, in this work, we design an explainable variational autoencoder framework (E-VAE) which leverages visual and textual item data to obtain disentangled item representations by jointly learning to disentangle the visual item data and to infer a two-level alignment of the visual and textual item data in a multimodal disentangled space. As such, E-VAE tackles the main challenges in disentangling multimodal e-commerce data. Firstly, with the weak supervision of the two-level alignment our E-VAE learns to steer the disentanglement process towards discovering the relevant factors of variations in the multimodal data and to ignore irrelevant visual variations which are abundant in e-commerce data. Secondly, to the best of our knowledge our E-VAE is the first VAE-based framework that has an automatic interpretation mechanism that allows to explain the components of the disentangled item representations with text. With our textual explanations we provide insight in the quality of the disentanglement. Furthermore, we demonstrate that with our explainable disentangled item representations we achieve state-of-the-art outfit recommendation results on the Polyvore Outfits dataset and report new state-of-the-art cross-modal search results on the Amazon Dresses dataset.

Graph representation learning based on deep generative gaussian mixture models

Graph representation learning is an effective tool for facilitating graph analysis with machine learning methods. Most GNNs, including Graph Convolutional Networks (GCN), Graph Recurrent Neural Networks (GRNN), and Graph Auto-Encoders (GAE), employ vectors to represent nodes in a deterministic way without exploiting the uncertainty in hidden variables. Deep generative models are combined with GAE in the Variational Graph Auto-Encoder (VGAE) framework to address this issue. While traditional VGAE-based methods can capture hidden and hierarchical dependencies in latent spaces, they are limited by the data’s multimodality. Here, we propose the Gaussian Mixture Model (GMM) to model the prior distribution in VGAE. Furthermore, an adversarial regularization is incorporated into the proposed approach to ensure the fruitful impact of the latent representations on the results. We demonstrate the performance of the proposed method on clustering and link prediction tasks. Our experimental results on real datasets show remarkable performance compared to state-of-the-art methods.

Temporal Network Embedding for Link Prediction via VAE Joint Attention Mechanism.

Network representation learning or embedding aims to project the network into a low-dimensional space that can be devoted to different network tasks. Temporal networks are an important type of network whose topological structure changes over time. Compared with methods on static networks, temporal network embedding (TNE) methods are facing three challenges: 1) it cannot describe the temporal dependence across network snapshots; 2) the node embedding in the latent space fails to indicate changes in the network topology; and 3) it cannot avoid a lot of redundant computation via parameter inheritance on a series of snapshots. To overcome these problems, we propose a novel TNE method named temporal network embedding method based on the VAE framework (TVAE), which is based on a variational autoencoder (VAE) to capture the evolution of temporal networks for link prediction. It not only generates low-dimensional embedding vectors for nodes but also preserves the dynamic nonlinear features of temporal networks. Through the combination of a self-attention mechanism and recurrent neural networks, TVAE can update node representations and keep the temporal dependence of vectors over time. We utilize parameter inheritance to keep the new embedding close to the previous one, rather than explicitly using regularization, and thus, it is effective for large-scale networks. We evaluate our model and several baselines on synthetic data sets and real-world networks. The experimental results demonstrate that TVAE has superior performance and lower time cost compared with the baselines.