Low-dimensional Latent Space Research Articles

Multi-source information fusion based heterogeneous network embedding

Heterogeneous network embedding aims to learn a mapping between network data in original topological space and vectored data in low dimensional latent space, while encoding valuable information, such as structural and semantic information. The resulting vector representation has shown promising performance for extensive real-world applications, such as node classification and node clustering. However, most of existing methods merely focus on modeling network structural information, ignoring the rich multi-source information of different types of nodes. In this paper, we propose a novel Multi-source Information Fusion based Heterogeneous Network Embedding (MIFHNE) approach. We first capture the semantic information using the strategy of meta-graph based random walk. Subsequently, we jointly model the structural proximity, attribute information and label information in the framework of Nonnegative Matrix Factorization (NMF). Theoretical proofs and comprehensive experiments on two real-world heterogeneous network datasets demonstrate the feasibility and effectiveness of our approach.

Machine-learning-based reduced-order modeling for unsteady flows around bluff bodies of various shapes

We propose a method to construct a reduced order model with machine learning for unsteady flows. The present machine-learned reduced order model (ML-ROM) is constructed by combining a convolutional neural network autoencoder (CNN-AE) and a long short-term memory (LSTM), which are trained in a sequential manner. First, the CNN-AE is trained using direct numerical simulation (DNS) data so as to map the high-dimensional flow data into low-dimensional latent space. Then, the LSTM is utilized to establish a temporal prediction system for the low-dimensionalized vectors obtained by CNN-AE. As a test case, we consider flows around a bluff body whose shape is defined using a combination of trigonometric functions with random amplitudes. The present ML-ROMs are trained on a set of 80 bluff body shapes and tested on a different set of 20 bluff body shapes not used for training, with both training and test shapes chosen from the same random distribution. The flow fields are confirmed to be well reproduced by the present ML-ROM in terms of various statistics. We also focus on the influence of two main parameters: (1) the latent vector size in the CNN-AE, and (2) the time step size between the mapped vectors used for the LSTM. The present results show that the ML-ROM works well even for unseen shapes of bluff bodies when these parameters are properly chosen, which implies great potential for the present type of ML-ROM to be applied to more complex flows

Single-Cell Transcriptome Data Clustering via Multinomial Modeling and Adaptive Fuzzy K-Means Algorithm.

Single-cell RNA sequencing technologies have enabled us to study tissue heterogeneity at cellular resolution. Fast-developing sequencing platforms like droplet-based sequencing make it feasible to parallel process thousands of single cells effectively. Although a unique molecular identifier (UMI) can remove bias from amplification noise to a certain extent, clustering for such sparse and high-dimensional large-scale discrete data remains intractable and challenging. Most existing deep learning-based clustering methods utilize the mean square error or negative binomial distribution with or without zero inflation to denoise single-cell UMI count data, which may underfit or overfit the gene expression profiles. In addition, neglecting the molecule sampling mechanism and extracting representation by simple linear dimension reduction with a hard clustering algorithm may distort data structure and lead to spurious analytical results. In this paper, we combined the deep autoencoder technique with statistical modeling and developed a novel and effective clustering method, scDMFK, for single-cell transcriptome UMI count data. ScDMFK utilizes multinomial distribution to characterize data structure and draw support from neural network to facilitate model parameter estimation. In the learned low-dimensional latent space, we proposed an adaptive fuzzy k-means algorithm with entropy regularization to perform soft clustering. Various simulation scenarios and the analysis of 10 real datasets have shown that scDMFK outperforms other state-of-the-art methods with respect to data modeling and clustering algorithms. Besides, scDMFK has excellent scalability for large-scale single-cell datasets.

Exponential Family Graph Embeddings

Representing networks in a low dimensional latent space is a crucial task with many interesting applications in graph learning problems, such as link prediction and node classification. A widely applied network representation learning paradigm is based on the combination of random walks for sampling context nodes and the traditional Skip-Gram model to capture center-context node relationships. In this paper, we emphasize on exponential family distributions to capture rich interaction patterns between nodes in random walk sequences. We introduce the generic exponential family graph embedding model, that generalizes random walk-based network representation learning techniques to exponential family conditional distributions. We study three particular instances of this model, analyzing their properties and showing their relationship to existing unsupervised learning models. Our experimental evaluation on real-world datasets demonstrates that the proposed techniques outperform well-known baseline methods in two downstream machine learning tasks.

Learning Cross-Aligned Latent Embeddings for Zero-Shot Cross-Modal Retrieval

Zero-Shot Cross-Modal Retrieval (ZS-CMR) is an emerging research hotspot that aims to retrieve data of new classes across different modality data. It is challenging for not only the heterogeneous distributions across different modalities, but also the inconsistent semantics across seen and unseen classes. A handful of recently proposed methods typically borrow the idea from zero-shot learning, i.e., exploiting word embeddings of class labels (i.e., class-embeddings) as common semantic space, and using generative adversarial network (GAN) to capture the underlying multimodal data structures, as well as strengthen relations between input data and semantic space to generalize across seen and unseen classes. In this paper, we propose a novel method termed Learning Cross-Aligned Latent Embeddings (LCALE) as an alternative to these GAN based methods for ZS-CMR. Unlike using the class-embeddings as the semantic space, our method seeks for a shared low-dimensional latent space of input multimodal features and class-embeddings by modality-specific variational autoencoders. Notably, we align the distributions learned from multimodal input features and from class-embeddings to construct latent embeddings that contain the essential cross-modal correlation associated with unseen classes. Effective cross-reconstruction and cross-alignment criterions are further developed to preserve class-discriminative information in latent space, which benefits the efficiency for retrieval and enable the knowledge transfer to unseen classes. We evaluate our model using four benchmark datasets on image-text retrieval tasks and one large-scale dataset on image-sketch retrieval tasks. The experimental results show that our method establishes the new state-of-the-art performance for both tasks on all datasets.

Representing Closed Transformation Paths in Encoded Network Latent Space

Deep generative networks have been widely used for learning mappings from a low-dimensional latent space to a high-dimensional data space. In many cases, data transformations are defined by linear paths in this latent space. However, the Euclidean structure of the latent space may be a poor match for the underlying latent structure in the data. In this work, we incorporate a generative manifold model into the latent space of an autoencoder in order to learn the low-dimensional manifold structure from the data and adapt the latent space to accommodate this structure. In particular, we focus on applications in which the data has closed transformation paths which extend from a starting point and return to nearly the same point. Through experiments on data with natural closed transformation paths, we show that this model introduces the ability to learn the latent dynamics of complex systems, generate transformation paths, and classify samples that belong on the same transformation path.

Embedding high-dimensional Bayesian optimization via generative modeling: Parameter personalization of cardiac electrophysiological models.

The estimation of patient-specific tissue properties in the form of model parameters is important for personalized physiological models. Because tissue properties are spatially varying across the underlying geometrical model, it presents a significant challenge of high-dimensional (HD) optimization at the presence of limited measurement data. A common solution to reduce the dimension of the parameter space is to explicitly partition the geometrical mesh. In this paper, we present a novel concept that uses a generative variational auto-encoder (VAE) to embed HD Bayesian optimization into a low-dimensional (LD) latent space that represents the generative code of HD parameters. We further utilize VAE-encoded knowledge about the generative code to guide the exploration of the search space. The presented method is applied to estimating tissue excitability in a cardiac electrophysiological model in a range of synthetic and real-data experiments, through which we demonstrate its improved accuracy and substantially reduced computational cost in comparison to existing methods that rely on geometry-based reduction of the HD parameter space.

The hourglass organization of the Caenorhabditis elegans connectome.

We approach the C. elegans connectome as an information processing network that receives input from about 90 sensory neurons, processes that information through a highly recurrent network of about 80 interneurons, and it produces a coordinated output from about 120 motor neurons that control the nematode’s muscles. We focus on the feedforward flow of information from sensory neurons to motor neurons, and apply a recently developed network analysis framework referred to as the “hourglass effect”. The analysis reveals that this feedforward flow traverses a small core (“hourglass waist”) that consists of 10-15 interneurons. These are mostly the same interneurons that were previously shown (using a different analytical approach) to constitute the “rich-club” of the C. elegans connectome. This result is robust to the methodology that separates the feedforward from the feedback flow of information. The set of core interneurons remains mostly the same when we consider only chemical synapses or the combination of chemical synapses and gap junctions. The hourglass organization of the connectome suggests that C. elegans has some similarities with encoder-decoder artificial neural networks in which the input is first compressed and integrated in a low-dimensional latent space that encodes the given data in a more efficient manner, followed by a decoding network through which intermediate-level sub-functions are combined in different ways to compute the correlated outputs of the network. The core neurons at the hourglass waist represent the information bottleneck of the system, balancing the representation accuracy and compactness (complexity) of the given sensory information.

Trust-embedded collaborative deep generative model for social recommendation

Social networks can provide massive amounts of information for communication among users and communities. The trust relationships in social networks can be utilized to reveal user preferences for improving the quality of social recommendation, which aims to mitigate information overload and provide users with the most attractive and relevant items or services. However, the data sparsity and cold-start issue degrade recommendation performance significantly. To address these issues, a novel trust-embedded collaborative deep generative model (TCDG) is proposed for exploiting multisource information (content, rating and trust) to predict ratings. TCDG employs deep generative model to unsupervisedly learn deep latent representations for item content through an inference network in latent space instead of observation space. Meanwhile, TCDG adopts probabilistic matrix factorization to map users into low-dimensional latent feature spaces by trust relationships, which can reflect users’ mutual influence on the formation of users’ opinions more accurately and learn better implicit relationships between items and users from content, rating and trust. In addition, an approach with an annealing parameter to calculate the maximum a posteriori estimates is proposed to learn model parameters. Experiments using four real-world datasets are conducted to evaluate the prediction and top-ranking performance of our model. The results indicate that TDCG has better accuracy and robustness than other methods for making recommendations.

Evolving Matrix-Factorization-Based Collaborative Filtering Using Genetic Programming

Recommender systems aim to estimate the judgment or opinion that a user might offer to an item. Matrix-factorization-based collaborative filtering typifies both users and items as vectors of factors inferred from item rating patterns. This method finds latent structure in the data, assuming that observations lie close to a low-dimensional latent space. However, matrix factorizations have been traditionally designed by hand. Here, we present Evolutionary Matrix Factorization (EMF), an evolutionary approach that automatically generates matrix factorizations aimed at improving the performance of recommender systems. Initial experiments using this approach show that EMF generally outperforms baseline methods when applied to MovieLens and FilmTrust datasets, having a similar performance to those baselines on the worst cases. These results serve as an incentive to continue improving and studying the application of an evolutionary approach to collaborative filtering based on Matrix Factorization.

Deep heterogeneous network embedding based on Siamese Neural Networks

Heterogeneous network embedding aims at mapping a heterogeneous network into a low-dimensional latent space. There exist diverse relations among different types of objects in heterogeneous networks. However, most existing heterogeneous network embedding methods focus on exploring network structures instead of relations among different objects, so some redundant and fuzzy relations are inevitably captured. To address the problem, we propose a Relation-Oriented Deep Embedding (RODE) framework for heterogeneous networks that explores different relations among nodes. The captured relations are modeled through node similarity and dissimilarity. Based on the similarity and dissimilarity, a multi-task Siamese Neural Network is formulated to perform network embedding and optimize embedding representations. Extensive experiments are conducted on four heterogeneous networks. Experimental results demonstrate our method outperforms state-of-the-art embedding algorithms on several network mining tasks, such as link prediction, node classification and node clustering.

Probabilistic Transfer Factor Analysis for Machinery Autonomous Diagnosis Cross Various Operating Conditions

The variability of machinery fault signatures causes the data samples to follow different distributions under various operating conditions, which poses significant challenges on autonomous diagnosis based on machine learning techniques. This article presents a new transfer learning method for cross-domain feature learning by mitigating the domain difference caused by various operating conditions for machinery autonomous diagnosis. More specifically, a factor analysis (FA)-based transfer learning method is formulated and named as probabilistic transfer FA (PTFA). It seeks a new feature space across different domains corresponding to various operating conditions and then transfers the original features into a low-dimensional latent space via feature extraction to minimize domain difference and preserve data properties. The learned features by PTFA minimize the domain difference and are then used to construct the machinery autonomous diagnosis model based on machine learning techniques [e.g., support vector machine (SVM)]. The effectiveness of the PTFA method is demonstrated in the experimental tests for a gearbox diagnosis under various operating conditions comparing with traditional feature extraction and transfer learning techniques.

Large-scale semantic exploration of scientific literature using topic-based hashing algorithms

Searching for similar documents and exploring major themes covered across groups of documents are common activities when browsing collections of scientific papers. This manual knowledge-intensive task can become less tedious and even lead to unexpected relevant findings if unsupervised algorithms are applied to help researchers. Most text mining algorithms represent documents in a common feature space that abstract them away from the specific sequence of words used in them. Probabilistic Topic Models reduce that feature space by annotating documents with thematic information. Over this low-dimensional latent space some locality-sensitive hashing algorithms have been proposed to perform document similarity search. However, thematic information gets hidden behind hash codes, preventing thematic exploration and limiting the explanatory capability of topics to justify content-based similarities. This paper presents a novel hashing algorithm based on approximate nearest-neighbor techniques that uses hierarchical sets of topics as hash codes. It not only performs efficient similarity searches, but also allows extending those queries with thematic restrictions explaining the similarity score from the most relevant topics. Extensive evaluations on both scientific and industrial text datasets validate the proposed algorithm in terms of accuracy and efficiency.

A Data-driven Model of Nucleosynthesis with Chemical Tagging in a Lower-dimensional Latent Space

AbstractChemical tagging seeks to identify unique star formation sites from present-day stellar abundances. Previous techniques have treated each abundance dimension as being statistically independent, despite theoretical expectations that many elements can be produced by more than one nucleosynthetic process. In this work, we introduce a data-driven model of nucleosynthesis, where a set of latent factors (e.g., nucleosynthetic yields) contribute to all stars with different scores and clustering (e.g., chemical tagging) is modeled by a mixture of multivariate Gaussians in a lower-dimensional latent space. We use an exact method to simultaneously estimate the factor scores for each star, the partial assignment of each star to each cluster, and the latent factors common to all stars, even in the presence of missing data entries. We use an information-theoretic Bayesian principle to estimate the number of latent factors and clusters. Using the secondGalahdata release, we find that six latent factors are preferred to explainN = 2566 stars with 17 chemical abundances. We identify the rapid- and slow neutron-capture processes, as well as latent factors consistent with Fe-peak andα-element production, and another where K and Zn dominate. When we considerN ∼ 160,000 stars with missing abundances, we find another seven factors, as well as 16 components in latent space. Despite these components showing separation in chemistry, which is explained through different yield contributions, none show significant structure in their positions or motions. We argue that more data and joint priors on cluster membership that are constrained by dynamical models are necessary to realize chemical tagging at a galactic-scale. We release accompanying software that scales well with the available data, allowing for the model’s parameters to be optimized in seconds given a fixed number of latent factors, components, and ∼107abundance measurements.

Biological representation of chemicals using latent target interaction profile

BackgroundComputational prediction of a phenotypic response upon the chemical perturbation on a biological system plays an important role in drug discovery, and many other applications. Chemical fingerprints are a widely used feature to build machine learning models. However, the fingerprints that are derived from chemical structures ignore the biological context, thus, they suffer from several problems such as the activity cliff and curse of dimensionality. Fundamentally, the chemical modulation of biological activities is a multi-scale process. It is the genome-wide chemical-target interactions that modulate chemical phenotypic responses. Thus, the genome-scale chemical-target interaction profile will more directly correlate with in vitro and in vivo activities than the chemical structure. Nevertheless, the scope of direct application of the chemical-target interaction profile is limited due to the severe incompleteness, biasness, and noisiness of bioassay data.ResultsTo address the aforementioned problems, we developed a novel chemical representation method: Latent Target Interaction Profile (LTIP). LTIP embeds chemicals into a low dimensional continuous latent space that represents genome-scale chemical-target interactions. Subsequently LTIP can be used as a feature to build machine learning models. Using the drug sensitivity of cancer cell lines as a benchmark, we have shown that the LTIP robustly outperforms chemical fingerprints regardless of machine learning algorithms. Moreover, the LTIP is complementary with the chemical fingerprints. It is possible for us to combine LTIP with other fingerprints to further improve the performance of bioactivity prediction.ConclusionsOur results demonstrate the potential of LTIP in particular and multi-scale modeling in general in predictive modeling of chemical modulation of biological activities.

Deciphering protein evolution and fitness landscapes with latent space models

Protein sequences contain rich information about protein evolution, fitness landscapes, and stability. Here we investigate how latent space models trained using variational auto-encoders can infer these properties from sequences. Using both simulated and real sequences, we show that the low dimensional latent space representation of sequences, calculated using the encoder model, captures both evolutionary and ancestral relationships between sequences. Together with experimental fitness data and Gaussian process regression, the latent space representation also enables learning the protein fitness landscape in a continuous low dimensional space. Moreover, the model is also useful in predicting protein mutational stability landscapes and quantifying the importance of stability in shaping protein evolution. Overall, we illustrate that the latent space models learned using variational auto-encoders provide a mechanism for exploration of the rich data contained in protein sequences regarding evolution, fitness and stability and hence are well-suited to help guide protein engineering efforts.

Deep learning for high-dimensional reliability analysis

High-dimensional reliability analysis remains a grand challenge since most of the existing methods suffer from the curse of dimensionality. This paper introduces a novel high-dimensional data abstraction (HDDA) framework for dimension reduction in reliability analysis. It first involves training of a failure-informed autoencoder network to reduce the dimensionality of the high-dimensional input space, aiming at creating a distinguishable failure surface in a low-dimensional latent space. Then a deep feedforward neural network is constructed to connect the high-dimensional input parameters with the low-dimensional latent variables. With the HDDA framework, the high-dimensional reliability can be estimated by capturing the limit state function in the latent space using Gaussian process regression. To manage the uncertainty due to lack of training data, a distance-based sampling strategy is developed for iteratively identifying critical training samples, which improves the accuracy of the high-dimensional reliability estimations. Three high-dimensional examples are used to demonstrate the effectiveness of the proposed approach.

MLNE: Multi-Label Network Embedding.

Network embedding aims to preserve topological structures of a network using low-dimensional vectors and has shown to be effective for driving a myriad of graph mining tasks (e.g., link prediction or classification) free of the stressful feature extraction procedure. Many methods have been proposed to integrate node content and/or label information, with nodes sharing similar content/labels being close to each other in the learned latent space. To date, existing methods either consider networked instances with a single label or consider a set of labels as a whole for node representation learning. Therefore, they cannot handle network of instances containing multiple labels (i.e. multi-labels), which are ubiquitous in describing complex concepts of instances. In this article, we formulate a new multi-label network embedding (MLNE) problem to learn feature representation for networked multi-label instances. We argue that the key to MLNE learning is to aggregate node topology structures, node content, and multi-label correlations. We propose a two-layer network embedding framework to couple information for effective learning. To capture higher order label correlations, we use labels to form a high-level label-label network over a low-level node-node network, in which the label network interacts with the node network through multi-labeling relations. The low-level node-node network can be enhanced by latent label-specific features from high-level label network with well-captured high-order correlations between labels. To enable the multi-label informed network embedding, we force both node and label representations being optimized under the same low-dimensional latent space by a unified training objective. Experiments on real-world data sets demonstrate that MLNE achieves better performance compared with methods with or without considering label information.

Incorporating prior shape knowledge via data-driven loss model to improve 3D liver segmentation in deep CNNs.

Convolutional neural networks (CNNs) have obtained enormous success in liver segmentation. However, there are several challenges, including low-contrast images, and large variations in the shape, and appearance of the liver. Incorporating prior knowledge in deep CNN models improves their performance and generalization. A convolutional denoising auto-encoder is utilized to learn global information about 3D liver shapes in a low-dimensional latent space. Then, the deep data-driven knowledge is used to define a loss function and combine it with the Dice loss in the main segmentation model. The resultant hybrid model would be forced to learn the global shape information as prior knowledge, while it tries to produce accurate results and increase the Dice score. The proposed training strategy improved the performance of the 3D U-Net model and reached the Dice score of 97.62% on the Sliver07-I liver dataset, which is competitive to the state-of-the-art automatic segmentation methods. The proposed algorithm enhanced the generalization and robustness of the hybrid model and outperformed the 3D U-Net model in the prediction of unseen images. The results indicate that the incorporation of prior shape knowledge enhances liver segmentation tasks in deep CNN models. The proposed method improves the generalization and robustness of the hybrid model due to the abstract features provided by the data-driven loss model.

Efficient dimension reduction and surrogate-based sensitivity analysis for expensive models with high-dimensional outputs

Sensitivity analysis has been widely used to gain more insights on complex system behavior, to facilitate model reduction, system design and decision making. Typically, sensitivity analysis entails many evaluations of the system model. For expensive system models with high-dimensional outputs, direct adoption of such models for sensitivity analysis poses significant challenges in computational effort and memory requirements. To address these challenges, this paper proposes an efficient sensitivity analysis approach. The proposed method uses surrogate model to replace the expensive model for sensitivity analysis, and tackle the problem of building surrogate models for high-dimensional outputs through surrogate model integrated with dimension reduction. More specifically, the proposed method first uses surrogate models in low-dimensional latent output space to efficiently calculate the relevant covariance matrices for the low-dimensional latent outputs, and then directly establishes the sensitivity indices for the original high-dimensional output based on these covariance matrices and the derived transformation. Two examples are presented to demonstrate the efficiency and accuracy of the proposed method.