Learning Low-dimensional Representations Research Articles

ScZAG: Integrating ZINB-Based Autoencoder with Adaptive Data Augmentation Graph Contrastive Learning for scRNA-seq Clustering.

Single-cell RNA sequencing (scRNA-seq) is widely used to interpret cellular states, detect cell subpopulations, and study disease mechanisms. In scRNA-seq data analysis, cell clustering is a key step that can identify cell types. However, scRNA-seq data are characterized by high dimensionality and significant sparsity, presenting considerable challenges for clustering. In the high-dimensional gene expression space, cells may form complex topological structures. Many conventional scRNA-seq data analysis methods focus on identifying cell subgroups rather than exploring these potential high-dimensional structures in detail. Although some methods have begun to consider the topological structures within the data, many still overlook the continuity and complex topology present in single-cell data. We propose a deep learning framework that begins by employing a zero-inflated negative binomial (ZINB) model to denoise the highly sparse and over-dispersed scRNA-seq data. Next, scZAG uses an adaptive graph contrastive representation learning approach that combines approximate personalized propagation of neural predictions graph convolution (APPNPGCN) with graph contrastive learning methods. By using APPNPGCN as the encoder for graph contrastive learning, we ensure that each cell's representation reflects not only its own features but also its position in the graph and its relationships with other cells. Graph contrastive learning exploits the relationships between nodes to capture the similarity among cells, better representing the data's underlying continuity and complex topology. Finally, the learned low-dimensional latent representations are clustered using Kullback-Leibler divergence. We validated the superior clustering performance of scZAG on 10 common scRNA-seq datasets in comparison to existing state-of-the-art clustering methods.

Learning dynamic embeddings for temporal attributed networks

Learning low-dimensional representations for networks has attracted considerable research interest due to the ever-increasing volume and variety of network-based information. It becomes especially important and challenging for temporal networks whose nodes and attributes evolve as time changes. To address the problem of temporal network embedding, we propose a novel Dynamic Bayesian Temporal Network Embedding (DBTNE) method that adopts a probabilistic framework and represents network embeddings as Gaussian distributions for learning effective embeddings for temporal networks. Unlike the existing temporal network embedding methods that learn dynamic embeddings only for nodes, our DBTNE jointly learns the dynamic embeddings of both nodes and attributes in temporal attributed networks for effectively capturing the similarities among them, which greatly facilitates attribute-related learning tasks such as link prediction, object classification and user profiling. In addition, we model the temporal dynamics of both nodes and attributes with neural ordinary differential equations, which supports effective and efficient capture of long-range dependencies among embeddings at different timestamps. Extensive experimental results on real-world temporal networks demonstrate that our DBTNE achieves better performance than popular state-of-the-art methods, including DeepWalk, Node2Vec, CAN, HTNE, M2DNE, MTSN, Dane-ATT, for typical graph learning tasks, such as dynamic link prediction and node classification.

DynG2G: An Efficient Stochastic Graph Embedding Method for Temporal Graphs

Dynamic graph embedding has gained great attention recently due to its capability of learning low-dimensional and meaningful graph representations for complex temporal graphs with high accuracy. However, recent advances mostly focus on learning node embeddings as deterministic "vectors" for static graphs, hence disregarding the key graph temporal dynamics and the evolving uncertainties associated with node embedding in the latent space. In this work, we propose an efficient stochastic dynamic graph embedding method (DynG2G) that applies an inductive feedforward encoder trained with node triplet energy-based ranking loss. Every node per timestamp is encoded as a time-dependent probabilistic multivariate Gaussian distribution in the latent space, and, hence, we are able to quantify the node embedding uncertainty on-the-fly. We have considered eight different benchmarks that represent diversity in size (from 96 nodes to 87 626 and from 13 398 edges to 4 870 863) as well as diversity in dynamics, from slowly changing temporal evolution to rapidly varying multirate dynamics. We demonstrate through extensive experiments based on these eight dynamic graph benchmarks that DynG2G achieves new state-of-the-art performance in capturing the underlying temporal node embeddings. We also demonstrate that DynG2G can simultaneously predict the evolving node embedding uncertainty, which plays a crucial role in quantifying the intrinsic dimensionality of the dynamical system over time. In particular, we obtain a "universal" relation of the optimal embedding dimension, Lo , versus the effective dimensionality of uncertainty, Du , and infer that Lo=Du for all cases. This, in turn, implies that the uncertainty quantification approach we employ in the DynG2G algorithm correctly captures the intrinsic dimensionality of the dynamics of such evolving graphs despite the diverse nature and composition of the graphs at each timestamp. In addition, this L0 - Du correlation provides a clear path to selecting adaptively the optimum embedding size at each timestamp by setting L ≥ Du .

Inductive Link Prediction via Interactive Learning Across Relations in Multiplex Networks

Network embedding is an important class of link prediction methods, which can use the distance between learned low-dimensional node representations to characterize the similarity between nodes. Traditional network embedding methods focus on single-layer networks, while in reality, a large part of complex networks are not isolated, but interdependent and interrelated, forming multiplex complex networks. Also, how to effectively exploit layer correlations in multiplex networks to learn more robust and valuable representations, to improve link prediction performance, has been a hot research topic in the field of complex network analysis. However, previous studies mainly focus on inferring intralinks in each layer of complex networks or anchor links among layers. Another issue that has not been discussed is how to predict potential links or reconstruct the network in unobserved relations based on existing multiplex networks. To this issue, we define a novel inductive link prediction problem in multiplex networks, in which most existing multichannel network embedding methods fail to solve. This is either because they only emphasize the specific structure information of an individual layer or only capture the common information for all layers. To effectively address this problem, we propose a novel embedding method termed interactive learning across relations (ILAR), to capture and fully exploit the multiple relations and complex layer correlations in multiplex networks. We leverage two convolutional modules and ILAR to capture the sufficient complementary and correlations in multiplex networks. Moreover, during interactive learning, a disparity constraint is introduced, which enforces the features encoded from two convolutional modules to be different and prevents information redundancy. Finally, the extensive experiments in several real-world datasets show that our model can significantly outperform the existing state-of-the-art network embedding methods on the novel link prediction problem in multiplex networks.

Deep neural network for learning wave scattering and interference of underwater acoustics

It is challenging to construct generalized physical models of underwater wave propagation owing to their complex physics and widely varying environmental parameters and dynamical scales. In this article, we present a deep convolutional recurrent autoencoder network (CRAN) for data-driven learning of complex underwater wave scattering and interference. We specifically consider the dynamics of underwater acoustic scattering from various non-uniform seamount shapes leading to complex wave interference patterns of back-scattered and forward-propagated waves. The CRAN consists of a convolutional autoencoder for learning low-dimensional system representation and a long short-term memory (LSTM)-based recurrent neural network for predicting system evolution in low dimensions. The convolutional autoencoder enables efficient dimension reduction of wave propagation by independently learning global and localized wave features. To improve the time horizon of wave dynamics prediction, we introduce an LSTM architecture with a single-shot learning mechanism and optimal time-delayed data embedding. On training the CRAN over 30 cases containing various seamount geometries and acoustic source frequencies, we can predict wave propagation up to a time horizon of 5 times the initiation sequence length for 15 out-of-training cases with a mean L2 error of approximately 10%. For selected out-of-training cases, the prediction time horizon could be increased to 6 times the initiation sequence length. Importantly, such predictions are obtained with physically consistent wave scattering and wave interference patterns and at 50% lower L2 error compared to routinely use standard LSTMs. These results demonstrate the potential of employing such deep neural networks for learning complex underwater ocean acoustic propagation physics.

Semisupervised Subspace Learning With Adaptive Pairwise Graph Embedding.

Graph-based semisupervised learning can explore the graph topology information behind the samples, becoming one of the most attractive research areas in machine learning in recent years. Nevertheless, existing graph-based methods also suffer from two shortcomings. On the one hand, the existing methods generate graphs in the original high-dimensional space, which are easily disturbed by noisy and redundancy features, resulting in low-quality constructed graphs that cannot accurately portray the relationships between data. On the other hand, most of the existing models are based on the Gaussian assumption, which cannot capture the local submanifold structure information of the data, thus reducing the discriminativeness of the learned low-dimensional representations. This article proposes a semisupervised subspace learning with adaptive pairwise graph embedding (APGE), which first builds a k1 -nearest neighbor graph on the labeled data to learn local discriminant embeddings for exploring the intrinsic structure of the non-Gaussian labeled data, i.e., the submanifold structure. Then, a k2 -nearest neighbor graph is constructed on all samples and mapped to GE learning to adaptively explore the global structure of all samples. Clustering unlabeled data and its corresponding labeled neighbors into the same submanifold, sharing the same label information, improves embedded data's discriminative ability. And the adaptive neighborhood learning method is used to learn the graph structure in the continuously optimized subspace to ensure that the optimal graph matrix and projection matrix are finally learned, which has strong robustness. Meanwhile, the rank constraint is added to the Laplacian matrix of the similarity matrix of all samples so that the connected components in the obtained similarity matrix are precisely equal to the number of classes in the sample, which makes the structure of the graph clearer and the relationship between the near-neighbor sample points more explicit. Finally, multiple experiments on several synthetic and real-world datasets show that the method performs well in exploring local structure and classification tasks.

Spectral clustering and embedding with inter-class topology-preserving

Spectral clustering and embedding are critical in data mining and rely heavily on pre-computed graph structure quality. Some methods simultaneously learn graph structures and low-dimensional embeddings to achieve high-quality graph structures with separable categories. However, existing approaches prioritize inter-class separability while overlooking the inter-class topological structure. Consequently, the learned low-dimensional manifold representations may fail to capture inter-class semantic information, causing the learned graph structures to represent categories partitioning inaccurately. To address these issues, we come up with the inter-class topology and utilize category anchors to indicate class locations in low-dimensional space. The inter-class topology reflects the elaborate structural neighbor information of class, which makes the neighbor classes rationally separable and helps to learn better low-dimensional embedding representations. The spectral embedded clustering method and some variants are proposed to preserve inter-class topology while making categories separable. Furthermore, we ensure intra-class sample compactness in low-dimensional space by assigning sample embeddings near the anchors. We analyze the importance of inter-class topology preserving and intra-class compactness learning. Extensive experiments on publicly available datasets demonstrate that our method is competitive with popular graph clustering and embedding baselines and effectively captures the inter-class topology.

L-BGNN: Layerwise Trained Bipartite Graph Neural Networks.

Learning low-dimensional representations of bipartite graphs enables e-commerce applications, such as recommendation, classification, and link prediction. A layerwise-trained bipartite graph neural network (L-BGNN) embedding method, which is unsupervised, efficient, and scalable, is proposed in this work. To aggregate the information across and within two partitions of a bipartite graph, a customized interdomain message passing (IDMP) operation and an intradomain alignment (IDA) operation are adopted by the proposed L-BGNN method. Furthermore, we develop a layerwise training algorithm for L-BGNN to capture the multihop relationship of large bipartite networks and improve training efficiency. We conduct extensive experiments on several datasets and downstream tasks of various scales to demonstrate the effectiveness and efficiency of the L-BGNN method as compared with state-of-the-art methods. Our codes are publicly available at https://github.com/TianXieUSC/L-BGNN.

Graphfool: Targeted Label Adversarial Attack on Graph Embedding

Graph embedding learns low-dimensional representations for nodes or edges on the graph, which is widely applied in many real-world applications. Excessive graph mining promotes the research of attack methods on graph embedding. Most attack methods generate perturbations that maximize the deviation of the prediction confidence. They are difficult to accurately misclassify the instances into the target label, and the nonminimized perturbations are more easily detected by defense methods. To address these problems, we propose Graphfool, a novel targeted label adversarial attack on graph embedding. It can generate adversarial graphs to attack graph embedding methods via classification boundary and gradient information in the target graph embedding method. Graphfool first estimates the classification boundaries of different categories. Then, it calculates the minimum perturbation matrix to misclassify the attacked node according to the target classification boundary. Finally, the adjacency matrix is modified according to the maximum absolute value of the perturbation matrix. Extensive experiments demonstrate that Graphfool achieves the state-of-the-art attack performance with minimum perturbations. Besides, the possible defense experiments further prove that the perturbations generated by Graphfool are more imperceptible.

The Deep Generative Decoder: MAP estimation of representations improves modeling of single-cell RNA data.

Learning low-dimensional representations of single-cell transcriptomics has become instrumental to its downstream analysis. The state of the art is currently represented by neural network models such as variational autoencoders (VAEs) which use a variational approximation of the likelihood for inference. We here present the Deep Generative Decoder (DGD), a simple generative model that computes model parameters and representations directly via maximum a posteriori (MAP) estimation. The DGD handles complex parameterized latent distributions naturally unlike VAEs which typically use a fixed Gaussian distribution, because of the complexity of adding other types. We first show its general functionality on a commonly used benchmark set, Fashion-MNIST. Secondly, we apply the model to multiple single-cell data sets. Here the DGD learns low-dimensional, meaningful and well-structured latent representations with sub-clustering beyond the provided labels. The advantages of this approach are its simplicity and its capability to provide representations of much smaller dimensionality than a comparable VAE. The code is made available in this GitHub repository. scDGD is available as a python package at https://github.com/Center-for-Health-Data-Science/scDGD. Supplementary data are available at Bioinformatics online.

A drug combination prediction framework based on graph convolutional network and heterogeneous information.

Combination therapy, which can improve therapeutic efficacy and reduce side effects, plays an important role in the treatment of complex diseases. Yet, a large number of possible combinations among candidate compounds limits our ability to identify effective combinations. Though many studies have focused on predicting potential drug combinations, the existing methods are not entirely satisfactory in terms of performance and scalability. In this study, we propose a new computational pipeline, called DCMGCN, which integrates diverse drug-related information, to predict novel drug combinations. Specifically, DCMGCN first learns low-dimensional representations of drugs from the drug attributes and similarity networks. Then, by quantifying the degree of the nodes in the known drug-drug network and the similarity between connected nodes, we found the drug-drug network has heterophily and sparseness, which may limit the effectiveness of the graph convolutional network (GCN). Therefore, we introduce two designs to modify GCN. Finally, the drug representations are optimized using modified GCN (MGCN) and used to predict drug combinations. The tests on multiple drug combination datasets show that DCMGCN achieved substantial improvements over state-of-the-art methods. Importantly, our model may embed the mechanism of ground-truth drug pairs into the low-dimensional representation of each drug, which may help to further clarify the understanding of mechanisms of drug action.

Identifying B-cell epitopes using AlphaFold2 predicted structures and pretrained language model.

Identifying the B-cell epitopes is an essential step for guiding rational vaccine development and immunotherapies. Since experimental approaches are expensive and time-consuming, many computational methods have been designed to assist B-cell epitope prediction. However, existing sequence-based methods have limited performance since they only use contextual features of the sequential neighbors while neglecting structural information. Based on the recent breakthrough of AlphaFold2 in protein structure prediction, we propose GraphBepi, a novel graph-based model for accurate B-cell epitope prediction. For one protein, the predicted structure from AlphaFold2 is used to construct the protein graph, where the nodes/residues are encoded by ESM-2 learning representations. The graph is input into the edge-enhanced deep graph neural network (EGNN) to capture the spatial information in the predicted 3D structures. In parallel, a bidirectional long short-term memory neural networks (BiLSTM) is employed to capture long-range dependencies in the sequence. The learned low-dimensional representations by EGNN and BiLSTM are then combined into a multilayer perceptron for predicting B-cell epitopes. Through comprehensive tests on the curated epitope dataset, GraphBepi was shown to outperform the state-of-the-art methods by more than 5.5% and 44.0% in terms of AUC and AUPR, respectively. A web server is freely available at http://bio-web1.nscc-gz.cn/app/graphbepi. The datasets, pre-computed features, source codes, and the trained model are available at https://github.com/biomed-AI/GraphBepi.

SGAEMDA: Predicting miRNA-Disease Associations Based on Stacked Graph Autoencoder.

MicroRNA (miRNA)-disease association (MDA) prediction is critical for disease prevention, diagnosis, and treatment. Traditional MDA wet experiments, on the other hand, are inefficient and costly.Therefore, we proposed a multi-layer collaborative unsupervised training base model called SGAEMDA (Stacked Graph Autoencoder-Based Prediction of Potential miRNA-Disease Associations). First, from the original miRNA and disease data, we defined two types of initial features: similarity features and association features. Second, stacked graph autoencoder is then used to learn unsupervised low-dimensional representations of meaningful higher-order similarity features, and we concatenate the association features with the learned low-dimensional representations to obtain the final miRNA-disease pair features. Finally, we used a multilayer perceptron (MLP) to predict scores for unknown miRNA-disease associations. SGAEMDA achieved a mean area under the ROC curve of 0.9585 and 0.9516 in 5-fold and 10-fold cross-validation, which is significantly higher than the other baseline methods. Furthermore, case studies have shown that SGAEMDA can accurately predict candidate miRNAs for brain, breast, colon, and kidney neoplasms.

BmVAE: a variational autoencoder method for clustering single-cell mutation data.

Genetic intra-tumor heterogeneity (ITH) characterizes the differences in genomic variations between tumor clones, and accurately unmasking ITH is important for personalized cancer therapy. Single-cell DNA sequencing now emerges as a powerful means for deciphering underlying ITH based on point mutations of single cells. However, detecting tumor clones from single-cell mutation data remains challenging due to the error-prone and discrete nature of the data. We introduce bmVAE, a bioinformatics tool for learning low-dimensional latent representation of single cell based on a variational autoencoder and then clustering cells into subpopulations in the latent space. bmVAE takes single-cell binary mutation data as inputs, and outputs inferred cell subpopulations as well as their genotypes. To achieve this, the bmVAE framework is designed to consist of three modules including dimensionality reduction, cell clustering and genotype estimation. We assess the method on various synthetic datasets where different factors including false negative rate, data size and data heterogeneity are considered in simulation, and further demonstrate its effectiveness on two real datasets. The results suggest bmVAE is highly effective in reasoning ITH, and performs competitive to existing methods. bmVAE is freely available at https://github.com/zhyu-lab/bmvae. Supplementary data are available at Bioinformatics online.

Walking With Attention: Self-Guided Walking for Heterogeneous Graph Embedding

Heterogeneous graph embedding aims at learning low-dimensional representations from a graph featuring nodes and edges of diverse natures, and meanwhile preserving the underlying topology. Existing approaches along this line have largely relied on <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">meta-paths</i> , which are by nature hand-crafted and pre-defined transition rules, so as to explore the semantics of a graph. Despite the promising results, defining meta-paths requires domain knowledge, and thus when the test distribution deviates from the priors, such methods are prone to errors. In this paper, we propose a self-learning scheme for heterogeneous graph embedding, termed as self-guided walk (SILK), that bypasses meta-paths and learns adaptive attentions for node walking. SILK assumes no prior knowledge or annotation is provided, and conducts a customized random walk to encode the contexts of the heterogeneous graph of interest. Specifically, this is achieved via maintaining a dynamically-updated <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">guidance matrix</i> that records the node-conditioned transition potentials. Experimental results on four real-world datasets demonstrate that SILK significantly outperforms state-of-the-art methods.

MUSE: Multi-faceted attention for signed network embedding

Signed network embedding is an approach to learning low-dimensional representations of nodes in signed networks with both positive and negative links, which facilitates downstream tasks such as link prediction with general data mining frameworks. Due to the distinct properties and significant added value of negative links, existing signed network embedding methods usually design dedicated methods based on social theories such as balance theory and status theory. However, existing signed network embedding methods ignore the characteristics of multiple facets of each node and mix them up in one single representation, which limits the ability to capture the fine-grained attentions between node pairs. In this paper, we propose MUSE, a MUlti-faceted attention-based Signed network Embedding framework to tackle this problem. Specifically, a joint intra- and inter-facet attention mechanism is introduced to aggregate fine-grained information from neighbor nodes. Moreover, balance theory is also utilized to guide information aggregation from multi-order balanced and unbalanced neighbors. Experimental results on four real-world signed network datasets demonstrate the effectiveness of our proposed framework.

A dynamic graph representation learning based on temporal graph transformer

The graph neural network has received significant attention in recent years because of its unique role in mining graph-structure data and its ubiquitous application in various fields, such as social networking and recommendation systems. Although most work focuses on learning low-dimensional node representation in static graphs, the dynamic nature of real-world networks makes temporal graphs more practical and significant. In this paper, we propose a dynamic graph representation learning method based on a temporal graph transformer (TGT), which can efficiently preserve high-order information and temporally evolve structural properties by incorporating an update module, an aggregation module, and a propagation module in a single model. The experimental results on three real-world networks demonstrate that the TGT outperforms state-of-the-art baselines for dynamic link prediction and edge classification tasks in terms of both accuracy and efficiency.

Neighborhood Geometric Structure-Preserving Variational Autoencoder for Smooth and Bounded Data Sources.

Many data sources, such as human poses, lie on low-dimensional manifolds that are smooth and bounded. Learning low-dimensional representations for such data is an important problem. One typical solution is to utilize encoder-decoder networks. However, due to the lack of effective regularization in latent space, the learned representations usually do not preserve the essential data relations. For example, adjacent video frames in a sequence may be encoded into very different zones across the latent space with holes in between. This is problematic for many tasks such as denoising because slightly perturbed data have the risk of being encoded into very different latent variables, leaving output unpredictable. To resolve this problem, we first propose a neighborhood geometric structure-preserving variational autoencoder (SP-VAE), which not only maximizes the evidence lower bound but also encourages latent variables to preserve their structures as in ambient space. Then, we learn a set of small surfaces to approximately bound the learned manifold to deal with holes in latent space. We extensively validate the properties of our approach by reconstruction, denoising, and random image generation experiments on a number of data sources, including synthetic Swiss roll, human pose sequences, and facial expression images. The experimental results show that our approach learns more smooth manifolds than the baselines. We also apply our approach to the tasks of human pose refinement and facial expression image interpolation where it gets better results than the baselines.

Leveraging network structure for efficient dynamic negative sampling in network embedding

Unsupervised network embedding learns low-dimensional vector representations of nodes based on the network structure. However, typical graphs only contain positive edges. Hence, most network embedding models take sampled negative edges as input to avoid trivial solutions. However, conventional negative sampling methods follow fixed distributions to generate negative samples. Such a static sampling strategy could dramatically hurt the performance of node embeddings in downstream tasks. Researchers have proposed several adaptive sampling strategies to improve the quality of negative samples. However, existing methods do not sufficiently use the network structure to explore high-quality samples, resulting in complex and inefficient negative sampling. This paper develops a novel dynamic negative sampling scheme, which maintains a candidate sample population for each node. We update the populations dynamically and retain high-quality samples in each iteration. We develop a novel network embedding algorithm based on the proposed model, which selects high-quality negative samples adaptively from the population. We conduct extensive experiments to evaluate its effectiveness based on several benchmark datasets. The experimental results verify the superiority of the proposed methods over state-of-the-art approaches.

NSL2CD: identifying potential circRNA-disease associations based on network embedding and subspace learning.

Many studies have evidenced that circular RNAs (circRNAs) are important regulators in various pathological processes and play vital roles in many human diseases, which could serve as promising biomarkers for disease diagnosis, treatment and prognosis. However, the functions of most of circRNAs remain to be unraveled, and it is time-consuming and costly to uncover those relationships between circRNAs and diseases by conventional experimental methods. Thus, identifying candidate circRNAs for human diseases offers new opportunities to understand the functional properties of circRNAs and the pathogenesis of diseases. In this study, we propose a novel network embedding-based adaptive subspace learning method (NSL2CD) for predicting potential circRNA-disease associations and discovering those disease-related circRNA candidates. The proposed method first calculates disease similarities and circRNA similarities by fully utilizing different data sources and learns low-dimensional node representations with network embedding methods. Then, we adopt an adaptive subspace learning model to discover potential associations between circRNAs and diseases. Meanwhile, an integrated weighted graph regularization term is imposed to preserve local geometric structures of data spaces, and L1,2-norm constraint is also incorporated into the model to realize the smoothness and sparsity of projection matrices. The experiment results show that NSL2CD achieves comparable performance under different evaluation metrics, and case studies further confirm its ability to discover potential candidate circRNAs for human diseases.