Low-dimensional Embedding Research Articles

Graph-Graph Similarity Network.

Graph learning aims to predict the label for an entire graph. Recently, graph neural network (GNN)-based approaches become an essential strand to learning low-dimensional continuous embeddings of entire graphs for graph label prediction. While GNNs explicitly aggregate the neighborhood information and implicitly capture the topological structure for graph representation, they ignore the relationships among graphs. In this article, we propose a graph-graph (G2G) similarity network to tackle the graph learning problem by constructing a SuperGraph through learning the relationships among graphs. Each node in the SuperGraph represents an input graph, and the weights of edges denote the similarity between graphs. By this means, the graph learning task is then transformed into a classical node label propagation problem. Specifically, we use an adversarial autoencoder to align embeddings of all the graphs to a prior data distribution. After the alignment, we design the G2G similarity network to learn the similarity between graphs, which functions as the adjacency matrix of the SuperGraph. By running node label propagation algorithms on the SuperGraph, we can predict the labels of graphs. Experiments on five widely used classification benchmarks and four public regression benchmarks under a fair setting demonstrate the effectiveness of our method.

Redefining the Game: MVAE-DFDPnet's Low-Dimensional Embeddings for Superior Drug-Protein Interaction Predictions

Redefining the Game: MVAE-DFDPnet's Low-Dimensional Embeddings for Superior Drug-Protein Interaction Predictions

Bidirectional Probabilistic Subspaces Approximation for Multiview Clustering.

The existing multiview clustering models learn a consistent low-dimensional embedding either from multiple feature matrices or multiple similarity matrices, which ignores the interaction between the two procedures and limits the improvement of clustering performance on multiview data. To address this issue, a bidirectional probabilistic subspaces approximation (BPSA) model is developed in this article to learn a consistently orthogonal embedding from multiple feature matrices and multiple similarity matrices simultaneously via the disturbed probabilistic subspace modeling and approximation. A skillful bidirectional fusion strategy is designed to guarantee the parameter-free property of the BPSA model. Two adaptively weighted learning mechanisms are introduced to ensure the inconsistencies among multiple views and the inconsistencies between bidirectional learning processes. To solve the optimization problem involved in the BPSA model, an iterative solver is derived, and a rigorous convergence guarantee is provided. Extensive experimental results on both toy and real-world datasets demonstrate that our BPSA model achieves state-of-the-art performance even if it is parameter-free.

Oncotree2vec - a method for embedding and clustering of tumor mutation trees.

Understanding the genomic heterogeneity of tumors is an important task in computational oncology, especially in the context of finding personalized treatments based on the genetic profile of each patient's tumor. Tumor clustering that takes into account the temporal order of genetic events, as represented by tumor mutation trees, is a powerful approach for grouping together patients with genetically and evolutionarily similar tumors and can provide insights into discovering tumor subtypes, for more accurate clinical diagnosis and prognosis. Here, we propose oncotree2vec, a method for clustering tumor mutation trees by learning vector representations of mutation trees that capture the different relationships between subclones in an unsupervised manner. Learning low-dimensional tree embeddings facilitates the visualization of relations between trees in large cohorts and can be used for downstream analyses, such as deep learning approaches for single-cell multi-omics data integration. We assessed the performance and the usefulness of our method in three simulation studies and on two real datasets: a cohort of 43 trees from six cancer types with different branching patterns corresponding to different modes of spatial tumor evolution and a cohort of 123 AML mutation trees. https://github.com/cbg-ethz/oncotree2vec.

Molecular sequence classification using efficient kernel based embedding

The alarming spread of diseases across the globe has become a major concern for global healthcare agencies. The research community is actively involved in inventing better and more efficient ways of detecting and treating diseases to solve this global challenge. The abundance of molecular sequence data has eased the path for researchers to develop Machine Learning (ML) based solutions. The performance of the ML models used to classify molecular sequences depends heavily on the type of embedding used to obtain an appropriate numerical representation of the molecular sequences. In recent years, many embedding approaches have been introduced for molecular sequence analysis. However, there is still a need for improvement as far as the efficiency of the methods is concerned (i.e., the ability to capture pairwise relationships and patterns effectively, which could affect the classification performance). To provide a solution to this problem, we propose an efficient kernel-based technique for embedding generation from molecular sequences, which involves computing a kernel matrix using the Sinkhorn-Knopp algorithm and the normalized pairwise distances between k-mers in a manner that satisfies the constraints of a probability distribution. Further, kernel principal component analysis (PCA) is applied to get the top PCs, which are then used as the final embedding. As a result of the experiments, we obtained an ROC-AUC score of 0.657 for our method, which is higher than the scores obtained using baselines. This clearly shows that the low-dimensional embedding obtained through the proposed approach provides an efficient and effective solution for molecular sequence analysis.

Information Bottleneck Approach for Markov Model Construction.

Markov state models (MSMs) have proven valuable in studying the dynamics of protein conformational changes via statistical analysis of molecular dynamics simulations. In MSMs, the complex configuration space is coarse-grained into conformational states, with dynamics modeled by a series of Markovian transitions among these states at discrete lag times. Constructing the Markovian model at a specific lag time necessitates defining states that circumvent significant internal energy barriers, enabling internal dynamics relaxation within the lag time. This process effectively coarse-grains time and space, integrating out rapid motions within metastable states. Thus, MSMs possess a multiresolution nature, where the granularity of states can be adjusted according to the time-resolution, offering flexibility in capturing system dynamics. This work introduces a continuous embedding approach for molecular conformations using the state predictive information bottleneck (SPIB), a framework that unifies dimensionality reduction and state space partitioning via a continuous, machine learned basis set. Without explicit optimization of the VAMP-based scores, SPIB demonstrates state-of-the-art performance in identifying slow dynamical processes and constructing predictive multiresolution Markovian models. Through applications to well-validated mini-proteins, SPIB showcases unique advantages compared to competing methods. It autonomously and self-consistently adjusts the number of metastable states based on a specified minimal time resolution, eliminating the need for manual tuning. While maintaining efficacy in dynamical properties, SPIB excels in accurately distinguishing metastable states and capturing numerous well-populated macrostates. This contrasts with existing VAMP-based methods, which often emphasize slow dynamics at the expense of incorporating numerous sparsely populated states. Furthermore, SPIB's ability to learn a low-dimensional continuous embedding of the underlying MSMs enhances the interpretation of dynamic pathways. With these benefits, we propose SPIB as an easy-to-implement methodology for end-to-end MSM construction.

Metric multidimensional scaling for large single-cell datasets using neural networks

Metric multidimensional scaling is one of the classical methods for embedding data into low-dimensional Euclidean space. It creates the low-dimensional embedding by approximately preserving the pairwise distances between the input points. However, current state-of-the-art approaches only scale to a few thousand data points. For larger data sets such as those occurring in single-cell RNA sequencing experiments, the running time becomes prohibitively large and thus alternative methods such as PCA are widely used instead. Here, we propose a simple neural network-based approach for solving the metric multidimensional scaling problem that is orders of magnitude faster than previous state-of-the-art approaches, and hence scales to data sets with up to a few million cells. At the same time, it provides a non-linear mapping between high- and low-dimensional space that can place previously unseen cells in the same embedding.

BP-MoE: Behavior Pattern-aware Mixture-of-Experts for Temporal Graph Representation Learning

Temporal graph representation learning aims to develop low-dimensional embeddings for nodes in a graph that can effectively capture their structural and temporal properties. Prior approaches primarily focus on devising sophisticated neural architectures to encode the historical interactions of nodes, enabling capturing the specific behavior patterns. However, such methods often overlook the fact that different nodes in a graph may exhibit distinct evolutionary preferences, resulting in a failure to adequately model the variations in the node evolution behavior. To address these issues, we propose a novel temporal graph learning framework called Behavior Pattern-aware Mixture-of-Experts (BP-MoE), which leverages multiple expert encoder networks with diverse architectures to comprehensively capture the behavior patterns for nodes. In detail, we first design a multi-perspective graph encoder that employs experts based on the gated recurrent units, graph attention units, and multilayer perceptron units to encode a node’s long-term memory, high-order neighborhood features, and short-term behavior preference, respectively. Subsequently, to achieve the personalized evolution preference learning for each individual node, we incorporate a behavior pattern-aware MoE mechanism to activate the pattern-specific experts for each node via a gating network. Finally, to optimize the training process of expert networks, we introduce two auxiliary losses that prevent the imbalanced training across experts. Extensive experiments conducted on multiple benchmark datasets verify the superiority of our approach over the state-of-the-art baselines. Specifically, BP-MoE consistently outperforms the baseline models for the link prediction and node classification tasks under various experimental settings.

Sparse multi-view image clustering with complete similarity information

Multi-view image clustering aims to efficiently divide the collection of images by studying the characteristics of different views. Many studies performed Laplacian dimensionality reduction on the original image to avoid noise interference in constructing the similarity matrix. However, they ignored the problem that local similarity information and isolated image information are lost due to dimensionality reduction. These problems will result in the similarity matrix being insufficient to accurately describe the similarity between images, which in turn affects the clustering accuracy. Therefore, we propose a sparse multi-view spectral clustering model with complete similarity information between images (SMSC-CSI). The model combines the original image space and the low-dimensional spectral embedding space based on the adaptive neighbors method to learn the initial similarity matrices of each view jointly. On the one hand, the original space can retain all the similarity information between images so that the initial similarity matrix can accurately describe the similarity between images, which is conducive to accurate clustering. On the other hand, the low-dimensional space can avoid noise interference and retain the main structure of high-dimensional data, improving the robustness of the initial similarity matrix. Meanwhile, to ensure consistency among the views, the model minimizes the difference between the initial similarity matrix and the central fusion matrix by alternately updating to obtain the optimal weights and central fusion matrix for each view. Finally, we can obtain ideal clustering results directly with low model complexity and without post-processing steps such as K-means by adding non-negative Laplace rank constraints and ℓ0-norm constraints to the model objective function. Experimental results on different real image datasets show that SMSC-CSI can outperform some traditional clustering models and recent models on multi-view image clustering.

A high-throughput synthetic biology approach for studying combinatorial chromatin-based transcriptional regulation

The construction of synthetic gene circuits requires the rational combination of multiple regulatory components, but predicting their behavior can be challenging due to poorly understood component interactions and unexpected emergent behaviors. In eukaryotes, chromatin regulators (CRs) are essential regulatory components that orchestrate gene expression. Here, we develop a screening platform to investigate the impact of CR pairs on transcriptional activity in yeast. We construct a combinatorial library consisting of over 1,900 CR pairs and use a high-throughput workflow to characterize the impact of CR co-recruitment on gene expression. We recapitulate known interactions and discover several instances of CR pairs with emergent behaviors. We also demonstrate that supervised machine learning models trained with low-dimensional amino acid embeddings accurately predict the impact of CR co-recruitment on transcriptional activity. This work introduces a scalable platform and machine learning approach that can be used to study how networks of regulatory components impact gene expression.

Clustering and visualization of single-cell RNA-seq data using path metrics.

Recent advances in single-cell technologies have enabled high-resolution characterization of tissue and cancer compositions. Although numerous tools for dimension reduction and clustering are available for single-cell data analyses, these methods often fail to simultaneously preserve local cluster structure and global data geometry. To address these challenges, we developed a novel analyses framework, Single-Cell Path Metrics Profiling (scPMP), using power-weighted path metrics, which measure distances between cells in a data-driven way. Unlike Euclidean distance and other commonly used distance metrics, path metrics are density sensitive and respect the underlying data geometry. By combining path metrics with multidimensional scaling, a low dimensional embedding of the data is obtained which preserves both the global data geometry and cluster structure. We evaluate the method both for clustering quality and geometric fidelity, and it outperforms current scRNAseq clustering algorithms on a wide range of benchmarking data sets.

Multi-view data visualisation via manifold learning

Non-linear dimensionality reduction can be performed by manifold learning approaches, such as stochastic neighbour embedding (SNE), locally linear embedding (LLE) and isometric feature mapping (ISOMAP). These methods aim to produce two or three latent embeddings, primarily to visualise the data in intelligible representations. This manuscript proposes extensions of Student’s t-distributed SNE (t-SNE), LLE and ISOMAP, for dimensionality reduction and visualisation of multi-view data. Multi-view data refers to multiple types of data generated from the same samples. The proposed multi-view approaches provide more comprehensible projections of the samples compared to the ones obtained by visualising each data-view separately. Commonly, visualisation is used for identifying underlying patterns within the samples. By incorporating the obtained low-dimensional embeddings from the multi-view manifold approaches into the K-means clustering algorithm, it is shown that clusters of the samples are accurately identified. Through extensive comparisons of novel and existing multi-view manifold learning algorithms on real and synthetic data, the proposed multi-view extension of t-SNE, named multi-SNE, is found to have the best performance, quantified both qualitatively and quantitatively by assessing the clusterings obtained. The applicability of multi-SNE is illustrated by its implementation in the newly developed and challenging multi-omics single-cell data. The aim is to visualise and identify cell heterogeneity and cell types in biological tissues relevant to health and disease. In this application, multi-SNE provides an improved performance over single-view manifold learning approaches and a promising solution for unified clustering of multi-omics single-cell data.

KDViT: COVID-19 diagnosis on CT-scans with knowledge distillation of vision transformer

This paper introduces Knowledge Distillation of Vision Transformer (KDViT), a novel approach for medical image classification. The Vision Transformer architecture incorporates a self-attention mechanism to autonomously learn image structure. The input medical image is segmented into patches and transformed into low-dimensional linear embeddings. Position information is integrated into each patch, and a learnable classification token is appended for classification, thereby preserving spatial relationships within the image. The output vectors are then fed into a Transformer encoder to extract both local and global features, leveraging the inherent attention mechanism for robust feature extraction across diverse medical imaging scenarios. Furthermore, knowledge distillation is employed to enhance performance by transferring insights from a large teacher model to a small student model. This approach reduces the computational requirements of the larger model and improves overall effectiveness. Integrating knowledge distillation with two Vision Transformer models not only showcases the novelty of the proposed solution for medical image classification but also enhances model interpretability, reduces computational complexity, and improves generalization capabilities. The proposed KDViT model achieved high accuracy rates of 98.39%, 88.57%, and 99.15% on the SARS-CoV-2-CT, COVID-CT, and iCTCF datasets respectively, surpassing the performance of other state-of-the-art methods.

M[formula omitted]ixKG: Mixing for harder negative samples in knowledge graph

Knowledge graph embedding (KGE) involves mapping entities and relations to low-dimensional dense embeddings, enabling a wide range of real-world applications. The mapping is achieved via distinguishing the positive and negative triplets in knowledge graphs. Therefore, how to design high-quality negative triplets is critical in the effectiveness of KEG models. Existing KGE models face challenges in generating high-quality negative triplets. Some models employ simple static distributions, i.e. uniform or Bernoulli distribution, and it is difficult for these methods to be trained distinguishably because of the sampled uninformative negative triplets. Furthermore, current methods are confined to constructing negative triplets from existing entities within the knowledge graph, limiting their ability to explore harder negatives. We introduce a novel mixing strategy in knowledge graphs called M2ixKG. M2ixKG adopts mixing operation in generating harder negative samples from two aspects: one is mixing among the heads and tails in triplets with the same relation to strengthen the robustness and generalization of the entity embeddings; the other is mixing the negatives with high scores to generate harder negatives. Our experiments, utilizing three datasets and four classical score functions, highlight the exceptional performance of M2ixKG in comparison to previous negative sampling algorithms.

Discrete latent embedding of single-cell chromatin accessibility sequencing data for uncovering cell heterogeneity.

Single-cell epigenomic data has been growing continuously at an unprecedented pace, but their characteristics such as high dimensionality and sparsity pose substantial challenges to downstream analysis. Although deep learning models-especially variational autoencoders-have been widely used to capture low-dimensional feature embeddings, the prevalent Gaussian assumption somewhat disagrees with real data, and these models tend to struggle to incorporate reference information from abundant cell atlases. Here we propose CASTLE, a deep generative model based on the vector-quantized variational autoencoder framework to extract discrete latent embeddings that interpretably characterize single-cell chromatin accessibility sequencing data. We validate the performance and robustness of CASTLE for accurate cell-type identification and reasonable visualization compared with state-of-the-art methods. We demonstrate the advantages of CASTLE for effective incorporation of existing massive reference datasets in a weakly supervised or supervised manner. We further demonstrate CASTLE's capacity for intuitively distilling cell-type-specific feature spectra that unveil cell heterogeneity and biological implications quantitatively.

Deep embedding based tensor incomplete multi-view clustering

The majority of multi-view data are extracted from real life, which often lose information in some views. To solve this problem, existing incomplete multi-view clustering algorithms explore the valuable information from incomplete data while populating the missing information. Nevertheless, they suffer from the following two limitations: 1) The non-linear relationships of high-dimensional available data are not considered. 2) Although some methods utilize information from different views to fill in missing information, they still cannot precisely explore the missing information of each view. To this end, this article proposes a novel one-step incomplete multi-view framework, referred to as deep embedding based tensor incomplete multi-view clustering (DETIMC). Concretely, in this framework, the high-dimensional available data are projected into the low-dimensional embedding space by deep non-negative matrix factorization (NMF), which can obtain a clean space while capturing the complex non-linear relationship of available data. Moreover, a novel double-enhanced missing-view inferring strategy is developed, in which the weighted higher-order information of different views and the clustering structure information are simultaneously exploited. Comprehensive experiments on several benchmark datasets exhibit the superiority of DETIMC over conventional and state-of-the-art algorithms.

Inferring gene regulatory networks from single-cell transcriptomics based on graph embedding.

Gene regulatory networks (GRNs) encode gene regulation in living organisms, and have become a critical tool to understand complex biological processes. However, due to the dynamic and complex nature of gene regulation, inferring GRNs from scRNA-seq data is still a challenging task. Existing computational methods usually focus on the close connections between genes, and ignore the global structure and distal regulatory relationships. In this study, we develop a supervised deep learning framework, IGEGRNS, to infer GRNs from scRNA-seq data based on graph embedding. In the framework, contextual information of genes is captured by GraphSAGE, which aggregates gene features and neighborhood structures to generate low-dimensional embedding for genes. Then, the k most influential nodes in the whole graph are filtered through Top-k pooling. Finally, potential regulatory relationships between genes are predicted by stacking CNNs. Compared with nine competing supervised and unsupervised methods, our method achieves better performance on six time-series scRNA-seq datasets. Our method IGEGRNS is implemented in Python using the Pytorch machine learning library, and it is freely available at https://github.com/DHUDBlab/IGEGRNS.

Identifying somatic fingerprints of cancers defined by germline and environmental risk factors.

Numerous studies over the past generation have identified germline variants that increase specific cancer risks. Simultaneously, a revolution in sequencing technology has permitted high-throughput annotations of somatic genomes characterizing individual tumors. However, examining the relationship between germline variants and somatic alteration patterns is hugely challenged by the large numbers of variants in a typical tumor, the rarity of most individual variants, and the heterogeneity of tumor somatic fingerprints. In this article, we propose statistical methodology that frames the investigation of germline-somatic relationships in an interpretable manner. The method uses meta-features embodying biological contexts of individual somatic alterations to implicitly group rare mutations. Our team has used this technique previously through a multilevel regression model to diagnose with high accuracy tumor site of origin. Herein, we further leverage topic models from computational linguistics to achieve interpretable lower-dimensional embeddings of the meta-features. We demonstrate how the method can identify distinctive somatic profiles linked to specific germline variants or environmental risk factors. We illustrate the method using The Cancer Genome Atlas whole-exome sequencing data to characterize somatic tumor fingerprints in breast cancer patients with germline BRCA1/2 mutations and in head and neck cancer patients exposed to human papillomavirus.

An orchestra of machine learning methods reveals landmarks in single-cell data exemplified with aging fibroblasts.

In this work, a Python framework for characteristic feature extraction is developed and applied to gene expression data of human fibroblasts. Unlabeled feature selection objectively determines groups and minimal gene sets separating groups. ML explainability methods transform the features correlating with phenotypic differences into causal reasoning, supported by further pipeline and visualization tools, allowing user knowledge to boost causal reasoning. The purpose of the framework is to identify characteristic features that are causally related to phenotypic differences of single cells. The pipeline consists of several data science methods enriched with purposeful visualization of the intermediate results in order to check them systematically and infuse the domain knowledge about the investigated process. A specific focus is to extract a small but meaningful set of genes to facilitate causal reasoning for the phenotypic differences. One application could be drug target identification. For this purpose, the framework follows different steps: feature reduction (PFA), low dimensional embedding (UMAP), clustering ((H)DBSCAN), feature correlation (chi-square, mutual information), ML validation and explainability (SHAP, tree explainer). The pipeline is validated by identifying and correctly separating signature genes associated with aging in fibroblasts from single-cell gene expression measurements: PLK3, polo-like protein kinase 3; CCDC88A, Coiled-Coil Domain Containing 88A; STAT3, signal transducer and activator of transcription-3; ZNF7, Zinc Finger Protein 7; SLC24A2, solute carrier family 24 member 2 and lncRNA RP11-372K14.2. The code for the preprocessing step can be found in the GitHub repository https://github.com/AC-PHD/NoLabelPFA, along with the characteristic feature extraction https://github.com/LauritzR/characteristic-feature-extraction.

An active learning method using deep adversarial autoencoder-based sufficient dimension reduction neural network for high-dimensional reliability analysis

Reliability analysis often requires time-consuming evaluations, especially when dealing with high-dimensional and nonlinear problems. To address this challenge, surrogate model methods are frequently employed. One way to improve the efficiency of surrogate model methods involves selecting informative samples that significantly enhance the accuracy of the surrogate model. This paper introduces a novel approach to facilitate the construction of surrogate models and selection of informative samples in high-dimensional reliability analysis, through an active learning method based on a deep adversarial autoencoder-based sufficient dimension reduction (AAE-SDR) neural network. The AAE-SDR neural network serves as a surrogate model, transforming complex high-dimensional variables into tractable, low-dimensional embeddings relevant to the target. These embeddings are Gaussian-distributed with a distinct latent limit state boundary. A new sampling strategy is proposed to select informative misclassified samples by iteratively identifying candidate samples near the latent limit state boundary and uniformly sampling from the candidate sample dataset based on the latent Gaussian distribution. The effectiveness of the proposed approach is demonstrated through two high-dimensional numerical examples and a cable-stayed bridge case study. Results show that the proposed method simplifies complex high-dimensional reliability problems and provides a relatively accurate estimated failure probability with a limited number of samples.