Representation Learning Framework Research Articles

Designing meaningful continuous representations of T cell receptor sequences with deep generative models

T Cell Receptor (TCR) antigen binding underlies a key mechanism of the adaptive immune response yet the vast diversity of TCRs and the complexity of protein interactions limits our ability to build useful low dimensional representations of TCRs. To address the current limitations in TCR analysis we develop a capacity-controlled disentangling variational autoencoder trained using a dataset of approximately 100 million TCR sequences, that we name TCR-VALID. We design TCR-VALID such that the model representations are low-dimensional, continuous, disentangled, and sufficiently informative to provide high-quality TCR sequence de novo generation. We thoroughly quantify these properties of the representations, providing a framework for future protein representation learning in low dimensions. The continuity of TCR-VALID representations allows fast and accurate TCR clustering and is benchmarked against other state-of-the-art TCR clustering tools and pre-trained language models.

Knowledge Graphs and Pretrained Language Models Enhanced Representation Learning for Conversational Recommender Systems.

Conversational recommender systems (CRSs) utilize natural language interactions and dialog history to infer user preferences and provide accurate recommendations. Due to the limited conversation context and background knowledge, existing CRSs rely on external sources such as knowledge graphs (KGs) to enrich the context and model entities based on their interrelations. However, these methods ignore the rich intrinsic information within entities. To address this, we introduce the knowledge-enhanced entity representation learning (KERL) framework, which leverages both the KG and a pretrained language model (PLM) to improve the semantic understanding of entities for CRS. In our KERL framework, entity textual descriptions are encoded via a PLM, while a KG helps reinforce the representation of these entities. We also employ positional encoding to effectively capture the temporal information of entities in a conversation. The enhanced entity representation is then used to develop a recommender component that fuses both entity and contextual representations for more informed recommendations, as well as a dialog component that generates informative entity-related information in the response text. A high-quality KG with aligned entity descriptions is constructed to facilitate this study, namely, the Wiki Movie Knowledge Graph (WikiMKG). The experimental results show that KERL achieves state-of-the-art results in both recommendation and response generation tasks. Our code is publicly available at the link: https://github.com/icedpanda/KERL.

UltraCLR: Contrastive Representation Learning Framework for Ultrasound-based Sensing

We propose UltraCLR, a new contrastive learning framework that fuses dual modulation ultrasonic sensing signals to enhance gesture representation. Most existing ultrasound-based gesture recognition tasks rely on a large amount of manually labeled samples to learn task-specific representations via end-to-end training. However, they cannot exploit unlabeled continuous gesture signals that are easy to collect. Inspired by recent self-supervised learning techniques, UltraCLR aims to autonomously learn a ubiquitous gesture signal representation that can benefit all tasks from low-cost unlabeled signals. We use the STFT heatmap as a secondary input and leverage the contrastive learning framework to improve the high-quality Channel Impulsive Response heatmap input representations. The learned representations can better represent the spatial-position information and intermediate states of gesture movement. With the representation learned by UltraCLR, we can greatly reduce the complexity of downstream gesture recognition tasks so that they can be completed using a simple classifier trained with a small training set and a lower computational cost. Our experimental results show that UltraCLR outperforms state-of-the-art gesture recognition systems with only a few labeled samples and achieves more than 85% reduction in computational complexity and over 9× improvement in inference speed.

VOLTA: an enVironment-aware cOntrastive ceLl represenTation leArning for histopathology

In clinical oncology, many diagnostic tasks rely on the identification of cells in histopathology images. While supervised machine learning techniques necessitate the need for labels, providing manual cell annotations is time-consuming. In this paper, we propose a self-supervised framework (enVironment-aware cOntrastive cell represenTation learning: VOLTA) for cell representation learning in histopathology images using a technique that accounts for the cell’s mutual relationship with its environment. We subject our model to extensive experiments on data collected from multiple institutions comprising over 800,000 cells and six cancer types. To showcase the potential of our proposed framework, we apply VOLTA to ovarian and endometrial cancers and demonstrate that our cell representations can be utilized to identify the known histotypes of ovarian cancer and provide insights that link histopathology and molecular subtypes of endometrial cancer. Unlike supervised models, we provide a framework that can empower discoveries without any annotation data, even in situations where sample sizes are limited.

A multi-view representation learning framework for commonsense knowledge bases

Commonsense knowledge bases play an essential role in a wide range of natural language processing tasks. This paper studies the problem of representation learning for commonsense knowledge bases to effectively incorporate their knowledge into numerical models. Most existing knowledge base representation learning methods are difficult to apply to commonsense knowledge bases since they are much sparser than general knowledge bases. Hence, in this paper, we propose a novel method for commonsense knowledge base representation learning. Specifically, we first model the nodes from multiple views, including word/phase information, context information, and graph information. Then, we design a scoring function to measure whether the commonsense triplets are established through relation representation learning. We conduct extensive experiments on two tasks and the results show that our proposed model outperforms other knowledge base representation learning methods.

CosUKG: A Representation Learning Framework for Uncertain Knowledge Graphs

Knowledge graphs have been extensively studied and applied, but most of these studies assume that the relationship facts in the knowledge graph are correct and deterministic. However, in the objective world, there inevitably exist uncertain relationship facts. The existing research lacks effective representation of such uncertain information. In this regard, we propose a novel representation learning framework called CosUKG, which is specifically designed for uncertain knowledge graphs. This framework models uncertain information by measuring the cosine similarity between transformed vectors and actual target vectors, effectively integrating uncertainty into the embedding process of the knowledge graph while preserving its structural information. Through multiple experiments on three public datasets, the superiority of the CosUKG framework in representing uncertain knowledge graphs is demonstrated. It achieves improved representation accuracy of uncertain information without increasing model complexity or weakening structural information.

Biometric identity recognition based on contrastive positive-unlabeled learning

With the development of neural networks, identity recognition techniques utilizing human physiological signals have found applications in healthcare systems. However, many current physiological signal datasets have to spend a lot of resources in the pre-labeling phase, which leads to longer training times. To reduce the cost of labeling, we propose a self-supervised time-series representation learning framework based on contrastive positive-unlabeled learning (TS − CPUL), to learn the representations of temporal neighborhood sequences. To obtain the representations, the unit root test divides the unlabeled time series into neighborhood samples and unlabeled samples. Secondly, we use neighborhood sample representations and unlabeled sample representations to do self-supervised contrastive learning and positive-unlabeled learning with a negative sample selector. Both approaches can deepen the model’s understanding of the sample representation. Lastly, to enhance the model’s capability in discriminating between ambiguous sample pairs, we design the distance-polarized matrix. The proposed TS-CPUL is evaluated through experiments on three physiological signal datasets. The results demonstrate its effectiveness in understanding time-series representations and performing well in classification. For the identity recognition task, we achieve accuracy comparable to the supervised model by fine-tuning the model with only 10% of the labeled data.

A Condition Knowledge Representation and Feedback Learning Framework for Dynamic Optimization of Integrated Energy Systems.

An optimal energy scheduling strategy for integrated energy systems (IESs) can effectively improve the energy utilization efficiency and reduce carbon emissions. Due to the large-scale state space of IES caused by uncertain factors, it would be beneficial for the model training process to formulate a reasonable state-space representation. Thus, a condition knowledge representation and feedback learning framework based on contrastive reinforcement learning is designed in this study. Considering that different state conditions would bring inconsistent daily economic costs, a dynamic optimization model based on deterministic deep policy gradient is established, so that the condition samples can be partitioned according to the preoptimized daily costs. In order to represent the overall conditions on a daily basis and constrain the uncertain states in the IES environment, the state-space representation is constructed by a contrastive network considering the time dependence of variables. A Monte-Carlo policy gradient-based learning architecture is further proposed to optimize the condition partition and improve the policy learning performance. To verify the effectiveness of the proposed method, typical load operation scenarios of an IES are used in our simulations. The human experience strategies and state-of-the-art approaches are selected for comparisons. The results validate the advantages of the proposed approach in terms of cost effectiveness and ability to adapt in uncertain environments.

Graph Representation and Prototype Learning for webly supervised fine-grained image recognition

Webly supervised fine-grained image recognition (FGIR) learns to distinguish sub-ordinate categories based on webly-retrieved data, which can dramatically alleviate the dependency on manually annotated labels. This is quite a challenging task due to the heavy noise labels and the inherent dilemma of small inter-class variance and large intra-class variance. Current webly supervised algorithms learn holistic category prototypes to help correct noisy labels but ignore local features that can distinguish different sub-ordinate categories. In this work, we propose a Graph Representation and Prototype Learning (GRPL) framework to automatically mine discriminative local regions and their interactions with holistic image to learn instance graph representation both category graph prototype to help correct noisy labels and retrieve out-of-distribution (OOD) samples. Specifically, an attention-focused module is designed to extract the discriminative regions and then build a structured graph to correlate them with the holistic image for each instance and an identical graph to model holistic-local correlations for each category. Next, we apply two stacked graph convolution networks to explore holistic-local interaction within each graph and across two graphs to learn graph representation for each instance and graph prototype for each category. Finally, the similarities between the instance-level and prototype-level graph representation are learned to help correct noisy labels and exclude OOD samples. Extensive experiments conducted on several datasets show the proposed approach achieves superior performance compared with current leading algorithms.

Contrastive BiLSTM-enabled Health Representation Learning for Remaining Useful Life Prediction

Remaining useful life (RUL) prediction is of vital significance in prognostics health management tasks. Due to powerful learning capabilities, deep learning methods, particularly long short-term memory (LSTM) have been widely applied in RUL prediction. However, many existing deep learning approaches overlook the inherent ordered relationship between samples in the direct mapping from sliced data to RUL pattern. To capture the faithful and ordered health representation of a given system, a Contrastive Bidirectional LSTM-enabled Health Representation Learning (CBHRL) framework is proposed. Firstly, the supervised contrastive regression loss (SupCR) is implemented to extract continuous health representation. The SupCR is designed to rank the similarity among health representations from different samples, prompting them highly correlated with linear RUL label. Among the process of contrastive learning, the series odd-even decomposition (SOED) method is devised to construct multi-view degradation data, which improves generalization ability. Finally, since the health representation is constructed on basis of similarity, a new similarity prediction method is proposed as the complement of regression prediction method. Experimental results show the health representations extracted by CBHRL achieve improved ratio ranging from a minimum of 17.19% to a maximum of 291.30% in monotonicity, smoothness and trendability.

Self-attention empowered graph convolutional network for structure learning and node embedding

In representation learning on graph-structured data, many popular graph neural networks (GNNs) fail to capture long-range dependencies, leading to performance degradation. Furthermore, this weakness is magnified when the concerned graph is characterized by heterophily (low homophily). To solve this issue, this paper proposes a novel graph representation learning framework called the graph convolutional network with self-attention (GCN-SA). The proposed scheme exhibits an exceptional generalization capability in node-level representation learning. The proposed GCN-SA contains two enhancements corresponding to edges and node features. For edges, we utilize a self-attention mechanism to design a stable and effective graph-structure-learning module that can capture the internal correlation between any pair of nodes. This graph-structure-learning module can identify reliable neighbors for each node from the entire graph. Regarding the node features, we modify the transformer block to enable it to assist the GCN in integrating valuable information from the entire graph. These two enhancements work in distinct ways to help our GCN-SA capture long-range dependencies, enabling it to perform representation learning on graphs with varying levels of homophily. The experimental results on real-world benchmark datasets demonstrate the effectiveness of the proposed GCN-SA. Compared to other outstanding GNN counterparts, the proposed GCN-SA is competitive. The source code is available at https://github.com/mengyingjiang/GCN-SA.

Unified cross-modality integration and analysis of T cell receptors and T cell transcriptomes by low-resource-aware representation learning

Single-cell RNA sequencing (scRNA-seq) and Tcell receptor sequencing (TCR-seq) are pivotal for investigating Tcell heterogeneity. Integrating these modalities, which is expected to uncover profound insights in immunology that might otherwise go unnoticed with a single modality, faces computational challenges due to the low-resource characteristics of the multimodal data. Herein, we present UniTCR, a novel low-resource-aware multimodal representation learning framework designed for the unified cross-modality integration, enabling comprehensive Tcell analysis. By designing a dual-modality contrastive learning module and a single-modality preservation module to effectively embed each modality into a common latent space, UniTCR demonstrates versatility in connecting TCR sequences with Tcell transcriptomes across various tasks, including single-modality analysis, modality gap analysis, epitope-TCR binding prediction, and TCR profile cross-modality generation, in a low-resource-aware way. Extensive evaluations conducted on multiple scRNA-seq/TCR-seq paired datasets showed the superior performance of UniTCR, exhibiting the ability of exploring the complexity of immune system.

A novel self-supervised representation learning framework based on time-frequency alignment and interaction for mechanical fault diagnosis

Recently, supervised learning methods for intelligent mechanical fault diagnosis have achieved considerable advancements, while they heavily rely on labeled information and neglect massive unlabeled signals. Consequently, some self-supervised research has emerged and employs contrastive learning to mine general knowledge from unlabeled samples. However, these methods emphasize independent feature extraction and lack deep feature interaction, which limits better representation learning. Furthermore, it also restricts the performance improvement of fault diagnosis. To fill this gap, a novel self-supervised representation learning framework based on time-frequency alignment and interaction (TFAI) is proposed to improve the diagnosis reliability under limited labeled data. Comprising a dual encoder and a cross-modal encoder based on Transformers, the proposed TFAI model takes paired time-frequency data as input for self-supervised pretraining. Two self-supervised learning strategies, namely time-frequency alignment and time-frequency interaction, are proposed to enable the model to gain valuable knowledge from unlabeled data. The former leverages a time-frequency contrastive loss to drive the dual encoder to extract profound features from unlabeled data. The latter involves utilizing a time-frequency matching loss to guide the cross-modal encoder to perform deep feature fusion in an unsupervised manner. The well pretrained TFAI model can learn informative representations and can be fine-tuned for specific downstream fault diagnosis tasks to improve the diagnosis accuracy. By conducting three experimental cases on an axial flow pump system and a public gearbox platform system, the effectiveness, generalizability, and domain adaptability of the proposed method are validated.

A stacked deep multi-kernel learning framework for blast induced flyrock prediction

Blasting operations are widely and frequently used for rock excavation in Civil and Mining constructions. Flyrock is one of the most important issues induced by blasting operations in open pit mines, and therefore needs to be well predicted in order to identify the safety zone to prevent the potential injuries. For this purpose, 234 sets of blasting data were collected from Sungun Copper Mine site, and a stacked deep multi-kernel learning (SD-MKL) framework was proposed to estimate the blast induced flyrock with confidence accuracy. The proposed model uses the stacking-based representation learning framework (S-RL) to achieve deep learning on small-scale training sets. A multi-kernel learning model (MKL) is used as the base module of S-RL framework, which uses a multi-feature fusion strategy to generate multiple kernels with different kernel length in order to reduce the effort in tuning hyperparameters. In addition, this study further enhanced the predictive capability of SD-MKL by introducing the boosting method into the S-RL framework and hence proposed a boosted SD-MKL model. For comparison purpose, several existing machine learning models were implemented, i.e., kernel ridge regression (KRR), support vector machine (SVM), random forest (RF), gradient boosting decision tree (GBDT), ensemble deep random vector functional link (edRVFL), SD-KRR and SD-SVM. Our experimental results showed that the proposed boosted SD-MKL achieved the best overall performance, with the lowest RMSE of 0.21/1.73, MAE of 0.08/0.78, and the highest VAF of 99.98/99.24.

An Optimal Edge-weighted Graph Semantic Correlation Framework for Multi-view Feature Representation Learning

In this article, we present an optimal edge-weighted graph semantic correlation (EWGSC) framework for multi-view feature representation learning. Different from most existing multi-view representation methods, local structural information and global correlation in multi-view feature spaces are exploited jointly in the EWGSC framework, leading to a new and high-quality multi-view feature representation. Specifically, a novel edge-weighted graph model is first conceptualized and developed to preserve local structural information in each of the multi-view feature spaces. Then, the explored structural information is integrated with a semantic correlation algorithm, labeled multiple canonical correlation analysis (LMCCA), to form a powerful platform for effectively exploiting local and global relations across multi-view feature spaces jointly. We then theoretically verified the relation between the upper limit on the number of projected dimensions and the optimal solution to the multi-view feature representation problem. To validate the effectiveness and generality of the proposed framework, we conducted experiments on five datasets of different scales, including visual-based (University of California Irvine (UCI) iris database, Olivetti Research Lab (ORL) face database, and Caltech 256 database), text-image-based (Wiki database), and video-based (Ryerson Multimedia Lab (RML) audio-visual emotion database) examples. The experimental results show the superiority of the proposed framework on multi-view feature representation over state-of-the-art algorithms.

Multi-task self-supervised time-series representation learning

Time-series representation learning is crucial for extracting meaningful representations from time-series data with temporal dynamics and sparse labels. Contrastive learning, a powerful technique for exploiting the inherent data patterns, has been applied to explore the diverse consistencies in time-series data, achieved through careful selection of contrastive pairs and design of appropriate loss. Encouraging such consistency is essential for acquiring comprehensive representations of time-series data. In this paper, we propose a new framework for time-series representation learning that combines the advantages of contextual, temporal, and transformation consistencies. This framework enables the network to learn general representations suitable for different tasks and domains. First, positive and negative pairs are generated to establish a multi-task learning setup. Then, contrastive losses are formulated to explore contextual, temporal, and transformation consistencies, which are jointly optimized to learn general time-series representations. In addition, we investigate an uncertainty weighting approach to enhance the effectiveness of multi-task learning. To evaluate the performance of our framework, we conduct experiments on three downstream tasks: time-series classification, forecasting, and anomaly detection. The experimental results demonstrate the superior performance of our framework compared to benchmark models across different tasks. Furthermore, our framework shows efficiency in cross-domain transfer learning scenarios.

Q-SupCon: Quantum-Enhanced Supervised Contrastive Learning Architecture within the Representation Learning Framework

In the evolving landscape of data privacy regulations, the challenge of providing extensive data for robust deep classification models arises. The accuracy of these models relies on the amount of training data, due to the multitude of parameters that require tuning. Unfortunately, obtaining such ample data proves challenging, particularly in domains like medical applications, where there is a pressing need for robust models for early disease detection but a shortage of labeled data. Nevertheless, the classical supervised contrastive learning models, have shown the potential to address this challenge up to a certain limit, by utilizing deep encoder models. However, recent advancements in quantum machine learning enable the extraction of meaningful representations from extremely limited and simple data. Thus, replacing classical counterparts in classical or hybrid quantum-classical supervised contrastive models enhances feature learning capability with minimal data. Therefore, this work proposes the Q-SupCon model, a fully quantum-powered supervised contrastive learning model comprising a quantum data augmentation circuit, quantum encoder, quantum projection head, and quantum variational classifier, enabling efficient image classification with minimal labeled data. Furthermore, the novel model attains 80%, 60%, and 80% test accuracy on MNIST, KMNIST, and FMNIST datasets, marking a significant advancement in addressing the data scarcity challenge.

From Latent Dynamics to Meaningful Representations.

While representation learning has been central to the rise of machine learning and artificial intelligence, a key problem remains in making the learned representations meaningful. For this, the typical approach is to regularize the learned representation through prior probability distributions. However, such priors are usually unavailable or are ad hoc. To deal with this, recent efforts have shifted toward leveraging the insights from physical principles to guide the learning process. In this spirit, we propose a purely dynamics-constrained representation learning framework. Instead of relying on predefined probabilities, we restrict the latent representation to follow overdamped Langevin dynamics with a learnable transition density─a prior driven by statistical mechanics. We show that this is a more natural constraint for representation learning in stochastic dynamical systems, with the crucial ability to uniquely identify the ground truth representation. We validate our framework for different systems including a real-world fluorescent DNA movie data set. We show that our algorithm can uniquely identify orthogonal, isometric, and meaningful latent representations.

A Survey of Trustworthy Representation Learning Across Domains

As AI systems have obtained significant performance to be deployed widely in our daily live and human society, people both enjoy the benefits brought by these technologies and suffer many social issues induced by these systems. To make AI systems good enough and trustworthy, plenty of researches have been done to build guidelines for trustworthy AI systems. Machine learning is one of the most important parts for AI systems and representation learning is the fundamental technology in machine learning. How to make the representation learning trustworthy in real-world application, e.g., cross domain scenarios, is very valuable and necessary for both machine learning and AI system fields. Inspired by the concepts in trustworthy AI, we proposed the first trustworthy representation learning across domains framework which includes four concepts, i.e, robustness, privacy, fairness, and explainability, to give a comprehensive literature review on this research direction. Specifically, we first introduce the details of the proposed trustworthy framework for representation learning across domains. Second, we provide basic notions and comprehensively summarize existing methods for the trustworthy framework from four concepts. Finally, we conclude this survey with insights and discussions on future research directions.

MNESEDA: A prior-guided subgraph representation learning framework for predicting disease-related enhancers

Enhancers, as a class of functional genomic regulatory elements, activate transcription of their target genes and play significant roles in the pathogenesis of complex human diseases. Therefore, identifying associations between enhancers and diseases could help to illuminate the potential disease pathogenesis mechanism. However, current methods for identifying disease-related enhancers mainly focus on biological experiments, which are time-consuming and cost-intensive. It is crucial to develop an effective computational model for detecting novel enhancer-disease associations (EDAs). To this end, we propose a prior-guided maximum neighbor enclosing subgraph (MNES) representation learning framework named MNESEDA, for enhancer-disease association prediction, where maximum connected motif (MCM) discovery is a key ingredient that constrains the subsequent MNES extraction strategy. The proposed residual information fusion captures representation difference between motif subgraphs and alleviates the over-smoothing issue. Extensive experiments conducted on three datasets demonstrate the superior performance of MNESEDA over recent state-of-the-art methods and the satisfactory generalization and transfer ability. Further case studies exhibit the potential ability of MNESEDA to predict novel disease-related enhancers. Overall, MNESEDA could serve as an effective guide to elucidate pathogenesis and etiology of human diseases and decipher potential disease biomarkers.