Representation Learning Research Articles

Graph-guided imputation-free incomplete multi-view clustering

Recently, multi-view clustering methods have garnered considerable attention and have been applied in various domains. However, in practical scenarios, some samples may lack specific views, giving rise to the challenge of incomplete multi-view clustering. While some methods focus on completing missing data, incorrect completion can negatively affect representation learning. Moreover, separating completion and representation learning prevents the attainment of an optimal representation. Other methods eschew completion but singularly concentrate on either feature information or graph information, thus failing to achieve comprehensive representations. To address these challenges, we propose a graph-guided, imputation-free method for incomplete multi-view clustering. Unlike completion-based methods, our approach aims to maximize the utilization of existing information by simultaneously considering feature and graph information. This is realized through the feature learning component and the graph learning component. Introducing a degradation network, the former reconstructs view-specific representations proximate to available samples from a unified representation, seamlessly integrating feature information into the unified representation. Leveraging the semi-supervised idea, the latter utilizes reliable graph information from available samples to guide the learning of the unified representation. These two components collaborate to acquire a comprehensive unified representation for multi-view clustering. Extensive experiments conducted on real datasets demonstrate the effectiveness and competitiveness of the proposed method when compared with other state-of-the-art methods. Our code will be released on https://github.com/yff-java/GIMVC/.

A Derivative Topic Dissemination Model Based on Representation Learning and Topic Relevance

A Derivative Topic Dissemination Model Based on Representation Learning and Topic Relevance

Vision + X: A Survey on Multimodal Learning in the Light of Data.

We are perceiving and communicating with the world in a multisensory manner, where different information sources are sophisticatedly processed and interpreted by separate parts of the human brain to constitute a complex, yet harmonious and unified sensing system. To endow the machines with true intelligence, multimodal machine learning that incorporates data from various sources has become an increasingly popular research area with emerging technical advances in recent years. In this paper, we present a survey on multimodal machine learning from a novel perspective considering not only the purely technical aspects but also the intrinsic nature of different data modalities. We analyze the commonness and uniqueness of each data format mainly ranging from vision, audio, text, and motions, and then present the methodological advancements categorized by the combination of data modalities, such as Vision+Text, with slightly inclined emphasis on the visual data. We investigate the existing literature on multimodal learning from both the representation learning and downstream application levels, and provide an additional comparison in the light of their technical connections with the data nature, e.g., the semantic consistency between image objects and textual descriptions, and the rhythm correspondence between video dance moves and musical beats. We hope that the exploitation of the alignment as well as the existing gap between the intrinsic nature of data modality and the technical designs, will benefit future research studies to better address a specific challenge related to the concrete multimodal task, prompting a unified multimodal machine learning framework closer to a real human intelligence system.

Graph-DTI: A New Model for Drug-target Interaction Prediction Based on Heterogenous Network Graph Embedding.

In this study, we aimed to develop a new end-to-end learning model called Graph-Drug-Target Interaction (DTI), which integrates various types of information in the heterogeneous network data, and to explore automatic learning of the topology-maintaining representations of drugs and targets, thereby effectively contributing to the prediction of DTI. Precise predictions of DTI can guide drug discovery and development. Most machine learning algorithms integrate multiple data sources and combine them with common embedding methods. However, the relationship between the drugs and target proteins is not well reported. Although some existing studies have used heterogeneous network graphs for DTI prediction, there are many limitations in the neighborhood information between the nodes in the heterogeneous network graphs. We studied the drug-drug interaction (DDI) and DTI from DrugBank Version 3.0, protein-protein interaction (PPI) from the human protein reference database Release 9, drug structure similarity from Morgan fingerprints of radius 2 and calculated by RDKit, and protein sequence similarity from Smith-Waterman score. Our study consists of three major components. First, various drugs and target proteins were integrated, and a heterogeneous network was established based on a series of data sets. Second, the graph neural networks-inspired graph auto-encoding method was used to extract high-order structural information from the heterogeneous networks, thereby revealing the description of nodes (drugs and proteins) and their topological neighbors. Finally, potential DTI prediction was made, and the obtained samples were sent to the classifier for secondary classification. The performance of Graph-DTI and all baseline methods was evaluated using the sums of the area under the precision-recall curve (AUPR) and the area under the receiver operating characteristic curve (AUC). The results indicated that Graph-DTI outperformed the baseline methods in both performance results. Compared with other baseline DTI prediction methods, the results showed that Graph-DTI had better prediction performance. Additionally, in this study, we effectively classified drugs corresponding to different targets and vice versa. The above findings showed that Graph-DTI provided a powerful tool for drug research, development, and repositioning. Graph- DTI can serve as a drug development and repositioning tool more effectively than previous studies that did not use heterogeneous network graph embedding.

HMS-TENet: A hierarchical multi-scale topological enhanced network based on EEG and EOG for driver vigilance estimation

Driving vigilance estimation is an important task for traffic safety. Nowadays, electroencephalography (EEG) and electrooculography (EOG) have made some achievements in vigilance estimation, but there are still some challenges: 1) The traditional approachs with direct multimodal fusion may face the problems of information redundancy and data dimensionality mismatch; 2) Capture key discriminative features during multimodal fusion without losing specific patterns to each modality. In order to solve the above problems, this paper proposes a approach with fusion of EEG and EOG features in split bands, which not only preserves the information about brain activities in different bands of EEG, but also effectively integrates the relevant information of EOG. On this basis, we further propose a hierarchical multi-scale topological enhanced network (HMS-TENet). This network first introduces a pyramid pooling structure (PPS) to capture contextual relationships from different discriminative perspectives. And then we design a selective convolutional structure (SCS) for adaptive sense-field selection, which enables us to mine the desired discriminative information in small-size features. In addition, we design a topology self-aware attention to enhance the learning of representations of complex topological relationships among EEG channels. Finally, the output of the model can be selected for both regression and classification tasks, providing higher flexibility and adaptability. We demonstrate the robustness, generalizability, and utility of the proposed method based on intra-subject and cross-subject experiments on the SEED-VIG public dataset. Codes are available at https://github.com/tangmeng28/HMS-TENet.

When Invariant Representation Learning Meets Label Shift: Insufficiency and Theoretical Insights.

As a crucial step toward real-world learning scenarios with changing environments, dataset shift theory and invariant representation learning algorithm have been extensively studied to relax the identical distribution assumption in classical learning setting. Among the different assumptions on the essential of shifting distributions, generalized label shift (GLS) is the latest developed one which shows great potential to deal with the complex factors within the shift. In this paper, we aim to explore the limitations of current dataset shift theory and algorithm, and further provide new insights by presenting a comprehensive understanding of GLS. From theoretical aspect, two informative generalization bounds are derived, and the GLS learner are proved to be sufficiently close to optimal target model from the Bayesian perspective. The main results show the insufficiency of invariant representation learning, and prove the sufficiency and necessity of GLS correction for generalization, which provide theoretical supports and innovations for exploring generalizable model under dataset shift. From methodological aspect, we provide a unified view of existing shift correction frameworks, and propose a kernel embedding-based correction algorithm (KECA) to minimize the generalization error and achieve successful knowledge transfer. Both theoretical results and extensive experiment evaluations demonstrate the sufficiency and necessity of GLS correction for addressing dataset shift and the superiority of proposed algorithm.

UniMiSS+: Universal Medical Self-Supervised Learning From Cross-Dimensional Unpaired Data.

Self-supervised learning (SSL) opens up huge opportunities for medical image analysis that is well known for its lack of annotations. However, aggregating massive (unlabeled) 3D medical images like computerized tomography (CT) remains challenging due to its high imaging cost and privacy restrictions. In our pilot study, we advocated bringing a wealth of 2D images like X-rays as compensation for the lack of 3D data, aiming to build a universal medical self-supervised representation learning framework, called UniMiSS. Especially, we designed a pyramid U-like medical Transformer (MiT) as the backbone to make UniMiSS possible to perform SSL with both 2D and 3D images. UniMiSS surpasses current 3D-specific SSL in effectiveness and versatility, excelling in various downstream tasks and overcoming the limitations of dimensionality. However, the initial version did not fully explore the anatomical correlations between 2D and 3D images due to the absence of paired multi-modal patient data. In this extension, we introduce UniMiSS+, which leverages digitally reconstructed radiographs (DRR) technology to simulate X-rays from CT volumes, providing access to paired data. Benefiting from the paired group, we introduce an extra pair-wise constraint to boost the cross modality correlation learning, which also can be adopted as a cross dimension regularization to further improve the representations. We conduct expensive experiments on multiple 3D/2D medical image analysis tasks, including segmentation and classification. The results show that our UniMiSS+ achieves promising performance on various downstream tasks, not only outperforming ImageNet pre-training and other advanced SSL counterparts but also improving the predecessor UniMiSS pre-training.

Object-Centric Representation Learning for Video Scene Understanding.

Depth-aware Video Panoptic Segmentation (DVPS) is a challenging task that requires predicting the semantic class and 3D depth of each pixel in a video, while also segmenting and consistently tracking objects across frames. Predominant methodologies treat this as a multi-task learning problem, tackling each constituent task independently, thus restricting their capacity to leverage interrelationships amongst tasks and requiring parameter tuning for each task. To surmount these constraints, we present Slot-IVPS, a new approach employing an object-centric model to acquire unified object representations, thereby facilitating the model's ability to simultaneously capture semantic and depth information. Specifically, we introduce a novel representation, Integrated Panoptic Slots (IPS), to capture both semantic and depth information for all panoptic objects within a video, encompassing background semantics and foreground instances. Subsequently, we propose an integrated feature generator and enhancer to extract depth-aware features, alongside the Integrated Video Panoptic Retriever (IVPR), which iteratively retrieves spatial-temporal coherent object features and encodes them into IPS. The resulting IPS can be effortlessly decoded into an array of video outputs, including depth maps, classifications, masks, and object instance IDs. We undertake comprehensive analyses across four datasets, attaining state-of-the-art performance in both Depth-aware Video Panoptic Segmentation and Video Panoptic Segmentation tasks.

SaaN 2L-GRL: Two-Level Graph Representation Learning Empowered With Subgraph-as-a-Node

SaaN 2L-GRL: Two-Level Graph Representation Learning Empowered With Subgraph-as-a-Node

Geometric Understanding of Discriminability and Transferability for Visual Domain Adaptation.

To overcome the restriction of identical distribution assumption, invariant representation learning for unsupervised domain adaptation (UDA) has made significant advances in computer vision and pattern recognition communities. In UDA scenario, the training and test data belong to different domains while the task model is learned to be invariant. Recently, empirical connections between transferability and discriminability have received increasing attention, which is the key to understand the invariant representations. However, theoretical study of these abilities and in-depth analysis of the learned feature structures are unexplored yet. In this work, we systematically analyze the essentials of transferability and discriminability from the geometric perspective. Our theoretical results provide insights into understanding the co-regularization relation and prove the possibility of learning these abilities. From methodology aspect, the abilities are formulated as geometric properties between domain/cluster subspaces (i.e., orthogonality and equivalence) and characterized as the relation between the norms/ranks of multiple matrices. Two optimization-friendly learning principles are derived, which also ensure some intuitive explanations. Moreover, a feasible range for the co-regularization parameters is deduced to balance the learning of geometric structures. Based on the theoretical results, a geometry-oriented model is proposed for enhancing the transferability and discriminability via nuclear norm optimization. Extensive experiment results validate the effectiveness of the proposed model in empirical applications, and verify that the geometric abilities can be sufficiently learned in the derived feasible range.

Label Deconvolution for Node Representation Learning on Large-Scale Attributed Graphs Against Learning Bias.

Node representation learning on attributed graphs-whose nodes are associated with rich attributes (e.g., texts and protein sequences)-plays a crucial role in many important downstream tasks. To encode the attributes and graph structures simultaneously, recent studies integrate pre-trained models with graph neural networks (GNNs), where pre-trained models serve as node encoders (NEs) to encode the attributes. As jointly training large NEs and GNNs on large-scale graphs suffers from severe scalability issues, many methods propose to train NEs and GNNs separately. Consequently, they do not take feature convolutions in GNNs into consideration in the training phase of NEs, leading to a significant learning bias relative to the joint training. To address this challenge, we propose an efficient label regularization technique, namely Label Deconvolution (LD), to alleviate the learning bias by a novel and highly scalable approximation to the inverse mapping of GNNs. The inverse mapping leads to an objective function that is equivalent to that by the joint training, while it can effectively incorporate GNNs in the training phase of NEs against the learning bias. More importantly, we show that LD converges to the optimal objective function values by the joint training under mild assumptions. Experiments demonstrate LD significantly outperforms state-of-the-art methods on Open Graph Benchmark datasets.

Disentangled Representation Learning.

Disentangled Representation Learning (DRL) aims to learn a model capable of identifying and disentangling the underlying factors hidden in the observable data in representation form. The process of separating underlying factors of variation into variables with semantic meaning benefits in learning explainable representations of data, which imitates the meaningful understanding process of humans when observing an object or relation. As a general learning strategy, DRL has demonstrated its power in improving the model explainability, controlability, robustness, as well as generalization capacity in a wide range of scenarios such as computer vision, natural language processing, and data mining. In this article, we comprehensively investigate DRL from various aspects including motivations, definitions, methodologies, evaluations, applications, and model designs. We first present two well-recognized definitions, i.e., Intuitive Definition and Group Theory Definition for disentangled representation learning. We further categorize the methodologies for DRL into four groups from the following perspectives, the model type, representation structure, supervision signal, and independence assumption. We also analyze principles to design different DRL models that may benefit different tasks in practical applications. Finally, we point out challenges in DRL as well as potential research directions deserving future investigations. We believe this work may provide insights for promoting the DRL research in the community.

IGCN: A Provably Informative GCN Embedding for Semi-Supervised Learning With Extremely Limited Labels.

Graph Neural Networks (GNNs) have gained much more attention in the representation learning for the graph-structured data. However, the labels are always limited in the graph, which easily leads to the overfitting problem and causes the poor performance. To solve this problem, we propose a new framework called IGCN, short for Informative Graph Convolutional Network, where the objective of IGCN is designed to obtain the informative embeddings via discarding the task-irrelevant information of the graph data based on the mutual information. As the mutual information for irregular data is intractable to compute, our framework is optimized via a surrogate objective, where two terms are derived to approximate the original objective. For the former term, it demonstrates that the mutual information between the learned embeddings and the ground truth should be high, where we utilize the semi-supervised classification loss and the prototype based supervised contrastive learning loss for optimizing it. For the latter term, it requires that the mutual information between the learned node embeddings and the initial embeddings should be high and we propose to minimize the reconstruction loss between them to achieve the goal of maximizing the latter term from the feature level and the layer level, which contains the graph encoder-decoder module and a novel architecture GCN Info. Moreover, we provably show that the designed GCN Info can better alleviate the information loss and preserve as much useful information of the initial embeddings as possible. Experimental results show that the IGCN outperforms the state-of-the-art methods on 7 popular datasets.

Unsupervised Part Discovery via Dual Representation Alignment.

Object parts serve as crucial intermediate representations in various downstream tasks, but part-level representation learning still has not received as much attention as other vision tasks. Previous research has established that Vision Transformer can learn instance-level attention without labels, extracting high-quality instance-level representations for boosting downstream tasks. In this paper, we achieve unsupervised part-specific attention learning using a novel paradigm and further employ the part representations to improve part discovery performance. Specifically, paired images are generated from the same image with different geometric transformations, and multiple part representations are extracted from these paired images using a novel module, named PartFormer. These part representations from the paired images are then exchanged to improve geometric transformation invariance. Subsequently, the part representations are aligned with the feature map extracted by a feature map encoder, achieving high similarity with the pixel representations of the corresponding part regions and low similarity in irrelevant regions. Finally, the geometric and semantic constraints are applied to the part representations through the intermediate results in alignment for part-specific attention learning, encouraging the PartFormer to focus locally and the part representations to explicitly include the information of the corresponding parts. Moreover, the aligned part representations can further serve as a series of reliable detectors in the testing phase, predicting pixel masks for part discovery. Extensive experiments are carried out on four widely used datasets, and our results demonstrate that the proposed method achieves competitive performance and robustness due to its part-specific attention.

ReiPool: Reinforced Pooling Graph Neural Networks for Graph-Level Representation Learning

ReiPool: Reinforced Pooling Graph Neural Networks for Graph-Level Representation Learning

Spatial-Temporal Cross-View Contrastive Pre-Training for Check-in Sequence Representation Learning

Spatial-Temporal Cross-View Contrastive Pre-Training for Check-in Sequence Representation Learning

Medical Image Segmentation Review: The Success of U-Net.

Automatic medical image segmentation is a crucial topic in the medical domain and successively a critical counterpart in the computer-aided diagnosis paradigm. U-Net is the most widespread image segmentation architecture due to its flexibility, optimized modular design, and success in all medical image modalities. Over the years, the U-Net model has received tremendous attention from academic and industrial researchers who have extended it to address the scale and complexity created by medical tasks. These extensions are commonly related to enhancing the U-Net's backbone, bottleneck, or skip connections, or including representation learning, or combining it with a Transformer architecture, or even addressing probabilistic prediction of the segmentation map. Having a compendium of different previously proposed U-Net variants makes it easier for machine learning researchers to identify relevant research questions and understand the challenges of the biological tasks that challenge the model. In this work, we discuss the practical aspects of the U-Net model and organize each variant model into a taxonomy. Moreover, to measure the performance of these strategies in a clinical application, we propose fair evaluations of some unique and famous designs on well-known datasets. Furthermore, we provide a comprehensive implementation library with trained models. In addition, for ease of future studies, we created an online list of U-Net papers with their possible official implementation.

Towards Context-Aware Emotion Recognition Debiasing From a Causal Demystification Perspective via De-Confounded Training.

Understanding emotions from diverse contexts has received widespread attention in computer vision communities. The core philosophy of Context-Aware Emotion Recognition (CAER) is to provide valuable semantic cues for recognizing the emotions of target persons by leveraging rich contextual information. Current approaches invariably focus on designing sophisticated structures to extract perceptually critical representations from contexts. Nevertheless, a long-neglected dilemma is that a severe context bias in existing datasets results in an unbalanced distribution of emotional states among different contexts, causing biased visual representation learning. From a causal demystification perspective, the harmful bias is identified as a confounder that misleads existing models to learn spurious correlations based on likelihood estimation, limiting the models' performance. To address the issue, we embrace causal inference to disentangle the models from the impact of such bias, and formulate the causalities among variables in the CAER task via a customized causal graph. Subsequently, we present a Contextual Causal Intervention Module (CCIM) to de-confound the confounder, which is built upon backdoor adjustment theory to facilitate seeking approximate causal effects during model training. As a plug-and-play component, CCIM can easily integrate with existing approaches and bring significant improvements. Systematic experiments on three datasets demonstrate the effectiveness of our CCIM.

Practical Vertical Federated Learning With Unsupervised Representation Learning

As societal concerns on data privacy recently increase, we have witnessed data silos among multiple parties in various applications. Federated learning emerges as a new learning paradigm that enables multiple parties to collaboratively train a machine learning model without sharing their raw data. Vertical federated learning, where each party owns different features of the same set of samples and only a single party has the label, is an important and challenging topic in federated learning. Communication costs among different parties have been a major hurdle for practical vertical learning systems. In this paper, we propose a novel communication-efficient vertical federated learning algorithm named FedOnce, which requires only one-shot communication among parties. To improve model accuracy and provide privacy guarantee, FedOnce features unsupervised learning representations in the federated setting and privacy-preserving techniques based on moments accountant. The comprehensive experiments on 10 datasets demonstrate that FedOnce achieves close performance compared to state-of-the-art vertical federated learning algorithms with much lower communication costs. Meanwhile, our privacy-preserving technique significantly outperforms the state-of-the-art approaches under the same privacy budget.

Graph Diffusion-Based Representation Learning for Sequential Recommendation

Graph Diffusion-Based Representation Learning for Sequential Recommendation