Latent Representation Learning Research Articles

Oscillating latent dynamics in robot systems during walking and reaching.

Sensorimotor control of complex, dynamic systems such as humanoids or quadrupedal robots is notoriously difficult. While artificial systems traditionally employ hierarchical optimisation approaches or black-box policies, recent results in systems neuroscience suggest that complex behaviours such as locomotion and reaching are correlated with limit cycles in the primate motor cortex. A recent result suggests that, when applied to a learned latent space, oscillating patterns of activation can be used to control locomotion in a physical robot. While reminiscent of limit cycles observed in primate motor cortex, these dynamics are unsurprising given the cyclic nature of the robot's behaviour (walking). In this preliminary investigation, we consider how a similar approach extends to a less obviously cyclic behaviour (reaching). This has been explored in prior work using computational simulations. But simulations necessarily make simplifying assumptions that do not necessarily correspond to reality, so do not trivially transfer to real robot platforms. Our primary contribution is to demonstrate that we can infer and control real robot states in a learnt representation using oscillatory dynamics during reaching tasks. We further show that the learned latent representation encodes interpretable movements in the robot's workspace. Compared to robot locomotion, the dynamics that we observe for reaching are not fully cyclic, as they do not begin and end at the same position of latent space. However, they do begin to trace out the shape of a cycle, and, by construction, they are driven by the same underlying oscillatory mechanics.

Latent Representation Learning for Geospatial Entities

Representation learning has been instrumental in the success of machine learning, offering compact and performant data representations for diverse downstream tasks. In the spatial domain, it has been pivotal in extracting latent patterns from various data types, including points, polylines, polygons, and networked structures. However, existing approaches often fall short of explicitly capturing both semantic and spatial information, relying on proxies and synthetic features. This paper presents GeoNN, a novel graph neural network-based model designed to learn spatially-aware embeddings for geospatial entities. GeoNN leverages edge features generated from geodesic functions, dynamically selecting relevant features based on relative locations. It introduces both transductive (GeoNN-T) and inductive (GeoNN-I) models, ensuring effective encoding of geospatial features and scalability with entity changes. Extensive experiments demonstrate GeoNN’s effectiveness in location-sensitive superpixel-based graphs and real-world points of interest, outperforming baselines across various evaluation measures.

A Principled Framework for Explainable Multimodal Disentanglement

Learning effective representations for data from multiple modalities is crucial in machine learning. Recent efforts focus on learning latent representations that integrate information from various modalities. These approaches generally assume simple or implicit relationships between different modalities and as a result are not able to accurately and explicitly depict the correlations among these modalities and lack explainability. To address this, we propose definitions and conditions for unsupervised multimodal disentanglement, offering guidelines for explicit disentanglement between modalities to enhance explainability. Furthermore, we have derived a novel objective function to explicitly separate multimodal data into components shared across modalities and components exclusive to each modality. The explicit guaranteed disentanglement is of great potential for downstream tasks. Benefiting from a cleverly designed network structure, we can visualize these disentangled representations, providing intuitive explainability. Experiments on a variety of multimodal datasets demonstrate that our objective can effectively disentangle information from different modalities while satisfying the disentangling conditions.

Long-term causal effects estimation via latent surrogates representation learning

Estimating long-term causal effects based on short-term surrogates is a significant but challenging problem in many real-world applications such as marketing and medicine. Most existing methods estimate causal effects in an idealistic and simplistic manner — disregarding unobserved surrogates and treating all short-term outcomes as surrogates. However, such methods are not well-suited to real-world scenarios where the partially observed surrogates are mixed with the proxies of unobserved surrogates among short-term outcomes. To address this issue, we develop our flexible method called LASER to estimate long-term causal effects in a more realistic situation where the surrogates are either observed or have observed proxies. In LASER, we employ an identifiable variational autoencoder to learn the latent surrogate representation by using all the surrogate candidates without the need to distinguish observed surrogates or proxies of unobserved surrogates. With the learned representation, we further devise a theoretically guaranteed and unbiased estimation of long-term causal effects. Extensive experimental results on the real-world and semi-synthetic datasets demonstrate the effectiveness of our proposed method.

Cluster structure augmented deep nonnegative matrix factorization with low-rank tensor learning

Clustering approaches based on deep nonnegative matrix factorization have received significant attention because it can learn hierarchical semantics from data in a layer-wise manner. However, latent representation learning and clustering are two separate processes in these models, leading to information loss and sub-optimal clusterings. Furthermore, if data contain complex structure or noise, the pre-defined graph in data space will not be precise enough for graph-regularized deep nonnegative matrix factorization models. Moreover, most of them maintain the similarity graph at the top layer, neglecting the information covered by other layers. In this paper, Cluster Structure Augmented Deep Nonnegative Matrix Factorization with Low-rank Tensor Learning is proposed. First, as representations at different layers cover different data abstractions, a learning mechanism is leveraged to generate multiple local similarity graphs based on representations from different layers, thus maintaining diversity across layers. Then, a low-rank third-order tensor is constructed by stacking these graphs to capture the high-order consistency across layers, thus capturing the intrinsic property of data. Third, clustering is integrated into the framework, allowing the cluster structure to facilitate multi-layer matrix factorization and graph structure learning, thereby generating discriminative representations. Experimental results on several datasets demonstrate that the proposed model outperforms several state-of-the-art methods.

Empowering Predictive Modeling by GAN-based Causal Information Learning

Generally speaking, we can easily specify many causal relationships in the prediction tasks of ubiquitous computing, such as human activity prediction, mobility prediction, and health prediction. However, most of the existing methods in these fields failed to take advantage of this prior causal knowledge. They typically make predictions only based on correlations in the data, which hinders the prediction performance in real-world scenarios because a distribution shift between training data and testing data generally exists. To fill in this gap, we proposed a G AN-based C ausal I nformation L earning prediction framework (GCIL), which can effectively leverage causal information to improve the prediction performance of existing ubiquitous computing deep learning models. Specifically, faced with a unique challenge that the treatment variable, referring to the intervention that influences the target in a causal relationship, is generally continuous in ubiquitous computing, the framework employs a representation learning approach with a GAN-based deep learning model. By projecting all variables except the treatment into a latent space, it effectively minimizes confounding bias and leverages the learned latent representation for accurate predictions. In this way, it deals with the continuous treatment challenge, and in the meantime, it can be easily integrated with existing deep learning models to lift their prediction performance in practical scenarios with causal information. Extensive experiments on two large-scale real-world datasets demonstrate its superior performance over multiple state-of-the-art baselines. We also propose an analytical framework together with extensive experiments to empirically show that our framework achieves better performance gain under two conditions: when the distribution differences between the training data and the testing data are more significant and when the treatment effects are larger. Overall, this work suggests that learning causal information is a promising way to improve the prediction performance of ubiquitous computing tasks. We open both our dataset and code 1 and call for more research attention in this area.

Multi-view clustering via pseudo-label guide learning and latent graph structure recovery

Multi-view clustering (MvC) accomplishes sample classification tasks by exploring information from different views. Currently, researchers have paid greater attention to graph-based MvC methods. However, most existing methods only consider the original graph structure and pay relatively little attention to the graph structure in the latent space. In addition, most methods need to pay more attention to the consistency of information on different labels. Otherwise, this may lead to the loss of label information. This paper presents a new multi-view clustering framework to address the above issues. The proposed method considers both the information in the latent space and the original data space, which firstly obtains the pseudo-label by latent representation learning and then lets the pseudo-label guide the learning of the complementary information between the raw data views. To ensure the integrity of the data structure, a latent graph structure recovery strategy is designed in the latent space. Finally, an enhanced label fusion strategy is designed to fusion the different types of labels, yielding an information-rich label matrix for clustering. Experimental results demonstrate the proposed method’s effectiveness compared to other advanced approaches.

Multi-label feature selection via latent representation learning and dynamic graph constraints

As an effective method to deal with the curse of dimensionality, multi-label feature selection aims to select the most representative subset of features by eliminating unfavorable features. Although great progress has been made in this field, how to mine adequate supervisory information from multi-label data remains a key challenge. Compared to the latent information of instances, the latent information of instance relevance contains both the basic information of instances and the latent relevance between instances. Base on this knowledge, we propose a novel multi-label feature selection method named LRDG that explores latent representation learning and dynamic graph constraints. Specifically, we introduce the latent representation of instance relevance as supervisory information for pseudo-label learning, and minimize information loss during pseudo-label learning by means of the label manifold, the non-negative constraints, and the minimization of the Frobenius norm between pseudo-labels and ground-truth labels. In addition, considering the shortcomings brought by traditional graph regularization, we propose to use the dynamic graph constructed from low-dimensional pseudo-labels to constrain feature weights. Extensive experiments on various multi-label datasets demonstrate the effectiveness of the proposed method. The code is available at https://github.com/yunbao520/LRDG.

Traceable Group-Wise Self-Optimizing Feature Transformation Learning: A Dual Optimization Perspective

Feature transformation aims to reconstruct an effective representation space by mathematically refining the existing features. It serves as a pivotal approach to combat the curse of dimensionality, enhance model generalization, mitigate data sparsity, and extend the applicability of classical models. Existing research predominantly focuses on domain knowledge-based feature engineering or learning latent representations. However, these methods, while insightful, lack full automation and fail to yield a traceable and optimal representation space. An indispensable question arises: Can we concurrently address these limitations when reconstructing a feature space for a machine learning task? Our initial work took a pioneering step towards this challenge by introducing a novel self-optimizing framework. This framework leverages the power of three cascading reinforced agents to automatically select candidate features and operations for generating improved feature transformation combinations. Despite the impressive strides made, there was room for enhancing its effectiveness and generalization capability. In this extended journal version, we advance our initial work from two distinct yet interconnected perspectives: 1) We propose a refinement of the original framework, which integrates a graph-based state representation method to capture the feature interactions more effectively and develop different Q-learning strategies to alleviate Q-value overestimation further. 2) We utilize a new optimization technique (actor-critic) to train the entire self-optimizing framework in order to accelerate the model convergence and improve the feature transformation performance. Finally, to validate the improved effectiveness and generalization capability of our framework, we perform extensive experiments and conduct comprehensive analyses. These provide empirical evidence of the strides made in this journal version over the initial work, solidifying our framework’s standing as a substantial contribution to the field of automated feature transformation. To improve the reproducibility, we have released the associated code and data by the Github link https://github.com/coco11563/TKDD2023_code.

Prediction of drug-disease associations based on reinforcement symmetric metric learning and graph convolution network.

Accurately identifying novel indications for drugs is crucial in drug research and discovery. Traditional drug discovery is costly and time-consuming. Computational drug repositioning can provide an effective strategy for discovering potential drug-disease associations. However, the known experimentally verified drug-disease associations is relatively sparse, which may affect the prediction performance of the computational drug repositioning methods. Moreover, while the existing drug-disease prediction method based on metric learning algorithm has achieved better performance, it simply learns features of drugs and diseases only from the drug-centered perspective, and cannot comprehensively model the latent features of drugs and diseases. In this study, we propose a novel drug repositioning method named RSML-GCN, which applies graph convolutional network and reinforcement symmetric metric learning to predict potential drug-disease associations. RSML-GCN first constructs a drug-disease heterogeneous network by integrating the association and feature information of drugs and diseases. Then, the graph convolutional network (GCN) is applied to complement the drug-disease association information. Finally, reinforcement symmetric metric learning with adaptive margin is designed to learn the latent vector representation of drugs and diseases. Based on the learned latent vector representation, the novel drug-disease associations can be identified by the metric function. Comprehensive experiments on benchmark datasets demonstrated the superior prediction performance of RSML-GCN for drug repositioning.

A Novel Feature Engineering Method Based on Latent Representation Learning for Radiomics: Application in NSCLC Subtype Classification.

Radiomics refers to the high-throughput extraction of quantitative features from medical images, and is widely used to construct machine learning models for the prediction of clinical outcomes, while feature engineering is the most important work in radiomics. However, current feature engineering methods fail to fully and effectively utilize the heterogeneity of features when dealing with different kinds of radiomics features. In this work, latent representation learning is first presented as a novel feature engineering approach to reconstruct a set of latent space features from original shape, intensity and texture features. This proposed method projects features into a subspace called latent space, in which the latent space features are obtained by minimizing a unique hybrid loss function including a clustering-like loss and a reconstruction loss. The former one ensures the separability among each class while the latter one narrows the gap between the original features and latent space features. Experiments were performed on a multi-center non-small cell lung cancer (NSCLC) subtype classification dataset from 8 international open databases. Results showed that compared with four traditional feature engineering methods (baseline, PCA, Lasso and L2,1-norm minimization), latent representation learning could significantly improve the classification performance of various machine learning classifiers on the independent test set (all p<0.001). Further on two additional test sets, latent representation learning also showed a significant improvement in generalization performance. Our research shows that latent representation learning is a more effective feature engineering method, which has the potential to be used as a general technology in a wide range of radiomics researches.

Semantics Disentangling for Cross-modal Retrieval.

Cross-modal retrieval (e.g., query a given image to obtain a semantically similar sentence, and vice versa) is an important but challenging task, as the heterogeneous gap and inconsistent distributions exist between different modalities. The dominant approaches struggle to bridge the heterogeneity by capturing the common representations among heterogeneous data in a constructed subspace which can reflect the semantic closeness. However, insufficient consideration is taken into the fact that learned latent representations are actually heavily entangled with those semantic-unrelated features, which obviously further compounds the challenges of cross-modal retrieval. To alleviate the difficulty, this work makes an assumption that the data are jointly characterized by two independent features: semantic-shared and semantic-unrelated representations. The former presents characteristics of consistent semantics shared by different modalities, while the latter reflects the characteristics with respect to the modality yet unrelated to semantics, such as background, illumination, and other low-level information. Therefore, this paper aims to disentangle the shared semantics from the entangled features, andthus the purer semantic representation can promote the closeness of paired data. Specifically, this paper designs a novel Semantics Disentangling approach for Cross-Modal Retrieval (termed as SDCMR) to explicitly decouple the two different features based on variational auto-encoder. Next, the reconstruction is performed by exchanging shared semantics to ensure the learning of semantic consistency. Moreover, a dual adversarial mechanism is designed to disentangle the two independent features via a pushing-and-pulling strategy. Comprehensive experiments on four widely used datasets demonstrate the effectiveness and superiority of the proposed SDCMR method by achieving a new bar on performance when compared against 15 state-of-the-art methods.

Low-Rank Tensor Regularized Views Recovery for Incomplete Multiview Clustering.

In real applications, it is often that the collected multiview data contain missing views. Most existing incomplete multiview clustering (IMVC) methods cannot fully utilize the underlying information of missing data or sufficiently explore the consistent and complementary characteristics. In this article, we propose a novel Low-rAnk Tensor regularized viEws Recovery (LATER) method for IMVC, which jointly reconstructs and utilizes the missing views and learns multilevel graphs for comprehensive similarity discovery in a unified model. The missing views are recovered from a common latent representation, and the recovered views conversely improve the learning of shared patterns. Based on the shared subspace representations and recovered complete multiview data, the multilevel graphs are learned by self-representation to fully exploit the consistent and complementary information among views. Besides, a tensor nuclear norm regularizer is introduced to pursue the global low-rank property and explore the interview correlations. An alternating direction minimization algorithm is presented to optimize the proposed model. Moreover, a new initialization method is proposed to promote the effectiveness of our method for latent representation learning and missing data recovery. Extensive experiments demonstrate that our method outperforms the state-of-the-art approaches.

Learning Latent Representation with Limited Labels for IoT Anomaly Detection

Abstract— Malware detection is a critical challenge in the current era, especially for IoT devices. Previous studies have applied analytic techniques to reduce data size and extract valuable information. However, most of these studies count on a considerable quantity of outliers to perform anomaly detection. In this paper, we propose an enhanced method (named FeaWAD*) that improves the data encoding strategy based on the FeaWAD network [1]. These models require only a small fraction of anomalies for training. We evaluate the FeaWAD* method on the N-BaIoT dataset with various test scenarios for detecting known attacks as well as unknown future attacks. The experimental results demonstrate that the FeaWAD* method outperforms the original model FeaWAD and other popular anomaly detection methods such as Isolation Forest, Local Outlier Factor, and One-class Support Vector Machine. We also measure the time-based effectiveness of the model to assess its practical applicability.

Adaptive-weighted deep multi-view clustering with uniform scale representation

Multi-view clustering has attracted growing attention owing to its powerful capacity of multi-source information integration. Although numerous advanced methods have been proposed in past decades, most of them generally fail to distinguish the unequal importance of multiple views to the clustering task and overlook the scale uniformity of learned latent representation among different views, resulting in blurry physical meaning and suboptimal model performance. To address these issues, in this paper, we propose a joint learning framework, termed Adaptive-weighted deep Multi-view Clustering with Uniform scale representation (AMCU). Specifically, to achieve more reasonable multi-view fusion, we introduce an adaptive weighting strategy, which imposes simplex constraints on heterogeneous views for measuring their varying degrees of contribution to consensus prediction. Such a simple yet effective strategy shows its clear physical meaning for the multi-view clustering task. Furthermore, a novel regularizer is incorporated to learn multiple latent representations sharing approximately the same scale, so that the objective for calculating clustering loss cannot be sensitive to the views and thus the entire model training process can be guaranteed to be more stable as well. Through comprehensive experiments on eight popular real-world datasets, we demonstrate that our proposal performs better than several state-of-the-art single-view and multi-view competitors.

A Comprehensive Survey on Multi-View Clustering

The development of information gathering and extraction technology has led to the popularity of multi-view data, which enables samples to be seen from numerous perspectives. Multi-view clustering (MVC), which groups data samples by leveraging complementary and consensual information from several views, is gaining popularity. Despite the rapid evolution of MVC approaches, there has yet to be a study that provides a full MVC roadmap for both stimulating technical improvements and orienting research newbies to MVC. In this article, we review recent MVC techniques with the purpose of exhibiting the concepts of popular methodologies and their advancements. This survey not only serves as a unique MVC comprehensive knowledge for researchers but also has the potential to spark new ideas in MVC research. We summarise a large variety of current MVC approaches based on two technical mechanisms: heuristic-based multi-view clustering (HMVC) and neural network-based multi-view clustering (NNMVC). We end with four technological approaches within the category of HMVC: nonnegative matrix factorisation, graph learning, latent representation learning, and tensor learning. Deep representation learning and deep graph learning are two technical methods that we demonstrate in NNMVC. We also show 15 publicly available multi-view datasets and examine how representative MVC approaches perform on them. In addition, this study identifies the potential research directions that may require further investigation in order to enhance the further development of MVC.

Semi‐supervised domain adaptation via subspace exploration

AbstractRecent methods of learning latent representations in Domain Adaptation (DA) often entangle the learning of features and exploration of latent space into a unified process. However, these methods can cause a false alignment problem and do not generalise well to the alignment of distributions with large discrepancy. In this study, the authors propose to explore a robust subspace for Semi‐Supervised Domain Adaptation (SSDA) explicitly. To be concrete, for disentangling the intricate relationship between feature learning and subspace exploration, the authors iterate and optimise them in two steps: in the first step, the authors aim to learn well‐clustered latent representations by aggregating the target feature around the estimated class‐wise prototypes; in the second step, the authors adaptively explore a subspace of an autoencoder for robust SSDA. Specially, a novel denoising strategy via class‐agnostic disturbance to improve the discriminative ability of subspace is adopted. Extensive experiments on publicly available datasets verify the promising and competitive performance of our approach against state‐of‐the‐art methods.

Trade-offs in the latent representation of microstructure evolution

Characterizing and quantifying microstructure evolution is critical to forming quantitative relationships between material processing conditions, resulting microstructure, and observed properties. Machine-learning methods are increasingly accelerating the development of these relationships by treating microstructure evolution as a pattern recognition problem, discovering relationships explicitly or implicitly. These methods often rely on identifying low-dimensional microstructural fingerprints as latent variables. However, using inappropriate latent variables can lead to challenges in learning meaningful relationships. In this work, we survey and discuss the ability of various linear and nonlinear dimensionality reduction methods including principal component analysis, autoencoders, and diffusion maps to quantify and characterize the learned latent space microstructural representations and their time evolution. We characterize latent spaces by their ability to represent high-dimensional microstructural data in terms of compression achieved as a function of the number of latent dimensions required to represent the data accurately, their accuracy based on their reconstruction performance, and the smoothness of the microstructural trajectories in latent dimension. We quantify these metrics for common microstructure evolution problems in material science including spinodal decomposition of a binary metallic alloy, thin film deposition of a binary metallic alloy, dendritic growth, and grain growth in a polycrystal. This study provides considerations and guidelines for choosing dimensionality reduction methods when considering materials problems that involve high dimensional data and a variety of features over a range of lengths and time scales.

Finite-Sample Analysis of Deep CCA-Based Unsupervised Post-Nonlinear Multimodal Learning.

Canonical correlation analysis (CCA) has been essential in unsupervised multimodal/multiview latent representation learning and data fusion. Classic CCA extracts shared information from multiple modalities of data using linear transformations. In recent years, deep neural networks-based nonlinear feature extractors were combined with CCA to come up with new variants, namely the ``DeepCCA'' line of work. These approaches were shown to have enhanced performance in many applications. However, theoretical supports of DeepCCA are often lacking. To address this challenge, the recent work of Lyu and Fu (2020) showed that, under a reasonable postnonlinear generative model, a carefully designed DeepCCA criterion provably removes unknown distortions in data generation and identifies the shared information across modalities. Nonetheless, a critical assumption used by Lyu and Fu (2020) for identifiability analysis was that unlimited data is available, which is unrealistic. This brief paper puts forth a finite-sample analysis of the DeepCCA method by Lyu and Fu (2020). The main result is that the finite-sample version of the method can still estimate the shared information with a guaranteed accuracy when the number of samples is sufficiently large. Our analytical approach is a nontrivial integration of statistical learning, numerical differentiation, and robust system identification, which may be of interest beyond the scope of DeepCCA and benefit other unsupervised learning paradigms.

Learning to predict prostate cancer recurrence from tissue images

Roughly 30% of men with prostate cancer who undergo radical prostatectomy will suffer biochemical cancer recurrence (BCR). Accurately predicting which patients will experience BCR could identify who would benefit from increased surveillance or adjuvant therapy. Unfortunately, no current method can effectively predict this. We develop and evaluate PathCLR, a novel semi-supervised method that learns a model that can use hematoxylin and eosin (H&E)-stained tissue microarrays (TMAs) to predict prostate cancer recurrence within 5 years after diagnosis. The learning process involves 2 sequential steps: PathCLR (a) first employs self-supervised learning to generate effective feature representations of the input images, then (b) feeds these learned features into a fully supervised neural network classifier to learn a model for predicting BCR. We conducted training and evaluation using 2 large prostate cancer datasets: (1) the Cooperative Prostate Cancer Tissue Resource (CPCTR) with 374 patients, including 189 who experienced BCR, and (2) the Johns Hopkins University (JHU) prostate cancer dataset of 646 patients, with 451 patients having BCR. PathCLR’s (10-fold cross-validation) F1 score was 0.61 for CPCTR and 0.85 for JHU. This was statistically superior (paired t-test with P <.05) to the best-learned model that relied solely on clinicopathological features, including PSA level, primary and secondary Gleason Grade, etc. We attribute the improvement of PathCLR over models using only clinicopathological features to its utilization of both learned latent representations of tissue core images and clinicopathological features. This finding suggests that there is essential predictive information in tissue images at the time of surgery that goes beyond the knowledge obtained from reported clinicopathological features, helping predict the patient’s 5-year outcome.