Latent Representation Learning Research Articles

Semantics Disentangling for Cross-modal Retrieval.

Cross-modal retrieval (e.g., query a given image to obtain a semantically similar sentence, and vice versa) is an important but challenging task, as the heterogeneous gap and inconsistent distributions exist between different modalities. The dominant approaches struggle to bridge the heterogeneity by capturing the common representations among heterogeneous data in a constructed subspace which can reflect the semantic closeness. However, insufficient consideration is taken into the fact that learned latent representations are actually heavily entangled with those semantic-unrelated features, which obviously further compounds the challenges of cross-modal retrieval. To alleviate the difficulty, this work makes an assumption that the data are jointly characterized by two independent features: semantic-shared and semantic-unrelated representations. The former presents characteristics of consistent semantics shared by different modalities, while the latter reflects the characteristics with respect to the modality yet unrelated to semantics, such as background, illumination, and other low-level information. Therefore, this paper aims to disentangle the shared semantics from the entangled features, andthus the purer semantic representation can promote the closeness of paired data. Specifically, this paper designs a novel Semantics Disentangling approach for Cross-Modal Retrieval (termed as SDCMR) to explicitly decouple the two different features based on variational auto-encoder. Next, the reconstruction is performed by exchanging shared semantics to ensure the learning of semantic consistency. Moreover, a dual adversarial mechanism is designed to disentangle the two independent features via a pushing-and-pulling strategy. Comprehensive experiments on four widely used datasets demonstrate the effectiveness and superiority of the proposed SDCMR method by achieving a new bar on performance when compared against 15 state-of-the-art methods.

Deep-learning enabled photonic nanostructure discovery in arbitrarily large shape sets via linked latent space representation learning

A cross-linked autoencoder neural network for photonics nanostructure discovery effectively links geometry and spectra representations. This approach aids in rapid multiple solution inverse design and assessing their fabrication sensitivity.

M 2 D-VAE: Self-Supervised Probabilistic Temporal- Spatial Latent Representation Learning for Unsupervised Industrial Operational Applications Under Missing Value Interference.

Due to sensor malfunctions and data transmission corruptions, the industrial process data collected commonly contain missing values. It poses a significant challenge for data-driven approaches in aggregating temporal-spatial correlations that reflect dependencies across both variables and times, which makes it difficult to directly carry out downstream industrial operational applications. In this study, a self-supervised representation learning model is proposed to extract probabilistic temporal-spatial latent variables (LVs) from sequential data under missing value interference. The extracted LVs can be utilized for typical industrial operational applications through a unified framework. First, a novel deep dynamic probabilistic latent variable model, named Markov dynamic variational autoencoder (MD-VAE), is proposed to explicitly model the temporal-spatial dependencies between LVs. The latent posteriors are Bayesian smoothed by global sequence information for effective variational inference (VI). Second, a self-supervised learning approach, termed masked MD-VAE (), is proposed to address the challenge of directly extracting temporal-spatial LVs under missing value interference. Controllable constraints with practical interpretations are introduced to balance the latent bottleneck capacity with reconstruction accuracy during model optimization. A unified framework is proposed to utilize the latent representations for typical industrial downstream tasks. Case studies conducted on a real-world multiphase flow process demonstrate the superiority of in unsupervised industrial operational applications including missing value imputation and dynamic process monitoring under missing value interference.

Learning Latent Representation with Limited Labels for IoT Anomaly Detection

Abstract— Malware detection is a critical challenge in the current era, especially for IoT devices. Previous studies have applied analytic techniques to reduce data size and extract valuable information. However, most of these studies count on a considerable quantity of outliers to perform anomaly detection. In this paper, we propose an enhanced method (named FeaWAD*) that improves the data encoding strategy based on the FeaWAD network [1]. These models require only a small fraction of anomalies for training. We evaluate the FeaWAD* method on the N-BaIoT dataset with various test scenarios for detecting known attacks as well as unknown future attacks. The experimental results demonstrate that the FeaWAD* method outperforms the original model FeaWAD and other popular anomaly detection methods such as Isolation Forest, Local Outlier Factor, and One-class Support Vector Machine. We also measure the time-based effectiveness of the model to assess its practical applicability.

Adaptive-weighted deep multi-view clustering with uniform scale representation

Multi-view clustering has attracted growing attention owing to its powerful capacity of multi-source information integration. Although numerous advanced methods have been proposed in past decades, most of them generally fail to distinguish the unequal importance of multiple views to the clustering task and overlook the scale uniformity of learned latent representation among different views, resulting in blurry physical meaning and suboptimal model performance. To address these issues, in this paper, we propose a joint learning framework, termed Adaptive-weighted deep Multi-view Clustering with Uniform scale representation (AMCU). Specifically, to achieve more reasonable multi-view fusion, we introduce an adaptive weighting strategy, which imposes simplex constraints on heterogeneous views for measuring their varying degrees of contribution to consensus prediction. Such a simple yet effective strategy shows its clear physical meaning for the multi-view clustering task. Furthermore, a novel regularizer is incorporated to learn multiple latent representations sharing approximately the same scale, so that the objective for calculating clustering loss cannot be sensitive to the views and thus the entire model training process can be guaranteed to be more stable as well. Through comprehensive experiments on eight popular real-world datasets, we demonstrate that our proposal performs better than several state-of-the-art single-view and multi-view competitors.

A Comprehensive Survey on Multi-View Clustering

The development of information gathering and extraction technology has led to the popularity of multi-view data, which enables samples to be seen from numerous perspectives. Multi-view clustering (MVC), which groups data samples by leveraging complementary and consensual information from several views, is gaining popularity. Despite the rapid evolution of MVC approaches, there has yet to be a study that provides a full MVC roadmap for both stimulating technical improvements and orienting research newbies to MVC. In this article, we review recent MVC techniques with the purpose of exhibiting the concepts of popular methodologies and their advancements. This survey not only serves as a unique MVC comprehensive knowledge for researchers but also has the potential to spark new ideas in MVC research. We summarise a large variety of current MVC approaches based on two technical mechanisms: heuristic-based multi-view clustering (HMVC) and neural network-based multi-view clustering (NNMVC). We end with four technological approaches within the category of HMVC: nonnegative matrix factorisation, graph learning, latent representation learning, and tensor learning. Deep representation learning and deep graph learning are two technical methods that we demonstrate in NNMVC. We also show 15 publicly available multi-view datasets and examine how representative MVC approaches perform on them. In addition, this study identifies the potential research directions that may require further investigation in order to enhance the further development of MVC.

Semi‐supervised domain adaptation via subspace exploration

AbstractRecent methods of learning latent representations in Domain Adaptation (DA) often entangle the learning of features and exploration of latent space into a unified process. However, these methods can cause a false alignment problem and do not generalise well to the alignment of distributions with large discrepancy. In this study, the authors propose to explore a robust subspace for Semi‐Supervised Domain Adaptation (SSDA) explicitly. To be concrete, for disentangling the intricate relationship between feature learning and subspace exploration, the authors iterate and optimise them in two steps: in the first step, the authors aim to learn well‐clustered latent representations by aggregating the target feature around the estimated class‐wise prototypes; in the second step, the authors adaptively explore a subspace of an autoencoder for robust SSDA. Specially, a novel denoising strategy via class‐agnostic disturbance to improve the discriminative ability of subspace is adopted. Extensive experiments on publicly available datasets verify the promising and competitive performance of our approach against state‐of‐the‐art methods.

Trade-offs in the latent representation of microstructure evolution

Characterizing and quantifying microstructure evolution is critical to forming quantitative relationships between material processing conditions, resulting microstructure, and observed properties. Machine-learning methods are increasingly accelerating the development of these relationships by treating microstructure evolution as a pattern recognition problem, discovering relationships explicitly or implicitly. These methods often rely on identifying low-dimensional microstructural fingerprints as latent variables. However, using inappropriate latent variables can lead to challenges in learning meaningful relationships. In this work, we survey and discuss the ability of various linear and nonlinear dimensionality reduction methods including principal component analysis, autoencoders, and diffusion maps to quantify and characterize the learned latent space microstructural representations and their time evolution. We characterize latent spaces by their ability to represent high-dimensional microstructural data in terms of compression achieved as a function of the number of latent dimensions required to represent the data accurately, their accuracy based on their reconstruction performance, and the smoothness of the microstructural trajectories in latent dimension. We quantify these metrics for common microstructure evolution problems in material science including spinodal decomposition of a binary metallic alloy, thin film deposition of a binary metallic alloy, dendritic growth, and grain growth in a polycrystal. This study provides considerations and guidelines for choosing dimensionality reduction methods when considering materials problems that involve high dimensional data and a variety of features over a range of lengths and time scales.

Finite-Sample Analysis of Deep CCA-Based Unsupervised Post-Nonlinear Multimodal Learning.

Canonical correlation analysis (CCA) has been essential in unsupervised multimodal/multiview latent representation learning and data fusion. Classic CCA extracts shared information from multiple modalities of data using linear transformations. In recent years, deep neural networks-based nonlinear feature extractors were combined with CCA to come up with new variants, namely the ``DeepCCA'' line of work. These approaches were shown to have enhanced performance in many applications. However, theoretical supports of DeepCCA are often lacking. To address this challenge, the recent work of Lyu and Fu (2020) showed that, under a reasonable postnonlinear generative model, a carefully designed DeepCCA criterion provably removes unknown distortions in data generation and identifies the shared information across modalities. Nonetheless, a critical assumption used by Lyu and Fu (2020) for identifiability analysis was that unlimited data is available, which is unrealistic. This brief paper puts forth a finite-sample analysis of the DeepCCA method by Lyu and Fu (2020). The main result is that the finite-sample version of the method can still estimate the shared information with a guaranteed accuracy when the number of samples is sufficiently large. Our analytical approach is a nontrivial integration of statistical learning, numerical differentiation, and robust system identification, which may be of interest beyond the scope of DeepCCA and benefit other unsupervised learning paradigms.

Learning to predict prostate cancer recurrence from tissue images

Roughly 30% of men with prostate cancer who undergo radical prostatectomy will suffer biochemical cancer recurrence (BCR). Accurately predicting which patients will experience BCR could identify who would benefit from increased surveillance or adjuvant therapy. Unfortunately, no current method can effectively predict this. We develop and evaluate PathCLR, a novel semi-supervised method that learns a model that can use hematoxylin and eosin (H&E)-stained tissue microarrays (TMAs) to predict prostate cancer recurrence within 5 years after diagnosis. The learning process involves 2 sequential steps: PathCLR (a) first employs self-supervised learning to generate effective feature representations of the input images, then (b) feeds these learned features into a fully supervised neural network classifier to learn a model for predicting BCR. We conducted training and evaluation using 2 large prostate cancer datasets: (1) the Cooperative Prostate Cancer Tissue Resource (CPCTR) with 374 patients, including 189 who experienced BCR, and (2) the Johns Hopkins University (JHU) prostate cancer dataset of 646 patients, with 451 patients having BCR. PathCLR’s (10-fold cross-validation) F1 score was 0.61 for CPCTR and 0.85 for JHU. This was statistically superior (paired t-test with P <.05) to the best-learned model that relied solely on clinicopathological features, including PSA level, primary and secondary Gleason Grade, etc. We attribute the improvement of PathCLR over models using only clinicopathological features to its utilization of both learned latent representations of tissue core images and clinicopathological features. This finding suggests that there is essential predictive information in tissue images at the time of surgery that goes beyond the knowledge obtained from reported clinicopathological features, helping predict the patient’s 5-year outcome.

Joint sparse latent representation learning and dual manifold regularization for unsupervised feature selection

As an effective dimensionality reduction method, unsupervised feature selection (UFS) focuses on the mutual correlations between high-dimensional data features but often overlooks the intrinsic relationships between instances. We also utilize pseudo-labels learned from the data to guide feature selection in UFS. However, the raw data space may contain noise and outliers, leading to a lower accuracy of the learned pseudo-label matrix. We propose a minimum-redundant UFS approach to tackle these problems through jointing sparse latent representation learning with dual manifold regularization (SLRDR). Firstly, SLRDR learns a subspace of latent representation by exploring the interconnection of original data. To enhance subspace sparsity, ℓ2,1-norm is applied to the residual matrix of latent representation learning. Pseudo-label matrix learning is then carried out in the high-quality latent space, resulting in effective pseudo-label information that can provide more useful guidance for sparse regression. Secondly, based on the manifold learning hypothesis, SLRDR exploits features’ local structural properties in feature space and explores the association between data and labels, allowing the model to learn richer and more accurate structural information. In addition, ℓ2,1/2-norm is imposed on the weight matrix to obtain a minimum-redundant solution and select more discriminative features. Finally, an alternating iterative method is used for SLRDR to solve the optimization problem of the objective function, and the convergence of the model is theoretically analyzed. Besides, a series of comparative experiments with ten existing algorithms on nine benchmark datasets are used to verify the model’s effectiveness.

Analyzing the Accuracy of Critical Micelle Concentration Predictions Using Deep Learning.

This paper presents a novel approach to predicting critical micelle concentrations (CMCs) by using graph neural networks (GNNs) augmented with Gaussian processes (GPs). The proposed model uses learned latent space representations of molecules to predict CMCs and estimate uncertainties. The performance of the model on a data set containing nonionic, cationic, anionic, and zwitterionic molecules is compared against a linear model that works with extended connectivity fingerprints (ECFPs). The GNN-based model performs slightly better than the linear ECFP model when there is enough well-balanced training data and achieves predictive accuracy that is comparable to published models that were evaluated on a smaller range of surfactant chemistries. We illustrate the applicability domain of our model using a molecular cartogram to visualize the latent space, which helps to identify molecules for which predictions are likely to be erroneous. In addition to accurately predicting CMCs for some surfactant classes, the proposed approach can provide valuable insights into the molecular properties that influence CMCs.

Bilateral Relation Distillation for Weakly Supervised Temporal Action Localization.

Weakly supervised temporal action localization (WSTAL), which aims to locate the time interval of actions in an untrimmed video with only video-level action labels, has attracted increasing research interest in the past few years. However, a model trained with such labels will tend to focus on segments that contributions most to the video-level classification, leading to inaccurate and incomplete localization results. In this paper, we tackle the problem from a novel perspective of relation modeling and propose a method dubbed Bilateral Relation Distillation (BRD). The core of our method involves learning representations by jointly modeling the relation at the category and sequence levels. Specifically, category-wise latent segment representations are first obtained by different embedding networks, one for each category. We then distill knowledge obtained from a pre-trained language model to capture the category-level relations, which is achieved by performing correlation alignment and category-aware contrast in an intra- and inter-video manner. To model the relations among segments at the sequence-level, we elaborate a gradient-based feature augmentation method and encourage the learned latent representation of the augmented feature to be consistent with that of the original one. Extensive experiments illustrate that our approach achieves state-of-the-art results on THUMOS14 and ActivityNet1.3 datasets.

MM-FRec: Multi-Modal Enhanced Fashion Item Recommendation

Existing studies on fashion item recommendation mainly focused on incorporating the visual signals of items to boost the user preference learning, while overlooking the semantic attributes (e.g., material and brand) of fashion items that also contain important cues about items' properties and users' preference. To bridge this gap, we aim to comprehensively explore the attribute and vision modalities of items to improve the fashion item recommendation performance. However, this is non-trivial due to the latent visual-semantic consistency, various relation types, and unique attributes with insufficient samples. To address these challenges, we propose a Multi-Modal enhanced Fashion item Recommendation scheme (MM-FRec). Specifically, to cope with the multi-modal data, we introduce a relation-oriented graph as well as a vision-oriented graph, and design MM-FRec with three key components: attribute-enhanced latent representation learning, visual representation learning, and multi-modal enhanced preference modeling. To deal with the various relation types, we present a new relation-aware propagation method for adaptively aggregating the information from neighbor nodes to promote the user and item representation learning. To cope with the unique attributes, we introduce the deep multi-task learning strategy in the relation-aware confidence assignment. Extensive experiments on a real-world dataset demonstrate the superiority of our model over state-of-the-art methods.

Common latent representation learning for low-resourced spoken language identification

Common latent representation learning for low-resourced spoken language identification

Improving drug discovery with a hybrid deep generative model using reinforcement learning trained on a Bayesian docking approximation.

Generative approaches to molecular design are an area of intense study in recent years as a method to generate new pharmaceuticals with desired properties. Often though, these types of efforts are constrained by limited experimental activity data, resulting in either models that generate molecules with poor performance or models that are overfit and produce close analogs of known molecules. In this paper, we reduce this data dependency for the generation of new chemotypes by incorporating docking scores of known and de novo molecules to expand the applicability domain of the reward function and diversify the compounds generated during reinforcement learning. Our approach employs a deep generative model initially trained using a combination of limited known drug activity and an approximate docking score provided by a second machine learned Bayes regression model, with final evaluation of high scoring compounds by a full docking simulation. This strategy results in molecules with docking scores improved by 10-20% compared to molecules of similar size, while being 130 × faster than a docking only approach on a typical GPU workstation. We also show that the increased docking scores correlate with (1) docking poses with interactions similar to known inhibitors and (2) result in higher MM-GBSA binding energies comparable to the energies of known DDR1 inhibitors, demonstrating that the Bayesian model contains sufficient information for the network to learn to efficiently interact with the binding pocket during reinforcement learning. This outcome shows that the combination of the learned latent molecular representation along with the feature-based docking regression is sufficient for reinforcement learning to infer the relationship between the molecules and the receptor binding site, which suggest that our method can be a powerful tool for the discovery of new chemotypes with potential therapeutic applications.

Double graphs regularized multi-view subspace clustering

In recent years, there has been an increasing interest in multi-view subspace clustering (MSC). However, existing MSC methods fail to take full advantage of the local geometric structure in each manifold throughout the data flow, which is essential for clustering. To remedy this drawback, in this paper, a novel Double Graphs Regularized Multi-view Subspace Clustering (DGRMSC) method is proposed, which aims to harness both global and local structural information of multi-view data in a unified framework. Specifically, DGRMSC first learns a latent representation to exploit the global complementary information of multiple views. Based on the learned latent representation, we learn a self-representation to explore its global cluster structure. Further, Double Graphs Regularization (DGR) is performed on both latent representation and self-representation to take advantage of their local manifold structures simultaneously. Then, we design an iterative algorithm to solve the optimization problem effectively. Comprehensive experiments on several popular multi-view datasets demonstrate the effectiveness of the proposed method.

Contrastively enforcing distinctiveness for multi-label image classification

Recently, as an effective way of learning latent representations, contrastive learning has been increasingly popular and successful in various domains. The success of contrastive learning in single-label classifications motivates us to leverage this learning framework to enhance distinctiveness for better performance in multi-label image classification. In this paper, we show that a direct application of contrastive learning can hardly improve in multi-label cases. Accordingly, we propose a novel framework for multi-label classification with contrastive learning in a fully supervised setting, which learns multiple representations of an image under the context of different labels. This introduces a simple yet intuitive adaption of contrastive learning into our model to boost its performance in multi-label image classification. Extensive experiments on four benchmark datasets show that the proposed framework achieves state-of-the-art performance in the comparison with the advanced methods in multi-label classification.

Semisupervised Fault Diagnosis for Gearboxes: A Novel Method Based on a Hybrid Classification Network and Weighted Pseudo-Labeling

To make the gearboxes reliable and secure, accurate and efficient fault diagnosis has received wide attention. Recently, many data-driven intelligent methods have made great progress due to their powerful feature extraction capability. However, most deep learning based methods can only perform well with sufficient labeled data and the shortage of labeled data has become a common problem in practical industrial applications. Therefore, a semi-supervised approach based on a hybrid classification network and weighted pseudo-labeling is proposed to relieve the above problem. The hybrid classification network is composed of an autoencoder and a softmax classifier. With this structure, the process of latent representation learning of autoencoder is guided by the supervised classification training. Therefore, the features extracted by the autoencoder are more suitable for the classification task. Moreover, a weighted pseudo-labeling method is developed to further enhance the generalization capability of the model. Firstly, pseudo labels for the unlabeled samples are generated according to the model predictions. Then, the proposed sample weighting scheme assigns a confidence-based weight to each pseudo-labeled sample, which is used to filter out the incorrect pseudo labels. Finally, these weighted pseudo-labeled samples are utilized to further optimize the model. Two gearbox datasets are utilized to validate the effectiveness of the proposed method. One is an experimental gearbox dataset and the other is an industrial wind turbine gearbox dataset. The experimental results show that the proposed method can still achieve high diagnostic accuracies for both the experimental and industrial datasets under limited labeled data.

Addressing Posterior Collapse in Variational Autoencoders with β-VAE

Posterior collapse is a pervasive issue in Variational Autoencoders (VAEs) that leads to the learned latent representations becoming trivial and devoid of meaningful information. To address this problem, this paper presents a novel β-VAE approach, which incorporates a hyperparameter β to strike an optimal balance between the reconstruction loss and the KL divergence loss. By conducting a comprehensive series of experiments and drawing comparisons with existing methods, robust evidence is provided that the proposed β-VAE method effectively mitigates posterior collapse and yields more expressive and informative latent representations.The experimental setup involves various architectures and datasets to demonstrate the versatility and efficacy of the β-VAE approach in diverse settings. Additionally, ablation studies are performed to investigate the impact of different β values on the model's performance, elucidating the role of this hyperparameter in controlling the trade-off between reconstruction quality and latent representation expressiveness. Furthermore, the disentanglement properties of the learned latent space are analyzed, which is a crucial aspect of VAEs, especially when applied to complex, real-world data.In-depth analysis of the results offers valuable insights into the underlying mechanisms of β-VAE, thereby contributing to a more profound understanding of VAEs and their inherent limitations. The findings not only establish the effectiveness of the β-VAE method in preventing posterior collapse but also pave the way for future research on improving VAEs' performance in various applications. Potential future work could explore alternative techniques for balancing the competing objectives of reconstruction and latent representation learning or delve into the theoretical properties of β-VAE, providing a more rigorous foundation for this approach.