Latent Space Representation Research Articles

Multimodal sensor fusion in the latent representation space

A new method for multimodal sensor fusion is introduced. The technique relies on a two-stage process. In the first stage, a multimodal generative model is constructed from unlabelled training data. In the second stage, the generative model serves as a reconstruction prior and the search manifold for the sensor fusion tasks. The method also handles cases where observations are accessed only via subsampling i.e. compressed sensing. We demonstrate the effectiveness and excellent performance on a range of multimodal fusion experiments such as multisensory classification, denoising, and recovery from subsampled observations.

Multiview Clustering via Proximity Learning in Latent Representation Space.

Most existing multiview clustering methods are based on the original feature space. However, the feature redundancy and noise in the original feature space limit their clustering performance. Aiming at addressing this problem, some multiview clustering methods learn the latent data representation linearly, while performance may decline if the relation between the latent data representation and the original data is nonlinear. The other methods which nonlinearly learn the latent data representation usually conduct the latent representation learning and clustering separately, resulting in that the latent data representation might be not well adapted to clustering. Furthermore, none of them model the intercluster relation and intracluster correlation of data points, which limits the quality of the learned latent data representation and therefore influences the clustering performance. To solve these problems, this article proposes a novel multiview clustering method via proximity learning in latent representation space, named multiview latent proximity learning (MLPL). For one thing, MLPL learns the latent data representation in a nonlinear manner which takes the intercluster relation and intracluster correlation into consideration simultaneously. For another, through conducting the latent representation learning and consensus proximity learning simultaneously, MLPL learns a consensus proximity matrix with k connected components to output the clustering result directly. Extensive experiments are conducted on seven real-world datasets to demonstrate the effectiveness and superiority of the MLPL method compared with the state-of-the-art multiview clustering methods.

Direct prediction of inelastic neutron scattering spectra from the crystal structure*

Inelastic neutron scattering (INS) is a powerful technique to study vibrational dynamics of materials with several unique advantages. However, analysis and interpretation of INS spectra often require advanced modeling that needs specialized computing resources and relevant expertise. This difficulty is compounded by the limited experimental resources available to perform INS measurements. In this work, we develop a machine-learning based predictive framework which is capable of directly predicting both one-dimensional INS spectra and two-dimensional INS spectra with additional momentum resolution. By integrating symmetry-aware neural networks with autoencoders, and using a large scale synthetic INS database, high-dimensional spectral data are compressed into a latent-space representation, and a high-quality spectra prediction is achieved by using only atomic coordinates as input. Our work offers an efficient approach to predict complex multi-dimensional neutron spectra directly from simple input; it allows for improved efficiency in using the limited INS measurement resources, and sheds light on building structure-property relationships in a variety of on-the-fly experimental data analysis scenarios.

Calibrated simplex-mapping classification.

We propose a novel methodology for general multi-class classification in arbitrary feature spaces, which results in a potentially well-calibrated classifier. Calibrated classifiers are important in many applications because, in addition to the prediction of mere class labels, they also yield a confidence level for each of their predictions. In essence, the training of our classifier proceeds in two steps. In a first step, the training data is represented in a latent space whose geometry is induced by a regular (n - 1)-dimensional simplex, n being the number of classes. We design this representation in such a way that it well reflects the feature space distances of the datapoints to their own- and foreign-class neighbors. In a second step, the latent space representation of the training data is extended to the whole feature space by fitting a regression model to the transformed data. With this latent-space representation, our calibrated classifier is readily defined. We rigorously establish its core theoretical properties and benchmark its prediction and calibration properties by means of various synthetic and real-world data sets from different application domains.

SS-VPE: Semi-Supervised Variational Prototyping Encoder With Student’s-t Mixture Model

As humans can identify new targets with one or a few images, machine learning can use a few images to generalize effective representations so as to achieve few-shot learning. Previous research has established relationships between visual prototype in input space and feature representations in latent space to capture data distributions. However, such visual prototype that can compactly represent class semantics are often lacking in most datasets. This study is aimed at distinguishing samples by using a latent prototype that represents class information in latent space. However, a latent prototype is mostly sensitive to outliers; therefore, a general method using a semi-supervised variational prototyping encoder (SS-VPE) is proposed. The contributions of this study are three-fold. First, a generative unsupervised learning is employed to optimize the prototypes in latent space, such that the data distribution can be captured accurately. Second, the Student’s-t mixture model is applied to the above clustering in the latent space, resulting in high robustness against outliers. Finally, the conventional variational autoencoder (VAE) is advanced by few-shot semi-supervised learning. Intensive experiments are conducted on four popular benchmark datasets, and the experimental results show that our proposed SS-VPE outperforms the state-of-the-art methods, significantly improving both the clustering and classification ability of few-shot learning tasks.

Unsupervised Multi-Latent Space RL Framework for Video Summarization in Ultrasound Imaging.

The COVID-19 pandemic has highlighted the need for a tool to speed up triage in ultrasound scans and provide clinicians with fast access to relevant information. To this end, we propose a new unsupervised reinforcement learning (RL) framework with novel rewards to facilitate unsupervised learning by avoiding tedious and impractical manual labelling for summarizing ultrasound videos. The proposed framework is capable of delivering video summaries with classification labels and segmentations of key landmarks which enhances its utility as a triage tool in the emergency department (ED) and for use in telemedicine. Using an attention ensemble of encoders, the high dimensional image is projected into a low dimensional latent space in terms of: a) reduced distance with a normal or abnormal class (classifier encoder), b) following a topology of landmarks (segmentation encoder), and c) the distance or topology agnostic latent representation (autoencoders). The summarization network is implemented using a bi-directional long short term memory (Bi-LSTM) which utilizes the latent space representation from the encoder. Validation is performed on lung ultrasound (LUS), that typically represent potential use cases in telemedicine and ED triage acquired from different medical centers across geographies (India and Spain). The proposed approach trained and tested on 126 LUS videos showed high agreement with the ground truth with an average precision of over 80% and average F1 score of well over 44 ±1.7 %. The approach resulted in an average reduction in storage space of 77% which can ease bandwidth and storage requirements in telemedicine.

NanoNET: machine learning platform for predicting nanoparticles distribution in a polymer matrix.

Polymer nanocomposites (PNCs) offer a broad range of thermophysical properties that are linked to their compositions. However, it is challenging to establish a universal composition-property relationship in PNCs due to their wide-ranging composition and chemical space. Here, we address this problem and develop a new method to model the composition-microstructure relation of a PNC through an intelligent machine-learning pipeline named nanoNET. The nanoNET is a nanoparticles (NPs) distribution predictor, built upon computer vision and image recognition concepts. It integrates unsupervised deep learning and regression in a fully automated pipeline. We conduct coarse-grained molecular dynamics simulations of PNCs and utilize the data to establish and validate the nanoNET. Within this framework, a random forest regression model predicts the distribution of NPs in a PNC in a latent space. Subsequently, a convolutional neural network-based decoder converts the latent space representation to the actual radial distribution function (RDF) of NPs in the given PNC. The nanoNET predicts NPs distribution in many unknown PNCs very accurately. This method is very generic and can accelerate the design, discovery, and fundamental understanding of composition-microstructure relationships in PNCs and other molecular systems.

Analysis of multimodal data fusion from an information theory perspective

Inspired by the McGurk effect, studies on multimodal data fusion start with audio-visual speech recognition tasks. Multimodal data fusion research was not popular for a period of time because the capacity of traditional machine learning is limited. Recently, advances in deep learning techniques have provided new opportunities for multimodal data fusion. Powerful deep learning models have the capacity to process high-dimensional and complex multimodal data, and multimodal deep learning has the potential to process multimodal data at the human level. However, there is still a lack of theoretical analytical methods relating data information with model performance. In this work, we propose basic concepts and principles to gain insight into the process of multimodal data fusion from an information theory perspective. We analyze different multimodal data fusion cases, such as redundant, noisy, consistent, and contradictory data fusion. We define the model accuracy upper bound for multimodal tasks and prove that a multimodal model with an extra modal channel can perform better in theory when extra modal data provide more effective information for prediction. We explicitly inspect the latent representation space and analyze the information loss of the representation space transformation in deep learning for the first time. From a naive example to a multimodal deep learning example, we demonstrate the theoretical analysis method for evaluating a multimodal data fusion model, and the experimental results validate the definitions and principles.

Subject-invariant feature learning for mTBI identification using LSTM-based variational autoencoder with adversarial regularization

Developing models for identifying mild traumatic brain injury (mTBI) has often been challenging due to large variations in data from subjects, resulting in difficulties for the mTBI-identification models to generalize to data from unseen subjects. To tackle this problem, we present a long short-term memory-based adversarial variational autoencoder (LSTM-AVAE) framework for subject-invariant mTBI feature extraction. In the proposed model, first, an LSTM variational autoencoder (LSTM-VAE) combines the representation learning ability of the variational autoencoder (VAE) with the temporal modeling characteristics of the LSTM to learn the latent space representations from neural activity. Then, to detach the subject’s individuality from neural feature representations, and make the model proper for cross-subject transfer learning, an adversary network is attached to the encoder in a discriminative setting. The model is trained using the 1 held-out approach. The trained encoder is then used to extract the representations from the held-out subject’s data. The extracted representations are then classified into normal and mTBI groups using different classifiers. The proposed model is evaluated on cortical recordings of Thy1-GCaMP6s transgenic mice obtained via widefield calcium imaging, prior to and after inducing injury. In cross-subject transfer learning experiment, the proposed LSTM-AVAE framework achieves classification accuracy results of 95.8% and 97.79%, without and with utilizing conditional VAE (cVAE), respectively, demonstrating that the proposed model is capable of learning invariant representations from mTBI data.

A deep learning-based intrusion detection system for in-vehicle networks

Modern vehicles are increasingly getting connected within the vehicles, with other systems, leading to more concerns about security. Controller area network (CAN) has become a de-facto standard for connecting internal vehicles’ components. However, it lacks security features. Conventional security mechanisms fail to protect in-vehicle networks from attacks, requiring the development of an effective intrusion detection system (IDS). This work develops an IDS for in-vehicle networks called IDS-IVN based on a compact representation of location invariant and time-variant traffic features using deep learning. The IDS-IVN uses convolutional neural and long–short-term memory networks as encoder/decoder functions of autoencoder networks to extract features from raw data and classify them using latent space representation into intrusive and non-intrusive classes. A benchmark real-time ROAD dataset is used to demonstrate the IDS-IVN’s performance compared to the existing methods. IDS-IVN reports 99% accuracy with a 0.32% low false-positive rate for detecting intrusions.

Adversarial Learning for Cross Domain Recommendations

Existing cross domain recommender systems typically assume homogeneous user preferences across multiple domains to capture similarities of user-item interactions and to provide cross domain recommendations accordingly. Meanwhile, the heterogeneity of user behaviors is usually not well studied and captured during the recommendation process, where users might have vastly different interests in different domains. In addition, previous models focus primarily on recommendation tasks between domain pairs, and cannot be naturally extended to serve for multiple domain recommendation applications. To address these challenges, we propose to utilize the idea of adversarial learning to intelligently incorporate global user preferences and domain-specific user preferences for providing satisfying cross domain recommendations. In particular, our proposed Adversarial Cross Domain Recommendation (ACDR) model first obtains the latent representations of global user preferences from their explicit feature information, and then transforms them into domain-specific user embeddings, where we take into account user behaviors and their heterogeneous preferences among different domains. By doing so, we address the differences among user representations in the domain-specific latent space while also preserving global user preferences, as we effectively segment the distributions of domain-specific user embeddings in the shared latent space. The convergence of our proposed model is theoretically guaranteed. The proposed ACDR model leads to significant and consistent improvements in cross domain recommendation performance over the state-of-the-art baseline models, which we demonstrate through extensive experiments on three real-world datasets. In addition, we show that the improvements are greater on those datasets that are smaller and more sparse, on those users that have fewer interaction records in the dataset, and when user interactions from more product domains are included in the cross domain recommendation model.

Non-Newtonian fluid simulation and reconstruction from monocular videos

In fluid simulation, the parameters of the constitutive model are difficult to predict accurately, which leads to the inconsistency between the simulation results and the visual effects of monocular videos. To solve this problem, a monocular video-guided non-Newtonian fluid reconstruction method is proposed. The model takes non-Newtonian fluid simulation videos as input for training and learns the best low-dimensional latent space representation of a single frame of fluid simulation images. Inter-frame prediction is then performed in this latent domain, employing a convolutional long–short-term memory network to predict latent vector representations of future frames. Finally, the reconstruction parameters of the constitutive model are predicted based on the frame-by-frame latent representation encoding and inter-frame temporal features. In the model verification stage, the real video of the non-Newtonian fluid is used as the input to predict the parameters of the fluid constitutive model, and realize the simulation and reconstruction of the non-Newtonian fluid based on the Cross model. Experimental results showed that the video-guided simulation reconstruction method could obtain fluid flow phenomena that were more consistent with those in the real video than the reconstruction method based on rheometer measurements. Furthermore, it provided higher pixel accuracy and precision at different times and visual effects that were more consistent with the actual fluid flow.

A Machine Learning Framework to Predict Images of Edge-on Protoplanetary Disks

The physical structure and properties of protoplanetary disks are typically derived from spatially resolved disk images. Edge-on disks in particular provide an important view point on the vertical structure and degree of settling of disks. Such analyses rely on radiative transfer (RT) calculations that are generally computationally intensive due to the high optical depth of disks. Here we present a machine learning framework that has the potential to dramatically speed up the forward modeling process by approximating the results of RT calculations. This framework, trained on an initial set of RT calculations, utilizes an autoencoder neural network to enable the generation of synthetic scattered light images of edge-on disks directly from a set of physical parameters. We demonstrate that this framework generates synthetic images 2–3 orders of magnitude faster than using RT calculations. These machine learning-generated images appear to approximate the RT images well, in particular preserving their size and shape. We also find a strong correlation between the latent space representations of the generated disk images and several of their associated physical parameters. Finally, we discuss potential changes to the framework, such as methods to further improve the image quality, extending the framework to multiple wavelengths, and inverting the process to infer physical parameters from observed images. Overall, these new tools have the potential to enable a more efficient and uniform analysis of edge-on disk properties and the initial conditions of planet formation.

Real-Time Magnetic Sensor Anomaly Detection Using Autoencoder Neural Networks on the DIII-D Tokamak

Magnetic diagnostics in tokamaks are key to plasma equilibrium control (plasma current, plasma shape, and position) and amelioration of plasma instabilities. Thus, real-time identification of the anomalous sensor is mandatory. A new real-time system based on autoencoder (AE) neural networks (NNs) for anomaly detection in magnetics signals, including both flux loops and magnetic probes, has been successfully implemented on the plasma control system (PCS) of the DIII-D tokamak. The AE NN is trained on over 4000 plasma discharges, with an optimized latent space representation of the input signals while minimizing the reconstruction loss. An algorithm determining anomalous sensors based on excessive deviations from accurate reconstruction from the trained NNs is constructed in a MATLAB/Simulink environment and is deployed on the real-time PCS using the embedded MATLAB coder (EMC) environment. The reconstruction performance of the algorithm is quantified in hardware-in-the-loop (HIL) simulation by injecting artificially common fault types, including stuck-at-zero fault, saturated fault, noise fault, and drift fault. The AE NN algorithm has been successfully commissioned on the DIII-D PCS.

Temporal RIT-Eyes: From Real Infrared Eye-Images to Synthetic Sequences of Gaze Behavior.

Current methods for segmenting eye imagery into skin, sclera, pupil, and iris cannot leverage information about eye motion. This is because the datasets on which models are trained are limited to temporally non-contiguous frames. We present Temporal RIT-Eyes, a Blender pipeline that draws data from real eye videos for the rendering of synthetic imagery depicting natural gaze dynamics. These sequences are accompanied by ground-truth segmentation maps that may be used for training image-segmentation networks. Temporal RIT-Eyes relies on a novel method for the extraction of 3D eyelid pose (top and bottom apex of eyelids/eyeball boundary) from raw eye images for the rendering of gaze-dependent eyelid pose and blink behavior. The pipeline is parameterized to vary in appearance, eye/head/camera/illuminant geometry, and environment settings (indoor/outdoor). We present two open-source datasets of synthetic eye imagery: sGiW is a set of synthetic-image sequences whose dynamics are modeled on those of the Gaze in Wild dataset, and sOpenEDS2 is a series of temporally non-contiguous eye images that approximate the OpenEDS-2019 dataset. We also analyze and demonstrate the quality of the rendered dataset qualitatively and show significant overlap between latent-space representations of the source and the rendered datasets.

Adversarial deep evolutionary learning for drug design

The design of a new therapeutic agent is a time-consuming and expensive process. The rise of machine intelligence provides a grand opportunity of expeditiously discovering novel drug candidates through smart search in the vast molecular structural space. In this paper, we propose a new approach called adversarial deep evolutionary learning (ADEL) to search for novel molecules in the latent space of an adversarial generative model and keep improving the latent representation space. In ADEL, a custom-made adversarial autoencoder (AAE) model is developed and trained under a deep evolutionary learning (DEL) process. This involves an initial training of the AAE model, followed by an integration of multi-objective evolutionary optimization in the continuous latent representation space of the AAE rather than the discrete structural space of molecules. By using the AAE, an arbitrary distribution can be provided to the training of AAE such that the latent representation space is set to that distribution. This allows for a starting latent space from which new samples can be produced. Throughout the process of learning, new samples of high quality are generated after each iteration of training and then added back into the full dataset, therefore, allowing for a more comprehensive procedure of understanding the data structure. This combination of evolving data and continuous learning not only enables improvement in the generative model, but the data as well. By comparing ADEL to the previous work in DEL, we see that ADEL can obtain better property distributions. We show that ADEL is able to design high-quality molecular structures which can be further used for virtual and experimental screenings.

Investigating internal migration with network analysis and latent space representations: an application to Turkey.

Human migration patterns influence the redistribution of population characteristics over the geography and since such distributions are closely related to social and economic outcomes, investigating the structure and dynamics of internal migration plays a crucial role in understanding and designing policies for such systems. We provide an in-depth investigation into the structure and dynamics of the internal migration in Turkey from 2008 to 2020. We identify a set of classical migration laws and examine them via various methods for signed network analysis, ego network analysis, representation learning, temporal stability analysis, community detection, and network visualization. The findings show that, in line with the classical migration laws, most migration links are geographically bounded with several exceptions involving cities with large economic activity, major migration flows are countered with migration flows in the opposite direction, there are well-defined migration routes, and the migration system is generally stable over the investigated period. Apart from these general results, we also provide unique and specific insights into Turkey. Overall, the novel toolset we employ for the first time in the literature allows the investigation of selected migration laws from a complex networks perspective and sheds light on future migration research on different geographies.

Multi-Perspective Feature Extraction and Fusion Based on Deep Latent Space for Diagnosis of Alzheimer's Diseases.

Resting-state functional magnetic resonance imaging (rs-fMRI) has been used to construct functional connectivity (FC) in the brain for the diagnosis and analysis of brain disease. Current studies typically use the Pearson correlation coefficient to construct dynamic FC (dFC) networks, and then use this as a network metric to obtain the necessary features for brain disease diagnosis and analysis. This simple observational approach makes it difficult to extract potential high-level FC features from the representations, and also ignores the rich information on spatial and temporal variability in FC. In this paper, we construct the Latent Space Representation Network (LSRNet) and use two stages to train the network. In the first stage, an autoencoder is used to extract potential high-level features and inner connections in the dFC representations. In the second stage, high-level features are extracted using two perspective feature parses. Long Short-Term Memory (LSTM) networks are used to extract spatial and temporal features from the local perspective. Convolutional neural networks extract global high-level features from the global perspective. Finally, the fusion of spatial and temporal features with global high-level features is used to diagnose brain disease. In this paper, the proposed method is applied to the ANDI rs-fMRI dataset, and the classification accuracy reaches 84.6% for NC/eMCI, 95.1% for NC/AD, 80.6% for eMCI/lMCI, 84.2% for lMCI/AD and 57.3% for NC/eMCI/lMCI/AD. The experimental results show that the method has a good classification performance and provides a new approach to the diagnosis of other brain diseases.

Data-driven model for divertor plasma detachment prediction

We present a fast and accurate data-driven surrogate model for divertor plasma detachment prediction leveraging the latent feature space concept in machine learning research. Our approach involves constructing and training two neural networks: an autoencoder that finds a proper latent space representation (LSR) of plasma state by compressing the multi-modal diagnostic measurements and a forward model using multi-layer perception (MLP) that projects a set of plasma control parameters to its corresponding LSR. By combining the forward model and the decoder network from autoencoder, this new data-driven surrogate model is able to predict a consistent set of diagnostic measurements based on a few plasma control parameters. In order to ensure that the crucial detachment physics is correctly captured, highly efficient 1D UEDGE model is used to generate training and validation data in this study. The benchmark between the data-driven surrogate model and UEDGE simulations shows that our surrogate model is capable of providing accurate detachment prediction (usually within a few per cent relative error margin) but with at least four orders of magnitude speed-up, indicating that performance-wise, it has the potential to facilitate integrated tokamak design and plasma control. Comparing with the widely used two-point model and/or two-point model formatting, the new data-driven model features additional detachment front prediction and can be easily extended to incorporate richer physics. This study demonstrates that the complicated divertor and scrape-off-layer plasma state has a low-dimensional representation in latent space. Understanding plasma dynamics in latent space and utilising this knowledge could open a new path for plasma control in magnetic fusion energy research.

Self-supervised anomaly detection for new physics

We investigate a method of model-agnostic anomaly detection through studying jets, collimated sprays of particles produced in high-energy collisions. We train a transformer neural network to encode simulated QCD ``event space'' dijets into a low-dimensional ``latent space'' representation. We optimize the network using the self-supervised contrastive loss, which encourages the preservation of known physical symmetries of the dijets. We then train a binary classifier to discriminate a beyond the standard model resonant dijet signal from a QCD dijet background both in the event space and the latent space representations. We find the classifier performances on the event and latent spaces to be comparable. We finally perform an anomaly detection search using a weakly supervised bump hunt on the latent space dijets, finding again a comparable performance to a search run on the physical space dijets. This opens the door to using low-dimensional latent representations as a computationally efficient space for resonant anomaly detection in generic particle collision events.