Latent Space Representation Research Articles

NCAE: data-driven representations using a deep network-coherent DNA methylation autoencoder identify robust disease and risk factor signatures.

Precision medicine relies on the identification of robust disease and risk factor signatures from omics data. However, current knowledge-driven approaches may overlook novel or unexpected phenomena due to the inherent biases in biological knowledge. In this study, we present a data-driven signature discovery workflow for DNA methylation analysis utilizing network-coherent autoencoders (NCAEs) with biologically relevant latent embeddings. First, we explored the architecture space of autoencoders trained on a large-scale pan-tissue compendium (n = 75272) of human epigenome-wide association studies. We observed the emergence of co-localized patterns in the deep autoencoder latent space representations that corresponded to biological network modules. We determined the NCAE configuration with the strongest co-localization and centrality signals in the human protein interactome. Leveraging the NCAE embeddings, we then trained interpretable deep neural networks for risk factor (aging, smoking) and disease (systemic lupus erythematosus) prediction and classification tasks. Remarkably, our NCAE embedding-based models outperformed existing predictors, revealing novel DNA methylation signatures enriched in gene sets and pathways associated with the studied condition in each case. Our data-driven biomarker discovery workflow provides a generally applicable pipeline to capture relevant risk factor and disease information. By surpassing the limitations of knowledge-driven methods, our approach enhances the understanding of complex epigenetic processes, facilitating the development of more effective diagnostic and therapeutic strategies.

Deep multimodality-disentangled association analysis network for imaging genetics in neurodegenerative diseases.

Imaging genetics is a crucial tool that is applied to explore potentially disease-related biomarkers, particularly for neurodegenerative diseases (NDs). With the development of imaging technology, the association analysis between multimodal imaging data and genetic data is gradually being concerned by a wide range of imaging genetics studies. However, multimodal data are fused first and then correlated with genetic data in traditional methods, which leads to an incomplete exploration of their common and complementary information. In addition, the inaccurate formulation in the complex relationships between imaging and genetic data and information loss caused by missing multimodal data are still open problems in imaging genetics studies. Therefore, in this study, a deep multimodality-disentangled association analysis network (DMAAN) is proposed to solve the aforementioned issues and detect the disease-related biomarkers of NDs simultaneously. First, the imaging data are nonlinearly projected into a latent space and imaging representations can be achieved. The imaging representations are further disentangled into common and specific parts by using a multimodal-disentangled module. Second, the genetic data are encoded to achieve genetic representations, and then, the achieved genetic representations are nonlinearly mapped to the common and specific imaging representations to build nonlinear associations between imaging and genetic data through an association analysis module. Moreover, modality mask vectors are synchronously synthesized to integrate the genetic and imaging data, which helps the following disease diagnosis. Finally, the proposed method achieves reasonable diagnosis performance via a disease diagnosis module and utilizes the label information to detect the disease-related modality-shared and modality-specific biomarkers. Furthermore, the genetic representation can be used to impute the missing multimodal data with our learning strategy. Two publicly available datasets with different NDs are used to demonstrate the effectiveness of the proposed DMAAN. The experimental results show that the proposed DMAAN can identify the disease-related biomarkers, which suggests the proposed DMAAN may provide new insights into the pathological mechanism and early diagnosis of NDs. The codes are publicly available at https://github.com/Meiyan88/DMAAN.

CGSA optimized LSTM auto encoder for outlier detection

In recent years, outlier detection has attained great attention with machine learning techniques due to its wide range of applications. By considering the input data’s distributive nature and large dimensionality, outlier detection becomes a challenging issue. Robust outlier detection systems are crucial for data pattern prediction without labeled data. This research develops a novel approach based on stacking auto encoders over Long-Short Term Memory (LSTM) for outlier prediction. The detection accuracy of outlier detection is improved with the hyperparameters optimized with the Chaotic Gravitational Search Algorithm (CGSA). CGSA minimizes the training loss with enhanced detection accuracy in the proposed outlier detection process. The auto encoder in outlier detection transforms the input into a latent space representation to generate the original input sequence. The involvement of learning parameters computes and minimizes the errors between input and generated sequences. The proposed work is experimented and compared with state-of-the-art approaches of recent research. Using the proposed approach, the performance of outlier prediction is improved with an accuracy of98.6%, sensitivity of 96.1%, specificity of 97.8%, G-mean of 96%, Area Under Curve (AUC) of 0.935, Hit rate of 92.3%. Also, the outlier detection errors are minimized, showing the proposed approach’s efficiency.

A detailed study of interpretability of deep neural network based top taggers

Recent developments in the methods of explainable artificial intelligence (XAI) allow researchers to explore the inner workings of deep neural networks (DNNs), revealing crucial information about input–output relationships and realizing how data connects with machine learning models. In this paper we explore interpretability of DNN models designed to identify jets coming from top quark decay in high energy proton–proton collisions at the Large Hadron Collider. We review a subset of existing top tagger models and explore different quantitative methods to identify which features play the most important roles in identifying the top jets. We also investigate how and why feature importance varies across different XAI metrics, how correlations among features impact their explainability, and how latent space representations encode information as well as correlate with physically meaningful quantities. Our studies uncover some major pitfalls of existing XAI methods and illustrate how they can be overcome to obtain consistent and meaningful interpretation of these models. We additionally illustrate the activity of hidden layers as neural activation pattern diagrams and demonstrate how they can be used to understand how DNNs relay information across the layers and how this understanding can help to make such models significantly simpler by allowing effective model reoptimization and hyperparameter tuning. These studies not only facilitate a methodological approach to interpreting models but also unveil new insights about what these models learn. Incorporating these observations into augmented model design, we propose the particle flow interaction network model and demonstrate how interpretability-inspired model augmentation can improve top tagging performance.

Comparison of reduced order models based on dynamic mode decomposition and deep learning for predicting chaotic flow in a random arrangement of cylinders

Three reduced order models are evaluated in their capacity to predict the future state of an unsteady chaotic flow field. A spatially fully developed flow generated in a random packing of cylinders at a solid fraction of 0.1 and a nominal Reynolds number of 50 is investigated. For deep learning (DL), convolutional autoencoders are used to reduce the high-dimensional data to lower dimensional latent space representations of size 16, which were then used for training the temporal architectures. To predict the future states, two DL based methods, long short-term memory and temporal convolutional neural networks, are used and compared to the linear dynamic mode decomposition (DMD). The predictions are tested in their capability to predict the spatiotemporal variations of velocity and pressure, flow statistics such as root mean squared values, and the capability to predict fluid forces on the cylinders. Relative errors between 15% and 20% are evident in predicting instantaneous velocities, chiefly resulting from phase differences between predictions and ground truth. The spatial distribution of statistical second moments is predicted to be within a maximum of 5%–10% of the ground truth with mean error in the range of 1%–2%. Using the predicted fields, instantaneous fluid drag force predictions on individual particles exhibit a mean relative error within 20%, time-averaged drag force predictions to within 5%, and total drag force over all particles to within 1% of the ground truth values. It is found that overall, the non-linear DL models are more accurate than the linear DMD algorithm for the prediction of future states.

Using deep LSD to build operators in GANs latent space with meaning in real space.

Generative models rely on the idea that data can be represented in terms of latent variables which are uncorrelated by definition. Lack of correlation among the latent variable support is important because it suggests that the latent-space manifold is simpler to understand and manipulate than the real-space representation. Many types of generative model are used in deep learning, e.g., variational autoencoders (VAEs) and generative adversarial networks (GANs). Based on the idea that the latent space behaves like a vector space Radford et al. (2015), we ask whether we can expand the latent space representation of our data elements in terms of an orthonormal basis set. Here we propose a method to build a set of linearly independent vectors in the latent space of a trained GAN, which we call quasi-eigenvectors. These quasi-eigenvectors have two key properties: i) They span the latent space, ii) A set of these quasi-eigenvectors map to each of the labeled features one-to-one. We show that in the case of the MNIST image data set, while the number of dimensions in latent space is large by design, 98% of the data in real space map to a sub-domain of latent space of dimensionality equal to the number of labels. We then show how the quasi-eigenvectors can be used for Latent Spectral Decomposition (LSD). We apply LSD to denoise MNIST images. Finally, using the quasi-eigenvectors, we construct rotation matrices in latent space which map to feature transformations in real space. Overall, from quasi-eigenvectors we gain insight regarding the latent space topology.

Forecasting passenger flows and headway at train level for a public transport line: Focus on atypical situations

This paper proposes a computer vision framework based on deep learning approaches for the real-time prediction of passenger loads and train headways in a metro line in an urban transit network. The short-term prediction problem is formulated as an image completion task. The train journeys on the metro line are represented as images, with the pixels denoting the train data, including the departure location, time, and load. Metro line applicative constraints, e.g., irregular time sampling of trains and univariate space, are considered in the images. Several deep learning-based architectures are investigated, including two new architectures based on transformers. Finally, an in-depth analysis and comparison of the models is performed based on a real test case of Paris metro line 9, for which a large database was collected over three years. This allowed us to include numerous instances with atypical transport system performance indicators (e.g., strikes, lockdowns, and disruptions) and to design a methodology (based on a latent space representation of the dataset) to identify and label interesting test cases. We evaluate and demonstrate the robustness of the proposed approaches numerically and study their limits for passenger loads and headways in atypical scenarios.

S3-VAE: A novel Supervised-Source-Separation Variational AutoEncoder algorithm to discriminate tumor cell lines in time-lapse microscopy images

The derivation of input–output relationships in deep learning architectures is mostly a black-box process, in which uninformative or confounding factors might bias the classification results, with no clue for the user. However, the analysis of living cells requires extracting latent space representations with interpretable meaning, which can be investigated not only for classification but mainly for understanding purposes. Using the properties of variational autoencoders in deriving unsupervised regular space representations, we propose a novel Supervised-Source-Separation Variational Autoencoder (S3-VAE) algorithm capable of guiding the encoding process of images toward a more effective class-separability. The scope has been reached, under the assumption of full-covariance Gaussian posterior, by introducing a term in the cost function that forces class-dependent sources of variations toward a one-hot encoding. The proposed approach has been designed for an automatic platform for single-cell biological investigations based on time-lapse microscopy, in which a dedicated video processing pipeline has been designed, as an additional contribution, to reduce the effects of confounding factors on the S3-VAE representations. The results obtained in a classification scenario with artificially generated phantom cells and with experimental data of human prostate cells, in non-neoplastic, neoplastic and metastatic neoplastic conditions, show that S3-VAE-based latent representations allow for investigation and recognition of the morphological differences among classes. Comparative analysis with other benchmark deep learning architectures demonstrates the effectiveness of the proposed approach and shows that S3-VAE is capable of achieving high performance in discriminating cell lines based on phenotypic variations.

ECG-NET: A deep LSTM autoencoder for detecting anomalous ECG

The electrocardiogram (ECG) is a standard test to monitor the activity of the heart. Many cardiac abnormalities are manifested in the ECG including arrhythmia that refers to an abnormal heart rhythm. The basis of arrhythmia diagnosis is the identification of normal versus abnormal heart beats, and their correct classification based on ECG morphology. This paper proposes a novel and robust approach for representation learning of ECG sequences using a LSTM autoencoder for anomaly detection.The encoder part encodes the ECG signal into a lower dimensional latent space representation and decoder part then tries to reconstruct the specified ECG signal. The model is trained only on normal (non-anomalous) ECG signals. Then reconstruction loss of test ECG signals are calculated. Next a reconstruction loss threshold value is determined from the frequency distribution of the reconstruction losses so that from the reconstruction loss value above a certain threshold is determined as anomaly, otherwise it will be treated as normal. Determination of threshold is done using manual and Kapur’s automated thresholding procedures.The aforementioned model has been applied on publicly available ECG5000 dataset. From the experimental results it is observed that the proposed model achieved more than 98% accuracy having precision, recall and F1 values more than 0.94, 0.97, 0.96 respectively. The performance of the proposed method is also found to be superior in most of the cases as compared to the results of seven other recent counter-part methods reported in the literature.

A Computer Method for Pronation-Supination Assessment in Parkinson's Disease Based on Latent Space Representations of Biomechanical Indicators.

One problem in the quantitative assessment of biomechanical impairments in Parkinson's disease patients is the need for scalable and adaptable computing systems. This work presents a computational method that can be used for motor evaluations of pronation-supination hand movements, as described in item 3.6 of the Unified Parkinson's Disease Rating Scale (MDS-UPDRS). The presented method can quickly adapt to new expert knowledge and includes new features that use a self-supervised training approach. The work uses wearable sensors for biomechanical measurements. We tested a machine-learning model on a dataset of 228 records with 20 indicators from 57 PD patients and eight healthy control subjects. The test dataset's experimental results show that the method's precision rates for the pronation and supination classification task achieved up to 89% accuracy, and the F1-scores were higher than 88% in most categories. The scores present a root mean squared error of 0.28 when compared to expert clinician scores. The paper provides detailed results for pronation-supination hand movement evaluations using a new analysis method when compared to the other methods mentioned in the literature. Furthermore, the proposal consists of a scalable and adaptable model that includes expert knowledge and affectations not covered in the MDS-UPDRS for a more in-depth evaluation.

Multi‐scale feature fusion pyramid attention network for single image dehazing

AbstractTexture and color distortion are common in existing learning‐based dehazing algorithms, and it is argued that one of the major reasons is that the shallow features of fog images are underutilized, and the deep features of fog images are insufficient for single image dehazing. In order to provide more texture and color information for image restoration, more shallow features need to be added in the process of image decoding. Therefore, a multi‐scale feature fusion pyramid attention network (PAN) for single image dehazing is proposed. In PAN, combined with the attention mechanism, a shallow and deep feature fusion (SDF) strategy is designed. SDF considers multi‐scale as well as channel‐level fusion to provide feature information under different receptive fields while also highlighting important channels, such as texture and color information. DC is designed as a latent space mapping module to learn a mapping relationship between the latent space representation of the hazy image at low resolution and the corresponding latent space representation of the haze‐free image. Additionally, network deconvolution (ND) and deformed convolution network (DCN) are introduced into PAN. The ND module can remove pixel‐wise and channel‐wise correlation of features, reduce data redundancy to obtain sparse representation of features, and speed up network convergence. The DCN module can use its adaptive receptive field to focus on the area of interest for calculation and play a role in texture feature enhancement. Finally, the perceptual loss is chosen as the regularization item of the loss function, which makes style features of the restored image closer to the real fog‐free image. Extensive experiments reveal that the proposed PAN outperforms other existing dehazing methods on real‐world and synthetic datasets.

Combining Representation Learning and Active Learning for Applications in Process Manufacturing

AbstractMultivariate time series classification (TSC) is an important data‐driven modeling task in process industries. A common challenge of TSC in the process industry is the unavailability of labels. Active learning (AL) tackles this problem by incrementally obtaining as few labeled instances as possible by querying an oracle. From our initial experience of building an AL application we derive requirements on the machine learning approach in the context of such an interactive application. To meet the requirements, we combine AL with representation learning (RL): a model learns a latent space representation of unlabeled data, which is then used to train a classification model on the labeled data.

Adaptive autoencoder latent space tuning for more robust machine learning beyond the training set for six-dimensional phase space diagnostics of a time-varying ultrafast electron-diffraction compact accelerator.

We present a general adaptive latent space tuning approach for improving the robustness of machine learning tools with respect to time variation and distribution shift. We demonstrate our approach by developing an encoder-decoder convolutional neural network-based virtual 6D phase space diagnostic of charged particle beams in the HiRES ultrafast electron diffraction (UED) compact particle accelerator with uncertainty quantification. Our method utilizes model-independent adaptive feedback to tune a low-dimensional 2D latent space representation of ∼1 million dimensional objects which are the 15 unique 2D projections (x,y),...,(z,p_{z}) of the 6D phase space (x,y,z,p_{x},p_{y},p_{z}) of the charged particle beams. We demonstrate our method with numerical studies of short electron bunches utilizing experimentally measured UED input beam distributions.

FLAN: feature-wise latent additive neural models for biological applications.

Interpretability has become a necessary feature for machine learning models deployed in critical scenarios, e.g. legal system, healthcare. In these situations, algorithmic decisions may have (potentially negative) long-lasting effects on the end-user affected by the decision. While deep learning models achieve impressive results, they often function as a black-box. Inspired by linear models, we propose a novel class of structurally constrained deep neural networks, which we call FLAN (Feature-wise Latent Additive Networks). Crucially, FLANs process each input feature separately, computing for each of them a representation in a common latent space. These feature-wise latent representations are then simply summed, and the aggregated representation is used for the prediction. These feature-wise representations allow a user to estimate the effect of each individual feature independently from the others, similarly to the way linear models are interpreted. We demonstrate FLAN on a series of benchmark datasets in different biological domains. Our experiments show that FLAN achieves good performances even in complex datasets (e.g. TCR-epitope binding prediction), despite the structural constraint we imposed. On the other hand, this constraint enables us to interpret FLAN by deciphering its decision process, as well as obtaining biological insights (e.g. by identifying the marker genes of different cell populations). In supplementary experiments, we show similar performances also on non-biological datasets. Code and example data are available at https://github.com/phineasng/flan_bio.

Interlayer Link Prediction in Multiplex Social Networks Based on Multiple Types of Consistency Between Embedding Vectors.

Online users are typically active on multiple social media networks (SMNs), which constitute a multiplex social network. With improvements in cybersecurity awareness, users increasingly choose different usernames and provide different profiles on different SMNs. Thus, it is becoming increasingly challenging to determine whether given accounts on different SMNs belong to the same user; this can be expressed as an interlayer link prediction problem in a multiplex network. To address the challenge of predicting interlayer links, feature or structure information is leveraged. Existing methods that use network embedding techniques to address this problem focus on learning a mapping function to unify all nodes into a common latent representation space for prediction; positional relationships between unmatched nodes and their common matched neighbors (CMNs) are not utilized. Furthermore, the layers are often modeled as unweighted graphs, ignoring the strengths of the relationships between nodes. To address these limitations, we propose a framework based on multiple types of consistency between embedding vectors (MulCEVs). In MulCEV, the traditional embedding-based method is applied to obtain the degree of consistency between the vectors representing the unmatched nodes, and a proposed distance consistency index based on the positions of nodes in each latent space provides additional clues for prediction. By associating these two types of consistency, the effective information in the latent spaces is fully utilized. In addition, MulCEV models the layers as weighted graphs to obtain representation. In this way, the higher the strength of the relationship between nodes, the more similar their embedding vectors in the latent representation space will be. The results of our experiments on several real-world and synthetic datasets demonstrate that the proposed MulCEV framework markedly outperforms current embedding-based methods, especially when the number of training iterations is small.

A hybrid deep learning framework for privacy preservation in edge computing

The number of connected devices in our world is continuously increasing at a rapid rate. The Internet of Things(IoT) civilization has resulted in the generation of enormous amounts of data. Analytics on this data pays off in various sectors like health care, manufacturing, and transportation. However, the data generated in the IoT environment is often sensitive, and hence, the need to address the privacy concerns of the data owners. Existing approaches incur a huge computation cost and there is also a gap between privacy preservation and data utility. In this work, a genetic algorithm is coupled with a deep learning network based on adversarial training to build a utility-privacy balanced, low computation solution. The proposal aims to prevent inference of implicit privacy labels present in the data while maintaining data utility. The first part of the proposed work leverages an optimized encoder architecture to learn latent space representation of the input and the second part is the incorporation of adversary for training the framework to prevent unintended sensitive inference. Both parts are governed by a genetic algorithm to output a fitting encoder. Numerical results carried on a benchmark dataset exhibit the capability to protect sensitive data by keeping the accuracy level of the adversary within 23%, and producing a maximum inference accuracy of 95% for the intended task.

Disentangling Geometric Deformation Spaces in Generative Latent Shape Models

A complete representation of 3D objects requires characterizing the space of deformations in an interpretable manner, from articulations of a single instance to changes in shape across categories. In this work, we improve on a prior generative model of geometric disentanglement for 3D shapes, wherein the space of object geometry is factorized into rigid orientation, non-rigid pose, and intrinsic shape. The resulting model can be trained from raw 3D shapes, without correspondences, labels, or even rigid alignment, using a combination of classical spectral geometry and probabilistic disentanglement of a structured latent representation space. Our improvements include more sophisticated handling of rotational invariance and the use of a diffeomorphic flow network to bridge latent and spectral space. The geometric structuring of the latent space imparts an interpretable characterization of the deformation space of an object. Furthermore, it enables tasks like pose transfer and pose-aware retrieval without requiring supervision. We evaluate our model on its generative modelling, representation learning, and disentanglement performance, showing improved rotation invariance and intrinsic-extrinsic factorization quality over the prior model.

Long–short-term memory encoder–decoder with regularized hidden dynamics for fault detection in industrial processes

The ability of recurrent neural networks (RNN) to model nonlinear dynamics of high dimensional process data has enabled data-driven RNN-based fault detection algorithms. Previous studies have focused on detecting faults by identifying the discrepancies in data distribution between the faulty and normal data, as reflected in prediction errors generated by RNN models. However, in industrial processes, variations in data distribution can also result from changes in normal control setpoints and compensatory control adjustments in response to disturbances, making it hard to differentiate between normal and faulty conditions. This paper proposes a fault detection method utilizing a long short-term memory (LSTM) encoder–decoder structure with regularized hidden dynamics and reversible instance normalization (RevIN) to compactly represent high-dimensional measurements for effective monitoring. During training, the hidden states of the model are regularized to form a low-dimensional latent space representation of the original multivariate time series data. As a result, the prediction errors of the latent states can be used to monitor the abnormal dynamic variations, while the reconstruction errors of the measured variables are used to monitor the abnormal static variations. Furthermore, the proposed indices can reflect operating conditions, even when the distribution of test data changes, which helps distinguish faults from normal adjustments and disturbances that controllers can settle. Data from numerical simulation and the Tennessee Eastman process are used to illustrate the effectiveness of the proposed fault detection method.

A Deep Learning Framework for Generation and Analysis of Driving Scenario Trajectories

We propose a unified deep learning framework for the generation and analysis of driving scenario trajectories, and validate its effectiveness in a principled way. To model and generate scenarios of trajectories with different lengths, we develop two approaches. First, we adapt the Recurrent Conditional Generative Adversarial Networks (RC-GAN) by conditioning on the length of the trajectories. This provides us the flexibility to generate variable-length driving trajectories, a desirable feature for scenario test case generation in the verification of autonomous driving. Second, we develop an architecture based on Recurrent Autoencoder with GANs to obviate the variable length issue, wherein we train a GAN to learn/generate the latent representations of original trajectories. In this approach, we train an integrated feed-forward neural network to estimate the length of the trajectories to be able to bring them back from the latent space representation. In addition to trajectory generation, we employ the trained autoencoder as a feature extractor, for the purpose of clustering and anomaly detection, to obtain further insights into the collected scenario dataset. We experimentally investigate the performance of the proposed framework on real-world scenario trajectories obtained from in-field data collection.

Attri-VAE: Attribute-based interpretable representations of medical images with variational autoencoders.

Deep learning (DL) methods where interpretability is intrinsically considered as part of the model are required to better understand the relationship of clinical and imaging-based attributes with DL outcomes, thus facilitating their use in the reasoning behind the medical decisions. Latent space representations built with variational autoencoders (VAE) do not ensure individual control of data attributes. Attribute-based methods enforcing attribute disentanglement have been proposed in the literature for classical computer vision tasks in benchmark data. In this paper, we propose a VAE approach, the Attri-VAE, that includes an attribute regularization term to associate clinical and medical imaging attributes with different regularized dimensions in the generated latent space, enabling a better-disentangled interpretation of the attributes. Furthermore, the generated attention maps explained the attribute encoding in the regularized latent space dimensions. Using the Attri-VAE approach we analyzed healthy and myocardial infarction patients with clinical, cardiac morphology, and radiomics attributes. The proposed model provided an excellent trade-off between reconstruction fidelity, disentanglement, and interpretability, outperforming state-of-the-art VAE approaches according to several quantitative metrics. The resulting latent space allowed the generation of realistic synthetic data in the trajectory between two distinct input samples or along a specific attribute dimension to better interpret changes between different cardiac conditions.