Low-dimensional Latent Space Research Articles

Unfolding the multiscale structure of networks with dynamical Ollivier-Ricci curvature

Describing networks geometrically through low-dimensional latent metric spaces has helped design efficient learning algorithms, unveil network symmetries and study dynamical network processes. However, latent space embeddings are limited to specific classes of networks because incompatible metric spaces generally result in information loss. Here, we study arbitrary networks geometrically by defining a dynamic edge curvature measuring the similarity between pairs of dynamical network processes seeded at nearby nodes. We show that the evolution of the curvature distribution exhibits gaps at characteristic timescales indicating bottleneck-edges that limit information spreading. Importantly, curvature gaps are robust to large fluctuations in node degrees, encoding communities until the phase transition of detectability, where spectral and node-clustering methods fail. Using this insight, we derive geometric modularity to find multiscale communities based on deviations from constant network curvature in generative and real-world networks, significantly outperforming most previous methods. Our work suggests using network geometry for studying and controlling the structure of and information spreading on networks.

Fully Learnable Model for Task-Driven Image Compressed Sensing.

This study primarily investigates image sensing at low sampling rates with convolutional neural networks (CNN) for specific applications. To improve the image acquisition efficiency in energy-limited systems, this study, inspired by compressed sensing, proposes a fully learnable model for task-driven image-compressed sensing (FLCS). The FLCS, based on Deep Convolution Generative Adversarial Networks (DCGAN) and Variational Auto-encoder (VAE), divides the image-compressed sensing model into three learnable parts, i.e., the Sampler, the Solver and the Rebuilder. To be specific, a measurement matrix suitable for a type of image is obtained by training the Sampler. The Solver calculates the image’s low-dimensional representation with the measurements. The Rebuilder learns a mapping from the low-dimensional latent space to the image space. All the mentioned could be trained jointly or individually for a range of application scenarios. The pre-trained FLCS reconstructs images with few iterations for task-driven compressed sensing. As indicated from the experimental results, compared with existing approaches, the proposed method could significantly improve the reconstructed images’ quality while decreasing the running time. This study is of great significance for the application of image-compressed sensing at low sampling rates.

Multiscale Reweighted Stochastic Embedding: Deep Learning of Collective Variables for Enhanced Sampling.

Machine learning methods provide a general framework for automatically finding and representing the essential characteristics of simulation data. This task is particularly crucial in enhanced sampling simulations. There we seek a few generalized degrees of freedom, referred to as collective variables (CVs), to represent and drive the sampling of the free energy landscape. In theory, these CVs should separate different metastable states and correspond to the slow degrees of freedom of the studied physical process. To this aim, we propose a new method that we call multiscale reweighted stochastic embedding (MRSE). Our work builds upon a parametric version of stochastic neighbor embedding. The technique automatically learns CVs that map a high-dimensional feature space to a low-dimensional latent space via a deep neural network. We introduce several new advancements to stochastic neighbor embedding methods that make MRSE especially suitable for enhanced sampling simulations: (1) weight-tempered random sampling as a landmark selection scheme to obtain training data sets that strike a balance between equilibrium representation and capturing important metastable states lying higher in free energy; (2) a multiscale representation of the high-dimensional feature space via a Gaussian mixture probability model; and (3) a reweighting procedure to account for training data from a biased probability distribution. We show that MRSE constructs low-dimensional CVs that can correctly characterize the different metastable states in three model systems: the Müller-Brown potential, alanine dipeptide, and alanine tetrapeptide.

Describing the Structural Phenotype of the Glaucomatous Optic Nerve Head Using Artificial Intelligence

To develop a novel deep-learning approach that can describe the structural phenotype of the glaucomatous optic nerve head (ONH) and can be used as a robust glaucoma diagnosis tool. Retrospective, deep-learning approach diagnosis study. We trained a deep-learning network to segment 3 neural-tissue and 4 connective-tissue layers of the ONH. The segmented optical coherence tomography images were then processed by a customized autoencoder network with an additional parallel branch for binary classification. The encoder part of the autoencoder reduced the segmented optical coherence tomography images into a low-dimensional latent space (LS), whereas the decoder and the classification branches reconstructed the images and classified them as glaucoma or nonglaucoma, respectively. We performed principal component analysis on the latent parameters and identified the principal components (PCs). Subsequently, the magnitude of each PC was altered in steps and reported how it impacted the morphology of the ONH. The image reconstruction quality and diagnostic accuracy increased with the size of the LS. With 54 parameters in the LS, the diagnostic accuracy was 92.0 ± 2.3% with a sensitivity of 90.0 ± 2.4% (at 95% specificity), and the corresponding Dice coefficient for the reconstructed images was 0.86 ± 0.04. By changing the magnitudes of PC in steps, we were able to reveal how the morphology of the ONH changes as one transitions from a "nonglaucoma" to a "glaucoma" condition. Our network was able to identify novel biomarkers of the ONH for glaucoma diagnosis. Specifically, the structural features identified by our algorithm were found to be related to clinical observations of glaucoma.

Vocal Feature Extraction-Based Artificial Intelligent Model for Parkinson’s Disease Detection

As a neurodegenerative disorder, Parkinson’s disease (PD) affects the nerve cells of the human brain. Early detection and treatment can help to relieve the symptoms of PD. Recent PD studies have extracted the features from vocal disorders as a harbinger for PD detection, as patients face vocal changes and impairments at the early stages of PD. In this study, two hybrid models based on a Support Vector Machine (SVM) integrating with a Principal Component Analysis (PCA) and a Sparse Autoencoder (SAE) are proposed to detect PD patients based on their vocal features. The first model extracted and reduced the principal components of vocal features based on the explained variance of each feature using PCA. For the first time, the second model used a novel Deep Neural Network (DNN) of an SAE, consisting of multiple hidden layers with L1 regularization to compress the vocal features into lower-dimensional latent space. In both models, reduced features were fed into the SVM as inputs, which performed classification by learning hyperplanes, along with projecting the data into a higher dimension. An F1-score, a Mathews Correlation Coefficient (MCC), and a Precision-Recall curve were used, along with accuracy to evaluate the proposed models due to highly imbalanced data. With its highest accuracy of 0.935, F1-score of 0.951, and MCC value of 0.788, the probing results show that the proposed model of the SAE-SVM surpassed not only the former model of the PCA-SVM and other standard models including Multilayer Perceptron (MLP), Extreme Gradient Boosting (XGBoost), K-Nearest Neighbor (KNN), and Random Forest (RF), but also surpassed two recent studies using the same dataset. Oversampling and balancing the dataset with SMOTE boosted the performance of the models.

Design subspace learning: Structural design space exploration using performance-conditioned generative modeling

Designers increasingly rely on parametric design studies to explore and improve structural concepts based on quantifiable metrics, generally either by generating design variations manually or using optimization methods. Unfortunately, both of these approaches have important shortcomings: effectively searching a large design space manually is infeasible, and design optimization overlooks qualitative aspects important in architectural and structural design. There is a need for methods that take advantage of computing intelligence to augment a designer's creativity while guiding—not forcing—their search for better-performing solutions. This research addresses this need by integrating conditional variational autoencoders in a performance-driven design exploration framework. First, a sampling algorithm generates a dataset of meaningful design options from an unwieldy design space. Second, a performance-conditioned variational autoencoder with a low-dimensional latent space is trained using the collected data. This latent space is intuitive to explore by designers even as it offers a diversity of high-performing design options.

Whole Heart Segmentation Using 3D FM-Pre-ResNet Encoder–Decoder Based Architecture with Variational Autoencoder Regularization

An accurate whole heart segmentation (WHS) on medical images, including computed tomography (CT) and magnetic resonance (MR) images, plays a crucial role in many clinical applications, such as cardiovascular disease diagnosis, pre-surgical planning, and intraoperative treatment. Manual whole-heart segmentation is a time-consuming process, prone to subjectivity and error. Therefore, there is a need to develop a quick, automatic, and accurate whole heart segmentation systems. Nowadays, convolutional neural networks (CNNs) emerged as a robust approach for medical image segmentation. In this paper, we first introduce a novel connectivity structure of residual unit that we refer to as a feature merge residual unit (FM-Pre-ResNet). The proposed connectivity allows the creation of distinctly deep models without an increase in the number of parameters compared to the pre-activation residual units. Second, we propose a three-dimensional (3D) encoder–decoder based architecture that successfully incorporates FM-Pre-ResNet units and variational autoencoder (VAE). In an encoding stage, FM-Pre-ResNet units are used for learning a low-dimensional representation of the input. After that, the variational autoencoder (VAE) reconstructs the input image from the low-dimensional latent space to provide a strong regularization of all model weights, simultaneously preventing overfitting on the training data. Finally, the decoding stage creates the final whole heart segmentation. We evaluate our method on the 40 test subjects of the MICCAI Multi-Modality Whole Heart Segmentation (MM-WHS) Challenge. The average dice values of whole heart segmentation are 90.39% (CT images) and 89.50% (MRI images), which are both highly comparable to the state-of-the-art.

Fusing stacked autoencoder and long short-term memory for regional multistep-ahead flood inundation forecasts

Reliable and accurate regional multistep-ahead flood forecasts during extreme events are crucial and beneficial to flood disaster management and preparedness. Hydrologic uncertainty associated with the nonlinear dependence structure of flood inundation dynamics makes flood inundation forecasting fundamentally challenging. This study proposes a novel machine learning-based model (SAE-RNN) that hybrids the stacked autoencoder (SAE) with a recurrent neural network (RNN) for providing accurate and timely information to support emergency management in areas impacted by flood hazards. The proposed SAE-RNN model uses SAE to compress (encode) the high-dimensional flood inundation depths in a wide region into a low-dimensional latent space representation (flood features), uses RNN to forecast multistep-ahead flood features based on regional rainfall patterns, and finally uses SAE to reconstruct (decode) the multistep-ahead forecasts of flood features into regional flood inundation depths. A large number of hourly datasets of flood inundation depths collected in Yilan County of Taiwan formed the case study, where each dataset contains 169,797 grids of inundation depth. The datasets were divided into three independent datasets for use in training, validating and testing stages. The models’ results showed that RMSE values were very small (<0.09 m) and R2 values were high (>0.95) in all the cases (1- up to 3-hour-ahead forecasts in three stages). We conclude that the reason why the proposed SAE-RNN models are capable of attaining favorable regional multistep-ahead flood inundation forecasts could be owing to two core strategies: the effective continual extraction of the nonlinear dependence structure from flood inundation dynamics for lessening hydrologic uncertainty by virtue of SAE; and the nonlinear conversion of rainfall sequences into future flood features by virtue of RNN.

Symbolic pregression: Discovering physical laws from distorted video.

We present a method for unsupervised learning of equations of motion for objects in raw and optionally distorted unlabeled synthetic video (or, more generally, for discovering and modeling predictable features in time-series data). We first train an autoencoder that maps each video frame into a low-dimensional latent space where the laws of motion are as simple as possible, by minimizing a combination of nonlinearity, acceleration, and prediction error. Differential equations describing the motion are then discovered using Pareto-optimal symbolic regression. We find that our pre-regression ("pregression") step is able to rediscover Cartesian coordinates of unlabeled moving objects even when the video is distorted by a generalized lens. Using intuition from multidimensional knot theory, we find that the pregression step is facilitated by first adding extra latent space dimensions to avoid topological problems during training and then removing these extra dimensions via principal component analysis. An inertial frame is autodiscovered by minimizing the combined equation complexity for multiple experiments.

Fast Multi-Objective Optimization of Magnetic Shield Shape by Combining Auto-Encoder and Level-Set Method

We proposed a new multi-objective optimization approach combining both gradient calculation in actual high-dimensional design space and that in a low-dimensional latent space that is an appropriately compressed design space with Auto-Encoder. The proposed approach enables us to quickly and precisely search for more global pareto solution group in multi-objective optimization by using the optimization results in the low-dimensional space as the initial value of the gradient calculation in the actual design space.

Predictive learning as a network mechanism for extracting low-dimensional latent space representations

Artificial neural networks have recently achieved many successes in solving sequential processing and planning tasks. Their success is often ascribed to the emergence of the task’s low-dimensional latent structure in the network activity – i.e., in the learned neural representations. Here, we investigate the hypothesis that a means for generating representations with easily accessed low-dimensional latent structure, possibly reflecting an underlying semantic organization, is through learning to predict observations about the world. Specifically, we ask whether and when network mechanisms for sensory prediction coincide with those for extracting the underlying latent variables. Using a recurrent neural network model trained to predict a sequence of observations we show that network dynamics exhibit low-dimensional but nonlinearly transformed representations of sensory inputs that map the latent structure of the sensory environment. We quantify these results using nonlinear measures of intrinsic dimensionality and linear decodability of latent variables, and provide mathematical arguments for why such useful predictive representations emerge. We focus throughout on how our results can aid the analysis and interpretation of experimental data.

Two-step hybrid collaborative filtering using deep variational Bayesian autoencoders

Existing recommender systems rely on user and item representations in a fixed continuous low-dimensional latent space. To predict ratings, they use only an implicit feedback matrix, whereas user and item side information is ignored. Furthermore, they use the same arbitrary priors for the user and item latent vectors, reducing the ability of the model to identify the actual latent vectors. Currently, the latent parameters should be learned directly for every user and movie. This is problematic, as it would require both model retraining and learning the latent vector representations when users or items are added to the underlying dataset. To address these issues, we propose a two-step hybrid variational Bayesian autoencoder to characterize the uncertainty of predicted ratings. An encoder is first trained to map data vectors to the latent space so that the latent representations can be dynamically computed. Subsequently, we use the generative process of users and items with their priors as side-specific information to handle matrix sparsity and better learn their latent vectors. Finally, we consider stochastic variational inference to approximate the posterior density of intractable user-item latent vectors. Experiments conducted on two real-world datasets demonstrate the effectiveness of the proposed method compared with state-of-the-art methods.

Convolutional neural network and long short-term memory based reduced order surrogate for minimal turbulent channel flow

We investigate the applicability of the machine learning based reduced order model (ML-ROM) to three-dimensional complex flows. As an example, we consider a turbulent channel flow at the friction Reynolds number of Reτ=110 in a minimum domain, which can maintain coherent structures of turbulence. Training datasets are prepared by direct numerical simulation (DNS). The present ML-ROM is constructed by combining a three-dimensional convolutional neural network autoencoder (CNN-AE) and a long short-term memory (LSTM). The CNN-AE works to map high-dimensional flow fields into a low-dimensional latent space. The LSTM is, then, utilized to predict a temporal evolution of the latent vectors obtained by the CNN-AE. The combination of the CNN-AE and LSTM can represent the spatiotemporal high-dimensional dynamics of flow fields by only integrating the temporal evolution of the low-dimensional latent dynamics. The turbulent flow fields reproduced by the present ML-ROM show statistical agreement with the reference DNS data in time-ensemble sense, which can also be found through an orbit-based analysis. Influences of the population of vortical structures contained in the domain and the time interval used for temporal prediction on the ML-ROM performance are also investigated. The potential and limitation of the present ML-ROM for turbulence analysis are discussed at the end of our presentation.

Generalization-Based Acquisition of Training Data for Motor Primitive Learning by Neural Networks

Autonomous robot learning in unstructured environments often faces the problem that the dimensionality of the search space is too large for practical applications. Dimensionality reduction techniques have been developed to address this problem and describe motor skills in low-dimensional latent spaces. Most of these techniques require the availability of a sufficiently large database of example task executions to compute the latent space. However, the generation of many example task executions on a real robot is tedious, and prone to errors and equipment failures. The main result of this paper is a new approach for efficient database gathering by performing a small number of task executions with a real robot and applying statistical generalization, e.g., Gaussian process regression, to generate more data. We have shown in our experiments that the data generated this way can be used for dimensionality reduction with autoencoder neural networks. The resulting latent spaces can be exploited to implement robot learning more efficiently. The proposed approach has been evaluated on the problem of robotic throwing at a target. Simulation and real-world results with a humanoid robot TALOS are provided. They confirm the effectiveness of generalization-based database acquisition and the efficiency of learning in a low-dimensional latent space.

Skeletonized Wave-Equation Refraction Inversion With Autoencoded Waveforms

We present a method that skeletonizes the first arriving seismic refractions by machine learning and inverts them for the subsurface velocity model. In this study, first arrivals can be compressed in a low-rank sense with their skeletal features extracted by a well-trained autoencoder. Empirical experiments suggest that the autoencoder’s <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$1\times 1$ </tex-math></inline-formula> or <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$2\times 1$ </tex-math></inline-formula> latent vectors vary continuously with respect to the input seismic data. It is, therefore, reasonable to introduce a misfit functional measuring the discrepancies between the predicted and the observed data in a low-dimensional latent space. The benefit of this approach is that an elaborated autoencoding neural network not only refines intrinsic information hidden in the refractions but also improves the quality of inversion for a reliable background velocity model. Numerical tests on both synthetic and field data demonstrate the effectiveness of this method, especially in recovering the low-to-intermediate wavenumber parts of the subsurface velocity distribution. Comparisons are made with the other three relevant methods, the wave-equation travel-time (WT) inversion, the envelope inversion, and the full waveform inversion (FWI). As expected, the cycle skipping problem is alleviated due to the reduction of dimensions of data space. This method outperforms the envelope inversion in resolution, and it is no worse than WT. Moreover, there is no need for careful manual travel-time picking with this methodology. In general, this inversion framework provides an extendable strategy to compress any input data for reconstructing high-dimensional physical parameters.

A Graph Gaussian Embedding Method for Predicting Alzheimer's Disease Progression With MEG Brain Networks.

Characterizing the subtle changes of functional brain networks associated with the pathological cascade of Alzheimer's disease (AD) is important for early diagnosis and prediction of disease progression prior to clinical symptoms. We developed a new deep learning method, termed multiple graph Gaussian embedding model (MG2G), which can learn highly informative network features by mapping high-dimensional resting-state brain networks into a low-dimensional latent space. These latent distribution-based embeddings enable a quantitative characterization of subtle and heterogeneous brain connectivity patterns at different regions, and can be used as input to traditional classifiers for various downstream graph analytic tasks, such as AD early stage prediction, and statistical evaluation of between-group significant alterations across brain regions. We used MG2G to detect the intrinsic latent dimensionality of MEG brain networks, predict the progression of patients with mild cognitive impairment (MCI) to AD, and identify brain regions with network alterations related to MCI.

Markov Chain Generative Adversarial Neural Networks for Solving Bayesian Inverse Problems in Physics Applications

In the context of solving inverse problems for physics applications within a Bayesian framework, we present a new approach, Markov Chain Generative Adversarial Neural Networks (MCGANs), to alleviate the computational costs associated with solving the Bayesian inference problem. GANs pose a very suitable framework to aid in the solution of Bayesian inference problems, as they are designed to generate samples from complicated high-dimensional distributions. By training a GAN to sample from a low-dimensional latent space and then embedding it in a Markov Chain Monte Carlo method, we can highly efficiently sample from the posterior, by replacing both the high-dimensional prior and the expensive forward map. We prove that the proposed methodology converges to the true posterior in the Wasserstein-1 distance and that sampling from the latent space is equivalent to sampling in the high-dimensional space in a weak sense. The method is showcased on three test cases where we perform both state and parameter estimation simultaneously. The approach is shown to be up to two orders of magnitude more accurate than alternative approaches while also being up to an order of magnitude computationally faster, in several test cases, including the important engineering setting of detecting leaks in pipelines.

PRIMAL-GMM: PaRametrIc MAnifold Learning of Gaussian Mixture Models.

We propose a ParametRIc MAnifold Learning (PRIMAL) algorithm for Gaussian mixtures models (GMM), assuming that GMMs lie on or near to a manifold of probability distributions that is generated from a low-dimensional hierarchical latent space through parametric mappings. Inspired by principal component analysis (PCA), the generative processes for priors, means and covariance matrices are modeled by their respective latent space and parametric mapping. Then, the dependencies between latent spaces are captured by a hierarchical latent space by a linear or kernelized mapping. The function parameters and hierarchical latent space are learned by minimizing the reconstruction error between ground-truth GMMs and manifold-generated GMMs, measured by Kullback-Leibler Divergence (KLD). Variational approximation is employed to handle the intractable KLD between GMMs and a variational EM algorithm is derived to optimize the objective function. Experiments on synthetic data, flow cytometry analysis, eye-fixation analysis and topic models show that PRIMAL learns a continuous and interpretable manifold of GMM distributions and achieves a minimum reconstruction error.

Depth-Aware Object Tracking With a Conditional Variational Autoencoder

Object tracking is a fundamental task in computer vision and artificial intelligence. However, state-of-the-art object tracking approaches are still prone to failures and are imprecise when applied to challenging scenarios, and their results are generally confidence agnostic. An imprecise deterministic output with low confidence may lead to disastrous consequences and a lack of proof for subsequent operations and human interventions. Deep network training with ambiguous data or the noise inherent in observations (i.e., data uncertainty or aleatoric uncertainty) will result in inherent uncertainties in predictions. In this paper, we exploit probabilistic depth-aware object tracking with a conditional variational autoencoder (CVAE). First, we build a bridge between the Siamese network and the variational autoencoder conditioned with depth images and propose a novel multimodal Bayesian object tracking method. Second, our proposed method yields a complete probability distribution that enables the production of multiple plausible features. Third, the variational autoencoder conditioned by depth images encodes a low-dimensional latent space that conducts depth-aware tracking, which has obvious advantages for challenging tracking scenarios. Our proposed tracking method outperformed the state-of-the-art trackers on the VOT 2016, VOT 2018, and VOT 2019 datasets.

A Hybrid Method for Remaining Useful Life Prediction of Proton Exchange Membrane Fuel Cell Stack

Proton exchange membrane fuel cell (PEMFC) is a clean and efficient alternative technology for transport applications. The degradation analysis of the PEFMC stack plays a vital role in electric vehicles. We propose a hybrid method based on a deep neural network model, which uses the Monte Carlo dropout approach called MC-DNN and a sparse autoencoder model to analyze the power degradation trend of the PEMFC stack. The sparse autoencoder can map high-dimensional data space to low-dimensional latent space and significantly reduce noise data. Under static and dynamic operating conditions, using two experimental PEMFC stack datasets the predictive performance of our proposed model is compared with some published models. The results show that the MC-DNN model is better than other models. Regarding the remaining useful life (RUL) prediction, the proposed model can obtain more accurate results under different training lengths, and the relative error between 0.19% and 1.82%. In addition, the prediction interval of the predicted RUL is derived by using the MC dropout approach.