Dimensional Latent Space Research Articles

Variational inference for the latent shrinkage position model

AbstractThe latent position model (LPM) is a popular method used in network data analysis where nodes are assumed to be positioned in a ‐dimensional latent space. The latent shrinkage position model (LSPM) is an extension of the LPM which automatically determines the number of effective dimensions of the latent space via a Bayesian nonparametric shrinkage prior. However, the LSPM's reliance on Markov chain Monte Carlo for inference, while rigorous, is computationally expensive, making it challenging to scale to networks with large numbers of nodes. We introduce a variational inference approach for the LSPM, aiming to reduce computational demands while retaining the model's ability to intrinsically determine the number of effective latent dimensions. The performance of the variational LSPM is illustrated through simulation studies and its application to real‐world network data. To promote wider adoption and ease of implementation, we also provide open‐source code.

Latent space representation method for structural acoustic assessments

When targeting structural acoustic objectives, engineering practitioners face epistemic uncertainties in the selection of optimal geometries and material distributions, particularly during early stages of the design process. Models built for simulating acoustic phenomena generally produce vector-valued output quantities of interest, such as autospectral density and frequency response functions. Given finite compute resources and time we seek computationally parsimonious ways to distill meaningful design information into actionable results from a limited set of model runs, and thus aim to use machine learning to perform model order reduction. Unlike time series data for which recurrent neural networks can learn from prior time steps to inform subsequent steps, frequency-dependent data demands a different machine learning paradigm. We thus evaluate the utility of autoencoders to represent structural acoustic results with a low dimensional latent space to enable such objectives as surrogate modeling for design optimization. We demonstrate the accuracy of autoencoder based methods of constructing a manifold representation for frequency dependent functions of varying modal density and damping, and discuss the predictive capability thereof.

Spatially Resolved Modeling of the Nonlinear Dynamics of a Laminar Premixed Flame With a Multilayer Perceptron—Convolution Autoencoder Network

Abstract This work presents a multilayer perceptron-convolutional auto-encoder (MLP-CAE) neural network, which accurately predicts the two-dimensional flame dynamics of an acoustically excited premixed laminar flame. The architecture maps the acoustic perturbation time series into a heat release rate field, capturing flame lengths and shapes. This extends previous neural network models, which predicted only the field-integrated value. The MLP-CAE comprises two submodels: an MLP and a CAE. The idea behind the CAE network is to find a lower dimensional latent space of the heat release rate field. The MLP is responsible for modeling the flame dynamics by transforming the acoustic forcing signal into this latent space, enabling the decoder to produce the flow field distributions. To train the MLP-CAE, computational fluid dynamics (CFD) flame simulations with a broadband acoustic forcing were used. Its normalized amplitude was set to 0.5 and 1.0, ensuring a nonlinear flame response. The network was found to accurately predict the perturbed flame shapes. Additionally, it conserved the correct frequency response as verified by the global and local flame describing functions. The MLP-CAE provides a building block toward a potential shift away from a “0D” flame analysis with the acoustic compactness assumption. Combined with an acoustic network, the generated flame fields could provide more physical insight into the thermoacoustic dynamics. Those capabilities do not come at an additional significant computational cost, as even previous nonspatial flame models had to train on the CFD data, which included field distributions.

The wave equation as a differentiable-programmable computer for the design of acoustic metamaterials

Designing new metamaterials enables fine-grain control of wave dynamics. Applications of these materials range from wave steering devices for seismic wave manipulation, to “super-focusing” devices for high-resolution ultrasound imaging and surgery. Control of systems governed by partial differential equations is an inherently hard problem. Specifically, control of wave dynamics is a challenging problem, because of the additional physical constraints and intrinsic properties of wave phenomena such as attenuation, reflection, and scattering. In this work, we propose a novel method for transient control of wave equations with Model Predictive Control. The proposed model incorporates the essential physical properties observed in the original dynamics directly into its lower dimensional latent space. We show that our model is capable of using its latent dynamics to predict scalar integral quantities over a variable length timespan. Moreover, it is fully interpretable by the corresponding physical quantities, which allows it to guarantee solution properties. We demonstrate this method by solving important cases of wave equation, which have not been solved before, such as control of transient waves. We make our code publicly available.

Comparison of reduced order models based on dynamic mode decomposition and deep learning for predicting chaotic flow in a random arrangement of cylinders

Three reduced order models are evaluated in their capacity to predict the future state of an unsteady chaotic flow field. A spatially fully developed flow generated in a random packing of cylinders at a solid fraction of 0.1 and a nominal Reynolds number of 50 is investigated. For deep learning (DL), convolutional autoencoders are used to reduce the high-dimensional data to lower dimensional latent space representations of size 16, which were then used for training the temporal architectures. To predict the future states, two DL based methods, long short-term memory and temporal convolutional neural networks, are used and compared to the linear dynamic mode decomposition (DMD). The predictions are tested in their capability to predict the spatiotemporal variations of velocity and pressure, flow statistics such as root mean squared values, and the capability to predict fluid forces on the cylinders. Relative errors between 15% and 20% are evident in predicting instantaneous velocities, chiefly resulting from phase differences between predictions and ground truth. The spatial distribution of statistical second moments is predicted to be within a maximum of 5%–10% of the ground truth with mean error in the range of 1%–2%. Using the predicted fields, instantaneous fluid drag force predictions on individual particles exhibit a mean relative error within 20%, time-averaged drag force predictions to within 5%, and total drag force over all particles to within 1% of the ground truth values. It is found that overall, the non-linear DL models are more accurate than the linear DMD algorithm for the prediction of future states.

ECG-NET: A deep LSTM autoencoder for detecting anomalous ECG

The electrocardiogram (ECG) is a standard test to monitor the activity of the heart. Many cardiac abnormalities are manifested in the ECG including arrhythmia that refers to an abnormal heart rhythm. The basis of arrhythmia diagnosis is the identification of normal versus abnormal heart beats, and their correct classification based on ECG morphology. This paper proposes a novel and robust approach for representation learning of ECG sequences using a LSTM autoencoder for anomaly detection.The encoder part encodes the ECG signal into a lower dimensional latent space representation and decoder part then tries to reconstruct the specified ECG signal. The model is trained only on normal (non-anomalous) ECG signals. Then reconstruction loss of test ECG signals are calculated. Next a reconstruction loss threshold value is determined from the frequency distribution of the reconstruction losses so that from the reconstruction loss value above a certain threshold is determined as anomaly, otherwise it will be treated as normal. Determination of threshold is done using manual and Kapur’s automated thresholding procedures.The aforementioned model has been applied on publicly available ECG5000 dataset. From the experimental results it is observed that the proposed model achieved more than 98% accuracy having precision, recall and F1 values more than 0.94, 0.97, 0.96 respectively. The performance of the proposed method is also found to be superior in most of the cases as compared to the results of seven other recent counter-part methods reported in the literature.

A multi-task learning-based optimization approach for finding diverse sets of microstructures with desired properties

Optimization along the chain processing-structure-properties-performance is one of the core objectives in data-driven materials science. In this sense, processes are supposed to manufacture workpieces with targeted material microstructures. These microstructures are defined by the material properties of interest and identifying them is a question of materials design. In the present paper, we addresse this issue and introduce a generic multi-task learning-based optimization approach. The approach enables the identification of sets of highly diverse microstructures for given desired properties and corresponding tolerances. Basically, the approach consists of an optimization algorithm that interacts with a machine learning model that combines multi-task learning with siamese neural networks. The resulting model (1) relates microstructures and properties, (2) estimates the likelihood of a microstructure of being producible, and (3) performs a distance preserving microstructure feature extraction in order to generate a lower dimensional latent feature space to enable efficient optimization. The proposed approach is applied on a crystallographic texture optimization problem for rolled steel sheets given desired properties.

Deep weighted Monte Carlo: a hybrid option pricing framework using neural networks

Recent studies have demonstrated the efficiency of Variational Autoencoders (VAE) to compress high-dimensional implied volatility surfaces into a low dimensional representation. Although this method can be effectively used for pricing vanilla options, it does not provide any explicit information about the dynamics of the underlying asset. In our work we present an effective way to overcome this problem. We use a Weighted Monte Carlo approach to first generate paths from a simple a priori Brownian dynamics, and then calculate path weights to price options correctly. We develop and successfully train a neural network that is able to assign these weights directly from the latent space. Combining the encoder network of the VAE and this new ”weight assigner” module, we are able to build a dynamic pricing framework which cleanses the volatility surface from irrelevant noise fluctuations, and then can price not just vanillas, but also exotic options on this idealized vol surface. This pricing method can provide relative value signals for option traders. Key findings of the paper: We confirmed the success of Variational Autoencoders in compressing volatility surfaces into low dimensional latent spaces in the swaption market. We successfully extracted the physically meaningful information from the latent representation of the market by translating the latent coordinates into weights of Monte Carlo paths. We developed a hybrid approach by combining Deep Learning and Weighted Monte Carlo that can price any type of option consistently with the latent model represented by the fully trained VAE.

Ultrafast generation of atomistic biomolecular trajectories for methods development.

Validation of new methods requires testing data of high (protein-like) complexity, that also can be exhaustively sampled to obtain a precise reference. In practice, however, these criteria are often mutually exclusive. The expense of integrating the equations of motion for molecular dynamics (MD) simulations, for example, typically makes high-quality sampling of complex systems infeasible. Simpler and faster models such as low dimensional potentials can be sampled extensively, but likely do not capture sufficient complexity to challenge cutting-edge new methods. Building on precedents in the literature, we demonstrate a general synthetic dynamics framework for efficiently propagating dynamics in a low dimensional latent space and projecting back to the full-dimensional coordinate space. This framework can efficiently produce extremely long trajectories, and enables rapid testing of enhanced sampling and analysis algorithms on highly non-trivial models. We demonstrate synthetic MD (synMD) trajectory generation based on a fine-grained Markov state model, stratified along slow coordinates to enforce realistic long-time dynamics. Each discrete state is mapped to an atomistic configuration, leading to reasonably realistic MD-mimicking trajectories. We show generation of atomistic synMD trajectories for the fast-folding miniprotein Trp-cage at a rate of over 8 seconds/day on a single core of a standard workstation CPU. We also demonstrate the use of synMD as a dynamics engine for the popular WESTPA implementation of the weighted ensemble enhanced sampling strategy. Although we currently use MSM generators, our implementation provides a generic interface for implementing different generators, such as have been proposed based on machine-learning approaches.

Surface Approximation by Means of Gaussian Process Latent Variable Models and Line Element Geometry

The close relation between spatial kinematics and line geometry has been proven to be fruitful in surface detection and reconstruction. However, methods based on this approach are limited to simple geometric shapes that can be formulated as a linear subspace of line or line element space. The core of this approach is a principal component formulation to find a best-fit approximant to a possibly noisy or impartial surface given as an unordered set of points or point cloud. We expand on this by introducing the Gaussian process latent variable model, a probabilistic non-linear non-parametric dimensionality reduction approach following the Bayesian paradigm. This allows us to find structure in a lower dimensional latent space for the surfaces of interest. We show how this can be applied in surface approximation and unsupervised segmentation to the surfaces mentioned above and demonstrate its benefits on surfaces that deviate from these. Experiments are conducted on synthetic and real-world objects.

Unsupervised Multi-Latent Space RL Framework for Video Summarization in Ultrasound Imaging.

The COVID-19 pandemic has highlighted the need for a tool to speed up triage in ultrasound scans and provide clinicians with fast access to relevant information. To this end, we propose a new unsupervised reinforcement learning (RL) framework with novel rewards to facilitate unsupervised learning by avoiding tedious and impractical manual labelling for summarizing ultrasound videos. The proposed framework is capable of delivering video summaries with classification labels and segmentations of key landmarks which enhances its utility as a triage tool in the emergency department (ED) and for use in telemedicine. Using an attention ensemble of encoders, the high dimensional image is projected into a low dimensional latent space in terms of: a) reduced distance with a normal or abnormal class (classifier encoder), b) following a topology of landmarks (segmentation encoder), and c) the distance or topology agnostic latent representation (autoencoders). The summarization network is implemented using a bi-directional long short term memory (Bi-LSTM) which utilizes the latent space representation from the encoder. Validation is performed on lung ultrasound (LUS), that typically represent potential use cases in telemedicine and ED triage acquired from different medical centers across geographies (India and Spain). The proposed approach trained and tested on 126 LUS videos showed high agreement with the ground truth with an average precision of over 80% and average F1 score of well over 44 ±1.7 %. The approach resulted in an average reduction in storage space of 77% which can ease bandwidth and storage requirements in telemedicine.

Designing catalysts with deep generative models and computational data. A case study for Suzuki cross coupling reactions.

The need for more efficient catalytic processes is ever-growing, and so are the costs associated with experimentally searching chemical space to find new promising catalysts. Despite the consolidated use of density functional theory (DFT) and other atomistic models for virtually screening molecules based on their simulated performance, data-driven approaches are rising as indispensable tools for designing and improving catalytic processes. Here, we present a deep learning model capable of generating new catalyst-ligand candidates by self-learning meaningful structural features solely from their language representation and computed binding energies. We train a recurrent neural network-based Variational Autoencoder (VAE) to compress the molecular representation of the catalyst into a lower dimensional latent space, in which a feed-forward neural network predicts the corresponding binding energy to be used as the optimization function. The outcome of the optimization in the latent space is then reconstructed back into the original molecular representation. These trained models achieve state-of-the-art predictive performances in catalysts' binding energy prediction and catalysts' design, with a mean absolute error of 2.42 kcal mol-1 and an ability to generate 84% valid and novel catalysts.

Latent space projection of brain FDG‐PET creates a powerful classifier for neurodegenerative diseases

AbstractBackgroundFluorodeoxyglucose positron emission tomography (FDG‐PET) is an established modality for differential diagnosis of dementia. Deriving phenotypic signatures from FDG‐PET via machine learning is challenging due to the high dimensional nature of brain images relative to the generally small number of labeled examples available for training, the class imbalance among those labels, and the cooccurrence of multiple pathologies. In this study, we developed a multi‐class, multi‐label framework to address these challenges.MethodA database of clinically acquired PET/CT images from 3,000 unique patients was used to develop a latent space model using matrix decomposition. This model was then applied to images from a separate cohort of Mayo Clinic Alzheimer’s Disease Research Center participants (n=1,745) labeled as cognitively unimpaired (CU) (n=1,436) or with the following potentially co‐occurring phenotypes: Alzheimer’s disease (AD) (n=165), Lewy body dementia (DLB) (n=92), behavioral variant frontotemporal dementia (bvFTD) (n=43), semantic (svPPA) (n=10) and logopenic (lvPPA) (n=13) variant PPA, and posterior cortical atrophy (PCA) (n=17). A k‐nearest neighbors classifier that is robust to these imbalanced and overlapping labels was then trained on these examples. The resulting classifier was evaluated by area under receiver‐operator characteristic curve (ROC‐AUC) via leave one out cross validation, using clinical diagnosis as the gold standard.ResultROC curves and AUC scores for each phenotype are illustrated in Fig. 1a. Because the classifier is based on a k‐nearest neighbors connectivity matrix, it has a convenient graphical representation, where images are nodes and edges are drawn between an image and its set of nearest neighbors in latent space. A self‐organizing force directed graph constructed in this way is illustrated in Fig 1b, highlighting the strong separation of CU and degenerative images, as well as the segregation of each phenotype within the neurodegenerative region of the graph.ConclusionIn this study, we developed a machine learning framework for classification of neurodegenerative disease based on k‐nearest neighbor analysis in a low dimensional latent space projection of FDG‐PET images. By leveraging low‐dimensional representations and k‐nearest neighbors analysis, this framework is robust in multi‐class, multi‐label tasks with strong class imbalance and provides a highly interpretable graphical representation.

Latent space projection of brain FDG‐PET creates a powerful classifier for neurodegenerative diseases

AbstractBackgroundFluorodeoxyglucose positron emission tomography (FDG‐PET) is an established modality for differential diagnosis of dementia. Deriving phenotypic signatures from FDG‐PET via machine learning is challenging due to the high dimensional nature of brain images relative to the generally small number of labeled examples available for training, the class imbalance among those labels, and the cooccurrence of multiple pathologies. In this study, we developed a multi‐class, multi‐label framework to address these challenges.MethodA database of clinically acquired PET/CT images from 3,000 unique patients was used to develop a latent space model using matrix decomposition. This model was then applied to images from a separate cohort of Mayo Clinic Alzheimer’s Disease Research Center participants (n=1,745) labeled as cognitively unimpaired (CU) (n=1,436) or with the following potentially co‐occurring phenotypes: Alzheimer’s disease (AD) (n=165), Lewy body dementia (DLB) (n=92), behavioral variant frontotemporal dementia (bvFTD) (n=43), semantic (svPPA) (n=10) and logopenic (lvPPA) (n=13) variant PPA, and posterior cortical atrophy (PCA) (n=17). A k‐nearest neighbors classifier that is robust to these imbalanced and overlapping labels was then trained on these examples. The resulting classifier was evaluated by area under receiver‐operator characteristic curve (ROC‐AUC) via leave one out cross validation, using clinical diagnosis as the gold standard.ResultROC curves and AUC scores for each phenotype are illustrated in Fig. 1a. Because the classifier is based on a k‐nearest neighbors connectivity matrix, it has a convenient graphical representation, where images are nodes and edges are drawn between an image and its set of nearest neighbors in latent space. A self‐organizing force directed graph constructed in this way is illustrated in Fig 1b, highlighting the strong separation of CU and degenerative images, as well as the segregation of each phenotype within the neurodegenerative region of the graph.ConclusionIn this study, we developed a machine learning framework for classification of neurodegenerative disease based on k‐nearest neighbor analysis in a low dimensional latent space projection of FDG‐PET images. By leveraging low‐dimensional representations and k‐nearest neighbors analysis, this framework is robust in multi‐class, multi‐label tasks with strong class imbalance and provides a highly interpretable graphical representation.

On the Dimensionality and Utility of Convolutional Autoencoder’s Latent Space Trained with Topology-Preserving Spectral EEG Head-Maps

Electroencephalography (EEG) signals can be analyzed in the temporal, spatial, or frequency domains. Noise and artifacts during the data acquisition phase contaminate these signals adding difficulties in their analysis. Techniques such as Independent Component Analysis (ICA) require human intervention to remove noise and artifacts. Autoencoders have automatized artifact detection and removal by representing inputs in a lower dimensional latent space. However, little research is devoted to understanding the minimum dimension of such latent space that allows meaningful input reconstruction. Person-specific convolutional autoencoders are designed by manipulating the size of their latent space. A sliding window technique with overlapping is employed to segment varied-sized windows. Five topographic head-maps are formed in the frequency domain for each window. The latent space of autoencoders is assessed using the input reconstruction capacity and classification utility. Findings indicate that the minimal latent space dimension is 25% of the size of the topographic maps for achieving maximum reconstruction capacity and maximizing classification accuracy, which is achieved with a window length of at least 1 s and a shift of 125 ms, using the 128 Hz sampling rate. This research contributes to the body of knowledge with an architectural pipeline for eliminating redundant EEG data while preserving relevant features with deep autoencoders.

Comparing Well and Geophysical Data for Temperature Monitoring Within a Bayesian Experimental Design Framework

AbstractTemperature logs are an important tool in the geothermal industry. Temperature measurements from boreholes are used for exploration, system design, and monitoring. The number of observations, however, is not always sufficient to fully determine the temperature field or explore the entire parameter space of interest. Drilling in the best locations is still difficult and expensive. It is therefore critical to optimize the number and location of boreholes. Due to its higher spatial resolution and lower cost, four‐dimensional (4D) temperature field monitoring via time‐lapse Electrical Resistivity Tomography has been investigated as a potential alternative. We use Bayesian Evidential Learning (BEL), a Monte Carlo‐based training approach, to optimize the design of a 4D temperature field monitoring experiment. We demonstrate how BEL can take into account various data source combinations (temperature logs combined with geophysical data) in the Bayesian optimal experimental design (BOED). To determine the optimal data source combination, we use the Root Mean Squared Error of the predicted target in the low dimensional latent space where BEL is solving the prediction problem. The parameter estimates are accurate enough to use in BOED. Furthermore, the method is not limited to monitoring temperature fields and can be applied to other similar experimental design problems. The method is computationally efficient and requires little training data. For the considered optimal design problem, a training set of only 200 samples and a test set of 50 samples is sufficient.

CF-AMVRGO: Collaborative Filtering based Adaptive Moment Variance Reduction Gradient Optimizer for Movie Recommendations

ABSTRACT In recent decades, recommendation systems are used in a variety of applications like social connections, movies, music, and venues. The existing algorithms has certain limitations like data sparsity, cold start problem, and poor scalability to degrade the movie recommendation performance. To address the aforementioned concerns, a new matrix factorization-based collaborative filtering algorithm is developed to enhance movie recommendation performance. In a neural collaborative filtering algorithm, an adaptive moment variance reduction gradient optimizer is applied in low-rank matrix factorization for representing the users and items in a low dimensional latent space to obtain effective movie recommendation performance. In this paper, the proposed algorithm performance is evaluated on three benchmark datasets; movielens 100K, 1M, and 10M datasets. Simulation results showed that the proposed algorithm obtained the difference of 0.08 and 0.056 roots mean square error value in movielens 1M and 10M datasets, which are better compared to the existing collaborative filtering-based deep learning algorithm.

Using the Kriging Correlation for unsupervised feature selection problems

This paper proposes a KC Score to measure feature importance in clustering analysis of high-dimensional data. The KC Score evaluates the contribution of features based on the correlation between the original features and the reconstructed features in the low dimensional latent space. A KC Score-based feature selection strategy is further developed for clustering analysis. We investigate the performance of the proposed strategy by conducting a study of four single-cell RNA sequencing (scRNA-seq) datasets. The results show that our strategy effectively selects important features for clustering. In particular, in three datasets, our proposed strategy selected less than 5% of the features and achieved the same or better clustering performance than when using all of the features.

Sampling Possible Reconstructions of Undersampled Acquisitions in MR Imaging With a Deep Learned Prior.

Undersampling the k-space during MR acquisitions saves time, however results in an ill-posed inversion problem, leading to an infinite set of images as possible solutions. Traditionally, this is tackled as a reconstruction problem by searching for a single "best" image out of this solution set according to some chosen regularization or prior. This approach, however, misses the possibility of other solutions and hence ignores the uncertainty in the inversion process. In this paper, we propose a method that instead returns multiple images which are possible under the acquisition model and the chosen prior to capture the uncertainty in the inversion process. To this end, we introduce a low dimensional latent space and model the posterior distribution of the latent vectors given the acquisition data in k-space, from which we can sample in the latent space and obtain the corresponding images. We use a variational autoencoder for the latent model and the Metropolis adjusted Langevin algorithm for the sampling. We evaluate our method on two datasets; with images from the Human Connectome Project and in-house measured multi-coil images. We compare to five alternative methods. Results indicate that the proposed method produces images that match the measured k-space data better than the alternatives, while showing realistic structural variability. Furthermore, in contrast to the compared methods, the proposed method yields higher uncertainty in the undersampled phase encoding direction, as expected.

Controllable and Identity-Aware Facial Attribute Transformation.

Modifying facial attributes without the paired dataset proves to be a challenging task. Previously, approaches either required supervision from a ground-truth transformed image or required training a separate model for mapping every pair of attributes. These limit the scalability of the models to accommodate a larger set of attributes since the number of models that we need to train grows exponentially large. Another major drawback of the previous approaches is the unintentional gain of the identity of the person as they transform the facial attributes. We propose a method that allows for controllable and identity-aware transformations across multiple facial attributes using only a single model. Our approach is to train a generative adversarial network (GAN) with a multitask conditional discriminator that recognizes the identity of the face, distinguishes real images from fake, as well as identifies facial attributes present in an image. This guides the generator into producing an output that is realistic while preserving the person's identity and facial attributes. Through this framework, our model also learns meaningful image representations in a lower dimensional latent space and semantically associate separate parts of the encoded vector with both the person's identity and facial attributes. This opens up the possibility of generating new faces and other transformations such as making the face thinner or chubbier. Furthermore, our model only encodes the image once and allows for multiple transformations using the encoded vector. This allows for faster transformations since it does not need to reprocess the entire image for every transformation. We show the effectiveness of our proposed method through both qualitative and quantitative evaluations, such as ablative studies, visual inspection, and face verification. Competitive results are achieved compared to the main competition (CycleGAN), however, at great space and extensibility gain by using a single model.