Low-dimensional Latent Space Research Articles

Detecting Microstructural Criticality/Degeneracy through Hybrid Learning Strategies Trained by Molecular Dynamics Simulations.

Efficient microstructure design can strongly accelerate the development of materials. However, the complexity of the microstructure-behavior relation renders the criticalities and degeneracies within the microstructure space highly possible. Criticality means that a slight microstructural change can lead to a dramatic transition in material behavior, while degeneracy means that very different microstructures may lead to similar behaviors. To investigate these microstructural characteristics of the fiber/matrix interface within composite materials, we have proposed a hybrid deep-learning-based framework by integrating the supervised feed-forward neural network and the unsupervised autoencoder, which are trained by the molecular dynamics (MD) simulation results. The well-trained model continuously maps the elemental density images within the interfacial area into a low-dimensional latent space. Assisted by the extracted latent features, we can easily detect the criticalities and degeneracies within the original microstructure space of the composite's interface. The predicted microstructural criticalities and degeneracies are validated by investigating their atomistic origins through MD simulations. The proposed framework can be employed for the interfacial microstructure design of composite materials by identifying certain interfacial microstructures that might lead to undesirable behaviors.

Ultrafast generation of atomistic biomolecular trajectories for methods development.

Validation of new methods requires testing data of high (protein-like) complexity, that also can be exhaustively sampled to obtain a precise reference. In practice, however, these criteria are often mutually exclusive. The expense of integrating the equations of motion for molecular dynamics (MD) simulations, for example, typically makes high-quality sampling of complex systems infeasible. Simpler and faster models such as low dimensional potentials can be sampled extensively, but likely do not capture sufficient complexity to challenge cutting-edge new methods. Building on precedents in the literature, we demonstrate a general synthetic dynamics framework for efficiently propagating dynamics in a low dimensional latent space and projecting back to the full-dimensional coordinate space. This framework can efficiently produce extremely long trajectories, and enables rapid testing of enhanced sampling and analysis algorithms on highly non-trivial models. We demonstrate synthetic MD (synMD) trajectory generation based on a fine-grained Markov state model, stratified along slow coordinates to enforce realistic long-time dynamics. Each discrete state is mapped to an atomistic configuration, leading to reasonably realistic MD-mimicking trajectories. We show generation of atomistic synMD trajectories for the fast-folding miniprotein Trp-cage at a rate of over 8 seconds/day on a single core of a standard workstation CPU. We also demonstrate the use of synMD as a dynamics engine for the popular WESTPA implementation of the weighted ensemble enhanced sampling strategy. Although we currently use MSM generators, our implementation provides a generic interface for implementing different generators, such as have been proposed based on machine-learning approaches.

Enabling Visual Action Planning for Object Manipulation Through Latent Space Roadmap

In this article, we present a framework for visual action planning of complex manipulation tasks with high-dimensional state spaces, focusing on manipulation of deformable objects. We propose a latent space roadmap (LSR) for task planning, which is a graph-based structure globally capturing the system dynamics in a low-dimensional latent space. Our framework consists of the following three parts. First, a mapping module (MM) that maps observations is given in the form of images into a structured latent space extracting the respective states as well as generates observations from the latent states. Second, the LSR, which builds and connects clusters containing similar states in order to find the latent plans between start and goal states, extracted by MM. Third, the action proposal module that complements the latent plan found by the LSR with the corresponding actions. We present a thorough investigation of our framework on simulated box stacking and rope/box manipulation tasks, and a folding task executed on a real robot.

Unsupervised model adaptation for multivariate calibration by domain adaptation-regularization based kernel partial least square

In chemometrics, calibration model adaptation is desired when training- and test-samples come from different distributions. Domain-invariant feature representation is currently a successful strategy to realize model adaptation and has received wide attention. The paper presents a nonlinear unsupervised model adaptation method termed as domain adaption regularization-based kernel partial least squares regression (DarKPLS). DarKPLS aims to minimize the source and target distributions in a low-dimensional latent space projected from the reproducing kernel Hilbert space (RKHS) generated with the labeled source data and unlabeled target data. Specially, the distributional means and variances between source and target latent variables are aligned in the RKHS. By extending existing domain invariant partial least square regression (di-PLS) with the projected maximum mean discrepancy (PMMD) to reduce the mean discrepancy in the RKHS further, DarKPLS could realize fine-grained domain alignment that further improves the adaptation performance. DarKPLS is applied to the γ-polyglutamic acid fermentation dataset, tobacco dataset and corn dataset, and it demonstrates improved prediction results in comparison with No adaptation partial least squares (PLS), null augmented regression (NAR), extended linear joint trained framework (ExtJT), scatter component analysis (SCA) and domain-invariant iterative partial least squares (DIPALS).

Surface Approximation by Means of Gaussian Process Latent Variable Models and Line Element Geometry

The close relation between spatial kinematics and line geometry has been proven to be fruitful in surface detection and reconstruction. However, methods based on this approach are limited to simple geometric shapes that can be formulated as a linear subspace of line or line element space. The core of this approach is a principal component formulation to find a best-fit approximant to a possibly noisy or impartial surface given as an unordered set of points or point cloud. We expand on this by introducing the Gaussian process latent variable model, a probabilistic non-linear non-parametric dimensionality reduction approach following the Bayesian paradigm. This allows us to find structure in a lower dimensional latent space for the surfaces of interest. We show how this can be applied in surface approximation and unsupervised segmentation to the surfaces mentioned above and demonstrate its benefits on surfaces that deviate from these. Experiments are conducted on synthetic and real-world objects.

Identifying Synthetic Faces through GAN Inversion and Biometric Traits Analysis

In the field of image forensics, notable attention has been recently paid toward the detection of synthetic contents created through Generative Adversarial Networks (GANs), especially face images. This work explores a classification methodology inspired by the inner architecture of typical GANs, where vectors in a low-dimensional latent space are transformed by the generator into meaningful high-dimensional images. In particular, the proposed detector exploits the inversion of the GAN synthesis process: given a face image under investigation, we identify the point in the GAN latent space which more closely reconstructs it; we project the vector back into the image space, and we compare the resulting image with the actual one. Through experimental tests on widely known datasets (including FFHQ, CelebA, LFW, and Caltech), we demonstrate that real faces can be accurately discriminated from GAN-generated ones by properly capturing the facial traits through different feature representations. In particular, features based on facial landmarks fed to a Support Vector Machine consistently yield a global accuracy of above 88% for each dataset. Furthermore, we experimentally prove that the proposed detector is robust concerning routinely applied post-processing operations.

Unsupervised Multi-Latent Space RL Framework for Video Summarization in Ultrasound Imaging.

The COVID-19 pandemic has highlighted the need for a tool to speed up triage in ultrasound scans and provide clinicians with fast access to relevant information. To this end, we propose a new unsupervised reinforcement learning (RL) framework with novel rewards to facilitate unsupervised learning by avoiding tedious and impractical manual labelling for summarizing ultrasound videos. The proposed framework is capable of delivering video summaries with classification labels and segmentations of key landmarks which enhances its utility as a triage tool in the emergency department (ED) and for use in telemedicine. Using an attention ensemble of encoders, the high dimensional image is projected into a low dimensional latent space in terms of: a) reduced distance with a normal or abnormal class (classifier encoder), b) following a topology of landmarks (segmentation encoder), and c) the distance or topology agnostic latent representation (autoencoders). The summarization network is implemented using a bi-directional long short term memory (Bi-LSTM) which utilizes the latent space representation from the encoder. Validation is performed on lung ultrasound (LUS), that typically represent potential use cases in telemedicine and ED triage acquired from different medical centers across geographies (India and Spain). The proposed approach trained and tested on 126 LUS videos showed high agreement with the ground truth with an average precision of over 80% and average F1 score of well over 44 ±1.7 %. The approach resulted in an average reduction in storage space of 77% which can ease bandwidth and storage requirements in telemedicine.

Designing catalysts with deep generative models and computational data. A case study for Suzuki cross coupling reactions.

The need for more efficient catalytic processes is ever-growing, and so are the costs associated with experimentally searching chemical space to find new promising catalysts. Despite the consolidated use of density functional theory (DFT) and other atomistic models for virtually screening molecules based on their simulated performance, data-driven approaches are rising as indispensable tools for designing and improving catalytic processes. Here, we present a deep learning model capable of generating new catalyst-ligand candidates by self-learning meaningful structural features solely from their language representation and computed binding energies. We train a recurrent neural network-based Variational Autoencoder (VAE) to compress the molecular representation of the catalyst into a lower dimensional latent space, in which a feed-forward neural network predicts the corresponding binding energy to be used as the optimization function. The outcome of the optimization in the latent space is then reconstructed back into the original molecular representation. These trained models achieve state-of-the-art predictive performances in catalysts' binding energy prediction and catalysts' design, with a mean absolute error of 2.42 kcal mol-1 and an ability to generate 84% valid and novel catalysts.

Atrial Fibrillation Detection With Spectral Manifolds in Low-Dimensional Latent Spaces

Atrial Fibrillation Detection With Spectral Manifolds in Low-Dimensional Latent Spaces

Conditional Injective Flows for Bayesian Imaging

Most deep learning models for computational imaging regress a single reconstructed image. In practice, however, ill-posedness, nonlinearity, model mismatch, and noise often conspire to make such point estimates misleading or insufficient. The Bayesian approach models images and (noisy) measurements as jointly distributed random vectors and aims to approximate the posterior distribution of unknowns. Recent variational inference methods based on conditional normalizing flows are a promising alternative to traditional MCMC methods, but they come with drawbacks: excessive memory and compute demands for moderate to high resolution images and underwhelming performance on hard nonlinear problems. In this work, we propose C-Trumpets -- conditional injective flows specifically designed for imaging problems, which greatly diminish these challenges. Injectivity reduces memory footprint and training time while low-dimensional latent space together with architectural innovations like fixed-volume-change layers and skip-connection revnet layers, C-Trumpets outperform regular conditional flow models on a variety of imaging and image restoration tasks, including limited-view CT and nonlinear inverse scattering, with a lower compute and memory budget. C-Trumpets enable fast approximation of point estimates like MMSE or MAP as well as physically-meaningful uncertainty quantification.

6D Phase space diagnostics based on adaptively tuned physics-informed generative convolutional neural networks

A physics-informed generative convolutional neural network (CNN)-based 6D phase space diagnostic is presented which generates all 15 unique 2D projections (x, y), (x, y′),...,(z, E) of a charged particle beam’s 6D phase space (x, y, z, x′, y′, E). The CNN is trained by supervised learning over a wide range of input beam distributions, accelerator parameters, and the associated 6D beam phase spaces at multiple accelerator locations. The CNN is applied in an un-supervised adaptive manner without knowledge of the input beam distribution or accelerator parameters and is robust to their unknown time variation. Adaptive feedback automatically tunes the low-dimensional latent space of the encoder-decoder CNN to predict the 6D phase space based only on 2D (z, E) longitudinal phase space measurements from a device such as a transverse deflecting RF cavity (TCAV). This method has the potential to provide diagnostics beyond the existing state of the art at many accelerator facilities. Studies are presented for two very different accelerators: the 5-meter-long ultra-fast electron diffraction (UED) HiRES compact accelerator at LBNL and the kilometer long plasma wakefield accelerator FACET-II at SLAC.

EQUIVARIANT GEOMETRIC LEARNING FOR DIGITAL ROCK PHYSICS: ESTIMATING FORMATION FACTOR AND EFFECTIVE PERMEABILITY TENSORS FROM MORSE GRAPH

We present a SE(3)-equivariant graph neural network (GNN) approach that directly predicts the formation factor and effective permeability from micro-CT images. Fast Fourier Transform (FFT) solvers are established to compute both the formation factor and effective permeability, while the topology and geometry of the pore space are represented by a persistence-based Morse graph. Together, they constitute the database for training, validating, and testing the neural networks. While the graph and Euclidean convolutional approaches both employ neural networks to generate low-dimensional latent space to represent the features of the microstructures for forward predictions, the SE(3) equivariant neural network is found to generate more accurate predictions, especially when the training data are limited. Numerical experiments have also shown that the new SE(3) approach leads to predictions that fulfill the material frame indifference whereas the predictions from classical convolutional neural networks (CNNs) may suffer from spurious dependence on the coordinate system of the training data. Comparisons among predictions inferred from training the CNN and those from graph convolutional neural networks with and without the equivariant constraint indicate that the equivariant graph neural network seems to perform better than the CNN and GNN without enforcing equivariant constraints.

Cross-Modal Variational Auto-Encoder for Content-Based Micro-Video Background Music Recommendation

In this paper, we propose a cross-modal variational auto-encoder (CMVAE) for content-based micro-video background music recommendation. CMVAE is a hierarchical Bayesian generative model that matches relevant background music to a micro-video by projecting these two multimodal inputs into a shared low-dimensional latent space, where the alignment of two corresponding embeddings of a matched video-music pair is achieved by cross-generation. Moreover, the multimodal information is fused by the product-of-experts (PoE) principle, where the semantic information in visual and textual modalities of the micro-video are weighted according to their variance estimations such that the modality with a lower noise level is given more weights. Therefore, the micro-video latent variables contain less irrelevant information that results in a more robust model generalization. Furthermore, we establish a large-scale content-based micro-video background music recommendation dataset, TT-150k, composed of approximately 3,000 different background music clips associated to 150,000 micro-videos from different users. Extensive experiments on the established TT-150k dataset demonstrate the effectiveness of the proposed method. A qualitative assessment of CMVAE by visualizing some recommendation results is also included.

A deep learning-based direct forecasting of CO2 plume migration

Accurate and timely forecasts of CO2 plume evolution in geological reservoirs are crucial for CO2 migration detection, leakage risk assessment, and operation decision support. Conventional forecasting usually adopts a two-step strategy, first calibrating reservoir model parameters against observations using iterative inverse modeling (or history matching) and then applying the calibrated model for predictions. This method impedes real-time forecasts due to the heavy computational demand in inverse modeling and may suffer from poor prediction accuracy because of the limited observation data. In this work, we propose a deep learning-based latent space mapping framework to forecast CO2 plume migration directly by avoiding the inverse modeling. We first use the convolutional autoencoder to map the high-dimensional complex plume extents onto low-dimensional latent space. Next, we use neural networks to learn the relationship between the observation variables and the prediction latent variables. And then for given observation data, we infer the prediction values directly. This one-step direct forecasting is computationally efficient which requires a few number of parallelizable reservoir simulations and it can provide accurate predictions with limited observations by learning the observation-prediction relationship in the reduced dimension. Therefore, our proposed method enables an in-time forecast of dynamic CO2 plume distributions. We demonstrate the effectiveness and accuracy of our method in predicting the CO2 plume migration using four metrics such as plume area, centroid movement distance, and plume spreading in the primary and secondary directions. And the spatio-temporal evolution patterns of plume migration under diverse geological complexities are also accurately quantified.

Parsimonious Bayesian factor analysis for modelling latent structures in spectroscopy data

In recent years, within the dairy sector, animal diet and management practices have been receiving increased attention, in particular, examining the impact of pasture-based feeding strategies on the composition and quality of milk and dairy products in line with the prevalence of premium grass-fed dairy products appearing on market shelves. To date, methods to thoroughly investigate the more relevant differences induced by the diet on milk chemical features are limited; enhanced statistical tools exploring these differences are required. Infrared spectroscopy techniques are widely used to collect data on milk samples and to predict milk related traits and characteristics. While these data are routinely used to predict the composition of the macro components of milk, each spectrum also provides a reservoir of unharnessed information about the sample. The accumulation and subsequent interpretation of these data present some challenges due to their high-dimensionality and the relationships amongst the spectral variables. In this work, directly motivated by a dairy application, we propose a modification of the standard factor analysis to induce a parsimonious summary of spectroscopic data. Our proposal maps the observations into a low-dimensional latent space while simultaneously clustering the observed variables. The method indicates possible redundancies in the data, and it helps disentangle the complex relationships among the wavelengths. A flexible Bayesian estimation procedure is proposed for model fitting, providing reasonable values for the number of latent factors and clusters. The method is applied on milk mid-infrared (MIR) spectroscopy data from dairy cows on distinctly different pasture and nonpasture based diets, providing accurate modelling of the correlation, clustering of variables, and information on differences among milk samples from cows on different diets.

Latent space projection of brain FDG‐PET creates a powerful classifier for neurodegenerative diseases

AbstractBackgroundFluorodeoxyglucose positron emission tomography (FDG‐PET) is an established modality for differential diagnosis of dementia. Deriving phenotypic signatures from FDG‐PET via machine learning is challenging due to the high dimensional nature of brain images relative to the generally small number of labeled examples available for training, the class imbalance among those labels, and the cooccurrence of multiple pathologies. In this study, we developed a multi‐class, multi‐label framework to address these challenges.MethodA database of clinically acquired PET/CT images from 3,000 unique patients was used to develop a latent space model using matrix decomposition. This model was then applied to images from a separate cohort of Mayo Clinic Alzheimer’s Disease Research Center participants (n=1,745) labeled as cognitively unimpaired (CU) (n=1,436) or with the following potentially co‐occurring phenotypes: Alzheimer’s disease (AD) (n=165), Lewy body dementia (DLB) (n=92), behavioral variant frontotemporal dementia (bvFTD) (n=43), semantic (svPPA) (n=10) and logopenic (lvPPA) (n=13) variant PPA, and posterior cortical atrophy (PCA) (n=17). A k‐nearest neighbors classifier that is robust to these imbalanced and overlapping labels was then trained on these examples. The resulting classifier was evaluated by area under receiver‐operator characteristic curve (ROC‐AUC) via leave one out cross validation, using clinical diagnosis as the gold standard.ResultROC curves and AUC scores for each phenotype are illustrated in Fig. 1a. Because the classifier is based on a k‐nearest neighbors connectivity matrix, it has a convenient graphical representation, where images are nodes and edges are drawn between an image and its set of nearest neighbors in latent space. A self‐organizing force directed graph constructed in this way is illustrated in Fig 1b, highlighting the strong separation of CU and degenerative images, as well as the segregation of each phenotype within the neurodegenerative region of the graph.ConclusionIn this study, we developed a machine learning framework for classification of neurodegenerative disease based on k‐nearest neighbor analysis in a low dimensional latent space projection of FDG‐PET images. By leveraging low‐dimensional representations and k‐nearest neighbors analysis, this framework is robust in multi‐class, multi‐label tasks with strong class imbalance and provides a highly interpretable graphical representation.

Latent space projection of brain FDG‐PET creates a powerful classifier for neurodegenerative diseases

AbstractBackgroundFluorodeoxyglucose positron emission tomography (FDG‐PET) is an established modality for differential diagnosis of dementia. Deriving phenotypic signatures from FDG‐PET via machine learning is challenging due to the high dimensional nature of brain images relative to the generally small number of labeled examples available for training, the class imbalance among those labels, and the cooccurrence of multiple pathologies. In this study, we developed a multi‐class, multi‐label framework to address these challenges.MethodA database of clinically acquired PET/CT images from 3,000 unique patients was used to develop a latent space model using matrix decomposition. This model was then applied to images from a separate cohort of Mayo Clinic Alzheimer’s Disease Research Center participants (n=1,745) labeled as cognitively unimpaired (CU) (n=1,436) or with the following potentially co‐occurring phenotypes: Alzheimer’s disease (AD) (n=165), Lewy body dementia (DLB) (n=92), behavioral variant frontotemporal dementia (bvFTD) (n=43), semantic (svPPA) (n=10) and logopenic (lvPPA) (n=13) variant PPA, and posterior cortical atrophy (PCA) (n=17). A k‐nearest neighbors classifier that is robust to these imbalanced and overlapping labels was then trained on these examples. The resulting classifier was evaluated by area under receiver‐operator characteristic curve (ROC‐AUC) via leave one out cross validation, using clinical diagnosis as the gold standard.ResultROC curves and AUC scores for each phenotype are illustrated in Fig. 1a. Because the classifier is based on a k‐nearest neighbors connectivity matrix, it has a convenient graphical representation, where images are nodes and edges are drawn between an image and its set of nearest neighbors in latent space. A self‐organizing force directed graph constructed in this way is illustrated in Fig 1b, highlighting the strong separation of CU and degenerative images, as well as the segregation of each phenotype within the neurodegenerative region of the graph.ConclusionIn this study, we developed a machine learning framework for classification of neurodegenerative disease based on k‐nearest neighbor analysis in a low dimensional latent space projection of FDG‐PET images. By leveraging low‐dimensional representations and k‐nearest neighbors analysis, this framework is robust in multi‐class, multi‐label tasks with strong class imbalance and provides a highly interpretable graphical representation.

On the Dimensionality and Utility of Convolutional Autoencoder’s Latent Space Trained with Topology-Preserving Spectral EEG Head-Maps

Electroencephalography (EEG) signals can be analyzed in the temporal, spatial, or frequency domains. Noise and artifacts during the data acquisition phase contaminate these signals adding difficulties in their analysis. Techniques such as Independent Component Analysis (ICA) require human intervention to remove noise and artifacts. Autoencoders have automatized artifact detection and removal by representing inputs in a lower dimensional latent space. However, little research is devoted to understanding the minimum dimension of such latent space that allows meaningful input reconstruction. Person-specific convolutional autoencoders are designed by manipulating the size of their latent space. A sliding window technique with overlapping is employed to segment varied-sized windows. Five topographic head-maps are formed in the frequency domain for each window. The latent space of autoencoders is assessed using the input reconstruction capacity and classification utility. Findings indicate that the minimal latent space dimension is 25% of the size of the topographic maps for achieving maximum reconstruction capacity and maximizing classification accuracy, which is achieved with a window length of at least 1 s and a shift of 125 ms, using the 128 Hz sampling rate. This research contributes to the body of knowledge with an architectural pipeline for eliminating redundant EEG data while preserving relevant features with deep autoencoders.

Non-Newtonian fluid simulation and reconstruction from monocular videos

In fluid simulation, the parameters of the constitutive model are difficult to predict accurately, which leads to the inconsistency between the simulation results and the visual effects of monocular videos. To solve this problem, a monocular video-guided non-Newtonian fluid reconstruction method is proposed. The model takes non-Newtonian fluid simulation videos as input for training and learns the best low-dimensional latent space representation of a single frame of fluid simulation images. Inter-frame prediction is then performed in this latent domain, employing a convolutional long–short-term memory network to predict latent vector representations of future frames. Finally, the reconstruction parameters of the constitutive model are predicted based on the frame-by-frame latent representation encoding and inter-frame temporal features. In the model verification stage, the real video of the non-Newtonian fluid is used as the input to predict the parameters of the fluid constitutive model, and realize the simulation and reconstruction of the non-Newtonian fluid based on the Cross model. Experimental results showed that the video-guided simulation reconstruction method could obtain fluid flow phenomena that were more consistent with those in the real video than the reconstruction method based on rheometer measurements. Furthermore, it provided higher pixel accuracy and precision at different times and visual effects that were more consistent with the actual fluid flow.

On compression rate of quantum autoencoders: Control design, numerical and experimental realization

Quantum autoencoders which aim at compressing quantum information in a low-dimensional latent space lie in the heart of automatic data compression in the field of quantum information. In this paper, we establish an upper bound of the compression rate for a given quantum autoencoder and present a learning control approach for training the autoencoder to achieve the maximal compression rate. The upper bound of the compression rate is theoretically proven using eigen-decomposition and matrix differentiation, which is determined by the eigenvalues of the density matrix representation of the input states. Numerical results on 2-qubit and 3-qubit systems are presented to demonstrate how to train the quantum autoencoder to achieve the theoretically maximal compression, and the training performance using different machine learning algorithms is compared. Experimental results of a quantum autoencoder using quantum optical systems are illustrated for compressing two 2-qubit states into two 1-qubit states.