Low-dimensional Latent Space Research Articles

An encoding generative modeling approach to dimension reduction and covariate adjustment in causal inference with observational studies

In this article, we develop CausalEGM, a deep learning framework for nonlinear dimension reduction and generative modeling of the dependency among covariate features affecting treatment and response. CausalEGM can be used for estimating causal effects in both binary and continuous treatment settings. By learning a bidirectional transformation between the high-dimensional covariate space and a low-dimensional latent space and then modeling the dependencies of different subsets of the latent variables on the treatment and response, CausalEGM can extract the latent covariate features that affect both treatment and response. By conditioning on these features, one can mitigate the confounding effect of the high dimensional covariate on the estimation of the causal relation between treatment and response. In a series of experiments, the proposed method is shown to achieve superior performance over existing methods in both binary and continuous treatment settings. The improvement is substantial when the sample size is large and the covariate is of high dimension. Finally, we established excess risk bounds and consistency results for our method, and discuss how our approach is related to and improves upon other dimension reduction approaches in causal inference.

Deep Batch Integration and Denoise of Single-Cell RNA-Seq Data.

Numerous single-cell transcriptomic datasets from identical tissues or cell lines are generated from different laboratories or single-cell RNA sequencing (scRNA-seq) protocols. The denoising of these datasets to eliminate batch effects is crucial for data integration, ensuring accurate interpretation and comprehensive analysis of biological questions. Although many scRNA-seq data integration methods exist, most are inefficient and/or not conducive to downstream analysis. Here, DeepBID, a novel deep learning-based method for batch effect correction, non-linear dimensionality reduction, embedding, and cell clustering concurrently, is introduced. DeepBID utilizes a negative binomial-based autoencoder with dual Kullback-Leibler divergence loss functions, aligning cell points from different batches within a consistent low-dimensional latent space and progressively mitigating batch effects through iterative clustering. Extensive validation on multiple-batch scRNA-seq datasets demonstrates that DeepBID surpasses existing tools in removing batch effects and achieving superior clustering accuracy. When integrating multiple scRNA-seq datasets from patients with Alzheimer's disease, DeepBID significantly improves cell clustering, effectively annotating unidentified cells, and detecting cell-specific differentially expressed genes.

Real-time prediction of fuel consumption and emissions based on deep autoencoding support vector regression for cylinder pressure-based feedback control of marine diesel engines

Predictive models serving as virtual sensors for online optimization and feedback control of diesel engines is gaining increasing attention. However, existing prediction models fall short in simultaneously achieving high prediction accuracy and fast computational speed. In this paper, a novel machine learning algorithm called deep autoencoding support vector regression (DASVR) was proposed, which combines the powerful non-linear feature extraction capability of artificial neural network (ANN) with the good adaptability of support vector regression (SVR) to low-dimensional input spaces. ANN-based autoencoder is firstly employed to extract features from the original high-dimensional input space, forming a low-dimensional latent variable space. SVR is then employed to perform non-linear mapping between latent variables and target output variables. Experiments were conducted on a 16-cylinder marine diesel engine under different load, rail pressure, and injection timing conditions. Prediction models for fuel consumption, NOx, PM, HC, and PM emissions were established based on experimental data and DASVR algorithm. The maximum error of DASVR-based models for the five desired output variables is less than 3.8 % under different operating conditions. DAVSR-based predictive models outperform conventional ANN-based and SVR-based models in terms of both prediction accuracy and real-time capability, and can theoretically meet the real-time requirement below 3000 r/min.

Interpretable Latent Space for Meteorological Out-of-Distribution Detection via Weak Supervision

Deep neural networks (DNNs) are effective tools for learning-enabled cyber-physical systems (CPSs) that handle high-dimensional image data. However, DNNs may make incorrect decisions when presented with inputs outside the distribution of their training data. These inputs can compromise the safety of CPSs. So, it becomes crucial to detect inputs as out-of-distribution (OOD) and interpret the reasons for their classification as OOD. In this study, we propose an interpretable learning method to detect OOD caused by meteorological features like darkness, lightness, and rain. To achieve this, we employ a variational autoencoder (VAE) to map high-dimensional image data to a lower-dimensional latent space. We then focus on a specific latent dimension and encourage it to classify different intensities of a particular meteorological feature in a monotonically increasing manner. This is accomplished by incorporating two additional terms into the VAE’s loss function: a classification loss and a positional loss. During training, we optimize the utilization of label information for classification. Remarkably, our results demonstrate that using only 25% of the training data labels is sufficient to train a single pre-selected latent dimension to classify different intensities of a specific meteorological feature. We evaluate the proposed method on two distinct datasets, CARLA and Duckietown, employing two different rain-generation methods. We show that our approach outperforms existing approaches by at least 15 in the F1 score and precision when trained and tested on CARLA dataset.

Bayesian inference of structured latent spaces from neural population activity with the orthogonal stochastic linear mixing model.

The brain produces diverse functions, from perceiving sounds to producing arm reaches, through the collective activity of populations of many neurons. Determining if and how the features of these exogenous variables (e.g., sound frequency, reach angle) are reflected in population neural activity is important for understanding how the brain operates. Often, high-dimensional neural population activity is confined to low-dimensional latent spaces. However, many current methods fail to extract latent spaces that are clearly structured by exogenous variables. This has contributed to a debate about whether or not brains should be thought of as dynamical systems or representational systems. Here, we developed a new latent process Bayesian regression framework, the orthogonal stochastic linear mixing model (OSLMM) which introduces an orthogonality constraint amongst time-varying mixture coefficients, and provide Markov chain Monte Carlo inference procedures. We demonstrate superior performance of OSLMM on latent trajectory recovery in synthetic experiments and show superior computational efficiency and prediction performance on several real-world benchmark data sets. We primarily focus on demonstrating the utility of OSLMM in two neural data sets: μECoG recordings from rat auditory cortex during presentation of pure tones and multi-single unit recordings form monkey motor cortex during complex arm reaching. We show that OSLMM achieves superior or comparable predictive accuracy of neural data and decoding of external variables (e.g., reach velocity). Most importantly, in both experimental contexts, we demonstrate that OSLMM latent trajectories directly reflect features of the sounds and reaches, demonstrating that neural dynamics are structured by neural representations. Together, these results demonstrate that OSLMM will be useful for the analysis of diverse, large-scale biological time-series datasets.

Encoding the space of protein-protein binding interfaces by artificial intelligence

The physical interactions between proteins are largely determined by the structural properties at their binding interfaces. It was found that the binding interfaces in distinctive protein complexes are highly similar. The structural properties underlying different binding interfaces could be further captured by artificial intelligence. In order to test this hypothesis, we broke protein-protein binding interfaces into pairs of interacting fragments. We employed a generative model to encode these interface fragment pairs in a low-dimensional latent space. After training, new conformations of interface fragment pairs were generated. We found that, by only using a small number of interface fragment pairs that were generated by artificial intelligence, we were able to guide the assembly of protein complexes into their native conformations. These results demonstrate that the conformational space of fragment pairs at protein-protein binding interfaces is highly degenerate. Features in this degenerate space can be well characterized by artificial intelligence. In summary, our machine learning method will be potentially useful to search for and predict the conformations of unknown protein-protein interactions.

Graph learning for particle accelerator operations.

Particle accelerators play a crucial role in scientific research, enabling the study of fundamental physics and materials science, as well as having important medical applications. This study proposes a novel graph learning approach to classify operational beamline configurations as good or bad. By considering the relationships among beamline elements, we transform data from components into a heterogeneous graph. We propose to learn from historical, unlabeled data via our self-supervised training strategy along with fine-tuning on a smaller, labeled dataset. Additionally, we extract a low-dimensional representation from each configuration that can be visualized in two dimensions. Leveraging our ability for classification, we map out regions of the low-dimensional latent space characterized by good and bad configurations, which in turn can provide valuable feedback to operators. This research demonstrates a paradigm shift in how complex, many-dimensional data from beamlines can be analyzed and leveraged for accelerator operations.

Bayesian imaging inverse problem with SA-Roundtrip prior via HMC-pCN sampler

Bayesian inference with deep generative prior has received considerable interest for solving imaging inverse problems in many scientific and engineering fields. The selection of the prior distribution is learned from, and therefore an important representation learning of, available prior measurements. The SA-Roundtrip, a novel deep generative prior, is introduced to enable controlled sampling generation and identify the data's intrinsic dimension. This prior incorporates a self-attention structure within a bidirectional generative adversarial network. Subsequently, Bayesian inference is applied to the posterior distribution in the low-dimensional latent space using the Hamiltonian Monte Carlo with preconditioned Crank-Nicolson (HMC-pCN) algorithm, which is proven to be ergodic under specific conditions. Experiments conducted on computed tomography (CT) reconstruction with the MNIST and TomoPhantom datasets reveal that the proposed method outperforms state-of-the-art comparisons, consistently yielding a robust and superior point estimator along with precise uncertainty quantification.

A Novel Obstacle Traversal Method for Multiple Robotic Fish Based on Cross-Modal Variational Autoencoders and Imitation Learning.

Precision control of multiple robotic fish visual navigation in complex underwater environments has long been a challenging issue in the field of underwater robotics. To address this problem, this paper proposes a multi-robot fish obstacle traversal technique based on the combination of cross-modal variational autoencoder (CM-VAE) and imitation learning. Firstly, the overall framework of the robotic fish control system is introduced, where the first-person view of the robotic fish is encoded into a low-dimensional latent space using CM-VAE, and then different latent features in the space are mapped to the velocity commands of the robotic fish through imitation learning. Finally, to validate the effectiveness of the proposed method, experiments are conducted on linear, S-shaped, and circular gate frame trajectories with both single and multiple robotic fish. Analysis reveals that the visual navigation method proposed in this paper can stably traverse various types of gate frame trajectories. Compared to end-to-end learning and purely unsupervised image reconstruction, the proposed control strategy demonstrates superior performance, offering a new solution for the intelligent navigation of robotic fish in complex environments.

An encoder-decoder ConvLSTM surrogate model for simulating geological CO2 sequestration with dynamic well controls

In Geological Carbon Sequestration (GCS), effectively managing the project requires predicting state variables such as pressure and saturation. However, numerical simulation of multiphase flow in subsurface porous media involves solving large linear algebra systems, resulting in a substantial computational burden. This can make it impractical for real-time history matching or optimization. Meanwhile, deep learning-based surrogate models are emerging as fast and accurate approximators. This study delves into the application of the Encoder-Decoder Convolutional Long Short-Term Memory (ED-ConvLSTM) neural network for predicting the complex evolution of state variables under dynamic CO2 injection schemes. Inception blocks enhanced with light-weighted attention modules are introduced in the encoder to extract high-dimensional input features. ConvLSTM is employed to propagate spatial temporal information in the low-dimensional latent space. Further, progressive upsampling blocks are used to reconstruct the latent features for the desired output. Instead of taking discrete time steps as an input feature, the proposed network captures the dynamic dependencies with the inherent ConvLSTM cell. The network has access to data at only portion of the initial time steps during training stage, while it is used to predict the state variables at unseen time steps during testing stage. Results show that the network can produce excellent predictions for both pressure and saturation, even at unseen future time steps. The remarkable generalizability to different geological permeability fields is also evaluated. ED-ConvLSTM outperforms the standard U-Net by far, especially when predicting beyond the training time period. These numerical experiments demonstrate the advantages of ED-ConvLSTM in terms of prediction accuracy, extrapolability and generalizability. This study highlights the importance of incorporating recurrent connections into the deep neural networks for simulating time-dependent multiphase flow problems. The proposed methodology has great potential in GCS surrogate modeling and offers a possible approach for real-time optimization of CO2 injection.

An accurate and efficient deep learning model for spatio-temporal prediction of interfacial flows

Aiming at fast and accurate spatiotemporal prediction of interfacial flow fields, a novel deep learning model combining Convolutional AutoEncoder (CAE) and long short-term memory with spatial and temporal attention (LSTM-STA) is proposed and named as CAE-LSTM-STA (hybrid model of CAE and LSTM-STA) in this article. To enable a fast calculation process, CAE is first utilized to compress the high-dimensional snapshots of flow fields into the low-dimensional latent space by its encoder. Then, the latent space serves as the input feature for LSTM-STA to temporally evolve the state of the low-dimensional latent space. Finally, the low-dimensional latent space at future time steps is fed back into the decoder of CAE to derive the full-order snapshots of the flow fields. The prediction performance of the proposed model is evaluated via two representative benchmark cases, including the dam break case and the rising bubble case. The capability of CAE in dimension reduction is found to be much better than that of the widely used Proper Orthogonal Decomposition, while LSTM-STA is observed that outperforms the original LSTM in multivariant temporal prediction. These promising results indicate that the proposed CAE-LSTM-STA model is able to effectively capture and advance the spatiotemporal characteristics of the interfacial flow fields, therefore making it an advanced surrogate model for fast and precise generation of the temporally continuous interfacial flow fields.

Spatio-temporal water height prediction for dam break flows using deep learning

Spatio-temporal prediction of the water height for dam break flow is of great significance in flood control projects. In this article, we propose a novel deep learning model named AE-LSTM-ATT that combines AutoEncoder (AE) and Long Short-Term Memory with Attention (LSTM-ATT) to predict the dynamic behavior of the dam-break water height field in spatial and temporal dimensions. The prediction performance of the AE-LSTM-ATT model was demonstrated using dam-break flow cases with different initial heights of the water column. The AE first compresses high-dimensional inputs into low-dimensional latent space using its encoder to enable a fast computation process. Then, the latent space is adopted as an input for the LSTM-ATT to predict low-dimensional representations at future time instances that can be restored to the water height field by the decoder of the AE. The major findings of this article include three parts: (1) AE-LSTM-ATT model can successfully achieve the spatio-temporal prediction of the water height for dam break flows with high precision; (2) The proposed AE presents much better performance than those of the widely used Principal Component Analysis (PCA) and Kernel Principal Component Analysis (KPCA) in the dimension reduction; (3) The developed LSTM-ATT performs far better than the commonly used Recurrent Neural Network (RNN) and Long Short-Term Memory (LSTM) in the time-series prediction. These promising results suggest that the proposed AE-LSTM-ATT model can play an essential role in capturing and advancing the spatio-temporal features of dam-break water height, which can be applied in the field of real-time decision making for dam-break floods.

Towards Detailed Text-to-Motion Synthesis via Basic-to-Advanced Hierarchical Diffusion Model

Text-guided motion synthesis aims to generate 3D human motion that not only precisely reflects the textual description but reveals the motion details as much as possible. Pioneering methods explore the diffusion model for text-to-motion synthesis and obtain significant superiority. However, these methods conduct diffusion processes either on the raw data distribution or the low-dimensional latent space, which typically suffer from the problem of modality inconsistency or detail-scarce. To tackle this problem, we propose a novel Basic-to-Advanced Hierarchical Diffusion Model, named B2A-HDM, to collaboratively exploit low-dimensional and high-dimensional diffusion models for high quality detailed motion synthesis. Specifically, the basic diffusion model in low-dimensional latent space provides the intermediate denoising result that to be consistent with the textual description, while the advanced diffusion model in high-dimensional latent space focuses on the following detail-enhancing denoising process. Besides, we introduce a multi-denoiser framework for the advanced diffusion model to ease the learning of high-dimensional model and fully explore the generative potential of the diffusion model. Quantitative and qualitative experiment results on two text-to-motion benchmarks (HumanML3D and KIT-ML) demonstrate that B2A-HDM can outperform existing state-of-the-art methods in terms of fidelity, modality consistency, and diversity.

Enhancing Hyperspectral Images via Diffusion Model and Group-Autoencoder Super-resolution Network

Existing hyperspectral image (HSI) super-resolution (SR) methods struggle to effectively capture the complex spectral-spatial relationships and low-level details, while diffusion models represent a promising generative model known for their exceptional performance in modeling complex relations and learning high and low-level visual features. The direct application of diffusion models to HSI SR is hampered by challenges such as difficulties in model convergence and protracted inference time. In this work, we introduce a novel Group-Autoencoder (GAE) framework that synergistically combines with the diffusion model to construct a highly effective HSI SR model (DMGASR). Our proposed GAE framework encodes high-dimensional HSI data into low-dimensional latent space where the diffusion model works, thereby alleviating the difficulty of training the diffusion model while maintaining band correlation and considerably reducing inference time. Experimental results on both natural and remote sensing hyperspectral datasets demonstrate that the proposed method is superior to other state-of-the-art methods both visually and metrically.

Utilizing Latent Diffusion Model to Accelerate Sampling Speed and Enhance Text Generation Quality

Diffusion models have achieved tremendous success in modeling continuous data modalities, such as images, audio, and video, yet their application in discrete data domains (e.g., natural language) has been limited. Existing methods primarily represent discrete text in a continuous diffusion space, incurring significant computational overhead during training and resulting in slow sampling speeds. This paper introduces LaDiffuSeq, a latent diffusion-based text generation model incorporating an encoder–decoder structure. Specifically, it first employs a pretrained encoder to map sequences composed of attributes and corresponding text into a low-dimensional latent vector space. Then, without the guidance of a classifier, it performs the diffusion process for the sequence’s corresponding latent space. Finally, a pretrained decoder is used to decode the newly generated latent vectors, producing target texts that are relevant to themes and possess multiple emotional granularities. Compared to the benchmark model, DiffuSeq, this model achieves BERTScore improvements of 0.105 and 0.009 on two public real-world datasets (ChnSentiCorp and a debate dataset), respectively; perplexity falls by 3.333 and 4.562; and it effectively quadruples the text generation sampling speed.

Image-based prediction and optimization of hysteresis properties of nanocrystalline permanent magnets using deep learning

We demonstrate the use of model order reduction and neural networks for estimating the hysteresis properties of nanocrystalline permanent magnets from microstructure. With a data-driven approach, we learn the demagnetization curve from data-sets created by grain growth and micromagnetic simulations. We show that the granular structure of a magnet can be encoded within a low-dimensional latent space. Latent codes are constructed using a variational autoencoder. The mapping of structure code to hysteresis properties is a multi-target regression problem. We apply deep neural network and use parameter sharing, in order to predict anchor points along the demagnetization curves from the magnet’s structure code. We present a proof of concept for microstructure design of nanocrystalline Nd2Fe14B permanent magnets using image-based machine learning. We show how new grain structures can be generated by interpolation between two points in the latent space and demonstrate inverse design. We show how to compute the granular microstructure that will give a demagnetization curve with a given remanence, squareness, and coercivity through optimization in latent space.

A novel hybrid machine learning model for rapid assessment of wave and storm surge responses over an extended coastal region

Storm surge and waves are responsible for a substantial portion of tropical and extratropical cyclones-related damages. While high-fidelity numerical models have significantly advanced the simulation accuracy of storm surge and waves, they are not practical to be employed for probabilistic analysis, risk assessment or rapid prediction due to their high computational demands. In this study, a novel hybrid model combining dimensionality reduction and data-driven techniques is developed for rapid assessment of waves and storm surge responses over an extended coastal region. Specifically, the hybrid model simultaneously identifies a low-dimensional representation of the high-dimensional spatial system based on a deep autoencoder (DAE) while mapping the storm parameters to the obtained low-dimensional latent space using a deep neural network (DNN). To train the hybrid model, a combined weighted loss function is designed to encourage a balance between DAE and DNN training and achieve the best accuracy. The performance of the hybrid model is evaluated through a case study using the synthetic data from the North Atlantic Comprehensive Coastal Study (NACCS) covering critical regions within New York and New Jersey. In addition, the proposed approach is compared with two decoupled models where the regression model is based on DNN and the reduction techniques are either principal component analysis (PCA) or DAE which are trained separately from the DNN model. High accuracy and computational efficiency are observed for the hybrid model which could be readily implemented as part of early warning systems or probabilistic risk assessment of waves and storm surge.

Intelligent detection of foreign objects over coal flow based on improved GANomaly

The mining belt conveyor is one of the most important modules in coal mine, whose safety always be threatened by the foreign objects. Although the traditional target detection methods achieve promising results in various computer vision tasks, the performance heavily depends on sufficient labelled data. However, in real-world production scenario, it is difficult to acquire huge number of images with foreign objects. The obtained datasets lacking of capacity and diversity are not suitable for training supervised learning-based foreign objects detection models. To address this concern, we propose a novel method for detecting the foreign objects on the surface of underground coal conveyor belt via improved GANomaly. The proposed foreign objects detection method employs generative adversarial networks (GAN) with attention gate to capture the distribution of normality in both high-dimensional image space and low-dimensional latent vector space. Only the normal images without foreign object are utilized to adversarially train the proposed network, including a U-shape generator to reconstruct the input image and a discriminator to classify real images from reconstructed ones. Then the combination of the difference between the input and generated images as well as the difference between latent representations are utilized as the anomaly score to evaluate whether the input image contain foreign objects. Experimental results over 707 images from real-world industrial scenarios demonstrate that the proposed method achieves an area under the receiver operating characteristic curve of 0.864 and is superior to the previous GAN-based anomaly detection methods.

Latent space representation method for structural acoustic assessments

When targeting structural acoustic objectives, engineering practitioners face epistemic uncertainties in the selection of optimal geometries and material distributions, particularly during early stages of the design process. Models built for simulating acoustic phenomena generally produce vector-valued output quantities of interest, such as autospectral density and frequency response functions. Given finite compute resources and time we seek computationally parsimonious ways to distill meaningful design information into actionable results from a limited set of model runs, and thus aim to use machine learning to perform model order reduction. Unlike time series data for which recurrent neural networks can learn from prior time steps to inform subsequent steps, frequency-dependent data demands a different machine learning paradigm. We thus evaluate the utility of autoencoders to represent structural acoustic results with a low dimensional latent space to enable such objectives as surrogate modeling for design optimization. We demonstrate the accuracy of autoencoder based methods of constructing a manifold representation for frequency dependent functions of varying modal density and damping, and discuss the predictive capability thereof.

Toward Deep Generation of Guided Wave Representations for Composite Materials

Laminated composite materials are widely used in most fields of engineering. Wave propagation analysis plays an essential role in understanding the short-duration transient response of composite structures. The forward physics-based models are utilized to map from elastic properties space to wave propagation behavior in a laminated composite material. Due to the high-frequency, multi-modal, and dispersive nature of the guided waves, the physics-based simulations are computationally demanding. It makes property prediction, generation, and material design problems more challenging. In this work, a forward physics-based simulator such as the stiffness matrix method is utilized to collect group velocities of guided waves for a set of composite materials. A variational autoencoder (VAE)-based deep generative model is proposed for the generation of new and realistic polar group velocity representations. It is observed that the deep generator is able to reconstruct unseen representations with very low mean square reconstruction error. Global Monte Carlo and directional equally-spaced samplers are used to sample the continuous, complete and organized low-dimensional latent space of VAE. The sampled point is fed into the trained decoder to generate new polar representations. The network has shown exceptional generation capabilities. It is also seen that the latent space forms a conceptual space where different directions and regions show inherent patterns related to the generated representations and their corresponding material properties.