Latent Space Research Articles

Articulated Motion-Aware NeRF for 3D Dynamic Appearance and Geometry Reconstruction by Implicit Motion States.

We propose a self-supervised approach for 3D dynamic reconstruction of articulated motions based on Generative Adversarial Networks and Neural Radiance Fields. Our method reconstructs articulated objects and recover their continuous motions and attributes from an unordered, discontinuous image set. Notably, we treat motion states as time-independent, recognizing that articulated objects can exhibit identical motions at different times. The key insight of our approach utilizes generative adversarial networks to create a continuous implicit motion state space. Initially, we employ a motion network extracts discrete motion states from images as anchors. These anchors are then expanded across the latent space using generative adversarial networks. Subsequently, motion state latent codes are input into motion-aware neural radiance fields for dynamic appearance and geometry reconstruction. To deduce motion attributes from the continuously generated motions, we adopt a cluster-based strategy. We thoroughly evaluate and validate our method on both synthesized and real data, demonstrating superior fidelity in appearances, geometries, and motion attributes of articulated objects compared to state-of-the-art methods.

Dispelling the Fake: Social Bot Detection Based on Edge Confidence Evaluation.

Social bot detection is essential for maintaining the safety and integrity of online social networks (OSNs). Graph neural networks (GNNs) have emerged as a promising solution. Mainstream GNN-based social bot detection methods learn rich user representations by recursively performing message passing along user-user interaction edges, where users are treated as nodes and their relationships as edges. However, these methods face challenges when detecting advanced bots interacting with genuine accounts. Interaction with real accounts results in the graph structure containing camouflaged and unreliable edges. These unreliable edges interfere with the differentiation between bot and human representations, and the iterative graph encoding process amplifies this unreliability. In this article, we propose a social Bot detection method based on Edge Confidence Evaluation (BECE). Our model incorporates an edge confidence evaluation module that assesses the reliability of the edges and identifies the unreliable edges. Specifically, we design features for edges based on the representation of user nodes and introduce parameterized Gaussian distributions to map the edge embeddings into a latent semantic space. We optimize these embeddings by minimizing Kullback-Leibler (KL) divergence from the standard distribution and evaluate their confidence based on edge representation. Experimental results on three real-world datasets demonstrate that BECE is effective and superior in social bot detection. Additionally, experimental results on six widely used GNN architectures demonstrate that our proposed edge confidence evaluation module can be used as a plug-in to improve detection performance.

Dynamic spatial‐temporal network for traffic forecasting based on joint latent space representation

Dynamic spatial‐temporal network for traffic forecasting based on joint latent space representation

Expediting structure–property analyses using variational autoencoders with regression

We present a machine learning approach that expedites structure–property analysis in materials, bypassing traditional feature extraction and exploratory data analysis techniques. This objective is accomplished by employing a variational autoencoder (VAE) structure that is modified to include a regressor network for property prediction (VAE-Regression). This modification allows for direct linkage of imaged features and quantitative part properties within the VAE latent space. We first demonstrate our approach using 2D optical micrographs and corresponding four-point bend fatigue life data from laser beam powder bed fusion additively manufactured Ti-6Al-4V coupons. The VAE-Regression model extracts spatial features, predicts fatigue life, and identifies features of porosity defect governing fatigue behavior such as pore clusters, pores near sample edges, and jagged pore morphologies. These features corroborate fatigue literature on physics-based modeling and experimentation. We then demonstrate the versatility of our methodology using binder jet additively manufactured WC-Co coupons, where porosity and microstructural discontinuities are known to lower the three-point bend transverse rupture strength, but the interaction between the WC and Co are yet to be completely understood. We attempted to understand these interactions using our VAE-Regression architecture. Within our dataset, we show that coarser WC grains surrounded by larger Co pools indicate lower strength, while finer WC grains with smaller Co pools indicate higher strength. This machine learning approach using image-based data will likely prove to be critical in understanding and identifying structure–property relationships in new materials and manufacturing processes.

Improving Anticancer Drug Selection and Prioritization via Neural Learning to Rank.

Personalized cancer treatment requires a thorough understanding of complex interactions between drugs and cancer cell lines in varying genetic and molecular contexts. To address this, high-throughput screening has been used to generate large-scale drug response data, facilitating data-driven computational models. Such models can capture complex drug-cell line interactions across various contexts in a fully data-driven manner. However, accurately prioritizing the most effective drugs for each cell line still remains a significant challenge. To address this, we developed multiple neural ranking approaches that leverage large-scale drug response data across multiple cell lines from diverse cancer types. Unlike existing approaches that primarily utilize regression and classification techniques for drug response prediction, we formulated the objective of drug selection and prioritization as a drug ranking problem. In this work, we proposed multiple pairwise and listwise neural ranking methods that learn latent representations of drugs and cell lines and then use those representations to score drugs in each cell line via a learnable scoring function. Specifically, we developed neural pairwise and listwise ranking methods, Pair-PushC and List-One on top of the existing methods, pLETORg and ListNet, respectively. Additionally, we proposed a novel listwise ranking method, List-All, that focuses on all the effective drugs instead of the top effective drug, unlike List-One. We also provide an exhaustive empirical evaluation with state-of-the-art regression and ranking baselines on large-scale data sets across multiple experimental settings. Our results demonstrate that our proposed ranking methods mostly outperform the best baselines with significant improvements of as much as 25.6% in terms of selecting truly effective drugs within the top 20 predicted drugs (i.e., hit@20) across 50% test cell lines. Furthermore, our analyses suggest that the learned latent spaces from our proposed methods demonstrate informative clustering structures and capture relevant underlying biological features. Moreover, our comprehensive evaluation provides a thorough and objective comparison of the performance of different methods (including our proposed ones).

Real‐time topology optimization via learnable mappings

AbstractIn traditional topology optimization, the computing time required to iteratively update the material distribution within a design domain strongly depends on the complexity or size of the problem, limiting its application in real engineering contexts. This work proposes a multi‐stage machine learning strategy that aims to predict an optimal topology and the related stress fields of interest, either in 2D or 3D, without resorting to any iterative analysis and design process. The overall topology optimization is treated as regression task in a low‐dimensional latent space, that encodes the variability of the target designs. First, a fully‐connected model is employed to surrogate the functional link between the parametric input space characterizing the design problem and the latent space representation of the corresponding optimal topology. The decoder branch of an autoencoder is then exploited to reconstruct the desired optimal topology from its latent representation. The deep learning models are trained on a dataset generated through a standard method of topology optimization implementing the solid isotropic material with penalization, for varying boundary and loading conditions. The underlying hypothesis behind the proposed strategy is that optimal topologies share enough common patterns to be compressed into small latent space representations without significant information loss. Results relevant to a 2D Messerschmitt‐Bölkow‐Blohm beam and a 3D bridge case demonstrate the capabilities of the proposed framework to provide accurate optimal topology predictions in a fraction of a second.

Variational inference for the latent shrinkage position model

AbstractThe latent position model (LPM) is a popular method used in network data analysis where nodes are assumed to be positioned in a ‐dimensional latent space. The latent shrinkage position model (LSPM) is an extension of the LPM which automatically determines the number of effective dimensions of the latent space via a Bayesian nonparametric shrinkage prior. However, the LSPM's reliance on Markov chain Monte Carlo for inference, while rigorous, is computationally expensive, making it challenging to scale to networks with large numbers of nodes. We introduce a variational inference approach for the LSPM, aiming to reduce computational demands while retaining the model's ability to intrinsically determine the number of effective latent dimensions. The performance of the variational LSPM is illustrated through simulation studies and its application to real‐world network data. To promote wider adoption and ease of implementation, we also provide open‐source code.

Fuser: An enhanced multimodal fusion framework with congruent reinforced perceptron for hateful memes detection

As a multimodal form of hate speech on social media, hateful memes are more aggressive and cryptic threats to the real life of humans. Automatic detection of hateful memes is crucial, but the images and texts in most memes are only weakly consistent or even irrelevant. Although existing works have achieved the initial goal of detecting hateful memes with pre-trained models, they are limited to monolithic inference methods while ignoring the semantic differences between multimodal representations. To strengthen the comprehension and reasoning of the hidden meaning behind the memes by combining real-world knowledge, we propose an enhanced multimodal fusion framework with congruent reinforced perceptron for hateful memes detection. Inspired by the human cognitive mechanism, we first divide the extracted multisource representations into main semantics and auxiliary contexts based on their strength and relevance, and then precode them into lightly correlated embeddings with unified spatial dimensions via a novel prefix uniform layer, respectively. To jointly learn the intrinsic correlation between primary and secondary semantics, a congruent reinforced perceptron with brain-like perceptual integration is designed to seamlessly fuse multimodal representations in a shared latent space while maintaining the feature integrity in the sub-fusion space, thereby implicitly reasoning about the subtle metaphors behind the memes. Extensive experiments on four benchmark datasets fully demonstrate the effectiveness and superiority of our architecture compared with previous state-of-the-art methods.

Response Item Network (ResIN): A network-based approach to explore attitude systems

Belief network analysis (BNA) refers to a class of methods designed to detect and outline structural organizations of complex attitude systems. BNA can be used to analyze attitude-structures of abstract concepts such as ideologies, worldviews, and norm systems that inform how people perceive and navigate the world. The present manuscript presents a formal specification of the Response-Item Network (or ResIN), a new methodological approach that advances BNA in at least two important ways. First, ResIN allows for the detection of attitude asymmetries between different groups, improving the applicability and validity of BNA in research contexts that focus on intergroup differences and/or relationships. Second, ResIN’s networks include a spatial component that is directly connected to item response theory (IRT). This allows for access to latent space information in which each attitude (i.e. each response option across items in a survey) is positioned in relation to the core dimension(s) of group structure, revealing non-linearities and allowing for a more contextual and holistic interpretation of the attitudes network. To validate the effectiveness of ResIN, we develop a mathematical model and apply ResIN to both simulated and real data. Furthermore, we compare these results to existing methods of BNA and IRT. When used to analyze partisan belief-networks in the US-American political context, ResIN was able to reliably distinguish Democrat and Republican attitudes, even in highly asymmetrical attitude systems. These results demonstrate the utility of ResIN as a powerful tool for the analysis of complex attitude systems and contribute to the advancement of BNA.

Multi-plane denoising diffusion-based dimensionality expansion for 2D-to-3D reconstruction of microstructures with harmonized sampling

Acquiring reliable microstructure datasets is a pivotal step toward the systematic design of materials with the aid of integrated computational materials engineering (ICME) approaches. However, obtaining three-dimensional (3D) microstructure datasets is often challenging due to high experimental costs or technical limitations, while acquiring two-dimensional (2D) micrographs is comparatively easier. To deal with this issue, this study proposes a novel framework called ‘Micro3Diff’ for 2D-to-3D reconstruction of microstructures using diffusion-based generative models (DGMs). Specifically, this approach solely requires pre-trained DGMs for the generation of 2D samples, and dimensionality expansion (2D-to-3D) takes place only during the generation process (i.e., reverse diffusion process). The proposed framework incorporates a concept referred to as ‘multi-plane denoising diffusion’, which transforms noisy samples (i.e., latent variables) from different planes into the data structure while maintaining spatial connectivity in 3D space. Furthermore, a harmonized sampling process is developed to address possible deviations from the reverse Markov chain of DGMs during the dimensionality expansion. Combined, we demonstrate the feasibility of Micro3Diff in reconstructing 3D samples with connected slices that maintain morphologically equivalence to the original 2D images. To validate the performance of Micro3Diff, various types of microstructures (synthetic or experimentally observed) are reconstructed, and the quality of the generated samples is assessed both qualitatively and quantitatively. The successful reconstruction outcomes inspire the potential utilization of Micro3Diff in upcoming ICME applications while achieving a breakthrough in comprehending and manipulating the latent space of DGMs.

Pretraining of attention-based deep learning potential model for molecular simulation

Machine learning-assisted modeling of the inter-atomic potential energy surface (PES) is revolutionizing the field of molecular simulation. With the accumulation of high-quality electronic structure data, a model that can be pretrained on all available data and finetuned on downstream tasks with a small additional effort would bring the field to a new stage. Here we propose DPA-1, a Deep Potential model with a gated attention mechanism, which is highly effective for representing the conformation and chemical spaces of atomic systems and learning the PES. We tested DPA-1 on a number of systems and observed superior performance compared with existing benchmarks. When pretrained on large-scale datasets containing 56 elements, DPA-1 can be successfully applied to various downstream tasks with a great improvement of sample efficiency. Surprisingly, for different elements, the learned type embedding parameters form a spiral in the latent space and have a natural correspondence with their positions on the periodic table, showing interesting interpretability of the pretrained DPA-1 model.

Physics-informed generator-encoder adversarial networks with latent space matching for stochastic differential equations

We propose a new class of physics-informed neural networks, called Physics-Informed Generator-Encoder Adversarial Networks, to effectively address the challenges posed by forward, inverse, and mixed problems in stochastic differential equations (SDEs). In these scenarios, while governing equations are known, the available data consist of only a limited set of snapshots for system parameters. Our model consists of two key components: the generator and the encoder, both updated alternately by gradient descent. In contrast to previous approaches that directly match approximated solutions with real snapshots, we employ an indirect matching operating within the lower-dimensional latent feature space. This method circumvents challenges associated with high-dimensional inputs and complex data distributions in solving SDEs. Numerical experiments indicate that, compared to existing deep learning solvers, our proposed approach not only demonstrates superior accuracy but also exhibits advantages in both computational efficiency and model complexity.

Real-time prediction of fuel consumption and emissions based on deep autoencoding support vector regression for cylinder pressure-based feedback control of marine diesel engines

Predictive models serving as virtual sensors for online optimization and feedback control of diesel engines is gaining increasing attention. However, existing prediction models fall short in simultaneously achieving high prediction accuracy and fast computational speed. In this paper, a novel machine learning algorithm called deep autoencoding support vector regression (DASVR) was proposed, which combines the powerful non-linear feature extraction capability of artificial neural network (ANN) with the good adaptability of support vector regression (SVR) to low-dimensional input spaces. ANN-based autoencoder is firstly employed to extract features from the original high-dimensional input space, forming a low-dimensional latent variable space. SVR is then employed to perform non-linear mapping between latent variables and target output variables. Experiments were conducted on a 16-cylinder marine diesel engine under different load, rail pressure, and injection timing conditions. Prediction models for fuel consumption, NOx, PM, HC, and PM emissions were established based on experimental data and DASVR algorithm. The maximum error of DASVR-based models for the five desired output variables is less than 3.8 % under different operating conditions. DAVSR-based predictive models outperform conventional ANN-based and SVR-based models in terms of both prediction accuracy and real-time capability, and can theoretically meet the real-time requirement below 3000 r/min.

Dimensionality reduction beyond neural subspaces with slice tensor component analysis.

Recent work has argued that large-scale neural recordings are often well described by patterns of coactivation across neurons. Yet the view that neural variability is constrained to a fixed, low-dimensional subspace may overlook higher-dimensional structure, including stereotyped neural sequences or slowly evolving latent spaces. Here we argue that task-relevant variability in neural data can also cofluctuate over trials or time, defining distinct 'covariability classes' that may co-occur within the same dataset. To demix these covariability classes, we develop sliceTCA (slice tensor component analysis), a new unsupervised dimensionality reduction method for neural data tensors. In three example datasets, including motor cortical activity during a classic reaching task in primates and recent multiregion recordings in mice, we show that sliceTCA can capture more task-relevant structure in neural data using fewer components than traditional methods. Overall, our theoretical framework extends the classic view of low-dimensional population activity by incorporating additional classes of latent variables capturing higher-dimensional structure.

A domain knowledge-informed design space exploration methodology for mechanical layout design

Layout designs have a large number of design variables and various physical constraints. Conventional design exploration approaches are time-consuming and may require human intervention. A unique feature about physical layout designs is the availability of domain knowledge, which can be utilised to speed up the design process. In this paper, we propose a generative deep learning-based design space exploration (DSE) methodology that is capable of learning design constraints in the layout design without explicit supervision. Moreover, it can incorporate domain knowledge in the generated layouts, thereby speeding up the design process. This is realised by constructing a layout generation variational autoencoder (LGVAE) model, which uses a latent space as an interface to generate the layouts. By training the LGVAE model, significantly lower-dimensional representations can be learned compared to the original dimensionality of the design space. Therefore, the number of design variables is greatly reduced. We showcase the performance of the proposed DSE approach by solving the heat source layout design problem encountered in thermal management of chips. Experiments demonstrate that the LGVAE model is capable of generating compressed latent representations that capture the characteristics of the input samples, which makes the DSE cost-effective.

Unsupervised low-dose CT denoising using bidirectional contrastive network

Background and Objective:Low-dose computed tomography (LDCT) scans significantly reduce radiation exposure, but introduce higher levels of noise and artifacts that compromise image quality and diagnostic accuracy. Supervised learning methods have proven effective in denoising LDCT images, but are hampered by the need for large, paired datasets, which pose significant challenges in data acquisition. This study aims to develop a robust unsupervised LDCT denoising method that overcomes the reliance on paired LDCT and normal-dose CT (NDCT) samples, paving the way for more accessible and practical denoising techniques. Methods:We propose a novel unsupervised network model, Bidirectional Contrastive Unsupervised Denoising (BCUD), for LDCT denoising. This model innovatively combines a bidirectional network structure with contrastive learning theory to map the precise mutual correspondence between the noisy LDCT image domain and the clean NDCT image domain. Specifically, we employ dual encoders and discriminators for domain-specific data generation, and use unique projection heads for each domain to adaptively learn customized embedded representations. We then align corresponding features across domains within the learned embedding spaces to achieve effective noise reduction. This approach fundamentally improves the model’s ability to match features in latent space, thereby improving noise reduction while preserving fine image detail. Results:Through extensive experimental validation on the AAPM-Mayo public dataset and real-world clinical datasets, the proposed BCUD method demonstrated superior performance. It achieved a peak signal-to-noise ratio (PSNR) of 31.387 dB, a structural similarity index measure (SSIM) of 0.886, an information fidelity criterion (IFC) of 2.305, and a visual information fidelity (VIF) of 0.373. Notably, subjective evaluation by radiologists resulted in a mean score of 4.23, highlighting its advantages over existing methods in terms of clinical applicability. Conclusions:This paper presents an innovative unsupervised LDCT denoising method using a bidirectional contrastive network, which greatly improves clinical applicability by eliminating the need for perfectly matched image pairs. The method sets a new benchmark in unsupervised LDCT image denoising, excelling in noise reduction and preservation of fine structural details.

Unsupervised deep-learning framework for 5D seismic denoising and interpolation

We develop an unsupervised framework to reconstruct missing data from noisy and incomplete five-dimensional (5D) seismic data. Our method comprises two main components: a deep-learning network and the projection onto the convex sets (POCS) method. The model works iteratively, passing the data between the two components and splitting the data into a group of patches. The developed deep-learning model is inspired by the transformer architecture, wherein the extracted patches are flattened, and an embedded layer (fully connected layer) is used to transform the 1D flattened vector into the feature latent space. Afterward, an attention network is used to highlight the important information within these features, which improves the denoising performance of the deep-learning model. Furthermore, we use several skip connections between the fully connected layers to enhance the learning capability of the network. Next, the output of the POCS algorithm is considered the input of the deep-learning model for the following iteration. The developed model iteratively works in an unsupervised scheme wherein labeled data are not required. A performance comparison with benchmark methods using several synthetic and field examples shows that our method outperforms the traditional methods.

Measurability of quality characteristics identified in latent spaces of Generative AI Models

Deep Learning can learn complex properties from image datasets, which are difficult to model with traditional machine vision algorithms, inherently in the form of disentangled latent spaces. With latent spaces of Generative AI models, a feature extraction method to access these properties can be implemented. This work evaluates whether the learned properties can be measured in the latent space. Quantity and quantity-value scale properties and the measurability of the dimensional quality characteristic ‘filling degree’ using a linear calibration function are demonstrated for an industrial machine vision application. An uncertainty indicator between 0.4–0.9 mm is estimated for the latent space measurements.

Self-supervised graph representations with generative adversarial learning

Graph Neural Networks (GNNs) are powerful neural models for representation learning on graphs. In this work we study a self-supervised learning GNN method GraphSSGAN (Graph Self-Supervised Generative Adversarial Network) that learns generalized node representations using only unlabeled data. The core idea lies in learning representations that capture both the local information of the links and the global information of the plausibility of the subgraph samples. Specifically, node features of the original graph are first encoded into latent representations by optimizing a link prediction objective. Then the generator converts noise vectors to node representations in the latent space and predicts the link probabilities from the generated node representations. The discriminator leverages the graph convolutional network (GCN) architecture to produce permutation-invariant graph-level embeddings, and the intermediate node representations are used by simple classifiers in the downstream tasks. In addition, we introduce several technical tricks including Gumbel-Top-k trick, Gumbel-Softmax trick and mini-batch training via subgraph sampling to improve the training process. Through extensive experiments on node classification and link prediction tasks, we demonstrate the effectiveness of the proposed model and the contribution of GAN framework on graph representation learning.

Talk-to-Edit: Fine-Grained 2D and 3D Facial Editing via Dialog.

Facial editing is to manipulate the facial attributes of a given face image. Nowadays, with the development of generative models, users can easily generate 2D and 3D facial images with high fidelity and 3D-aware consistency. However, existing works are incapable of delivering a continuous and fine-grained editing mode (e.g., editing a slightly smiling face to a big laughing one) with natural interactions with users. In this work, we propose Talk-to-Edit, an interactive facial editing framework that performs fine-grained attribute manipulation through dialog between the user and the system. Our key insight is to model a continual "semantic field" in the GAN latent space. 1) Unlike previous works that regard the editing as traversing straight lines in the latent space, here the fine-grained editing is formulated as finding a curving trajectory that respects fine-grained attribute landscape on the semantic field. 2) The curvature at each step is location-specific and determined by the input image as well as the users' language requests. 3) To engage the users in a meaningful dialog, our system generates language feedback by considering both the user request and the current state of the semantic field. We demonstrate the effectiveness of our proposed framework on both 2D and 3D-aware generative models. We term the semantic field for the 3D-aware models as "tri-plane" flow, as it corresponds to the changes not only in the color space but also in the density space. We also contribute CelebA-Dialog, a visual-language facial editing dataset to facilitate large-scale study. Specifically, each image has manually annotated fine-grained attribute annotations as well as template-based textual descriptions in natural language. Extensive quantitative and qualitative experiments demonstrate the superiority of our framework in terms of 1) the smoothness of fine-grained editing, 2) the identity/attribute preservation, and 3) the visual photorealism and dialog fluency. Notably, the user study validates that our overall system is consistently favored by around 80% of the participants.