Latent Space Representation Research Articles

Data-driven model for divertor plasma detachment prediction

We present a fast and accurate data-driven surrogate model for divertor plasma detachment prediction leveraging the latent feature space concept in machine learning research. Our approach involves constructing and training two neural networks: an autoencoder that finds a proper latent space representation (LSR) of plasma state by compressing the multi-modal diagnostic measurements and a forward model using multi-layer perception (MLP) that projects a set of plasma control parameters to its corresponding LSR. By combining the forward model and the decoder network from autoencoder, this new data-driven surrogate model is able to predict a consistent set of diagnostic measurements based on a few plasma control parameters. In order to ensure that the crucial detachment physics is correctly captured, highly efficient 1D UEDGE model is used to generate training and validation data in this study. The benchmark between the data-driven surrogate model and UEDGE simulations shows that our surrogate model is capable of providing accurate detachment prediction (usually within a few per cent relative error margin) but with at least four orders of magnitude speed-up, indicating that performance-wise, it has the potential to facilitate integrated tokamak design and plasma control. Comparing with the widely used two-point model and/or two-point model formatting, the new data-driven model features additional detachment front prediction and can be easily extended to incorporate richer physics. This study demonstrates that the complicated divertor and scrape-off-layer plasma state has a low-dimensional representation in latent space. Understanding plasma dynamics in latent space and utilising this knowledge could open a new path for plasma control in magnetic fusion energy research.

Self-supervised anomaly detection for new physics

We investigate a method of model-agnostic anomaly detection through studying jets, collimated sprays of particles produced in high-energy collisions. We train a transformer neural network to encode simulated QCD ``event space'' dijets into a low-dimensional ``latent space'' representation. We optimize the network using the self-supervised contrastive loss, which encourages the preservation of known physical symmetries of the dijets. We then train a binary classifier to discriminate a beyond the standard model resonant dijet signal from a QCD dijet background both in the event space and the latent space representations. We find the classifier performances on the event and latent spaces to be comparable. We finally perform an anomaly detection search using a weakly supervised bump hunt on the latent space dijets, finding again a comparable performance to a search run on the physical space dijets. This opens the door to using low-dimensional latent representations as a computationally efficient space for resonant anomaly detection in generic particle collision events.

Augmented Adversarial Learning for Human Activity Recognition with Partial Sensor Sets

Human activity recognition (HAR) plays an important role in a wide range of applications, such as health monitoring and gaming. Inertial sensors attached to body segments constitute a critical sensing system for HAR. Diverse inertial sensor datasets for HAR have been released with the intention of attracting collective efforts and saving the data collection burden. However, these datasets are heterogeneous in terms of subjects and sensor positions. The coupling of these two factors makes it hard to generalize the model to a new application scenario, where there are unseen subjects and new sensor position combinations. In this paper, we design a framework to combine heterogeneous data to learn a general representation for HAR, so that it can work for new applications. We propose an Augmented Adversarial Learning framework for HAR (AALH) to learn generalizable representations to deal with diverse combinations of sensor positions and subject discrepancies. We train an adversarial neural network to map various sensor sets' data into a common latent representation space which is domain-invariant and class-discriminative. We enrich the latent representation space by a hybrid missing strategy and complement each subject domain with a multi-domain mixup method, and they significantly improve model generalization. Experiment results on two HAR datasets demonstrate that the proposed method significantly outperforms previous methods on unseen subjects and new sensor position combinations.

The role of latent representations for design space exploration of floorplans

Floorplans often require considering numerous factors, from the layout size to cost, numeric attributes such as room sizes, and other intrinsic properties such as connectivity between visible regions. Representing these complex factors is challenging, but doing so in a representative and efficient way can enable new modes of design exploration. Existing image and graph-based approaches of floorplans’ representation often failed to consider low-level space semantics, structural features, and space utilization with respect to its future inhabitants, which are all the critical elements to analyze design layouts. We present a latent-space representation of floorplans using gated recurrent unit variational autoencoder (GRU-VAE), where floorplans are represented as attributed graphs (encoded with the abovementioned features). Two local search approaches are presented to efficiently explore the latent space for optimizing and generating new floorplans for the given environment. Semantic, structural, and visibility metrics are evaluated individually and as a combined objective for optimizations.

Multiperspective Progressive Structure Adaptation for JPEG Steganography Detection Across Domains.

The aim of steganography detection is to identify whether the multimedia data contain hidden information. Although many detection algorithms have been presented to solve tasks with inconsistent distributions between the source and target domains, effectively exploiting transferable correlation information across domains remains challenging. As a solution, we present a novel multiperspective progressive structure adaptation (MPSA) scheme based on active progressive learning (APL) for JPEG steganography detection across domains. First, the source and target data originating from unprocessed steganalysis features are clustered together to explore the structures in different domains, where the intradomain and interdomain structures can be captured to provide adequate information for cross-domain steganography detection. Second, the structure vectors containing the global and local modalities are exploited to reduce nonlinear distribution discrepancy based on APL in the latent representation space. In this way, the signal-to-noise ratio (SNR) of a weak stego signal can be improved by selecting suitable objects and adjusting the learning sequence. Third, the structure adaptation across multiple domains is achieved by the constraints for iterative optimization to promote the discrimination and transferability of structure knowledge. In addition, a unified framework for single-source domain adaptation (SSDA) and multiple-source domain adaptation (MSDA) in mismatched steganalysis can enhance the model's capability to avoid a potential negative transfer. Extensive experiments on various benchmark cross-domain steganography detection tasks show the superiority of the proposed approach over the state-of-the-art methods.

Feature selection via Non-convex constraint and latent representation learning with Laplacian embedding

In unsupervised feature selection, the relationship between pseudo-labels is often ignored, and the interconnection information between the data is not fully utilized. In order to solve these problems, this paper proposes a feature selection method via non-convex constraint and latent representation learning with Laplacian embedding (NLRL-LE). NLRL-LE keeps the correlation between the pseudo-labels to make the pseudo-label closer to the true label. And it combines with the interconnection information between data, learns the latent representation matrix to guide feature selection. Specifically, first, NLRL-LE regards each pseudo-label as a latent feature of the sample, constructs a latent feature graph, and retains the inherent attributes of the pseudo-labels. Second, latent representation learning is performed in the space which is made up of the latent feature space and data space. Since the latent feature graph retains the correlation between pseudo-labels, latent representation learning considers the interconnection information between data, and the information contained in the latent representation space is more complete. In addition, in order to make full use of pseudo-labels, the learned latent representation matrix is used as pseudo-label information to provide cluster labels in the latent representation space to guide feature selection. Finally, non-negative and l2,1-2-norm non-convex constraint are applied to the feature transformation matrix. The combination of non-negative constraint and non-convex constraint, compared with convex constraint, can ensure the row sparsity of the feature transformation matrix, select low-redundant features, and improve the feature selection effect. The experimental results show that the ACC and NMI of the NLRL-LE are better than the other seven compared algorithms on twelve datasets.

Hospital-Agnostic Image Representation Learning in Digital Pathology.

Whole Slide Images (WSIs) in digital pathology are used to diagnose cancer subtypes. The difference in procedures to acquire WSIs at various trial sites gives rise to variability in the histopathology images, thus making consistent diagnosis challenging. These differences may stem from variability in image acquisition through multi-vendor scanners, variable acquisition parameters, and differences in staining procedure; as well, patient demographics may bias the glass slide batches before image acquisition. These variabilities are assumed to cause a domain shift in the images of different hospitals. It is crucial to overcome this domain shift because an ideal machine-learning model must be able to work on the diverse sources of images, independent of the acquisition center. A domain generalization technique is leveraged in this study to improve the generalization capability of a Deep Neural Network (DNN), to an unseen histopathology image set (i.e., from an unseen hospital/trial site) in the presence of domain shift. According to experimental results, the conventional supervisedlearning regime generalizes poorly to data collected from different hospitals. However, the proposed hospital-agnostic learning can improve the generalization considering the lowdimensional latent space representation visualization, and classification accuracy results.

Soldering Data Classification with a Deep Clustering Approach: Case Study of an Academic-Industrial Cooperation

Modern industries still commonly use traditional methods to visually inspect products, even though automation has many advantages over the skills of human labour. The automation of redundant tasks is one of the greatest successes of Artificial Intelligence (AI). It employs human annotation and finds possible relationships between features within a particular dataset. However, until recently, this has always been the responsibility of AI specialists with a specific type of knowledge that is not available to the industrial domain experts. We documented the joint research of AI and domain experts as a case study on processing a soldering-related industrial dataset. Our image classification approach relies on the latent space representations of neural networks already trained on other databases. We perform dimensionality reduction of the representations of the new data and cluster the outputs in the lower dimension. This method requires little to no knowledge of the underlying architecture of neural networks by the domain experts, meaning it is easily manageable by them, supporting generalization to other use cases that can be investigated in future work. We also suggest a misclassification detecting method. We were able to achieve near-perfect test accuracy with minimal annotation work.

Improving Deep Learning for Maritime Remote Sensing through Data Augmentation and Latent Space

Training deep learning models requires having the right data for the problem and understanding both your data and the models’ performance on that data. Training deep learning models is difficult when data are limited, so in this paper, we seek to answer the following question: how can we train a deep learning model to increase its performance on a targeted area with limited data? We do this by applying rotation data augmentations to a simulated synthetic aperture radar (SAR) image dataset. We use the Uniform Manifold Approximation and Projection (UMAP) dimensionality reduction technique to understand the effects of augmentations on the data in latent space. Using this latent space representation, we can understand the data and choose specific training samples aimed at boosting model performance in targeted under-performing regions without the need to increase training set sizes. Results show that using latent space to choose training data significantly improves model performance in some cases; however, there are other cases where no improvements are made. We show that linking patterns in latent space is a possible predictor of model performance, but results require some experimentation and domain knowledge to determine the best options.

Employing Artificial Neural Networks to Identify Reaction Coordinates and Pathways for Self-Assembly.

Capturing the autonomous self-assembly of molecular building blocks in computer simulations is a persistent challenge, requiring to model complex interactions and to access long time scales. Advanced sampling methods allow to bridge these time scales but typically need accurate low-dimensional representations of the transition pathways. In this work, we demonstrate for the self-assembly of two single-stranded DNA fragments into a ring-like structure how autoencoder neural networks can be employed to reliably provide a suitable low-dimensional representation and to expose transition pathways: The assembly proceeds through a two-step process with two distinct half-bound states, which are correctly identified by the neural net. We exploit this latent space representation to construct a Markov state model for predicting the four molecular conformations and their transition rates. We present a detailed comparison with two other low-dimensional representations based on empirically determined order parameters and a time-lagged independent component analysis (TICA). Our work opens up new avenues for the computational modeling of multistep and hierarchical self-assembly, which has proven challenging so far.

Generative Incomplete Multi-View Prognosis Predictor for Breast Cancer: GIMPP.

In today's digital world, we are equipped with modern computer-based data collection sources and feature extraction methods. It enhances the availability of the multi-view data and corresponding researches. Multi-view prediction models form a mainstream research direction in the healthcare and bioinformatics domain. While these models are designed with the assumption that there is no missing data for any views, in the real world, certain views of the data are often not having the same number of samples, resulting in the incomplete multi-view dataset. The studies performed over these datasets are termed incomplete multi-view clustering or prediction. Here, we develop a two-stage generative incomplete multi-view prediction model named GIMPP to address the missing view problem of breast cancer prognosis prediction by explicitly generating the missing data. The first stage incorporates the multi-view encoder networks and the bi-modal attention scheme to learn common latent space representations by leveraging complementary knowledge between different views. The second stage generates missing view data using view-specific generative adversarial networks conditioned on the shared representations and encoded features given by other views. Experimental results on TCGA-BRCA and METABRIC datasets proves the usefulness of the developed method over the state-of-the-art methods.

Image De-Noising Using Convolutional Variational Autoencoders

Abstract: Typically, image noise is random colour information in picture pixels that serves as an unfavourable by-product of the image, obscuring the intended information. In most cases, noise is injected into photographs during the transfer or reception of the image, or during the capture of an image when an object is moving quickly. To improve the noisy picture predictions, autoencoders that denoise the input images are employed. Autoencoders are a sort of unsupervised machine learning that compresses the input and reconstructs an output that is very similar to the original input. The autoencoder tries to figure out non-linear correlations between data points. An encoder, a latent space, and a decoder all exist in autoencoders. The encoder reduces the dimensionality of an original picture to its latent space representation, which is then used by the decoder to reconstruct the reduced dimensional image back to its original image. Basic Autoencoder, Variational Autoencoder, and Convolutional Autoencoder are the three approaches that were employed to denoise the picture. In the basic and convolutional autoencoders, there is only one loss parameter, however in the variational autoencoder, there are two losses: generative loss and latent loss. TensorFlow as the frontend and Keras as the backend are used to implement autoencoders in this project. The noisy pictures are trained on every convolutional variational autoencoder techniques to produce a decent prediction of noisy test data.

SimSR: Simple Distance-Based State Representations for Deep Reinforcement Learning

This work explores how to learn robust and generalizable state representation from image-based observations with deep reinforcement learning methods. Addressing the computational complexity, stringent assumptions and representation collapse challenges in existing work of bisimulation metric, we devise Simple State Representation (SimSR) operator. SimSR enables us to design a stochastic approximation method that can practically learn the mapping functions (encoders) from observations to latent representation space. In addition to the theoretical analysis and comparison with the existing work, we experimented and compared our work with recent state-of-the-art solutions in visual MuJoCo tasks. The results shows that our model generally achieves better performance and has better robustness and good generalization.

Graph regularized spatial–spectral subspace clustering for hyperspectral band selection

Hyperspectral band selection, which aims to select a small number of bands to reduce data redundancy and noisy bands, has attracted widespread attention in recent years. Many effective clustering-based band selection methods have been proposed to accomplish the band selection task and have achieved satisfying performance. However, most of the previous methods reshape the original hyperspectral images (HSIs) into a set of stretched band vectors, which ignore the spatial information of HSIs and the difference between diverse regions. To address these issues, a graph regularized spatial–spectral subspace clustering method for hyperspectral band selection is proposed in this paper, referred to as GRSC. Specifically, the proposed method adopts superpixel segmentation to preserve the spatial information of HSIs by segmenting their first principal component into diverse homogeneous regions. Then the discriminative latent features are generated from each segmented region to represent the whole band, which can mitigate the effect of noise on the band selection. Finally, a self-representation subspace clustering model and an l2,1-norm regularization are utilized to explore the spectral correlation among all bands. In addition, a similarity graph between region-aware latent features is adaptively learned to preserve the spatial structure of HSIs in the latent representation space. Extensive classification experimental results on three public datasets verify the effectiveness of GRSC over several state-of-the-art methods. The demo code of this work is publicly available at https://github.com/WangJun2023/GRSC.

Adversarial Autoencoder and Multi-Armed Bandit for Dynamic Difficulty Adjustment in Immersive Virtual Reality for Rehabilitation: Application to Hand Movement.

Motor rehabilitation is used to improve motor control skills to improve the patient’s quality of life. Regular adjustments based on the effect of therapy are necessary, but this can be time-consuming for the clinician. This study proposes to use an efficient tool for high-dimensional data by considering a deep learning approach for dimensionality reduction of hand movement recorded using a wireless remote control embedded with the Oculus Rift S. This latent space is created as a visualization tool also for use in a reinforcement learning (RL) algorithm employed to provide a decision-making framework. The data collected consists of motions drawn with wireless remote control in an immersive VR environment for six different motions called “Cube”, “Cylinder”, “Heart”, “Infinity”, “Sphere”, and “Triangle”. From these collected data, different artificial databases were created to simulate variations of the data. A latent space representation is created using an adversarial autoencoder (AAE), taking into account unsupervised (UAAE) and semi-supervised (SSAAE) training. Then, each test point is represented by a distance metric and used as a reward for two classes of Multi-Armed Bandit (MAB) algorithms, namely Boltzmann and Sibling Kalman filters. The results showed that AAE models can represent high-dimensional data in a two-dimensional latent space and that MAB agents can efficiently and quickly learn the distance evolution in the latent space. The results show that Sibling Kalman filter exploration outperforms Boltzmann exploration with an average cumulative weighted probability error of 7.9 versus 19.9 using the UAAE latent space representation and 8.0 versus 20.0 using SSAAE. In conclusion, this approach provides an effective approach to visualize and track current motor control capabilities regarding a target in order to reflect the patient’s abilities in VR games in the context of DDA.

A practical guide for generating unsupervised, spectrogram-based latent space representations of animal vocalizations.

The manual detection, analysis and classification of animal vocalizations in acoustic recordings is laborious and requires expert knowledge. Hence, there is a need for objective, generalizable methods that detect underlying patterns in these data, categorize sounds into distinct groups and quantify similarities between them. Among all computational methods that have been proposed to accomplish this, neighbourhood-based dimensionality reduction of spectrograms to produce a latent space representation of calls stands out for its conceptual simplicity and effectiveness. Goal of the study/what was done: Using a dataset of manually annotated meerkat Suricata suricatta vocalizations, we demonstrate how this method can be used to obtain meaningful latent space representations that reflect the established taxonomy of call types. We analyse strengths and weaknesses of the proposed approach, give recommendations for its usage and show application examples, such as the classification of ambiguous calls and the detection of mislabelled calls. What this means: All analyses are accompanied by example code to help researchers realize the potential of this method for the study of animal vocalizations.

Controllable and Identity-Aware Facial Attribute Transformation.

Modifying facial attributes without the paired dataset proves to be a challenging task. Previously, approaches either required supervision from a ground-truth transformed image or required training a separate model for mapping every pair of attributes. These limit the scalability of the models to accommodate a larger set of attributes since the number of models that we need to train grows exponentially large. Another major drawback of the previous approaches is the unintentional gain of the identity of the person as they transform the facial attributes. We propose a method that allows for controllable and identity-aware transformations across multiple facial attributes using only a single model. Our approach is to train a generative adversarial network (GAN) with a multitask conditional discriminator that recognizes the identity of the face, distinguishes real images from fake, as well as identifies facial attributes present in an image. This guides the generator into producing an output that is realistic while preserving the person's identity and facial attributes. Through this framework, our model also learns meaningful image representations in a lower dimensional latent space and semantically associate separate parts of the encoded vector with both the person's identity and facial attributes. This opens up the possibility of generating new faces and other transformations such as making the face thinner or chubbier. Furthermore, our model only encodes the image once and allows for multiple transformations using the encoded vector. This allows for faster transformations since it does not need to reprocess the entire image for every transformation. We show the effectiveness of our proposed method through both qualitative and quantitative evaluations, such as ablative studies, visual inspection, and face verification. Competitive results are achieved compared to the main competition (CycleGAN), however, at great space and extensibility gain by using a single model.

PriorVAE: encoding spatial priors with variational autoencoders for small-area estimation.

Gaussian processes (GPs), implemented through multivariate Gaussian distributions for a finite collection of data, are the most popular approach in small-area spatial statistical modelling. In this context, they are used to encode correlation structures over space and can generalize well in interpolation tasks. Despite their flexibility, off-the-shelf GPs present serious computational challenges which limit their scalability and practical usefulness in applied settings. Here, we propose a novel, deep generative modelling approach to tackle this challenge, termed PriorVAE: for a particular spatial setting, we approximate a class of GP priors through prior sampling and subsequent fitting of a variational autoencoder (VAE). Given a trained VAE, the resultant decoder allows spatial inference to become incredibly efficient due to the low dimensional, independently distributed latent Gaussian space representation of the VAE. Once trained, inference using the VAE decoder replaces the GP within a Bayesian sampling framework. This approach provides tractable and easy-to-implement means of approximately encoding spatial priors and facilitates efficient statistical inference. We demonstrate the utility of our VAE two-stage approach on Bayesian, small-area estimation tasks.

Adaptive Clustering of Robust Semantic Representations for Adversarial Image Purification on Social Networks

Advances in Artificial Intelligence (AI) have made it possible to automate human-level visual search and perception tasks on the massive sets of image data shared on social media on a daily basis. However, AI-based automated filters are highly susceptible to deliberate image attacks that can lead to content misclassification of cyberbulling, child sexual abuse material (CSAM), adult content, and deepfakes. One of the most effective methods to defend against such disturbances is adversarial training, but this comes at the cost of generalization for unseen attacks and transferability across models. In this article, we propose a robust defense against adversarial image attacks, which is model agnostic and generalizable to unseen adversaries. We begin with a baseline model, extracting the latent representations for each class and adaptively clustering the latent representations that share a semantic similarity. Next, we obtain the distributions for these clustered latent representations along with their originating images. We then learn semantic reconstruction dictionaries (SRD). We adversarially train a new model constraining the latent space representation to minimize the distance between the adversarial latent representation and the true cluster distribution. To purify the image, we decompose the input into low and high-frequency components. The high-frequency component is reconstructed based on the best SRD from the clean dataset. In order to evaluate the best SRD, we rely on the distance between the robust latent representations and semantic cluster distributions. The output is a purified image with no perturbations. Evaluations using comprehensive datasets including image benchmarks and social media images demonstrate that our proposed purification approach guards and enhances the accuracy of AI-based image filters for unlawful and harmful perturbed images considerably.

BRULÈ: Barycenter-Regularized Unsupervised Landmark Extraction

Unsupervised retrieval of image features is vital for many computer vision tasks where the annotation is missing or scarce. In this work, we propose a new unsupervised approach to detect the landmarks in images, validating it on the popular task of human face key-points extraction. The method is based on the idea of auto-encoding the wanted landmarks in the latent space while discarding the non-essential information (and effectively preserving the interpretability). The interpretable latent space representation (the bottleneck containing nothing but the wanted key-points) is achieved by a new two-step regularization approach. The first regularization step evaluates transport distance from a given set of landmarks to some average value (the barycenter by Wasserstein distance). The second regularization step controls deviations from the barycenter by applying random geometric deformations synchronously to the initial image and to the encoded landmarks. We demonstrate the effectiveness of the approach both in unsupervised and semi-supervised training scenarios using 300-W, CelebA, and MAFL datasets. The proposed regularization paradigm is shown to prevent overfitting, and the detection quality is shown to improve beyond the state-of-the-art face models.