Low-dimensional Latent Space Research Articles

A novel and systematic approach to identify the design space of pharmaceutical processes

Feasibility analysis is a mathematical technique that can be used to assist the identification of the design space (DS) of a pharmaceutical process, given the availability of a process model. One of its main drawbacks is that it suffers from the curse of dimensionality, i.e. simulations can potentially become computationally extremely expensive and very cumbersome when the number of input factors is large. Additionally, giving a graphical and compact representation of the high-dimensional design space is difficult. In this study, we propose a novel and systematic methodology to exploit partial least-squares (PLS) regression modelling to reduce the dimensionality of a feasibility problem. We use PLS to obtain a linear transformation between the original multidimensional input space and a lower dimensional latent space. We then apply a Radial Basis Function (RBF) adaptive sampling feasibility analysis on this lower dimensional space to identify the feasible region of the process. We assess the accuracy and robustness of the results with three metrics, and we critically discuss the criteria that should be adopted for the choice of the number of latent variables. The performance of the methodology is tested on three simulated case studies, one of which involving the continuous direct compaction of a pharmaceutical powder. In all case studies, the methodology shows to be effective in reducing the computational burden while maintaining an accurate and robust identification of the design space.

Multi-view manifold learning with locality alignment

Manifold learning aims to discover the low dimensional space where the input high dimensional data are embedded by preserving the geometric structure. Unfortunately, almost all the existing manifold learning methods were proposed under single view scenario, and they cannot be straightforwardly applied to multiple feature sets. Although concatenating multiple views into a single feature provides a plausible solution, it remains a question on how to better explore the independence and interdependence of different views while conducting manifold learning. In this paper, we propose a multi-view manifold learning with locality alignment (MVML-LA) framework to learn a common yet discriminative low-dimensional latent space that contain sufficient information of original inputs. Both supervised algorithm (S-MVML-LA) and unsupervised algorithm (U-MVML-LA) are developed. Experiments on benchmark real-world datasets demonstrate the superiority of our proposed S-MVML-LA and U-MVML-LA over existing state-of-the-art methods.

Nonnegative matrix factorization with mixed hypergraph regularization for community detection

Community structure is the most significant attribute of networks, which is often identified to help discover the underlying organization of networks. Currently, nonnegative matrix factorization (NMF) based community detection method makes use of the related topology information and assumes that networks are able to be projected onto a latent low-dimensional space, in which the nodes can be efficiently clustered. In this paper, we propose a novel framework named mixed hypergraph regularized nonnegative matrix factorization (MHGNMF), which takes higher-order information among the nodes into consideration to enhance the clustering performance. The hypergraph regularization term forces the nodes within the identical hyperedge to be projected onto the same latent subspace, so that a more discriminative representation is achieved. In the proposed framework, we generate a set of hyperedges by mixing two kinds of neighbors for each centroid, which makes full use of topological connection information and structural similarity information. By testing on two artificial benchmarks and eight real-world networks, the proposed framework demonstrates better detection results than the other state-of-the-art methods.

Unsupervised Locality-Preserving Robust Latent Low-Rank Recovery-Based Subspace Clustering for Fault Diagnosis

With the increasing demand for unsupervised learning for fault diagnosis, the subspace clustering has been considered as a promising technique enabling unsupervised fault diagnosis. Although various subspace clustering methods have been developed to deal with high-dimensional and non-linear data, analyzing the intrinsic structure from the data is still challenging. To address this issue, a new subspace clustering method based on locality-preserving robust latent low-rank recovery (L2PLRR) was developed. Unlike conventional subspace clustering methods, the developed method maps the high-dimensional and non-linear data into a low-dimensional latent space by preserving local similarities of the data with the goal of resolving the difficulty in analyzing the high-dimensional data. Likewise, in the developed L2PLRR method, learned features correspond to low-rank coefficients of the data in the latent space, which will be further used for fault diagnosis (e.g., identification of health states of an object system). The efficacy of the developed L2PLRR method was verified with a bearing fault diagnosis application by comparing with conventional and state-of-the-art subspace clustering methods in terms of diagnostic performance.

Wellness Representation of Users in Social Media: Towards Joint Modelling of Heterogeneity and Temporality

The increasing popularity of social media has encouraged health consumers to share, explore, and validate health and wellness information on social networks, which provide a rich repository of Patient Generated Wellness Data (PGWD). While data-driven healthcare has attracted a lot of attention from academia and industry for improving care delivery through personalized healthcare, limited research has been done on harvesting and utilizing PGWD available on social networks. Recently, representation learning has been widely used in many applications to learn low-dimensional embedding of users. However, existing approaches for representation learning are not directly applicable to PGWD due to its domain nature as characterized by longitudinality, incompleteness, and sparsity of observed data as well as heterogeneity of the patient population. To tackle these problems, we propose an approach which directly learns the embedding from longitudinal data of users, instead of vector-based representation. In particular, we simultaneously learn a low-dimensional latent space as well as the temporal evolution of users in the wellness space. The proposed method takes into account two types of wellness prior knowledge: (1) temporal progression of wellness attributes; and (2) heterogeneity of wellness attributes in the patient population. Our approach scales well to large datasets using parallel stochastic gradient descent. We conduct extensive experiments to evaluate our framework at tackling three major tasks in wellness domain: attribute prediction, success prediction, and community detection. Experimental results on two real-world datasets demonstrate the ability of our approach in learning effective user representations.

Linear latent low dimensional space for online early action recognition and prediction

Recognition and prediction of human actions is one of the important tasks in various computer vision applications including video surveillance, human computer interaction and home entertainment that require online and real time approaches. In this work, we propose a novel approach that utilises continuous streams of joint motion data for recognising and predicting actions in linear latent spaces operating online and in real time. Our approach is based on supervised learning and dimensionality reduction techniques that allow the representation of high dimensional nonlinear actions to linear latent low dimensional spaces. Our methodology has been evaluated using well-known datasets and performance metrics specifically designed for online and real time action recognition and prediction. We demonstrate the performance of the proposed approach in a comparative study showing high accuracy and low latency.

A surrogate modeling approach for reliability analysis of a multidisciplinary system with spatio-temporal output

Reliability analysis of a multidisciplinary system is computationally intensive due to the involvement of multiple disciplinary models and coupling between the individual models. When the system inputs and outputs are varying over time and space, the reliability analysis is even more challenging. This paper proposes a surrogate model-based method for the reliability analysis of a multidisciplinary system with spatio-temporal output. The transient characteristics of the multidisciplinary system output under time-dependent variability are analyzed first. Based on the transient analysis, surrogate models are built for individual disciplinary analyses instead of a single surrogate model for the fully coupled analysis. To address the challenge introduced by the high-dimensionality of spatially varying inter-disciplinary coupling variables, a data compression method is first employed to convert the high-dimensional coupling variables into low-dimensional latent space. Kriging surrogate modeling is then used to build surrogates for the individual disciplinary models in the latent space. Based on the individual disciplinary surrogate models, reliability analysis of the coupled multidisciplinary system under time-dependent uncertainty is investigated. Further, epistemic uncertainty sources, such as data uncertainty and model uncertainty, lead to uncertainty in the reliability estimate. Therefore, an auxiliary variable approach is used to efficiently include the epistemic uncertainty sources within the reliability analysis. An aircraft panel subjected to hypersonic flow conditions is used to demonstrate the proposed method. The analysis involves four interacting disciplinary models, namely, aerodynamics, aerothermal analysis, heat transfer, and structural analysis. The results show that the proposed method is able to effectively perform reliability analysis of a multidisciplinary system with spatio-temporal output.

Characterizing Dynamic Walking Patterns and Detecting Falls with Wearable Sensors Using Gaussian Process Methods.

By incorporating a growing number of sensors and adopting machine learning technologies, wearable devices have recently become a prominent health care application domain. Among the related research topics in this field, one of the most important issues is detecting falls while walking. Since such falls may lead to serious injuries, automatically and promptly detecting them during daily use of smartphones and/or smart watches is a particular need. In this paper, we investigate the use of Gaussian process (GP) methods for characterizing dynamic walking patterns and detecting falls while walking with built-in wearable sensors in smartphones and/or smartwatches. For the task of characterizing dynamic walking patterns in a low-dimensional latent feature space, we propose a novel approach called auto-encoded Gaussian process dynamical model, in which we combine a GP-based state space modeling method with a nonlinear dimensionality reduction method in a unique manner. The Gaussian process methods are fit for this task because one of the most import strengths of the Gaussian process methods is its capability of handling uncertainty in the model parameters. Also for detecting falls while walking, we propose to recycle the latent samples generated in training the auto-encoded Gaussian process dynamical model for GP-based novelty detection, which can lead to an efficient and seamless solution to the detection task. Experimental results show that the combined use of these GP-based methods can yield promising results for characterizing dynamic walking patterns and detecting falls while walking with the wearable sensors.

Semisupervised, Multilabel, Multi-Instance Learning for Structured Data.

Many classification tasks require both labeling objects and determining label associations for parts of each object. Example applications include labeling segments of images or determining relevant parts of a text document when the training labels are available only at the image or document level. This task is usually referred to as multi-instance (MI) learning, where the learner typically receives a collection of labeled (or sometimes unlabeled) bags, each containing several segments (instances). We propose a semisupervised MI learning method for multilabel classification. Most MI learning methods treat instances in each bag as independent and identically distributed samples. However, in many practical applications, instances are related to each other and should not be considered independent. Our model discovers a latent low-dimensional space that captures structure within each bag. Further, unlike many other MI learning methods, which are primarily developed for binary classification, we model multiple classes jointly, thus also capturing possible dependencies between different classes. We develop our model within a semisupervised framework, which leverages both labeled and, typically, a larger set of unlabeled bags for training. We develop several efficient inference methods for our model. We first introduce a Markov chain Monte Carlo method for inference, which can handle arbitrary relations between bag labels and instance labels, including the standard hard-max MI assumption. We also develop an extension of our model that uses stochastic variational Bayes methods for inference, and thus scales better to massive data sets. Experiments show that our approach outperforms several MI learning and standard classification methods on both bag-level and instance-level label prediction. All code for replicating our experiments is available from https://github.com/hsoleimani/MLTM .

READER: Robust Semi-Supervised Multi-Label Dimension Reduction

Multi-label classification is an appealing and challenging supervised learning problem, where multiple labels, rather than a single label, are associated with an unseen test instance. To remove possible noises in labels and features of high-dimensionality, multi-label dimension reduction has attracted more and more attentions in recent years. The existing methods usually suffer from several problems, such as ignoring label outliers and label correlations. In addition, most of them emphasize on conducting dimension reduction in an unsupervised or supervised way, therefore, unable to utilize the label information or a large amount of unlabeled data to improve the performance. In order to cope with these problems, we propose a novel method termed Robust sEmi-supervised multi-lAbel DimEnsion Reduction, shortly READER. From the viewpoint of empirical risk minimization, READER selects most discriminative features for all the labels in a semi-supervised way. Specifically, the ℓ2,1-norm induced loss function and regularization term make READER robust to the outliers in the data points. READER finds a feature subspace so as to keep originally neighbor instances close and embeds labels into a low-dimensional latent space nonlinearly. To optimize the objective function, an efficient algorithm is developed with convergence property. Extensive empirical studies on real-world datasets demonstrate the superior performance of the proposed method.

Social Collaborative Filtering by Trust.

Recommender systems are used to accurately and actively provide users with potentially interesting information or services. Collaborative filtering is a widely adopted approach to recommendation, but sparse data and cold-start users are often barriers to providing high quality recommendations. To address such issues, we propose a novel method that works to improve the performance of collaborative filtering recommendations by integrating sparse rating data given by users and sparse social trust network among these same users. This is a model-based method that adopts matrix factorization technique that maps users into low-dimensional latent feature spaces in terms of their trust relationship, and aims to more accurately reflect the users reciprocal influence on the formation of their own opinions and to learn better preferential patterns of users for high-quality recommendations. We use four large-scale datasets to show that the proposed method performs much better, especially for cold start users, than state-of-the-art recommendation algorithms for social collaborative filtering based on trust.

Hidden-Markov Factor analysis as a spatiotemporal model for electrocorticography.

We present a new approach to extracting low-dimensional neural trajectories that summarize the electrocorticographic (ECoG) signals recorded with high-channel-count electrode arrays implanted subdurally. In our approach, Hidden-Markov Factor Analysis (HMFA), a finite set of factor analyzers are used to model the relationship between the high-dimensional ECoG neural space and a low-dimensional latent neural space; the factor analyzers at different time points are in turn linked together with a hidden Markov model. The recorded ECoG signals were band-pass filtered such that our analysis was focused on a sub-band (76-100Hz) of high gamma. HMFA affords the quantization of the ECoG neural space and dimensionality reduction in a common probabilistic space. We applied this method to the ECoG recordings of 2 subjects who responded with button presses to audiovisual stimuli in an experimental task. Using a goodness-of-fit metric that measures how well the ECoG activity of each electrode can be predicted by all the other electrodes, we found that HMFA performed best when compared with Gaussian-Process Factor Analysis (GPFA) and other related spatiotemporal modeling techniques. In contradistinction to HMFA, GPFA and the other techniques integrate temporal smoothing with dimensionality reduction. We believe that this method will provide a powerful tool for relating high-channel-count ECoG signals to the perception and behavior of subjects.

Sparse Latent Space Policy Search

Computational agents often need to learn policies that involve many control variables, e.g., a robot needs to control several joints simultaneously. Learning a policy with a high number of parameters, however, usually requires a large number of training samples. We introduce a reinforcement learning method for sample-efficient policy search that exploits correlations between control variables. Such correlations are particularly frequent in motor skill learning tasks. The introduced method uses Variational Inference to estimate policy parameters, while at the same time uncovering a low-dimensional latent space of controls. Prior knowledge about the task and the structure of the learning agent can be provided by specifying groups of potentially correlated parameters. This information is then used to impose sparsity constraints on the mapping between the high-dimensional space of controls and a lower-dimensional latent space. In experiments with a simulated bi-manual manipulator, the new approach effectively identifies synergies between joints, performs efficient low-dimensional policy search, and outperforms state-of-the-art policy search methods.

Discriminative Subspace Alignment for Unsupervised Visual Domain Adaptation

We address the problem of unsupervised visual domain adaptation for transferring category models from one visual domain or image data set to another. We present a new unsupervised domain adaptation algorithm based on subspace alignment. The core idea of our approach is to reduce the discrepancy between the source domain and the target domain in a latent discriminative subspace. Specifically, we first generate pseudo-labels for the target data by applying spectral clustering to a cross-domain similarity matrix, which is built from sparse coefficients found in a low-dimensional latent space. This coarse alignment between the two domains exploits the assumption that the collection of data of different classes from both domains can be viewed as samples from a union of low-dimensional subspaces. Then, we create discriminative subspaces for both domains using partial least squares correlation. Finally, a mapping which aligns the discriminative source subspace into the target one is learned by minimizing a Bregman matrix divergence function. Experimental results on benchmark cross-domain visual object recognition data sets and cross-view scene classification data sets demonstrate that the proposed method outperforms the baselines and several state-of-the-art competing methods.

Latent Space Sparse and Low-Rank Subspace Clustering

We propose three novel algorithms for simultaneous dimensionality reduction and clustering of data lying in a union of subspaces. Specifically, we describe methods that learn the projection of data and find the sparse and/or low-rank coefficients in the low-dimensional latent space. Cluster labels are then assigned by applying spectral clustering to a similarity matrix built from these representations. Efficient optimization methods are proposed and their non-linear extensions based on kernel methods are presented. Various experiments show that the proposed methods perform better than many competitive subspace clustering methods.

Bayesian principal geodesic analysis for estimating intrinsic diffeomorphic image variability.

In this paper, we present a generative Bayesian approach for estimating the low-dimensional latent space of diffeomorphic shape variability in a population of images. We develop a latent variable model for principal geodesic analysis (PGA) that provides a probabilistic framework for factor analysis in the space of diffeomorphisms. A sparsity prior in the model results in automatic selection of the number of relevant dimensions by driving unnecessary principal geodesics to zero. To infer model parameters, including the image atlas, principal geodesic deformations, and the effective dimensionality, we introduce an expectation maximization (EM) algorithm. We evaluate our proposed model on 2D synthetic data and the 3D OASIS brain database of magnetic resonance images, and show that the automatically selected latent dimensions from our model are able to reconstruct unobserved testing images with lower error than both linear principal component analysis (LPCA) in the image space and tangent space principal component analysis (TPCA) in the diffeomorphism space.

GTM-based service map to identify new service opportunities

This study proposes a unique approach for developing and utilising generative topographic mapping (GTM)–based service maps to discover new opportunities of smartphone application service (SAS) with a case of App Store. Since GTM is a probabilistic approach of mapping multidimensional data space onto a low–dimensional latent space and vice versa, it contributes to the automatic detection and interpretation of derived service vacuums that must be intensively investigated to discover new opportunities of SAS. The main contributions and potential utilities of this study are threefold. First, this study theoretically contributes to new service development (NSD) research in mobile service area by proposing an intelligent approach. Secondly, this study is exploratory in that the inverse mapping of a GTM model is first proposed. Lastly, since all activities in the process could be computerised, researchers, engineers, managers, etc., who are interested in NSD can save considerable time and effort when uncovering new service opportunities in terms of managerial implications.

Nonparametric Bayes dynamic modelling of relational data

Symmetric binary matrices representing relations among entities are commonly collected in many areas. Our focus is on dynamically evolving binary relational matrices, with interest being in inference on the relationship structure and prediction. We propose a nonparametric Bayesian dynamic model, which reduces dimensionality in characterizing the binary matrix through a lower-dimensional latent space representation, with the latent coordinates evolving in continuous time via Gaussian processes. By using a logistic mapping function from the probability matrix space to the latent relational space, we obtain a flexible and computational tractable formulation. Employing P\`olya-Gamma data augmentation, an efficient Gibbs sampler is developed for posterior computation, with the dimension of the latent space automatically inferred. We provide some theoretical results on flexibility of the model, and illustrate performance via simulation experiments. We also consider an application to co-movements in world financial markets.

Forecasting Potential Diabetes Complications

Diabetes complications often afflict diabetes patients seriously: over 68% of diabetes-related mortality is caused by diabetes complications. In this paper, we study the problem of automatically diagnosing diabetes complications from patients' lab test results. The objective problem has two main challenges: 1) feature sparseness: a patient only undergoes 1.26% lab tests on average, and 65.5% types of lab tests are performed on samples from less than 10 patients; 2) knowledge skewness: it lacks comprehensive detailed domain knowledge of the association between diabetes complications and lab tests. To address these challenges, we propose a novel probabilistic model called Sparse Factor Graph Model (SparseFGM). SparseFGM projects sparse features onto a lower-dimensional latent space, which alleviates the problem of sparseness. SparseFGM is also able to capture the associations between complications and lab tests, which help handle the knowledge skewness. We evaluate the proposed model on a large collections of real medical records. SparseFGM outperforms (+20% by F1) baselines significantly and gives detailed associations between diabetes complications and lab tests.

Supervised embedding of textual predictors with applications in clinical diagnostics for pediatric cardiology.

Electronic health records possess critical predictive information for machine-learning-based diagnostic aids. However, many traditional machine learning methods fail to simultaneously integrate textual data into the prediction process because of its high dimensionality. In this paper, we present a supervised method using Laplacian Eigenmaps to enable existing machine learning methods to estimate both low-dimensional representations of textual data and accurate predictors based on these low-dimensional representations at the same time. We present a supervised Laplacian Eigenmap method to enhance predictive models by embedding textual predictors into a low-dimensional latent space, which preserves the local similarities among textual data in high-dimensional space. The proposed implementation performs alternating optimization using gradient descent. For the evaluation, we applied our method to over 2000 patient records from a large single-center pediatric cardiology practice to predict if patients were diagnosed with cardiac disease. In our experiments, we consider relatively short textual descriptions because of data availability. We compared our method with latent semantic indexing, latent Dirichlet allocation, and local Fisher discriminant analysis. The results were assessed using four metrics: the area under the receiver operating characteristic curve (AUC), Matthews correlation coefficient (MCC), specificity, and sensitivity. The results indicate that supervised Laplacian Eigenmaps was the highest performing method in our study, achieving 0.782 and 0.374 for AUC and MCC, respectively. Supervised Laplacian Eigenmaps showed an increase of 8.16% in AUC and 20.6% in MCC over the baseline that excluded textual data and a 2.69% and 5.35% increase in AUC and MCC, respectively, over unsupervised Laplacian Eigenmaps. As a solution, we present a supervised Laplacian Eigenmap method to embed textual predictors into a low-dimensional Euclidean space. This method allows many existing machine learning predictors to effectively and efficiently capture the potential of textual predictors, especially those based on short texts.