Nonlinear Latent Variable Model Research Articles

Bayesian analysis of two-part nonlinear latent variable model: Semiparametric method

Two-part model (TPM) is a widely appreciated statistical method for analyzing semi-continuous data. Semi-continuous data can be viewed as arising from two distinct stochastic processes: one governs the occurrence or binary part of data and the other determines the intensity or continuous part. In the regression setting with the semi-continuous outcome as functions of covariates, the binary part is commonly modelled via logistic regression and the continuous component via a log-normal model. The conventional TPM, still imposes assumptions such as log-normal distribution of the continuous part, with no unobserved heterogeneity among the response, and no collinearity among covariates, which are quite often unrealistic in practical applications. In this article, we develop a two-part nonlinear latent variable model (TPNLVM) with mixed multiple semi-continuous and continuous variables. The semi-continuous variables are treated as indicators of the latent factor analysis along with other manifest variables. This reduces the dimensionality of the regression model and alleviates the potential multicollinearity problems. Our TPNLVM can accommodate the nonlinear relationships among latent variables extracted from the factor analysis. To downweight the influence of distribution deviations and extreme observations, we develop a Bayesian semiparametric analysis procedure. The conventional parametric assumptions on the related distributions are relaxed and the Dirichlet process (DP) prior is used to improve model fitting. By taking advantage of the discreteness of DP, our method is effective in capturing the heterogeneity underlying population. Within the Bayesian paradigm, posterior inferences including parameters estimates and model assessment are carried out through Markov Chains Monte Carlo (MCMC) sampling method. To facilitate posterior sampling, we adapt the Polya-Gamma stochastic representation for the logistic model. Using simulation studies, we examine properties and merits of our proposed methods and illustrate our approach by evaluating the effect of treatment on cocaine use and examining whether the treatment effect is moderated by psychiatric problems.

Internet Gaming Disorder in Early Adolescents: Gender and Depression Differences in a Latent Growth Model.

Background: Studies have shown that many Internet game players may have experienced Internet gaming disorder (IGD), which is thought to coexist with a variety of cognitive and psychological disorders, such as depression. A three-year, four-point longitudinal tracking study was conducted to examine the trajectory of IGD of Chinese early adolescents, as well as the predictive level of gender and depression for IGD. Participants (N = 316) completed questionnaires assessing IGD and depression at the time of the T1, T2, T3, and T4. This study adopted the widely used latent growth model for data analysis. The results showed that, in the early stage of adolescence, (1) the development trajectory of individual IGD was shown as a nonlinear latent variable growth model; the IGD was significantly higher than zero when teenagers were first measured (sixth grade); and, while on the rise, its growth rate is gradually slowing. (2) Gender can significantly predict the development trajectory of individual IGD. (3) Individuals with a high initial level of depression usually have a high initial level of IGD, individuals with a large range of depression display a large range of IGD, and those with a rapid rate of depression change show similar change in their IGD. In sum, this study provided an empirical basis for the prevention and intervention of IGD in early adolescents.

Assessing the Accuracy of Errors of Measurement. Implications for Assessing Reliable Change in Clinical settings.

Item response theory (IRT) models are non-linear latent variable models for discrete measures, whereas factor analysis (FA) is a latent variable model for continuous measures. In FA, the standard error (SE) of individuals' scores is common for all individuals. In IRT, the SE depends on the individual's score, and the SE function is to be provided. The empirical standard deviation of the scores across discrete ranges should also be computed to inform the extent to which IRT SEs overestimate or underestimate the variability of the scores. Within the target range of scores the test was designed to measure, one should expect IRT SEs to be smaller and more precise than FA SEs, and therefore preferable to assess clinical change. Outside the target range, IRT SEs may be too large and more imprecise than FA SEs, and FA more precise to assess change. As a result, whether FA or IRT characterize reliable change more accurately in a sample will depend on the proportion of individuals within or outside the IRT target score range. An application is provided to illustrate these concepts.

Recent Advances in Variational Autoencoders With Representation Learning for Biomedical Informatics: A Survey

Variational autoencoders (VAEs) are deep latent space generative models that have been immensely successful in multiple exciting applications in biomedical informatics such as molecular design, protein design, medical image classification and segmentation, integrated multi-omics data analyses, and large-scale biological sequence analyses, among others. The fundamental idea in VAEs is to learn the distribution of data in such a way that new meaningful data with more intra-class variations can be generated from the encoded distribution. The ability of VAEs to synthesize new data with more representation variance at state-of-art levels provides hope that the chronic scarcity of labeled data in the biomedical field can be resolved. Furthermore, VAEs have made nonlinear latent variable models tractable for modeling complex distributions. This has allowed for efficient extraction of relevant biomedical information from learned features for biological data sets, referred to as unsupervised feature representation learning. In this article, we review the various recent advancements in the development and application of VAEs for biomedical informatics. We discuss challenges and future opportunities for biomedical research with respect to VAEs.

A robust nonlinear low-dimensional manifold for single cell RNA-seq data

BackgroundModern developments in single-cell sequencing technologies enable broad insights into cellular state. Single-cell RNA sequencing (scRNA-seq) can be used to explore cell types, states, and developmental trajectories to broaden our understanding of cellular heterogeneity in tissues and organs. Analysis of these sparse, high-dimensional experimental results requires dimension reduction. Several methods have been developed to estimate low-dimensional embeddings for filtered and normalized single-cell data. However, methods have yet to be developed for unfiltered and unnormalized count data that estimate uncertainty in the low-dimensional space. We present a nonlinear latent variable model with robust, heavy-tailed error and adaptive kernel learning to estimate low-dimensional nonlinear structure in scRNA-seq data.ResultsGene expression in a single cell is modeled as a noisy draw from a Gaussian process in high dimensions from low-dimensional latent positions. This model is called the Gaussian process latent variable model (GPLVM). We model residual errors with a heavy-tailed Student’s t-distribution to estimate a manifold that is robust to technical and biological noise found in normalized scRNA-seq data. We compare our approach to common dimension reduction tools across a diverse set of scRNA-seq data sets to highlight our model’s ability to enable important downstream tasks such as clustering, inferring cell developmental trajectories, and visualizing high throughput experiments on available experimental data.ConclusionWe show that our adaptive robust statistical approach to estimate a nonlinear manifold is well suited for raw, unfiltered gene counts from high-throughput sequencing technologies for visualization, exploration, and uncertainty estimation of cell states.

Monitoring and prediction of big process data with deep latent variable models and parallel computing

Process monitoring and quality prediction are crucial for maintaining favorable operating conditions and have received considerable attention in previous decades. For majority complicated cases in chemical and biological industrial processes with particular nonlinear characteristics, traditional latent variable models, such as principal component analysis (PCA), principal component regression (PCR), partial least squares (PLS), may not work well. In this paper, various nonlinear latent variable models based on autoencoder (AE) are developed. In order to extract deeper nonlinear features from process data, the basic shallow AE models are extended to the deep latent variable models, which provides a deep generative structure for nonlinear process monitoring and quality prediction. Meanwhile, with the ever increasing scale of industrial data, the computational burden for process modeling and analytics has becoming more and more tremendous, particularly for large-scale processes. To handle the big data problem, the parallel computing strategy is further applied to the above model, which partitions the whole computational task into a few sub-tasks and assigns them to parallel computing nodes. Then the parallel models are utilized for process monitoring and quality prediction applications. The effectiveness of the developed methods are evaluated through the Tennessee Eastman (TE) benchmark process and a real-life industrial process in an ammonia synthesis plant (ASP).

The intergenerational transmission of health in the United States: A latent variables analysis.

Social scientists have long documented that many components of socioeconomic status such as income and education have strong ties across generations. However, health status, arguably a more critical component of welfare, has largely been ignored. We fill this void by providing the first estimates of the Intergenerational Health Association (IHA) that are explicitly based on a nonlinear latent variable model. We develop an estimation procedure for a nonlinear model with categorical outcomes in which the latent index is a mixed linear model and contains covariates that might not vary within cross-sectional units. Adjusting for only age and gender, we estimate an IHA of 0.3 indicating that about one third of a parent's health status gets transmitted to their children. Once we add additional mediators to the model, we show that education, and particularly children's education, is an important transmission channel in that it reduces the IHA by one third. Finally, we show that estimates of the IHA from nonlinear models are only moderately higher than those from linear models, whereas rank-based mobility estimates are identical.

Monitoring Distillation Column Systems Using Improved Nonlinear Partial Least Squares-Based Strategies

Fault detection in industrial systems plays a core role in improving their safety, productivity and avoiding expensive maintenance. This paper proposed and verified data-driven anomaly detection schemes based on a nonlinear latent variable model and statistical monitoring algorithms. Integrating both the suitable characteristics of partial least squares (PLS) and adaptive neural network fuzzy inference systems (ANFIS) procedure, PLS-ANFIS model is employed to allow for flexible modeling of multivariable nonlinear processes. Furthermore, PLS-ANFIS modeling was connected with k-nearest neighbors (kNN)-based data mining schemes and employed for nonlinear process monitoring. Specifically, residuals generated from the PLS-ANFIS model are used as the input to the kNN-based mechanism to uncover anomalies in the data. Moreover, kNN-based exponentially smoothing with parametric and nonparametric thresholds is adopted to better anomaly detection. The effectiveness of the proposed approach is evaluated using real measurements from an actual bubble cap distillation column.

Bayesian Nonlinear Latent variable Models with Mixed Non-normal Variables and Covariates for Multi-sample Psychological Data

The purpose of this paper is to develop a latent variable model with nonlinear covariates and latent variables. Mixed ordered categorical and dichotomous variables and covariates with two different types of thresholds (with equal and unequal spaces) are used in Bayesian multi-sample nonlinear latent variable models and the Gibbs sampling method is applied for estimation and model comparison. Hidden continuous normal distribution (censored normal distribution) and (truncated normal distribution with known parameters) are used to handle the problem of mixed ordered categorical and dichotomous data. Hidden continuous normal distribution (truncated normal distribution with known parameters) is used to handle the problem of mixed ordered categorical and dichotomous data in covariates. Statistical analysis, which involves the estimation of parameters, standard deviations and their highest posterior density, are discussed. The proposed procedure is illustrated using psychological data with the results obtained from the OpenBUGS program.

A time‐series model for estimating temporal variation in phenotypic selection on laying dates in a Dutch great tit population

Abstract Temporal and spatial variation in phenotypic selection due to changing environmental conditions is of great interest to evolutionary biologists, but few existing methods estimating its magnitude take into account the temporal autocorrelation. We use state‐space models (SSMs) to analyse phenotypic selection processes that cannot be observed directly and use Template Model builder (TMB), an R package for computing and maximizing the Laplace approximation of the marginal likelihood for SSM and other complex, nonlinear latent variable model via automatic differentiation. Using a long‐term great tit (Parus major) dataset, we fit several SSMs and conduct model selection based on Akaike information criterion (AIC) to assess the support for fluctuated directional or autocorrelated stabilizing selection on breeding time of the great tit population. Our results show that there is directional selection on the probability of breeding failure, and stabilizing selection on the mean number of fledglings. This selection for early laying date is consistent with a previous study of the same population. We also estimate the variation and autocorrelation in other parameters of the fitness functions, including the width and height, and found the height and location of annual fitness function are autocorrelated with significant variation, while the width can be assumed to constant over time. Using TMB to fit SSMs, we are able to estimate additional parameters compared to previous methods, all without requiring a substantial increase in computational resources. Furthermore, our specification of complex nonlinear model structure benefits greatly from the flexibility of model formulation with TMB. Therefore, our approach could be directly applied to estimating even more complicated phenotypic selection processes induced by environmental change for other species.

The Intergenerational Transmission of Health in the United States: A Latent Variables Analysis

Social scientists have long documented that many components of socioeconomic status such as income and education have strong ties across generations. However, health status, arguably a more critical component of welfare, has largely been ignored. We fill this void by providing the first estimates of the Intergenerational Health Association (IHA) that are explicitly based on a non-linear latent variable model. Adjusting for only age and gender, we estimate an IHA of 0.3 indicating that about one third of a parent's health status gets transmitted to their children. Once we add additional mediators to the model, we show that education, and particularly children's education, is an important transmission channel in that it reduces the IHA by one third. Finally, we show that estimates of the IHA from non-linear models are only moderately higher than those from linear models, while rank-based mobility estimates are identical.

Probabilistic Nonlinear Soft Sensor Modeling Based on Generative Topographic Mapping Regression

Projection regression is an important tool for process soft sensing in order to eliminate redundant information and obtain proper data features. As most industrial process is intrinsically nonlinear and process variables are collected in random noise environment, it is significant to adopt probabilistic nonlinear latent variable model to carry out dimension reduction for feature extraction before regression modeling. Generative topographic mapping (GTM) is such a probabilistic nonlinear model. However, GTM is an unsupervised method, in which the extracted features may include irrelevant ones with the output information. Thus, it may result in inaccuracy of soft sensor performance. To deal with this problem, a generative topographic mapping regression is developed based on supervised GTM in this paper, which incorporates the output information to guide feature extraction and projection regression. By utilizing the output to jointly generate the latent variables, output-related features can be extracted for output prediction. The effectiveness and flexibility of the proposed method are validated on a numerical example and an industrial process.

Matrix completion by deep matrix factorization

Conventional methods of matrix completion are linear methods that are not effective in handling data of nonlinear structures. Recently a few researchers attempted to incorporate nonlinear techniques into matrix completion but there still exists considerable limitations. In this paper, a novel method called deep matrix factorization (DMF) is proposed for nonlinear matrix completion. Different from conventional matrix completion methods that are based on linear latent variable models, DMF is on the basis of a nonlinear latent variable model. DMF is formulated as a deep-structure neural network, in which the inputs are the low-dimensional unknown latent variables and the outputs are the partially observed variables. In DMF, the inputs and the parameters of the multilayer neural network are simultaneously optimized to minimize the reconstruction errors for the observed entries. Then the missing entries can be readily recovered by propagating the latent variables to the output layer. DMF is compared with state-of-the-art methods of linear and nonlinear matrix completion in the tasks of toy matrix completion, image inpainting and collaborative filtering. The experimental results verify that DMF is able to provide higher matrix completion accuracy than existing methods do and DMF is applicable to large matrices.

Deep learning based matrix completion

Previous matrix completion methods are generally based on linear and shallow models where the given incomplete matrices are of low-rank and the data are assumed to be generated by linear latent variable models. In this paper, we first propose a novel method called AutoEncoder based matrix completion (AEMC). The main idea of AEMC is to utilize the partially observed data to learn and construct a nonlinear latent variable model in the form of AutoEncoder. The hidden layer of the AutoEncoder has much fewer units than the visible layers do. Meanwhile, the unknown entries of the data are recovered to fit the nonlinear latent variable model. Based on AEMC, we further propose a deep learning based matrix completion (DLMC) method. In DLMC, AEMC is used as a pre-training step for both the missing entries and network parameters; the hidden layer of AEMC is then used to learn stacked AutoEncoders (SAEs) with greedy layer-wise training; finally, fine-tuning is carried out on the deep network formed by AEMC and SAEs to obtain the missing entries of the data and the parameters of the network. In addition, we also provide out-of-sample extensions for AEMC and DLMC to recover online incomplete data. AEMC and DLMC are compared with state-of-the-art methods in the tasks of synthetic matrix completion, image inpainting, and collaborative filtering. The experimental results verify the effectiveness and superiority of the proposed methods.

Fitting Nonlinear Structural Equation Models in R with Package nlsem

Structural equation mixture modeling (SEMM) has become a standard procedure in latent variable modeling over the last two decades (Jedidi, Jagpal, and DeSarbo 1997b; Muthén and Shedden 1999; Muthén 2001, 2004; Muthén and Asparouhov 2009). SEMM was proposed as a technique for the approximation of nonlinear latent variable relationships by finite mixtures of linear relationships (Bauer 2005, 2007; Bauer, Baldasaro, and Gottfredson 2012). In addition to this semiparametric approach to nonlinear latent variable modeling, there are numerous parametric nonlinear approaches for normally distributed variables (e.g., LMS in Mplus; Klein and Moosbrugger 2000). Recently, an additional semiparametric nonlinear structural equation mixture modeling (NSEMM) approach was proposed by Kelava, Nagengast, and Brandt (2014) that is capable of dealing with nonnormal predictors. In the nlsem package presented here, the SEMM, two distribution analytic (QML and LMS) and NSEMM approaches can be specified and estimated. We provide examples of how to use the package in the context of nonlinear latent variable modeling.

Bayesian Analysis of Linear and Nonlinear Latent Variable Models with Fixed Covariate and Ordered Categorical Data

In this paper, ordered categorical variables are used to compare between linear and nonlinear interactions of fixed covariate and latent variables Bayesian structural equation models. Gibbs sampling method is applied for estimation and model comparison. Hidden continuous normal distribution (censored normal distribution) is used to handle the problem of ordered categorical data. Statistical inferences, which involve estimation of parameters and their standard deviations, and residuals analyses for testing the selected model, are discussed. The proposed procedure is illustrated by a simulation data obtained from R program. Analysis are done by using OpenBUGS program.

Expert-Guided Generative Topographical Modeling with Visual to Parametric Interaction

Introduced by Bishop et al. in 1996, Generative Topographic Mapping (GTM) is a powerful nonlinear latent variable modeling approach for visualizing high-dimensional data. It has shown useful when typical linear methods fail. However, GTM still suffers from drawbacks. Its complex parameterization of data make GTM hard to fit and sensitive to slight changes in the model. For this reason, we extend GTM to a visual analytics framework so that users may guide the parameterization and assess the data from multiple GTM perspectives. Specifically, we develop the theory and methods for Visual to Parametric Interaction (V2PI) with data using GTM visualizations. The result is a dynamic version of GTM that fosters data exploration. We refer to the new version as V2PI-GTM. In this paper, we develop V2PI-GTM in stages and demonstrate its benefits within the context of a text mining case study.

Comparison Of Linear And Non-Linear Calibration Models For Non-Destructive Firmness Determining Of ‘Mazafati’ Date Fruit By Near Infrared Spectroscopy

The selection of calibration method is one of the most important factors affecting the measurement accuracy with near infrared spectroscopy. In this research, the performance of two general calibration methods, namely, linear partial least squares regression and nonlinear back propagation artificial neural networks for firmness predicting of ‘Mazafati’ date fruit was investigated. A total of 175 date samples harvested during fruit ripening were selected as the data set. Optical scanning was performed with a fiber optic near infrared spectrometer with a range of 900–1700 nm. The inputs of back propagation artificial neural networks were the first five principal components resulting from the principal component analysis, namely, principal component analysis artificial neural network modeling or the first six latent variables obtained from partial least squares regression, namely, latent variable artificial neural network modeling. Both the leave-one-out cross validation and test set validation showed the priority of latent variable artificial neural network model with respect to partial least squares and also principal component analysis artificial neural network models. The best latent variable artificial neural network model could predict the firmness of Mazafati date fruits with Rp 2 of 0.90, RMSEP of 1.30 N, and SDRp of 3.28. The results also recommended the adoption of nonlinear latent variable artificial neural network modeling for applying light scattering properties of fruits in order to predict indirect property of firmness with near infrared spectroscopy.

Constructing Generative Topographic Mapping by Variational Bayes with ARD Hierarchical Prior

Generative Topographic Mapping (GTM) is a nonlinear latent variable model introduced as a data visualization technique by Bishop et al. In this paper, we focus on variational Bayesian inference in GTM. Variational Bayesian GTM, first proposed by Olier et al., uses a single regularization term and regularization parameter to avoid overfitting and therefore cannot be used to control the degree of regularization locally. To overcome this problem, we propose variational Bayesian inference with Automatic Relevance Determination (ARD) hierarchical prior for use with GTM. The proposed model uses multiple regularization parameters and therefore can be used to control the degree of regularization in local areas of data space individually. Several experiments show that GTM that we propose provides better visualization than conventional GTM approaches.

Modeling of Travel Time Variations on Urban Links in London

An econometric framework was developed to combine data from various sources to identify the key factors contributing to travel time variations in Central London. Nonlinear latent variable regression models that explicitly accounted for measurement errors in the data were developed to combine data extracted from automatic number plate recognition cameras and automatic traffic counters. This procedure significantly differed from previous research in this area that was based primarily on traffic flow data and ignored measurement errors. The results indicate that nonlinear latent variable regression models can effectively explain travel time variations on a regular day by using variables related to vehicle type, traffic density, and traffic composition. Test results indicate that the proposed framework for correcting measurement errors yields significant improvements over base models, where such errors are ignored. The findings from the study validate some key hypotheses regarding influences of various factors on speed of urban traffic streams and can serve as a tool for investigation of the causes of traffic congestion. The model framework is general enough for application in other cases in which traffic data have similar measurement errors.