Distribution Of Latent Variables Research Articles

Score Matching for Compositional Distributions

Compositional data are challenging to analyse due to the non-negativity and sum-to-one constraints on the sample space. With real data, it is often the case that many of the compositional components are highly right-skewed, with large numbers of zeros. Major limitations of currently available models for compositional data include one or more of the following: insufficient flexibility in terms of distributional shape; difficulty in accommodating zeros in the data in estimation; and lack of computational viability in moderate to high dimensions. In this article, we propose a new model, the polynomially tilted pairwise interaction (PPI) model, for analysing compositional data. Maximum likelihood estimation is difficult for the PPI model. Instead, we propose novel score matching estimators, which entails extending the score matching approach to Riemannian manifolds with boundary. These new estimators are available in closed form and simulation studies show that they perform well in practice. As our main application, we analyse real microbiome count data with fixed totals using a multinomial latent variable model with a PPI model for the latent variable distribution. We prove that, under certain conditions, the new score matching estimators are consistent for the parameters in the new multinomial latent variable model.

Estimating Distributions of Parameters in Nonlinear State Space Models with Replica Exchange Particle Marginal Metropolis-Hastings Method.

Extracting latent nonlinear dynamics from observed time-series data is important for understanding a dynamic system against the background of the observed data. A state space model is a probabilistic graphical model for time-series data, which describes the probabilistic dependence between latent variables at subsequent times and between latent variables and observations. Since, in many situations, the values of the parameters in the state space model are unknown, estimating the parameters from observations is an important task. The particle marginal Metropolis–Hastings (PMMH) method is a method for estimating the marginal posterior distribution of parameters obtained by marginalization over the distribution of latent variables in the state space model. Although, in principle, we can estimate the marginal posterior distribution of parameters by iterating this method infinitely, the estimated result depends on the initial values for a finite number of times in practice. In this paper, we propose a replica exchange particle marginal Metropolis–Hastings (REPMMH) method as a method to improve this problem by combining the PMMH method with the replica exchange method. By using the proposed method, we simultaneously realize a global search at a high temperature and a local fine search at a low temperature. We evaluate the proposed method using simulated data obtained from the Izhikevich neuron model and Lévy-driven stochastic volatility model, and we show that the proposed REPMMH method improves the problem of the initial value dependence in the PMMH method, and realizes efficient sampling of parameters in the state space models compared with existing methods.

Transfer learning for regression via latent variable represented conditional distribution alignment

Since labelled data are expensive to generate both computationally and experimentally, how to establish data-driven models with limited data has become an important challenge in various engineering and scientific applications. Transfer learning has been used to improve the performance of many tasks by leveraging rich labelled data from related domains. This paper focuses on the transfer learning problem for regression under the situation of conditional distribution shift, which is a common scenario in manufacturing industry and has not been fully studied. Hence, we first propose a Latent variable represented Conditional Distribution Alignment method (LCDA), which exploits the low-dimensional latent representation to describe the global conditional distribution discrepancy. Then the properties of the latent variables are optimised by matching the kernel embedding conditional distribution between domains, so that the global distribution discrepancy can be aligned with the residual function generated by the latent variables. By combining domain priors with machine learning, the proposed method provides a potential direction to reduce labelling consumption for the intelligent manufacturing. Series of experiments on two industrial applications are conducted to validate the effectiveness and analyse the characteristics of the proposed method.

Bayesian longitudinal item response modeling with multivariate asymmetric serial dependencies

It is usually impossible to impose experimental conditions in large-scale longitudinal (observational) studies in education. This increases the risk of bias due to for instance unobserved heterogeneity, missing background variables, and dropouts. A flexible statistical model is required for the nature of the observational assessment data and to account for the unexplained heterogeneity. A general class of longitudinal item response theory (IRT) models is proposed, where growth in performance can be monitored using a skewed multivariate normal distribution for the latent variables. Change in performance and unexplained heterogeneity is addressed through structured covariance patterns and skewed multivariate latent variable distributions. The Cholesky decomposition of the covariance matrix is considered to model the dependence structure. A novel multivariate skew-normal distribution is defined by the antedependence model with centered skew-normal distributed errors. A hybrid MCMC approach is developed for parameter estimation, model-fit assessment, and model comparison. Results of simulation studies show good parameter recovery. A longitudinal assessment study by the Brazilian federal government is considered to show the performance of the general LIRT model.

CLASS-IMBALANCED CLASSIFIERS USING ENSEMBLES OF GAUSSIAN PROCESSES AND GAUSSIAN PROCESS LATENT VARIABLE MODELS.

Classification with imbalanced data is a common and challenging problem in many practical machine learning problems. Ensemble learning is a popular solution where the results from multiple base classifiers are synthesized to reduce the effect of a possibly skewed distribution of the training set. In this paper, binary classifiers based on Gaussian processes are chosen as bases for inferring the predictive distributions of test latent variables. We apply a Gaussian process latent variable model where the outputs of the Gaussian processes are used for making the final decision. The tests of the new method in both synthetic and real data sets show improved performance over standard approaches.

Use of the Lagrange Multiplier Test for Assessing Measurement Invariance Under Model Misspecification.

This article studies the Type I error, false positive rates, and power of four versions of the Lagrange multiplier test to detect measurement noninvariance in item response theory (IRT) models for binary data under model misspecification. The tests considered are the Lagrange multiplier test computed with the Hessian and cross-product approach, the generalized Lagrange multiplier test and the generalized jackknife score test. The two model misspecifications are those of local dependence among items and nonnormal distribution of the latent variable. The power of the tests is computed in two ways, empirically through Monte Carlo simulation methods and asymptotically, using the asymptotic distribution of each test under the alternative hypothesis. The performance of these tests is evaluated by means of a simulation study. The results highlight that, under mild model misspecification, all tests have good performance while, under strong model misspecification, the tests performance deteriorates, especially for false positive rates under local dependence and power for small sample size under misspecification of the latent variable distribution. In general, the Lagrange multiplier test computed with the Hessian approach and the generalized Lagrange multiplier test have better performance in terms of false positive rates while the Lagrange multiplier test computed with the cross-product approach has the highest power for small sample sizes. The asymptotic power turns out to be a good alternative to the classic empirical power because it is less time consuming. The Lagrange tests studied here have been also applied to a real data set.

Dirichlet Process Prior for Student’s t Graph Variational Autoencoders

Graph variational auto-encoder (GVAE) is a model that combines neural networks and Bayes methods, capable of deeper exploring the influential latent features of graph reconstruction. However, several pieces of research based on GVAE employ a plain prior distribution for latent variables, for instance, standard normal distribution (N(0,1)). Although this kind of simple distribution has the advantage of convenient calculation, it will also make latent variables contain relatively little helpful information. The lack of adequate expression of nodes will inevitably affect the process of generating graphs, which will eventually lead to the discovery of only external relations and the neglect of some complex internal correlations. In this paper, we present a novel prior distribution for GVAE, called Dirichlet process (DP) construction for Student’s t (St) distribution. The DP allows the latent variables to adapt their complexity during learning and then cooperates with heavy-tailed St distribution to approach sufficient node representation. Experimental results show that this method can achieve a relatively better performance against the baselines.

Estimating Net Primary Productivity (NPP) and Debris-Fall in Forests Using Lidar Time Series

Temporal series of lidar, properly field-validated, can provide critical information allowing in-ferences about the dynamics of biomass and carbon in forest canopies. Forest canopies gain carbon through net primary production (NPP) and lose carbon through canopy component damage and death, such as fine and coarse woody debris and litterfall (collectively, debris-fall). We describe a statistical method to extract gamma distributions of NPP and debris-fall rates in forest canopies from lidar missions repeated through time and we show that the means of these distributions covary with ecologically meaningful variables: topography, canopy structure, and taxonomic composition. The method employed is the generalized method of moments that applies the R package gmm to uncover the distribution of latent variables. We present an example with eco-logical interpretations that support the method’s application to change in biomass estimated for a boreal forest in southcentral Alaska. The deconvolution of net change from remote sensing products as distributions of NPP and debris-fall rates can inform carbon cycling models of can-opy-level NPP and debris-fall rates.

Interpretable Variational Graph Autoencoder with Noninformative Prior

Variational graph autoencoder, which can encode structural information and attribute information in the graph into low-dimensional representations, has become a powerful method for studying graph-structured data. However, most existing methods based on variational (graph) autoencoder assume that the prior of latent variables obeys the standard normal distribution which encourages all nodes to gather around 0. That leads to the inability to fully utilize the latent space. Therefore, it becomes a challenge on how to choose a suitable prior without incorporating additional expert knowledge. Given this, we propose a novel noninformative prior-based interpretable variational graph autoencoder (NPIVGAE). Specifically, we exploit the noninformative prior as the prior distribution of latent variables. This prior enables the posterior distribution parameters to be almost learned from the sample data. Furthermore, we regard each dimension of a latent variable as the probability that the node belongs to each block, thereby improving the interpretability of the model. The correlation within and between blocks is described by a block–block correlation matrix. We compare our model with state-of-the-art methods on three real datasets, verifying its effectiveness and superiority.

Examining Nonnormal Latent Variable Distributions for Non-Ignorable Missing Data.

Missing not at random (MNAR) modeling for non-ignorable missing responses usually assumes that the latent variable distribution is a bivariate normal distribution. Such an assumption is rarely verified and often employed as a standard in practice. Recent studies for "complete" item responses (i.e., no missing data) have shown that ignoring the nonnormal distribution of a unidimensional latent variable, especially skewed or bimodal, can yield biased estimates and misleading conclusion. However, dealing with the bivariate nonnormal latent variable distribution with present MNAR data has not been looked into. This article proposes to extend unidimensional empirical histogram and Davidian curve methods to simultaneously deal with nonnormal latent variable distribution and MNAR data. A simulation study is carried out to demonstrate the consequence of ignoring bivariate nonnormal distribution on parameter estimates, followed by an empirical analysis of "don't know" item responses. The results presented in this article show that examining the assumption of bivariate nonnormal latent variable distribution should be considered as a routine for MNAR data to minimize the impact of nonnormality on parameter estimates.

Macroeconomic Forecasting with Neural Network Reinforced Factor Models

In many macroeconomic forecasting applications factor models are used to cope with large datasets. This study aligns variational autoencoders with macroeconomic factor modeling and proposes an extension to adapt this framework for forecasting exercises. Variational autoencoders are well suited for nonlinear dimensionality reduction. They estimate the distribution of the common latent variables by combining a statistical factor model with a purely data-driven neural network approach. It is demonstrated that the resulting deep factor model can be interpreted as a flexible nonlinear extension of the standard factor model. In the empirical part, it is analyzed whether factor models augmented by neural networks can achieve superior forecasting power. The results suggest significant improvements in the forecasting accuracy of four major US macroeconomic time series.

Bayesian Learning for Deep Neural Network Adaptation

A key task for speech recognition systems is to reduce the mismatch between training and evaluation data that is often attributable to speaker differences. Speaker adaptation techniques play a vital role to reduce the mismatch. Model-based speaker adaptation approaches often require sufficient amounts of target speaker data to ensure robustness. When the amount of speaker level data is limited, speaker adaptation is prone to overfitting and poor generalization. To address the issue, this paper proposes a full Bayesian learning based DNN speaker adaptation framework to model speaker-dependent (SD) parameter uncertainty given limited speaker specific adaptation data. This framework is investigated in three forms of model based DNN adaptation techniques: Bayesian learning of hidden unit contributions (BLHUC), Bayesian parameterized activation functions (BPAct), and Bayesian hidden unit bias vectors (BHUB). In the three methods, deterministic SD parameters are replaced by latent variable posterior distributions for each speaker, whose parameters are efficiently estimated using a variational inference based approach. Experiments conducted on 300-hour speed perturbed Switchboard corpus trained LF-MMI TDNN/CNN-TDNN systems suggest the proposed Bayesian adaptation approaches consistently outperform the deterministic adaptation on the NIST Hub5'00 and RT03 evaluation sets. When using only the first five utterances from each speaker as adaptation data, significant word error rate reductions up to 1.4% absolute (7.2% relative) were obtained on the CallHome subset. The efficacy of the proposed Bayesian adaptation techniques is further demonstrated in a comparison against the state-of-the-art performance obtained on the same task using the most recent systems reported in the literature.

Replica Exchange Particle-Gibbs Method with Ancestor Sampling

Particle-Gibbs (PG) method is a Markov chain Monte Carlo (MCMC) method for sampling from the joint posterior distribution of latent variables and parameters in a nonlinear state space model, which ...

Testing Latent Variable Distribution Fit in IRT Using Posterior Residuals

This research proposes a new statistic for testing latent variable distribution fit for unidimensional item response theory (IRT) models. If the typical assumption of normality is violated, then item parameter estimates will be biased, and dependent quantities such as IRT score estimates will be adversely affected. The proposed statistic compares the specified latent variable distribution to the sample average of latent variable posterior distributions commonly used in IRT scoring. Formally, the statistic is an instantiation of a generalized residual and is thus asymptotically distributed as standard normal. Also, the statistic naturally complements residual-based item-fit statistics, as both are conditional on the latent trait, and can be presented with graphical plots. In addition, a corresponding unconditional statistic, which controls for multiple comparisons, is proposed. The statistics are evaluated using a simulation study, and empirical analyses are provided.

A Primer on Machine Learning.

A Primer on Machine Learning.

An Efficient Image Categorization Method With Insufficient Training Samples.

Image classification is an important part of pattern recognition. With the development of convolutional neural networks (CNNs), many CNN methods are proposed, which have a large number of samples for training, which can have high performance. However, there may exist limited samples in some real-world applications. In order to improve the performance of CNN learning with insufficient samples, this article proposes a new method called the classifier method based on a variational autoencoder (CFVAE), which is comprised of two parts: 1) a standard CNN as a prior classifier and 2) a CNN based on variational autoencoder (VAE) as a posterior classifier. First, the prior classifier is utilized to generate the prior label and information about distributions of latent variables; and the posterior classifier is trained to augment some latent variables from regularized distributions to improve the performance. Second, we also present the uniform objective function of CFVAE and put forward an optimization method based on the stochastic gradient variational Bayes method to solve the objective model. Third, we analyze the feasibility of CFVAE based on Hoeffding's inequality and Chernoff's bounding method. This analysis indicates that the latent variables augmentation method based on regularized latent variables distributions can generate samples fitting well with the distribution of data such that the proposed method can improve the performance of CNN with insufficient samples. Finally, the experiments manifest that our proposed CFVAE can provide more accurate performance than state-of-the-art methods.

Wasserstein autoencoders for collaborative filtering

The recommender systems have long been studied in the literature. The collaborative filtering is one of the most widely adopted recommendation techniques which is usually applied to the explicit data, e.g., rating scores. However, the implicit data, e.g., click data, is believed to be able to discover user’s latent preferences. Consequently, a number of research attempts have been made toward this issue. To the best of our knowledge, this paper is the first attempt to adapt the Wasserstein autoencoders to collaborative filtering problem. Particularly, we propose a new loss function by introducing an $$L_1$$ regularization term to learn a sparse low-rank representation form to represent latent variables. Then, we carefully design (1) a new cost function to minimize the data reconstruction error, and (2) the appropriate distance metrics for the calculation of KL divergence between the learned distribution of latent variables and the underlying true data distribution. Rigorous experiments are performed on three widely adopted datasets. Both the state-of-the-art approaches, e.g., Mult-VAE and Mult-DAE, and the baseline models are evaluated on these datasets. The promising experimental results demonstrate that the proposed approach is superior to the compared approaches with respect to evaluation criteria Recall@R and NDCG@R.

A novel quadruple generative adversarial network for semi-supervised categorization of low-resolution images

In order to make utilization of unlabeled low-resolution (LR) images to shape discriminative models, we present quadruple generative adversarial network (Q-GAN), a game-theoretical framework for implementing semi-supervised categorization of LR images. It can realize photo-realistic image super-resolution (SR) and semi-supervised pattern recognition simultaneously. We consider our pipeline as a four-player optimization-based formulation, which consists of four vital components, i.e., a refiner for image SR and generation, a discriminator for identifying high-resolution (HR) samples and another for identifying true (original) samples, a classifier for label prediction. The refiner and two discriminators characterize the conditional distributions between images and labels, whilst the classifier solely focuses on predicting real image-label pairs. We select those high-quality super-solved images with ground-truth labels for data supplement. We customize the global optimization objective function as well as the training procedure to ensure model approximates the posterior distribution of latent variables given true data in a semi-supervised manner. Experimental results demonstrate that Q-GAN can simultaneously (1) deliver the promising categorization performance among state-of-the-arts, i.e., validation accuracy achieves 92.18% and testing accuracy achieves 90.63%, and (2) recover fine-grained textures with high peak signal-to-noise ratios (PNSRs) and structural similarities (SSIMs) from heavily downsampled testing images of hand-crafted dataset and public benchmarks.

Knowledge-aware Attentive Wasserstein Adversarial Dialogue Response Generation

Natural language generation has become a fundamental task in dialogue systems. RNN-based natural response generation methods encode the dialogue context and decode it into a response. However, they tend to generate dull and simple responses. In this article, we propose a novel framework, called KAWA-DRG (Knowledge-aware Attentive Wasserstein Adversarial Dialogue Response Generation) to model conversation-specific external knowledge and the importance variances of dialogue context in a unified adversarial encoder-decoder learning framework. In KAWA-DRG, a co-attention mechanism attends to important parts within and among context utterances with word-utterance-level attention. Prior knowledge is integrated into the conditional Wasserstein auto-encoder for learning the latent variable space. The posterior and prior distribution of latent variables are generated and trained through adversarial learning. We evaluate our model on Switchboard, DailyDialog, In-Car Assistant, and Ubuntu Dialogue Corpus. Experimental results show that KAWA-DRG outperforms the existing methods.

Multiview learning with variational mixtures of Gaussian processes

Gaussian processes (GPs) are powerful Bayesian nonparametric tools widely used in probabilistic modeling, and the mixture of GPs (MGPs) were introduced afterwards to make data modeling more flexible. However, MGPs are not directly applicable to multiview learning. In order to improve the modeling ability of MGPs, in this paper, we propose a new framework of multiview learning for the MGPs and instantiate it for classification. We make the divergence between views as small as possible while ensuring that the posterior probability of each view is as large as possible. Specifically, we regularize the posterior distribution of latent variables with the consistency of posterior distributions of the latent functions between different views. Since it is intractable to solve the model analytically, the variational inference and optimization algorithms of the classification model are also presented in this paper. Experimental results on multiple real-world datasets have shown that the proposed method has outperformed the original MGP model and several state-of-the-art multiview learning methods, which indicate the effectiveness of the proposed multiview learning framework for MGPs.