Dynamic Latent Variable Model Research Articles

Dirichlet Latent Variable Model: A Dynamic Model Based on Dirichlet Prior for Audio Processing

We propose a dynamic latent variable model for learning latent bases from time varying, non-negative data. We take a probabilistic approach to modeling the temporal dependence in data by introducing a dynamic Dirichlet prior—a Dirichlet distribution with dynamic parameters. This new distribution allows us to assure non-negativity and avoid intractability when sequential updates are performed otherwise encountered in using Dirichlet prior. We refer to the proposed model as the Dirichlet latent variable model DLVM. We develop an expectation maximization algorithm for the proposed model, and also derive a maximum a posteriori estimate of the parameters. Furthermore, we connect the proposed DLVM to two popular latent basis learning methods—probabilistic latent component analysis PLCA and non-negative matrix factorization NMF. We show that 1 PLCA is a special case of our DLVM, and 2 DLVM can be interpreted as a dynamic version of NMF. The usefulness of DLVM is demonstrated for three audio processing applications—speaker source separation, denoising, and bandwidth expansion. To this end, a new algorithm for source separation is also proposed. Through extensive experiments on benchmark databases, we show that the proposed model outperforms several relevant existing methods in all three applications.

A practical propagation path identification scheme for quality-related faults based on nonlinear dynamic latent variable model and partitioned Bayesian network

In this paper, a practical technology or solution of quality-related fault diagnosis is provided for nonlinear and dynamic process. Unlike traditional data-based fault diagnosis methods, the alternative approach is focused more on identifying the propagation path that combines diagnostic information and process knowledge. The new method addresses the quality-related fault detection issue with developed nonlinear dynamic latent variable model for extracting nonlinear latent variables that exhibit dynamic correlations, then the advantage of relative reconstruction based contribution approach is followed to analyze the potential root-cause variables. Meanwhile, a new partitioned Bayesian network methodology is proposed for propagation path identification of quality-related faults. Finally, the whole proposed framework is applied to a real hot strip mill process, where the effectiveness is further demonstrated from real industrial data.

Identification of robust probabilistic slow feature regression model for process data contaminated with outliers

Modeling of high dimensional dynamic process is considered as a challenging task. In this regard, probabilistic Slow Feature Analysis (PSFA), a dynamic latent variable model, is proven to be a useful tool which extracts temporally correlated dynamic features from the high-dimensional raw measurements. The extracted latent Slow Features (SFs) can capture process variations which are useful in developing dynamic models. Often times industrial data is affected by outliers, and modeling such data could result in inferior prediction performance. To deal with such scenarios, we propose a robust PSFA (RPSFA) based regression model that models outliers in the observation data using the Student's t-distribution. To estimate the parameters in RPSFA and to extract reduced dimension of SFs, we employ Expectation-Maximization (EM) algorithm under the Maximum Likelihood Estimation (MLE) framework considering SFs as hidden variables. To estimate the hidden SFs we propose a weighted gain Kalman filter based approach as the Normal distribution assumption of the observations is no longer valid. The validity and merits of the proposed approach are demonstrated though a simulated example, an industrial application and an experimental study.

Kernel dynamic latent variable model for process monitoring with application to hot strip mill process

Dynamic models are preferred rather than static models in the process monitoring of modern manufacturing. Compared with static models, dynamic models can reflect not only correlations but also causality among measurements and manipulated variables. Linear dynamic models are very common due to the simplicity of representation and parameter estimation. However, because of natural nonlinearity of a dynamic process, it is ineffective to apply linear models within a long term and varying condition. Nonlinear dynamic models are hence desired under such a circumstance. In this paper, a kernel dynamic latent variable (KDLV) model is proposed to describe the nonlinearity between original measurements and dynamic latent variables. This model is an extension of dynamic latent variable model in the aspect of nonlinearity, and keeps all merits of it. In order to build such a model, a KDLV search algorithm is proposed to acquire key model parameters from data, then a KDLV modeling procedure is derived to complete the whole model. After the KDLV model is trained from data, corresponding detection strategy is also developed to perform fault detection. The KDLV based fault detection is applied to the monitoring of hot strip mill process and comparison study is also conducted on both DLV and DKPCA models.

Monte Carlo profile confidence intervals for dynamic systems.

Monte Carlo methods to evaluate and maximize the likelihood function enable the construction of confidence intervals and hypothesis tests, facilitating scientific investigation using models for which the likelihood function is intractable. When Monte Carlo error can be made small, by sufficiently exhaustive computation, then the standard theory and practice of likelihood-based inference applies. As datasets become larger, and models more complex, situations arise where no reasonable amount of computation can render Monte Carlo error negligible. We develop profile likelihood methodology to provide frequentist inferences that take into account Monte Carlo uncertainty. We investigate the role of this methodology in facilitating inference for computationally challenging dynamic latent variable models. We present examples arising in the study of infectious disease transmission, demonstrating our methodology for inference on nonlinear dynamic models using genetic sequence data and panel time-series data. We also discuss applicability to nonlinear time-series and spatio-temporal data.

Autoregressive Dynamic Latent Variable Models for Process Monitoring

In most industrial processes, both autocorrelations and cross correlations in the data need to be extracted for the purpose of process monitoring and diagnosis. However, traditional dynamic modeling methods focus on the dynamic relationship while the cross correlations are at best implicit. In this brief, a new autoregressive dynamic latent variable model is proposed to capture both dynamic and static relationships simultaneously. The proposed method is a rather general dynamic model which can improve the performance of modeling and process monitoring. The Kalman filter and smoother are employed for inference while model parameters are estimated with an expectation-maximization algorithm. The corresponding fault detection method is also developed and a numerical example and the Tennessee Eastman benchmark process are used to evaluate the performance of the proposed model.

Exploring the Dynamics of Latent Variable Models

While latent variable models are increasingly common in political science research, there is little consensus on how these models are best adapted to accommodate time-series, cross-sectional data. In this paper, we investigate new approaches for modeling dynamic political processes. We begin by examining existing dynamic modeling strategies and outline the properties that make these models poorly suited to modeling time series data, particularly when the latent variables are subject to periods of volatility. We then develop two new models: a robust dynamic latent variable model and a finite mixture model that nests both the static and dynamic approaches. Simulation results indicate that the robust dynamic model outperforms all other modeling strategies. We then conduct replications of existing studies that model judicial preferences and democracy as latent traits. We find that the robust model performs as as well if not better than traditional models. In addition, for the case of judicial ideology, the robust model leads to new evidence about the strategic nature of judicial action.

Discriminating clinical phases of recovery from major depressive disorder using the dynamics of facial expression.

We used several metrics of variability to extract unsupervised features from video recordings of patients before and after deep brain stimulation (DBS) treatment for major depressive disorder (MDD). Our goal was to quantify the treatment effects on facial expressivity. Multiscale entropy (MSE) was used to capture the temporal variability in pixel intensity level at multiple time-scales. A dynamic latent variable model (DLVM) was used to learn a low dimensional (D = 20) set of dynamic factors that explain the observed covariance across the high-dimensional pixels (M = 30 × 30) within each video frame and across time. Our preliminary results indicate that unsupervised features learned from these video recordings can distinguish different phases of depression and recovery. The overarching goal of this research is to develop more refined markers of clinical response to treatment for depression.

Dynamic time warping based causality analysis for root-cause diagnosis of nonstationary fault processes

It is very important to diagnose abnormal events in industrial processes. Based on normal operating data in a dynamic process, dynamic latent variable model provides a clear view of separating dynamic and static variations. Recent work by Li et al. (2014a) has shown an effective diagnosis in faulty variables with multidirectional reconstruction based contributions. Their further work took Granger causality analysis into accounts to explore the casual relations instead of only correlations. Although Granger causality is a widely used method for many applications, it needs time series to be stationary to calculate the causality index, which is not applicable for nonstationary fault processes. In this paper, a new causality analysis index based on dynamic time warping is proposed to determine the causal direction between pairs of faulty variables. The case study on the Tennessee Eastman process with a step fault shows the effectiveness of the proposed approach.

A New Method of Dynamic Latent-Variable Modeling for Process Monitoring

Dynamic principal component analysis (DPCA) is widely used in the monitoring of dynamic multivariate processes. In traditional DPCA where a time window is used, the dynamic relations among process variables are implicit and difficult to interpret in terms of variables. To extract explicit latent variables that are dynamically correlated, a dynamic latent-variable model is proposed in this paper. The new structure can improve the modeling and the interpretation of dynamic processes and enhance the performance of monitoring. Fault detection strategies are developed, and contribution analysis is available for the proposed model. The case study on the Tennessee Eastman Process is used to illustrate the effectiveness of the proposed methods.

Uncertain Events: A Dynamic Latent Variable Model of Human Rights Respect and Government Killing with Binary, Ordered, and Count Outcomes

Counting repressive events is difficult because state leaders have an incentive to conceal the actions of their subordinates and destroy evidence of abuse. In this paper, I describe an existing latent variable model and then extend it to account for the uncertainty inherent in counting this type of difficult to observe event. To validate the model, I focus on one dataset, which defines one-sided government killing as government caused deaths of non-combatants. The model generates a more precise estimate of latent levels of repression for each country-year using several repression variables included in the model (1949-2010) and new estimates of the distribution of the number of individuals killed for each country-year in the original one-sided government killing dataset (1989-2010). These new event-based count estimates will be useful for researchers interested in this type of data but skeptical of the comparability of such events across countries and over time.

Efficient importance sampling in mixture frameworks

A flexible importance sampling procedure for the likelihood evaluation of dynamic latent variable models involving mixtures of distributions leading to possibly heavy tailed or multi-modal target densities is provided. The procedure is based upon the efficient importance sampling (EIS) approach and exploits the mixture structure of the model via data augmentation when constructing importance sampling distributions as mixtures of distributions. The proposed mixture EIS procedure is illustrated with ML estimation of a Student-t state space model for realized volatilities. MC simulations are used to characterize the sampling distribution of the ML estimator based upon the mixture EIS approach.

Estimating Party Positions across Countries and Time—A Dynamic Latent Variable Model for Manifesto Data

This article presents a new method for estimating positions of political parties across country- and time-specific contexts by introducing a latent variable model for manifesto data. We estimate latent positions and exploit bridge observations to make the scales comparable. We also incorporate expert survey data as prior information in the estimation process to avoid ex post facto interpretation of the latent space. To illustrate the empirical contribution of our method, we estimate the left-right positions of 388 parties competing in 238 elections across twenty-five countries and over sixty years. Compared to the puzzling volatility of existing estimates, we find that parties more modestly change their left-right positions over time. We also show that estimates without country- and time-specific bias parameters risk serious, systematic bias in about two-thirds of our data. This suggests that researchers should carefully consider the comparability of party positions across countries and/or time.

Estimation of dynamic latent variable models using simulated non‐parametric moments

Abstract. Given a model that can be simulated, conditional moments at a trial parameter value can be calculated with high accuracy by applying kernel smoothing methods to a long simulation. With such conditional moments in hand, standard method of moments techniques can be used to estimate the parameter. Because conditional moments are calculated using kernel smoothing rather than simple averaging, it is not necessary that the model be simulable subject to the conditioning information that is used to define the moment conditions. For this reason, the proposed estimator is applicable to general dynamic latent variable models. It is shown that as the number of simulations diverges, the estimator is consistent and a higher-order expansion reveals the stochastic difference between the infeasible GMM estimator based on the same moment conditions and the simulated version. In particular, we show how to adjust standard errors to account for the simulations. Monte Carlo results show how the estimator may be applied to a range of dynamic latent variable (DLV) models, and that it performs well in comparison to several other estimators that have been proposed for DLV models.

Efficient High-Dimensional Importance Sampling in Mixture Frameworks

This paper provides high-dimensional and flexible importance sampling procedures for the likelihood evaluation of dynamic latent variable models involving finite or infi nite mixtures leading to possibly heavy tailed and/or multi-modal target densities. Our approach is based upon the efficient importance sampling (EIS) approach of Richard and Zhang (2007) and exploits the mixture structure of the model when constructing importance sampling distributions as mixture of distributions. The proposed mixture EIS procedures are illustrated with ML estimation of a student-t state space model for realized volatilities and a stochastic volatility model with leverage effects and jumps for asset returns.

Dynamic latent variable modeling for statistical process monitoring

AbstractDynamic principal component analysis (DPCA) has been widely used in the monitoring of dynamic multivariate processes. In traditional DPCA, the dynamic relationship between process variables are implicit and hard to interpret. To extract explicit latent factors that are dynamically correlated, a new dynamic latent variable model is proposed. The new structure can improve modeling of dynamic data and enhance the process monitoring performance. Fault detection indices are developed based on the proposed model. A case study is given to illustrate the effectiveness of the proposed new dynamic factor model.

Estimation of Dynamic Latent Variable Models Using Simulated Nonparametric Moments

Given a model that can be simulated, conditional moments at a trial parameter value can be calculated with high accuracy by applying kernel smoothing methods to a long simulation. With such conditional moments in hand, standard method of moments techniques can be used to estimate the parameter. Because conditional moments are calculated using kernel smoothing rather than simple averaging, it is not necessary that the model be simulable subject to the conditioning information that is used to define the moment conditions. For this reason, the proposed estimator is applicable to general dynamic latent variable models. It is shown that as the number of simulations diverges, the estimator is consistent and a higher-order expansion reveals the stochastic difference between the infeasible GMM estimator based on the same moment conditions and the simulated version. In particular, we show how to adjust standard errors to account for the simulations. Monte Carlo results show how the estimator may be applied to a range of dynamic latent variable (DLV) models, and that it performs well in comparison to several other estimators that have been proposed for DLV models.

Indirect estimation of large conditionally heteroskedastic factor models, with an application to the Dow 30 stocks

We derive indirect estimators of conditionally heteroskedastic factor models in which the volatilities of common and idiosyncratic factors depend on their past unobserved values by calibrating the score of a Kalman-filter approximation with inequality constraints on the auxiliary model parameters. We also propose alternative indirect estimators for large-scale models, and explain how to apply our procedures to many other dynamic latent variable models. We analyse the small sample behaviour of our indirect estimators and several likelihood-based procedures through an extensive Monte Carlo experiment with empirically realistic designs. Finally, we apply our procedures to weekly returns on the Dow 30 stocks.

Bayesian Inference in Dynamic Disequilibrium Models: An Application to the Polish Credit Market

We review Bayesian inference for dynamic latent variable models using the data augmentation principle. We detail the difficulties of simulating dynamic latent variables in a Gibbs sampler. We propose an alternative specification of the dynamic disequilibrium model which leads to a simple simulation procedure and renders Bayesian inference fully operational. Identification issues are discussed. We conduct a specification search using the posterior deviance criterion of Spiegelhalter, Best, Carlin, and van der Linde (2002) for a disequilibrium model of the Polish credit market.

Kernel-Based Indirect Inference

The class of parametric dynamic latent variable models is becoming increasingly popular in finance and economics. Latent factor models, switching regimes models, stochastic volatility models, and dynamic disequilibrium models are only a few examples of this class of model. Inference in this class may be difficult since the computation of the likelihood function requires integrating out the unobservable components and calculating very high dimensional integrals. We propose an estimation procedure that could be applied to any dynamic latent model. The approach is based on the indirect inference principle and, in order to capture the dynamic features of these models, the binding functions are conditional expectations of functions of the endogenous variables given their past values. These conditional expectations are estimated by a nonparametric kernel-based approach. Unlike the indirect inference method, no optimization step is involved in the computation of the binding function and the approach is useful when no convenient auxiliary model is available. In spite of the nonparametric feature of the approach, the estimator is consistent and its convergence rate may be arbitrarily close to the classical parametric one. Moreover, a scoring method to select the best binding functions is proposed. Finally, some Monte Carlo experiments for factor ARCH and GARCH models show the feasibility of the approach.