The simulated online EM algorithm for latent factor models

Abstract

The estimation of latent factor models are treated in an integrated maximum likelihood context where one parameter is marginalized and another is estimated. An extension to the online Expectation Maximization (EM) algorithm is employed the simulated online Expectation Maximization algorithm. Both these algorithms apply to exponential family models, but the simulated version of the algorithm can make use of Monte Carlo simulation to compute the stochastic E-steps while maintaining the convergence properties of the original online EM algorithm. A class of important latent factor models are identified that can be expressed in complete data exponential family form, the algorithm is applied to one of these models Itakura-Saito Non-negative Matrix Factorisation. An additional parameter is introduced into this model and it is conjectured if this is set to a high value the posterior variance of the parameters is reduced and estimation becomes easier. Simulations are provided that support this conjecture, although online estimation for models with even a modest number of components continues to be hampered by the presence of local minima.