TLS-EM algorithm of Mixture Density Models for exponential families

Abstract

As a widely used model, Mixture Density Model (MDM) is traditionally solved by Expectation–Maximization (EM) algorithm. EM maximizes a lower bound function iteratively, especially for exponential families. This paper managed to improve EM by combining it with Total Least Squares (TLS), proposing a new algorithm called the TLS-EM algorithm. In this algorithm, parameters are divided into two groups, linear parameters and sub-model parameters. They are solved in each iteration separately. First, data set is separated in different intervals and the conditional maximizing question is transformed into the over-determined linear equations. TLS is adopted to solve these equations and calculate linear parameters, with sub-model parameters fixed. Second, sub-model parameters are solved with EM. Properties of TLS-EM have been provided with proofs. Combining the properties of TLS, EM and the properties of its own, TLS-EM not only inherits most advantages of EM but also improves it in most cases, especially in bad initial or bad model conditions. Numerical experiments confirm these properties.