Deterministic Annealing EM Research Articles

Sparse Count Data Clustering Using an Exponential Approximation to Generalized Dirichlet Multinomial Distributions.

Clustering frequency vectors is a challenging task on large data sets considering its high dimensionality and sparsity nature. Generalized Dirichlet multinomial (GDM) distribution is a competitive generative model for count data in terms of accuracy, yet its parameters estimation process is slow. The exponential-family approximation of the multivariate Polya distribution has shown to be efficient to train and cluster data directly, without dimensionality reduction. In this article, we derive an exponential-family approximation to the GDM distributions, and we call it (EGDM). A mixture model is developed based on the new member of the exponential-family of distributions, and its parameters are learned through the deterministic annealing expectation-maximization (DAEM) approach as a new clustering algorithm for count data. Moreover, we propose to estimate the optimal number of EGDM mixture components based on the minimum message length (MML) criterion. We have conducted a set of empirical experiments, concerning text, image, and video clustering, to evaluate the proposed approach performance. Results show that the new model attains a superior performance, and it is considerably faster than the corresponding method for GDM distributions.

Exact fisher information of generalized Dirichlet multinomial distribution for count data modeling

Despite the multiple benefits of the Fisher information matrix, it is generally disregarded and substituted by the identity matrix or an approximation format. However, when dealing with complicated real-world applications, ignoring the correlation between data features may compromise the modeling capability. To address this problem we present the exact calculation of the Fisher information matrix (EFIM) for the generalized Dirichlet multinomial (GDM) mixture that has proven its efficiency when modeling count data. We present a parametrization of GDM mixture model that allows the determination of the Fisher matrix’s elements by means of the Beta-binomial probability function. We also propose a novel count data modeling approach with the benefit of EFIM. In particular, we tackle the problem of mixture model estimation and selection using the Fisher scoring algorithm and minimum message length within the Deterministic Annealing Expectation-Maximization learning framework. Experiments on detecting depression in tweets, dialogue-based emotion recognition, and image-based sentiment analysis confirm the capability of the proposed approach and the merits of using the EFIM as compared with existing state-of-the-art methods and techniques that ignore the full determination of the Fisher information matrix’s elements.

Selective Linear Segmentation for Detecting Relevant Parameter Changes

Abstract Change-point (CP) processes are one flexible approach to model long time series. We propose a method to uncover which model parameters truly vary when a CP is detected. Given a set of breakpoints, we use a penalized likelihood approach to select the best set of parameters that changes over time and we prove that the penalty function leads to a consistent selection of the true model. Estimation is carried out via the deterministic annealing expectation-maximization algorithm. Our method accounts for model selection uncertainty and associates a probability to all the possible time-varying parameter specifications. Monte Carlo simulations highlight that the method works well for many time series models including heteroskedastic processes. For a sample of fourteen hedge fund (HF) strategies, using an asset-based style pricing model, we shed light on the promising ability of our method to detect the time-varying dynamics of risk exposures as well as to forecast HF returns.

Selective Linear Segmentation For Detecting Relevant Parameter Changes

Change-point processes are one flexible approach to model long time series. We propose a method to uncover which model parameter truly vary when a change-point is detected. Given a set of breakpoints, we use a penalized likelihood approach to select the best set of parameters that changes over time and we prove that the penalty function leads to a consistent selection of the true model. Estimation is carried out via the deterministic annealing expectation-maximization algorithm. Our method accounts for model selection uncertainty and associates a probability to all the possible time-varying parameter specifications. Monte Carlo simulations highlight that the method works well for many time series models including heteroskedastic processes. For a sample of 14 Hedge funds (HF) strategies, using an asset based style pricing model, we shed light on the promising ability of our method to detect the time-varying dynamics of risk exposures as well as to forecast HF returns.

Model selection and application to high-dimensional count data clustering

EDCM, the Exponential-family approximation to the Dirichlet Compound Multinomial (DCM), proposed by Elkan (2006), is an efficient statistical model for high-dimensional and sparse count data. EDCM models take into account the burstiness phenomenon correctly while being many times faster than DCM. This work proposes the use of Minimum Message Length (MML) criterion for determining the number of components that best describes the data with a finite EDCM mixture model. Parameters estimation is based on the previously proposed Deterministic Annealing Expectation-Maximization (DAEM) approach. The validation of the proposed unsupervised algorithm involves different real applications: text document modeling, topic novelty detection and hierarchical image clustering. A comparison with results obtained for other information-theory based selection criteria is provided.

On convergence and parameter selection of the EM and DA-EM algorithms for Gaussian mixtures

The expectation & maximization (EM) for Gaussian mixtures is popular as a clustering algorithm. However, the EM algorithm is sensitive to initial values, and so Ueda and Nakano [4] proposed the deterministic annealing EM (DA-EM) algorithm to improve it. In this paper, we investigate theoretical behaviors of the EM and DA-EM algorithms. We first derive a general Jacobian matrix of the DA-EM algorithm with respect to posterior probabilities. We then propose a theoretical lower bound for initialization of the annealing parameter in the DA-EM algorithm. On the other hand, some researches mentioned that the EM algorithm exhibits a self-annealing behavior, that is, the equal posterior probability with small random perturbations can avoid the EM algorithm to output the mass center for Gaussian mixtures. However, there is no theoretical analysis on this self-annealing property. Since the DA-EM will become the EM when the annealing parameter is 1, according to the Jacobian matrix of the DA-EM, we can prove the self-annealing property of the EM algorithm for Gaussian mixtures. Based on these results, we give not only convergence behaviors of the equal posterior probabilities and initialization lower bound of the temperature parameter of the DA-EM, but also a theoretical explanation why the EM algorithm for Gaussian mixtures exhibits a self-annealing behavior.

A parameter selection method of the deterministic anti-annealing algorithm for network exploring

The traditional expectation maximization (EM) algorithm for the mixture model can explore the structural regularities of a network efficiently. But it always traps into local maxima. A deterministic annealing EM (DAEM) algorithm is put forward to solve this problem. However, it brings about the problem of convergence speed. A deterministic anti-annealing expectation maximization (DAAEM) algorithm not only prevents poor local optima, but also improves the convergence speed. Thus, the DAAEM algorithm is used to estimate parameters of the mixture model. This algorithm always sets its initial parameter β0 by experience, which maybe get trapped into meaningless results due to too small β0, or converge to local maxima more frequently due to too large β0. A parameter selection method for β0 is designed. In our method, the convergence rate of the DAAEM algorithm for mixture model is first derived from Jacobian matrix of the posterior probabilities. Then the theoretical lower bound of β0 is computed based on the convergence rate at meaningless points. In our experiments we select β0 by rounding up the lower bound to the nearest tenth. Experiments on real and synthetic networks demonstrate that the parameter selection method is valid, and the performance of the DAAEM algorithm beginning from the selected parameter is better than the EM and DAEM algorithms for mixture model. In addition, we find that the convergence rate of the DAAEM algorithm is affected by assortative mixing by degree of a network.

Combining Active and Semisupervised Learning of Remote Sensing Data Within a Renyi Entropy Regularization Framework

Active and semisupervised learning are related techniques aiming at reducing the effort of creating training sets for classification and regression tasks. In this work, we present a framework for combining these two techniques on the basis of Renyi entropy regularization, enabling a synergy effect. We build upon the existing semisupervised learning model which attempts to balance the likelihood of labeled examples and the entropy of putative object probabilities within the unlabeled pool. To enable efficient optimization of the model, we generalize the deterministic annealing expectation–maximization (DAEM) algorithm, originally designed for Shannon entropy, to accommodate the use of Renyi entropies. The Renyi-regularized model is then applied to expected error reduction (EER), an active learning approach based on minimizing the entropy of unlabeled object probabilities. We investigate object preselection with a greedy approximation of the object feature matrix as a means to reduce computational complexity. To assess the performance of the proposed framework, we apply it to two real-world remote sensing problems with significantly different input data characteristics: detecting dead trees from color infrared aerial images (2-D) and detecting dead trunk stems in ALS point clouds (3-D). Our results show that for small training sets, the semisupervised Renyi-regularized classifier improves the classification rate by up to 11% and 10% points compared to the unregularized baseline for ALS and image data, respectively. This gain carries over to active learning, where the regularized EER achieves 90% of the final classification performance using 50% and 70% of the number of queries required by standard EER.

Emotional speech acoustic model for Malay: Iterative versus isolated unit training

The ability of speech synthesis system to synthesize emotional speech enhances the user's experience when using this kind of system and its related applications. However, the development of an emotional speech synthesis system is a daunting task in view of the complexity of human emotional speech. The more recent state-of-the-art speech synthesis systems, such as the one based on hidden Markov models, can synthesize emotional speech with acceptable naturalness with the use of a good emotional speech acoustic model. However, building an emotional speech acoustic model requires adequate resources including segment-phonetic labels of emotional speech, which is a problem for many under-resourced languages, including Malay. This research shows how it is possible to build an emotional speech acoustic model for Malay with minimal resources. To achieve this objective, two forms of initialization methods were considered: iterative training using the deterministic annealing expectation maximization algorithm and the isolated unit training. The seed model for the automatic segmentation is a neutral speech acoustic model, which was transformed to target emotion using two transformation techniques: model adaptation and context-dependent boundary refinement. Two forms of evaluation have been performed: an objective evaluation measuring the prosody error and a listening evaluation to measure the naturalness of the synthesized emotional speech.

Application of deterministic annealing EM algorithm to MAP/PH parameter estimation

This paper proposes a variant of EM (expectation-maximization) algorithm for Markovian arrival process (MAP) and phase-type distribution (PH) parameter estimation. Especially, we derive the deterministic annealing EM (DAEM) algorithm for MAP/PH parameter estimation. The DAEM algorithm is one of the methods to overcome a local maxima problem associated with the conventional EM algorithm. This paper derives concrete E- and M-step formulas for MAP parameter estimation from inter-arrival time data and PH parameter estimation from point samples in the framework of DAEM algorithm. Numerical examples demonstrate the DAEM algorithm for Markov-modulated Poisson process (MMPP) and several classes of PH distribution.

A Bayesian Framework Using Multiple Model Structures for Speech Recognition

This paper proposes an acoustic modeling technique based on Bayesian framework using multiple model structures for speech recognition. The aim of the Bayesian approach is to obtain good prediction of observation by marginalizing all variables related to generative processes. Although the effectiveness of marginalizing model parameters was recently reported in speech recognition, most of these systems use only “one” model structure, e.g., topologies of HMMs, the number of states and mixtures, types of state output distributions, and parameter tying structures. However, it is insufficient to represent a true model distribution, because a family of such models usually does not include a true distribution in most practical cases. One of solutions of this problem is to use multiple model structures. Although several approaches using multiple model structures have already been proposed, the consistent integration of multiple model structures based on the Bayesian approach has not seen in speech recognition. This paper focuses on integrating multiple phonetic decision trees based on the Bayesian framework in HMM based acoustic modeling. The proposed method is derived from a new marginal likelihood function which includes the model structures as a latent variable in addition to HMM state sequences and model parameters, and the posterior distributions of these latent variables are obtained using the variational Bayesian method. Furthermore, to improve the optimization algorithm, the deterministic annealing EM (DAEM) algorithm is applied to the training process. The proposed method effectively utilizes multiple model structures, especially in the early stage of training and this leads to better predictive distributions and improvement of recognition performance.

A Hybrid Algorithm with GA and DAEM

Although the expectation-maximization (EM) algorithm has been widely used for finding maximum likelihood estimation of parameters in probabilistic models, it has the problem of trapping by local maxima. To overcome this problem, the deterministic annealing EM (DAEM) algorithm was once proposed and had achieved better performance than EM algorithm, but it is not very effective at avoiding local maxima. In this paper, a solution is proposed by integrating GA and DAEM into one procedure to further improve the solution quality. The population based search of genetic algorithm will produce different solutions and thus can increase the search space of DAEM. Therefore, the proposed algorithm will reach better solution than just using DAEM. The algorithm retains the property of DAEM and gets the better solution by genetic operation. Experiment results on Gaussian mixture model parameter estimation demonstrate that the proposed algorithm can achieve better performance.

Advanced Probabilistic Load Flow Technique in Consideration of Non-Gaussianity and Nodal Correlation of Input Variables

AbstractThis paper proposes a new probabilistic load flow (PLF) method that considers the non-Gaussianity and the correlation of nodal input variables. The load flow calculation plays an important role to evaluate power system conditions. In recent years, power systems become more complicated due to renewable energy such as wind power generators and PV systems. As a result, the impotence of PLF has been reevaluated as one of the promising techniques for handling the network uncertainties. This paper focuses on the applications of Monte Carlo Simulation (MCS) to PLF so that the effect of the correlations of input variables and the network uncertainties is evaluated through the nonlinear equation. In this paper, new techniques are introduced into MCS-PLF to evaluate more accurate solutions. The maximum likelihood estimation for a probability density function (PDF) of input variables is carried out to construct the non-Gaussian distribution model of input variables with the correlation by the Deterministic Annealing Expectation Maximization (DAEM) algorithm. Also, the Metropolis-Hastings sampling is used to generate random numbers that correspond to complicated multivariate probability density functions. The proposed method is successively applied to a system with wind power generators.

Speech recognition based on statistical models including multiple phonetic decision trees

We propose a speech recognition technique using multiple model structures. In the use of context-dependent models, decision-tree-based context clustering is applied to find an appropriate parameter tying structure. However, context clustering is usually performed on the basis of unreliable statistics of hidden Markov model (HMM) state sequences because the estimation of reliable state sequences requires an appropriate model structures, that cannot be obtained prior to context clustering. Therefore, context clustering and the estimation of state sequences essentially cannot be performed independently. To overcome this problem, we propose an optimization technique of state sequences based on an annealing process using multiple decision trees. In this technique, a new likelihood function is defined in order to treat multiple model structures, and the deterministic annealing expectation maximization algorithm is used as the training algorithm. Experimental continuous phoneme recognition results show that the proposed method of using only two decision trees achieved about an 11.1% relative error reduction over the conventional method.

Count Data Modeling and Classification Using Finite Mixtures of Distributions

In this paper, we consider the problem of constructing accurate and flexible statistical representations for count data, which we often confront in many areas such as data mining, computer vision, and information retrieval. In particular, we analyze and compare several generative approaches widely used for count data clustering, namely multinomial, multinomial Dirichlet, and multinomial generalized Dirichlet mixture models. Moreover, we propose a clustering approach via a mixture model based on a composition of the Liouville family of distributions, from which we select the Beta-Liouville distribution, and the multinomial. The novel proposed model, which we call multinomial Beta-Liouville mixture, is optimized by deterministic annealing expectation-maximization and minimum description length, and strives to achieve a high accuracy of count data clustering and model selection. An important feature of the multinomial Beta-Liouville mixture is that it has fewer parameters than the recently proposed multinomial generalized Dirichlet mixture. The performance evaluation is conducted through a set of extensive empirical experiments, which concern text and image texture modeling and classification and shape modeling, and highlights the merits of the proposed models and approaches.

On Semi-Supervised Learning Genetic-Based and Deterministic Annealing EM Algorithm for Dirichlet Mixture Models

We propose a genetic-based and deterministic annealing expectation-maximization (GA&DA-EM) algorithm for learning Dirichlet mixture models from multivariate data. This algorithm is capable of selecting the number of components of the model using the minimum description length (MDL) criterion. Our approach benefits from the properties of Genetic algorithms and deterministic annealing algorithm by combination of both into a single procedure. The population-based stochastic search of the GA&DA explores the search space more thoroughly than the EM method. Therefore, our algorithm enables escaping from local optimal solutions since the algorithm becomes less sensitive to its initialization. The GA&DA-EM algorithm is elitist which maintains the monotonic convergence property of the EM algorithm. We conducted experiments on the WebKB and 20NEWSGROUPS. The results show that show that 1) the GA&DA-EM outperforms the EM method since: Our approach identifies the number of components which were used to generate the underlying data more often than the EM algorithm. 2) the algorithm alternatives to EM that overcoming the challenges of local maxima.

Clustering of protein domains for functional and evolutionary studies

BackgroundThe number of protein family members defined by DNA sequencing is usually much larger than those characterised experimentally. This paper describes a method to divide protein families into subtypes purely on sequence criteria. Comparison with experimental data allows an independent test of the quality of the clustering.ResultsAn evolutionary split statistic is calculated for each column in a protein multiple sequence alignment; the statistic has a larger value when a column is better described by an evolutionary model that assumes clustering around two or more amino acids rather than a single amino acid. The user selects columns (typically the top ranked columns) to construct a motif. The motif is used to divide the family into subtypes using a stochastic optimization procedure related to the deterministic annealing EM algorithm (DAEM), which yields a specificity score showing how well each family member is assigned to a subtype. The clustering obtained is not strongly dependent on the number of amino acids chosen for the motif. The robustness of this method was demonstrated using six well characterized protein families: nucleotidyl cyclase, protein kinase, dehydrogenase, two polyketide synthase domains and small heat shock proteins. Phylogenetic trees did not allow accurate clustering for three of the six families.ConclusionThe method clustered the families into functional subtypes with an accuracy of 90 to 100%. False assignments usually had a low specificity score.

Integrating spatial and color information in images using a statistical framework

Color histograms have been widely used successfully in many computer vision and image processing applications. However, they do not include any spatial information. In this paper, we propose a statistical model to integrate both color and spatial information. Our model is based on finite multiple-Bernoulli mixtures. For the estimation of the model’s parameters, we use a maximum a posteriori (MAP) approach through deterministic annealing expectation maximization (DAEM). Smoothing priors on the components parameters are introduced to stabilize the estimation. The selection of the number of clusters is based on stochastic complexity. The results show that our model achieves good performance in some image classification problems.

Model-Based Clustering by Probabilistic Self-Organizing Maps

In this paper, we consider the learning process of a probabilistic self-organizing map (PbSOM) as a model-based data clustering procedure that preserves the topological relationships between data clusters in a neural network. Based on this concept, we develop a coupling-likelihood mixture model for the PbSOM that extends the reference vectors in Kohonen's self-organizing map (SOM) to multivariate Gaussian distributions. We also derive three expectation-maximization (EM)-type algorithms, called the SOCEM, SOEM, and SODAEM algorithms, for learning the model (PbSOM) based on the maximum-likelihood criterion. SOCEM is derived by using the classification EM (CEM) algorithm to maximize the classification likelihood; SOEM is derived by using the EM algorithm to maximize the mixture likelihood; and SODAEM is a deterministic annealing (DA) variant of SOCEM and SOEM. Moreover, by shrinking the neighborhood size, SOCEM and SOEM can be interpreted, respectively, as DA variants of the CEM and EM algorithms for Gaussian model-based clustering. The experimental results show that the proposed PbSOM learning algorithms achieve comparable data clustering performance to that of the deterministic annealing EM (DAEM) approach, while maintaining the topology-preserving property.

A <formula formulatype="inline"><tex>$q$</tex></formula>-Parameterized Deterministic Annealing EM Algorithm Based on Nonextensive Statistical Mechanics

We propose a q-parameterized deterministic annealing expectation maximization (q-DAEM) algorithm for parameter estimation motivated by the concept of Tsallis entropy that originates from the nonextensive statistical mechanics. The q-DAEM algorithm combines the feature of annealing algorithms to reduce initialization sensitivity and that of q-EM algorithms to achieve fast convergence. The q-EM algorithm is a one-parameter generalized EM algorithm that has been previously proposed by the authors. By interpreting the EM algorithm via likelihood lower bound maximization, we build the fundamental interconnections among DAEM, q-DAEM, and statistical mechanics. To illustrate the benefits of using q-DAEM, we investigate two applications: finite mixture model estimation in data clustering; and joint channel estimation and data detection in communication systems, where we show that the q-DAEM algorithm achieves superior performance over the EM algorithm for both applications.