Deterministic Annealing EM Algorithm Research Articles

Selective Linear Segmentation for Detecting Relevant Parameter Changes

Abstract Change-point (CP) processes are one flexible approach to model long time series. We propose a method to uncover which model parameters truly vary when a CP is detected. Given a set of breakpoints, we use a penalized likelihood approach to select the best set of parameters that changes over time and we prove that the penalty function leads to a consistent selection of the true model. Estimation is carried out via the deterministic annealing expectation-maximization algorithm. Our method accounts for model selection uncertainty and associates a probability to all the possible time-varying parameter specifications. Monte Carlo simulations highlight that the method works well for many time series models including heteroskedastic processes. For a sample of fourteen hedge fund (HF) strategies, using an asset-based style pricing model, we shed light on the promising ability of our method to detect the time-varying dynamics of risk exposures as well as to forecast HF returns.

Selective Linear Segmentation For Detecting Relevant Parameter Changes

Change-point processes are one flexible approach to model long time series. We propose a method to uncover which model parameter truly vary when a change-point is detected. Given a set of breakpoints, we use a penalized likelihood approach to select the best set of parameters that changes over time and we prove that the penalty function leads to a consistent selection of the true model. Estimation is carried out via the deterministic annealing expectation-maximization algorithm. Our method accounts for model selection uncertainty and associates a probability to all the possible time-varying parameter specifications. Monte Carlo simulations highlight that the method works well for many time series models including heteroskedastic processes. For a sample of 14 Hedge funds (HF) strategies, using an asset based style pricing model, we shed light on the promising ability of our method to detect the time-varying dynamics of risk exposures as well as to forecast HF returns.

On convergence and parameter selection of the EM and DA-EM algorithms for Gaussian mixtures

The expectation & maximization (EM) for Gaussian mixtures is popular as a clustering algorithm. However, the EM algorithm is sensitive to initial values, and so Ueda and Nakano [4] proposed the deterministic annealing EM (DA-EM) algorithm to improve it. In this paper, we investigate theoretical behaviors of the EM and DA-EM algorithms. We first derive a general Jacobian matrix of the DA-EM algorithm with respect to posterior probabilities. We then propose a theoretical lower bound for initialization of the annealing parameter in the DA-EM algorithm. On the other hand, some researches mentioned that the EM algorithm exhibits a self-annealing behavior, that is, the equal posterior probability with small random perturbations can avoid the EM algorithm to output the mass center for Gaussian mixtures. However, there is no theoretical analysis on this self-annealing property. Since the DA-EM will become the EM when the annealing parameter is 1, according to the Jacobian matrix of the DA-EM, we can prove the self-annealing property of the EM algorithm for Gaussian mixtures. Based on these results, we give not only convergence behaviors of the equal posterior probabilities and initialization lower bound of the temperature parameter of the DA-EM, but also a theoretical explanation why the EM algorithm for Gaussian mixtures exhibits a self-annealing behavior.

Emotional speech acoustic model for Malay: Iterative versus isolated unit training

The ability of speech synthesis system to synthesize emotional speech enhances the user's experience when using this kind of system and its related applications. However, the development of an emotional speech synthesis system is a daunting task in view of the complexity of human emotional speech. The more recent state-of-the-art speech synthesis systems, such as the one based on hidden Markov models, can synthesize emotional speech with acceptable naturalness with the use of a good emotional speech acoustic model. However, building an emotional speech acoustic model requires adequate resources including segment-phonetic labels of emotional speech, which is a problem for many under-resourced languages, including Malay. This research shows how it is possible to build an emotional speech acoustic model for Malay with minimal resources. To achieve this objective, two forms of initialization methods were considered: iterative training using the deterministic annealing expectation maximization algorithm and the isolated unit training. The seed model for the automatic segmentation is a neutral speech acoustic model, which was transformed to target emotion using two transformation techniques: model adaptation and context-dependent boundary refinement. Two forms of evaluation have been performed: an objective evaluation measuring the prosody error and a listening evaluation to measure the naturalness of the synthesized emotional speech.

Application of deterministic annealing EM algorithm to MAP/PH parameter estimation

This paper proposes a variant of EM (expectation-maximization) algorithm for Markovian arrival process (MAP) and phase-type distribution (PH) parameter estimation. Especially, we derive the deterministic annealing EM (DAEM) algorithm for MAP/PH parameter estimation. The DAEM algorithm is one of the methods to overcome a local maxima problem associated with the conventional EM algorithm. This paper derives concrete E- and M-step formulas for MAP parameter estimation from inter-arrival time data and PH parameter estimation from point samples in the framework of DAEM algorithm. Numerical examples demonstrate the DAEM algorithm for Markov-modulated Poisson process (MMPP) and several classes of PH distribution.

Speech recognition based on statistical models including multiple phonetic decision trees

We propose a speech recognition technique using multiple model structures. In the use of context-dependent models, decision-tree-based context clustering is applied to find an appropriate parameter tying structure. However, context clustering is usually performed on the basis of unreliable statistics of hidden Markov model (HMM) state sequences because the estimation of reliable state sequences requires an appropriate model structures, that cannot be obtained prior to context clustering. Therefore, context clustering and the estimation of state sequences essentially cannot be performed independently. To overcome this problem, we propose an optimization technique of state sequences based on an annealing process using multiple decision trees. In this technique, a new likelihood function is defined in order to treat multiple model structures, and the deterministic annealing expectation maximization algorithm is used as the training algorithm. Experimental continuous phoneme recognition results show that the proposed method of using only two decision trees achieved about an 11.1% relative error reduction over the conventional method.

On Semi-Supervised Learning Genetic-Based and Deterministic Annealing EM Algorithm for Dirichlet Mixture Models

We propose a genetic-based and deterministic annealing expectation-maximization (GA&DA-EM) algorithm for learning Dirichlet mixture models from multivariate data. This algorithm is capable of selecting the number of components of the model using the minimum description length (MDL) criterion. Our approach benefits from the properties of Genetic algorithms and deterministic annealing algorithm by combination of both into a single procedure. The population-based stochastic search of the GA&DA explores the search space more thoroughly than the EM method. Therefore, our algorithm enables escaping from local optimal solutions since the algorithm becomes less sensitive to its initialization. The GA&DA-EM algorithm is elitist which maintains the monotonic convergence property of the EM algorithm. We conducted experiments on the WebKB and 20NEWSGROUPS. The results show that show that 1) the GA&DA-EM outperforms the EM method since: Our approach identifies the number of components which were used to generate the underlying data more often than the EM algorithm. 2) the algorithm alternatives to EM that overcoming the challenges of local maxima.

Clustering of protein domains for functional and evolutionary studies

BackgroundThe number of protein family members defined by DNA sequencing is usually much larger than those characterised experimentally. This paper describes a method to divide protein families into subtypes purely on sequence criteria. Comparison with experimental data allows an independent test of the quality of the clustering.ResultsAn evolutionary split statistic is calculated for each column in a protein multiple sequence alignment; the statistic has a larger value when a column is better described by an evolutionary model that assumes clustering around two or more amino acids rather than a single amino acid. The user selects columns (typically the top ranked columns) to construct a motif. The motif is used to divide the family into subtypes using a stochastic optimization procedure related to the deterministic annealing EM algorithm (DAEM), which yields a specificity score showing how well each family member is assigned to a subtype. The clustering obtained is not strongly dependent on the number of amino acids chosen for the motif. The robustness of this method was demonstrated using six well characterized protein families: nucleotidyl cyclase, protein kinase, dehydrogenase, two polyketide synthase domains and small heat shock proteins. Phylogenetic trees did not allow accurate clustering for three of the six families.ConclusionThe method clustered the families into functional subtypes with an accuracy of 90 to 100%. False assignments usually had a low specificity score.

Applying the extended mass-constraint EM algorithm to image retrieval

We extend the mass-constraint data clustering and vector quantization algorithm to estimate Gaussian Mixture Models (GMMs) as image features applying to the image retrieval problems. The GMM feature is an alternative method to histograms to represent data density distributions. Histograms are well known for their advantages including rotation invariance, low calculation load, and so on. The GMM maintains the rotation invariance properties; moreover, it addresses the high-dimensional problems due to which histograms usually suffer inefficiency problems. The extended mass-constraint (EMass) GMM estimation algorithm is compared with the typical Expectation–Maximization(EM) algorithm, and the deterministic annealing EM (DAEM) algorithm. The three algorithms are applied to train a GMM for a set of simulation data, and compared with the log-likelihood values. From the comparison results, we know that DAEM still has strong dependence on initial data point selection, which is the main problem we need to solve by taking advantage of the deterministic annealing methods. Thus the DAEM algorithm is not chosen to estimate GMM density functions for image retrieval. The EM and EMass algorithms are then applied to train GMMs from image RGB color features for the purpose of image retrieval. Finally the GMM features are combined with the Local Binary Pattern (LBP) features to achieve higher precision retrieval. After we compare the precision/recall curves and mean average precisions achieved by two algorithms, we conclude that the extended mass-constraint algorithm is a better solution for GMM estimation, and combining the GMM and Local Binary Pattern (LBP) provides a new promising feature for image retrieval.

Deterministic Annealing EM Algorithm in Acoustic Modeling for Speaker and Speech Recognition

This paper investigates the effectiveness of the DAEM (Deterministic Annealing EM) algorithm in acoustic modeling for speaker and speech recognition. Although the EM algorithm has been widely used to approximate the ML estimates, it has the problem of initialization dependence. To relax this problem, the DAEM algorithm has been proposed and confirmed the effectiveness in artificial small tasks. In this paper, we applied the DAEM algorithm to practical speech recognition tasks: speaker recognition based on GMMs and continuous speech recognition based on HMMs. Experimental results show that the DAEM algorithm can improve the recognition performance as compared to the standard EM algorithm with conventional initialization algorithms, especially in the flat start training for continuous speech recognition.

Deterministic annealing EM algorithm

This paper presents a deterministic annealing EM (DAEM) algorithm for maximum likelihood estimation problems to overcome a local maxima problem associated with the conventional EM algorithm. In our approach, a new posterior parameterized by `temperature' is derived by using the principle of maximum entropy and is used for controlling the annealing process. In the DAEM algorithm, the EM process is reformulated as the problem of minimizing the thermodynamic free energy by using a statistical mechanics analogy. Since this minimization is deterministically performed at each temperature, the total search is executed far more efficiently than in the simulated annealing. Moreover, the derived DAEM algorithm, unlike the conventional EM algorithm, can obtain better estimates free of the initial parameter values. We also apply the DAEM algorithm to the training of probabilistic neural networks using mixture models to estimate the probability density and demonstrate the performance of the DAEM algorithm.