Traditional Expectation Maximization Research Articles

Online state and unknown inputs estimation for nonlinear systems with particle filter based recursive expectation‐maximization algorithm

AbstractThe article presents an innovative approach to simultaneously estimate states and unknown inputs (UIs) in nonlinear systems using a particle filter (PF) based recursive expectation‐maximization (EM) algorithm. This method is distinct from traditional iterative EM algorithms. During the E‐step, it calculates the Q‐function recursively within the maximum likelihood framework, while the PF estimates the system states. The M‐step involves local maximization of the recursive Q‐function to online estimate the UIs. The effectiveness of the PF‐based recursive EM algorithm is demonstrated with a numerical example, and comparisons with the augmented state PF are made to highlight its advantages. Finally, the proposed algorithm is implemented in a real application for the estimation of the continuous fermentation process.

Offshore Wind Energy Assessment with a Clustering Approach to Mixture Model Parameter Estimation

In wind resource assessment research, mixture models are gaining importance due to the complex characteristics of wind data. The precision of parameter estimations for these models is paramount, as it directly affects the reliability of wind energy forecasts. Traditionally, the expectation–maximization (EM) algorithm has served as a primary tool for such estimations. However, challenges are often encountered with this method when handling complex probability distributions. Given these limitations, the objective of this study is to propose a new clustering algorithm, designed to transform mixture distribution models into simpler probability clusters. To validate its efficacy, a numerical experiment was conducted, and its outcomes were compared with those derived from the established EM algorithm. The results demonstrated a significant alignment between the new method and the traditional EM approach, indicating that comparable accuracy can be achieved without the need for solving complex nonlinear equations. Moreover, the new algorithm was utilized to examine the joint probabilistic structure of wind speed and air density in China’s coastal regions. Notably, the clustering algorithm demonstrated its robustness, with the root mean square error value being notably minimal and the coefficient of determination exceeding 0.9. The proposed approach is suggested as a compelling alternative for parameter estimation in mixture models, particularly when dealing with complex probability models.

Inference Based on the Stochastic Expectation Maximization Algorithm in a Kumaraswamy Model with an Application to COVID-19 Cases in Chile

Extensive research has been conducted on models that utilize the Kumaraswamy distribution to describe continuous variables with bounded support. In this study, we examine the trapezoidal Kumaraswamy model. Our objective is to propose a parameter estimation method for this model using the stochastic expectation maximization algorithm, which effectively tackles the challenges commonly encountered in the traditional expectation maximization algorithm. We then apply our results to the modeling of daily COVID-19 cases in Chile.

An efficient image segmentation method based on expectation maximization and Salp swarm algorithm.

Multilevel image thresholding using Expectation Maximization (EM) is an efficient method for image segmentation. However, it has two weaknesses: 1) EM is a greedy algorithm and cannot jump out of local optima. 2) it cannot guarantee the number of required classes while estimating the histogram by Gaussian Mixture Models (GMM). in this paper, to overcome these shortages, a novel thresholding approach by combining EM and Salp Swarm Algorithm (SSA) is developed. SSA suggests potential points to the EM algorithm to fly to a better position. Moreover, a new mechanism is considered to maintain the number of desired clusters. Twenty-four medical test images are selected and examined by standard metrics such as PSNR and FSIM. The proposed method is compared with the traditional EM algorithm, and an average improvement of 5.27% in PSNR values and 2.01% in FSIM values were recorded. Also, the proposed approach is compared with four existing segmentation techniques by using CT scan images that Qatar University has collected. Experimental results depict that the proposed method obtains the first rank in terms of PSNR and the second rank in terms of FSIM. It has been observed that the proposed technique performs better performance in the segmentation result compared to other considered state-of-the-art methods.

Analyzing the Rainfall Pattern in Honduras Through Non-Homogeneous Hidden Markov Models

One of the major climatic interests of the last decades has been to understand and describe the rainfall patterns of specific areas of the world as functions of other climate covariates. We do it for the historical climate monitoring data from Tegucigalpa, Honduras, using non-homogeneous hidden Markov models (NHMMs), which are dynamic models usually used to identify and predict heterogeneous regimes. For estimating the NHMM in an efficient and scalable way, we propose the stochastic Expectation-Maximization (EM) algorithm and a Bayesian method, and compare their performance in synthetic data. Although these methodologies have already been used for estimating several other statistical models, it is not the case of NHMMs which are still widely fitted by the traditional EM algorithm. We observe that, under tested conditions, the performance of the Bayesian and stochastic EM algorithms is similar and discuss their slight differences. Analyzing the Honduras rainfall data set, we identify three heterogeneous rainfall periods and select temperature and humidity as relevant covariates for explaining the dynamic relation among these periods.

GMM based low-complexity adaptive machine-learning equalizers for optical fiber communication

The demand for high-speed data transmission has increased rapidly over the past few years, leading to advanced optical communication techniques. However, most of machine learning based equalizers which are trained with offline dataset mainly focus on achieving low BER, neglecting the generalization ability when the properties of optical link change. In this paper, we propose Gaussian Mixture model (GMM) based low-complexity adaptive machine-learning equalizers. The proposed online training strategy can fine-tune the parameters in GMM with a small amount of training sequence by introducing Maximum a posteriori probability (MAP) algorithm and sliding window. The size of online training sequence can be reduced by utilizing the parameters trained from offline data as the priori information to reduce the size of online training sequence. The proposed adaptive GMM based equalizers can not only show an excellent capability of mitigating nonlinear distortions but also show the ability to update parameters automatically according to the channel conditions. In addition, experimental results show that by introducing MAP algorithm and sliding window, the convergence speed of the proposed equalizers have been accelerated by 2 times, compared with the case that the traditional Expectation Maximization (EM) algorithm is used for online parameter update. On the other hand, there is no significant increase in computational complexity during the online stage.

A Parameter Estimation Method for Stress- Strength Model Based on Extending Markov State-Space With Variable Transition Rates

A stress-strength model usually has more than one failure mode since the component suffers at least two types of stresses, complicating the expression of the likelihood function and increasing the computational complexity of the parameter estimation for general distributions (non-exponential distributions). A phase-type distribution (also known as a PH distribution) is dense and has closure properties, which makes it suitable to reduce the computational complexity of the stress-strength model. The traditional expectation-maximization (EM) method for estimating the parameters of the PH distribution cannot be used directly when the strength changes over time since the PH distribution is a continuous-time Markov process that must satisfy the relevant properties of the infinitesimal generator in the Markov state-space. Therefore, a parameter estimation method based on extending the Markov state-space with variable transition rates for the stress-strength model is proposed. Both failure and censored samples are considered. First, the stress-strength model based on the PH distribution is briefly introduced, and the likelihood functions for different failure modes are derived. Subsequently, the principle of the method is described in detail, the derivation process of the relevant equations is provided, and the limitations of the method are discussed. The performance of the method is evaluated using two simulation cases.

Motor Skills Learning and Generalization with Adapted Curvilinear Gaussian Mixture Model

This paper is intended to solve the motor skills learning, representation and generalization problems in robot imitation learning. To this end, we present an Adapted Curvilinear Gaussian Mixture Model (AdC-GMM), which is a general extension of the GMM. The proposed model can encode data more compactly. More critically, it is inherently suitable for representing data with strong non-linearity. To infer the parameters of this model, a Cross Entropy Optimization (CEO) algorithm is proposed, where the cross entropy loss of the training data is minimized. Compared with the traditional Expectation Maximization (EM) algorithm, the CEO can automatically infer the optimal number of components. Finally, the generalized trajectories are retrieved by an Adapted Curvilinear Gaussian Mixture Regression (AdC-GMR) model. To encode observations from different frames, the sophisticated task parameterization (TP) technique is introduced. All above proposed algorithms are verified by comprehensive tasks. The CEO is evaluated by a hand writing task. Another goal-directed reaching task is used to evaluate the AdC-GMM and AdC-GMR algorithm. A novel hammer-over-a-nail task is designed to verify the task parameterization technique. Experimental results demonstrate the proposed CEO is superior to the EM in terms of encoding accuracy and the AdC-GMM can achieve more compact representation by reducing the number of components by up to 50%. In addition, the trajectory retrieved by the AdC-GMR is smoother and the approximation error is comparable to the Gaussian process regression (GPR) even far fewer parameters need to be estimated. Because of this, the AdC-GMR is much faster than the GPR. Finally, simulation experiments on the hammer-over-a-nail task demonstrates the proposed methods can be deployed and used in real-world applications.

Integrating Differential Evolution Optimization to Cognitive Diagnostic Model Estimation.

A log-linear cognitive diagnostic model (LCDM) is estimated via a global optimization approach- differential evolution optimization (DEoptim), which can be used when the traditional expectation maximization (EM) fails. The application of the DEoptim to LCDM estimation is introduced, explicated, and evaluated via a Monte Carlo simulation study in this article. The aim of this study is to fill the gap between the field of psychometric modeling and modern machine learning estimation techniques and provide an alternative solution in the model estimation.

A Modified EM Algorithm for ISAR Scatterer Trajectory Matrix Completion

The anisotropy of radar cross section of scatterers makes the scatterer trajectory matrix incomplete in sequential inverse synthetic aperture radar images. As a result, factorization methods cannot be directly applied to reconstruct the 3-D geometry of scatterers without additional consideration. We propose a modified expectation-maximization (EM) algorithm to retrieve the complete scatterer trajectory matrix. First, we derive the motion dynamics of the projected scatterer, which approximates an ellipse. Then, based on the estimated ellipse parameters using the known data of each scatterer trajectory, we use the Kalman filter to initialize the missing data. To address the limitations of a traditional EM, which only considers the rank-deficient characteristics of the scatterer trajectory matrix, we propose to augment EM by using both the known rank-deficient and elliptical motion characteristics. Experimental results on simulated data verify the effectiveness of the proposed method.

Addressing overfitting and underfitting in Gaussian model-based clustering

The expectation–maximization (EM) algorithm is a common approach for parameter estimation in the context of cluster analysis using finite mixture models. This approach suffers from the well-known issue of convergence to local maxima, but also the less obvious problem of overfitting. These combined, and competing, concerns are illustrated through simulation and then addressed by introducing an algorithm that augments the traditional EM with the nonparametric bootstrap. Further simulations and applications to real data lend support for the usage of this bootstrap augmented EM-style algorithm to avoid both overfitting and local maxima.

Bayesian-Optimization-Based Peak Searching Algorithm for Clustering in Wireless Sensor Networks

We propose a new peak searching algorithm (PSA) that uses Bayesian optimization to find probability peaks in a dataset, thereby increasing the speed and accuracy of clustering algorithms. Wireless sensor networks (WSNs) are becoming increasingly common in a wide variety of applications that analyze and use collected sensing data. Typically, the collected data cannot be directly used in modern data analysis problems that adopt machine learning techniques because such data lacks additional information (such as data labels) specifying its purpose of users. Clustering algorithms that divide the data in a dataset into clusters are often used when additional information is not provided. However, traditional clustering algorithms such as expectation–maximization (EM) and k - m e a n s algorithms require massive numbers of iterations to form clusters. Processing speeds are therefore slow, and clustering results become less accurate because of the way such algorithms form clusters. The PSA addresses these problems, and we adapt it for use with the EM and k - m e a n s algorithms, creating the modified P S E M and P S k - m e a n s algorithms. Our simulation results show that our proposed P S E M and P S k - m e a n s algorithms significantly decrease the required number of clustering iterations (by 1.99 to 6.3 times), and produce clustering that, for a synthetic dataset, is 1.69 to 1.71 times more accurate than it is for traditional EM and enhanced k - m e a n s ( k - m e a n s ++) algorithms. Moreover, in a simulation of WSN applications aimed at detecting outliers, P S E M correctly identified the outliers in a real dataset, decreasing iterations by approximately 1.88 times, and P S E M was 1.29 times more accurate than EM at a maximum.

An adaptive approach for recovering overlapping echoes in oil film thickness measurement by ultrasound

PurposeFor oil film thickness measurement using ultrasonic spring model, obtaining the isolated reflection from the oil film layer is the key point. While for oil film thickness measurement in thrust bearings with thin liner, the reflection from the substrate-Babbitt interface will overlap with the reflection from the oil film layer. This overlapping will render the ultrasonic spring model invalid. To obtain the isolated reflected signal from the oil film layer accurately, an adaptive method was developed to recover the overlapping echoes.Design/methodology/approachA genetic-algorithm-based support matching pursuit (GA-based SMP) was developed to provide the optimal echo number and initial parameters guesses automatically and efficiently. Then, the traditional expectation maximization (EM) model was used to fine tune the accurate results.FindingsThe developed method was tested using both simulated echoes and the overlapping echoes encountered in the ultrasonic oil film thickness measurement of thrust bearings. The results demonstrated that the developed method performed well on recovering overlapping echoes adaptively.Originality/valueThe work shows an adaptive method to recover the ultrasonic overlapping echoes. When used in ultrasonic oil film thickness measurement, it can help extend the application of traditional ultrasonic spring model to objects with four or more layers.

A parameter selection method of the deterministic anti-annealing algorithm for network exploring

The traditional expectation maximization (EM) algorithm for the mixture model can explore the structural regularities of a network efficiently. But it always traps into local maxima. A deterministic annealing EM (DAEM) algorithm is put forward to solve this problem. However, it brings about the problem of convergence speed. A deterministic anti-annealing expectation maximization (DAAEM) algorithm not only prevents poor local optima, but also improves the convergence speed. Thus, the DAAEM algorithm is used to estimate parameters of the mixture model. This algorithm always sets its initial parameter β0 by experience, which maybe get trapped into meaningless results due to too small β0, or converge to local maxima more frequently due to too large β0. A parameter selection method for β0 is designed. In our method, the convergence rate of the DAAEM algorithm for mixture model is first derived from Jacobian matrix of the posterior probabilities. Then the theoretical lower bound of β0 is computed based on the convergence rate at meaningless points. In our experiments we select β0 by rounding up the lower bound to the nearest tenth. Experiments on real and synthetic networks demonstrate that the parameter selection method is valid, and the performance of the DAAEM algorithm beginning from the selected parameter is better than the EM and DAEM algorithms for mixture model. In addition, we find that the convergence rate of the DAAEM algorithm is affected by assortative mixing by degree of a network.

Causal Latent Markov Model for the Comparison of Multiple Treatments in Observational Longitudinal Studies

We extend to the longitudinal setting a latent class approach that was recently introduced by Lanza, Coffman, and Xu to estimate the causal effect of a treatment. The proposed approach enables an evaluation of multiple treatment effects on subpopulations of individuals from a dynamic perspective, as it relies on a latent Markov (LM) model that is estimated taking into account propensity score weights based on individual pretreatment covariates. These weights are involved in the expression of the likelihood function of the LM model and allow us to balance the groups receiving different treatments. This likelihood function is maximized through a modified version of the traditional expectation–maximization algorithm, while standard errors for the parameter estimates are obtained by a nonparametric bootstrap method. We study in detail the asymptotic properties of the causal effect estimator based on the maximization of this likelihood function, and we illustrate its finite sample properties through a series of simulations showing that the estimator has the expected behavior. As an illustration, we consider an application aimed at assessing the relative effectiveness of certain degree programs on the basis of three ordinal response variables in which the work path of a graduate is considered as the manifestation of his or her human capital-level across time.

Semi‐supervised Artificial Immune Mixture Models Clustering

Learning with partly labeled data aims at combining labeled and unlabeled data in order to boost the accuracy of an algorithm. The traditional Expectation maximization (EM) algorithm only produces locally optimal solutions, it is sensitive to initialization, and the number of components of mixture model must be known in advance. We propose a novel semi-supervised clustering algorithm that uses Gaussian mixture models (GMM) as the underlying clustering model. A novel adaptive global search mechanism is introduced into semi-supervised gaussian mixture model-based clustering, where the EM algorithm is incorporated with the ideas of an immune clonal selection technique. The new algorithm overcomes the various problems associated with the traditional EM algorithm. And it can improve the effectiveness in estimating the parameters and determining the optimal number of clusters automatically. The experimental results illustrate the proposed algorithm provides significantly better clustering results, when compared with other methods of incorporating equivalence constraints.

Mixture Model Analysis of Partially Rank‐Ordered Set Samples: Age Groups of Fish from Length‐Frequency Data

AbstractWe present a novel methodology for estimating the parameters of a finite mixture model (FMM) based on partially rank‐ordered set (PROS) sampling and use it in a fishery application. A PROS sampling design first selects a simple random sample of fish and creates partially rank‐ordered judgement subsets by dividing units into subsets of prespecified sizes. The final measurements are then obtained from these partially ordered judgement subsets. The traditional expectation–maximization algorithm is not directly applicable for these observations. We propose a suitable expectation–maximization algorithm to estimate the parameters of the FMMs based on PROS samples. We also study the problem of classification of the PROS sample into the components of the FMM. We show that the maximum likelihood estimators based on PROS samples perform substantially better than their simple random sample counterparts even with small samples. The results are used to classify a fish population using the length‐frequency data.

Bayesian analysis of multivariate t linear mixed models with missing responses at random

The multivariate t linear mixed model (MtLMM) has been recently proposed as a robust tool for analysing multivariate longitudinal data with atypical observations. Missing outcomes frequently occur in longitudinal research even in well controlled situations. As a powerful alternative to the traditional expectation maximization based algorithm employing single imputation, we consider a Bayesian analysis of the MtLMM to account for the uncertainties of model parameters and missing outcomes through multiple imputation. An inverse Bayes formulas sampler coupled with Metropolis-within-Gibbs scheme is used to effectively draw the posterior distributions of latent data and model parameters. The techniques for multiple imputation of missing values, estimation of random effects, prediction of future responses, and diagnostics of potential outliers are investigated as well. The proposed methodology is illustrated through a simulation study and an application to AIDS/HIV data.

Adaptive Rejection Metropolis Simulated Annealing for Detecting Global Maximum Regions

A finite mixture model has been used to fit the data from heterogeneous populations to many applications. An Expectation Maximization (EM) algorithm is the most popular method to estimate parameters in a finite mixture model. A Bayesian approach is another method for fitting a mixture model. However, the EM algorithm often converges to the local maximum regions, and it is sensitive to the choice of starting points. In the Bayesian approach, the Markov Chain Monte Carlo (MCMC) sometimes converges to the local mode and is difficult to move to another mode. Hence, in this paper we propose a new method to improve the limitation of EM algorithm so that the EM can estimate the parameters at the global maximum region and to develop a more effective Bayesian approach so that the MCMC chain moves from one mode to another more easily in the mixture model. Our approach is developed by using both simulated annealing (SA) and adaptive rejection metropolis sampling (ARMS). Although SA is a well-known approach for detecting distinct modes, the limitation of SA is the difficulty in choosing sequences of proper proposal distributions for a target distribution. Since ARMS uses a piecewise linear envelope function for a proposal distribution, we incorporate ARMS into an SA approach so that we can start a more proper proposal distribution and detect separate modes. As a result, we can detect the maximum region and estimate parameters for this global region. We refer to this approach as ARMS annealing. By putting together ARMS annealing with the EM algorithm and with the Bayesian approach, respectively, we have proposed two approaches: an EM-ARMS annealing algorithm and a Bayesian-ARMS annealing approach. We compare our two approaches with traditional EM algorithm alone and Bayesian approach alone using simulation, showing that our two approaches are comparable to each other but perform better than EM algorithm alone and Bayesian approach alone. Our two approaches detect the global maximum region well and estimate the parameters in this region. We demonstrate the advantage of our approaches using an example of the mixture of two Poisson regression models. This mixture model is used to analyze a survey data on the number of charitable donations.

Estimation of spatial autoregressive models with randomly missing data in the dependent variable

Summary We suggest and compare different methods for estimating spatial autoregressive models with randomly missing data in the dependent variable. Aside from the traditional expectation-maximization (EM) algorithm, a nonlinear least squares method is suggested and a generalized method of moments estimation is developed for the model. A two-stage least squares estimation with imputation is proposed as well. We analytically compare these estimation methods and find that generalized nonlinear least squares, best generalized two-stage least squares with imputation and best method of moments estimators have identical asymptotic variances. These methods are less efficient than maximum likelihood estimation implemented with the EM algorithm. When unknown heteroscedasticity exists, however, EM estimation produces inconsistent estimates. Under this situation, these methods outperform EM. We provide finite sample evidence through Monte Carlo experiments.