Standard Expectation-maximization Research Articles

Maximum likelihood estimation for discrete latent variable models via evolutionary algorithms

We propose an evolutionary optimization method for maximum likelihood and approximate maximum likelihood estimation of discrete latent variable models. The proposal is based on modified versions of the expectation–maximization (EM) and variational EM (VEM) algorithms, which are based on the genetic approach and allow us to accurately explore the parameter space, reducing the chance to be trapped into one of the multiple local maxima of the log-likelihood function. Their performance is examined through an extensive Monte Carlo simulation study where they are employed to estimate latent class, hidden Markov, and stochastic block models and compared with the standard EM and VEM algorithms. We observe a significant increase in the chance to reach global maximum of the target function and a high accuracy of the estimated parameters for each model. Applications focused on the analysis of cross-sectional, longitudinal, and network data are proposed to illustrate and compare the algorithms.

A Truncated Mixture Transition Model for Interval-Valued Time Series

Abstract We propose a model for interval-valued time series that specifies the conditional joint distribution of the upper and lower bounds as a mixture of truncated bivariate normal distributions. It preserves the interval natural order and provides great flexibility on capturing potential conditional heteroscedasticity and non-Gaussian features. The standard expectation maximization (EM) algorithm applied to truncated mixtures does not provide a closed-form solution in the M step. A new EM algorithm solves this problem. The model applied to the interval-valued IBM daily stock returns exhibits superior performance over competing models in-sample and out-of-sample evaluation. A trading strategy showcases the usefulness of our approach.

Tempered expectation-maximization algorithm for the estimation of discrete latent variable models

Maximum likelihood estimation of discrete latent variable (DLV) models is usually performed by the expectation-maximization (EM) algorithm. A well-known drawback is related to the multimodality of the log-likelihood function so that the estimation algorithm can converge to a local maximum, not corresponding to the global one. We propose a tempered EM algorithm to explore the parameter space adequately for two main classes of DLV models, namely latent class and hidden Markov. We compare the proposal with the standard EM algorithm by an extensive Monte Carlo simulation study, evaluating both the ability to reach the global maximum and the computational time. We show the results of the analysis of discrete and continuous cross-sectional and longitudinal data referring to some applications of interest. All the results provide supporting evidence that the proposal outperforms the standard EM algorithm, and it significantly improves the chance to reach the global maximum. The advantage is relevant even considering the overall computing time.

Factor models for large and incomplete data sets with unknown group structure

Most economic applications rely on a large number of time series, which typically have a remarkable clustering structure and they are available over different spans. To handle these databases, we combined the expectation–maximization (EM) algorithm outlined by Stock and Watson (JBES, 2002) and the estimation algorithm for large factor models with an unknown number of group structures and unknown membership described by Ando and Bai (JAE, 2016; JASA, 2017) . Several Monte Carlo experiments demonstrated the good performance of the proposed method at determining the correct number of clusters, providing the appropriate number of group-specific factors, identifying error-free group membership, and obtaining accurate estimates of unobserved missing data. In addition, we found that our proposed method performed substantially better than the standard EM algorithm when the data had a grouped factor structure. Using the Federal Reserve Economic Data FRED-QD, our method detected two distinct groups of macroeconomic indicators comprising the real activity indicators and nominal indicators. Thus, we demonstrated the usefulness of our group-specific factor model for studies of business cycle chronology and for forecasting purposes.

A kernel-based identification approach for a class of nonlinear systems with quantized output data

In this paper, we consider the problem of identification for a class of nonlinear systems with Hammerstein structure, which is a combination of some basis functions for the static nonlinearity and finite impulse response models for the linear block under quantized output data. Following the empirical Bayes method, we model the impulse response of the system using a Gaussian process whose covariance matrix depends on some suitable kernels. Then, we compute the minimum mean-square error estimation of the impulse response via a special case of Markov Chain Monte Carlo sampling approach, namely, Gibbs sampler. The estimation of the impulse response depends on the kernel hyperparameters, the noise variance and the coefficients of the static nonlinearity. These unknowns are estimated by the marginal likelihood maximization, which is solved via the expectation-conditional maximization algorithm combining the Gibbs sampler. The expectation-conditional maximization method is a variation of the standard expectation-maximization method for solving maximum-likelihood problems. Numerical simulations show the effectiveness of the proposed scheme.

Causal mediation analysis with latent subgroups.

In biomedical studies, the causal mediation effect might be heterogeneous across individuals in the study population due to each study subject's unique characteristics. While individuals' mediation effects may differ from each other, it is often reasonable and more interpretable to assume that individuals belong to several distinct latent subgroups with similar attributes. In this article, we first show that the subgroup-specific mediation effect can be identified under the group-specific sequential ignorability assumptions. Then, we propose a simple mixture modeling approach to account for the latent subgroup structure where each mixture component corresponds to one latent subgroup in the linear structural equation model framework. Model parameters can be estimated using the standard expectation-maximization (EM) algorithm. Each individual's subgroup membership can be inferred based on the posterior probability. We propose to use the singular Bayesian information criterion to consistently select the number of latent subgroups by recognizing that the Fisher information matrix for mixture models might be singular. We then propose to use nonparametric bootstrap method to compute standard errors and confidence intervals. We conducted simulation studies to evaluate the empirical performance of our proposed method named iMed. Finally, we reanalyzed a DNA methylation data set from the Normative Aging Study and found that the mediation effects of two well-documented DNA methylation CpG sites are heterogeneous across two latent subgroups in the causal pathway from smoking behavior to lung function. We also developed an R package iMed for public use.

Improving Gradient Flow with Unrolled Highway Expectation Maximization

Integrating model-based machine learning methods into deep neural architectures allows one to leverage both the expressive power of deep neural nets and the ability of model-based methods to incorporate domain-specific knowledge. In particular, many works have employed the expectation maximization (EM) algorithm in the form of an unrolled layer-wise structure that is jointly trained with a backbone neural network. However, it is difficult to discriminatively train the backbone network by backpropagating through the EM iterations as they are prone to the vanishing gradient problem. To address this issue, we propose Highway Expectation Maximization Networks (HEMNet), which is comprised of unrolled iterations of the generalized EM (GEM) algorithm based on the Newton-Rahpson method. HEMNet features scaled skip connections, or highways, along the depths of the unrolled architecture, resulting in improved gradient flow during backpropagation while incurring negligible additional computation and memory costs compared to standard unrolled EM. Furthermore, HEMNet preserves the underlying EM procedure, thereby fully retaining the convergence properties of the original EM algorithm. We achieve significant improvement in performance on several semantic segmentation benchmarks and empirically show that HEMNet effectively alleviates gradient decay.

Multi-scale Self-Organizing Map assisted Deep Autoencoding Gaussian Mixture Model for unsupervised intrusion detection

In an age when the Internet has become the backbone of communications, a robust and safe network environment is critical. Intrusion detection techniques are thus valuable for IT infrastructure. The state of the art (SOTA) solution, Deep Autoencoding Gaussian Mixture Model (DAGMM), outperforms those approaches relying on decoupled two-stage training and the standard Expectation–Maximization optimization algorithm. However, DAGMM suffers from the failure in preserving the input topology, caused by the bottleneck layer of the adopted deep autoencoder as well as the method of constructing the input for the follow-up density estimation. This research first presents a Self-Organizing Map assisted Deep Autoencoding Gaussian Mixture Model (SOM-DAGMM) for overcoming the above-mentioned shortcoming of DAGMM, through well balancing the low-dimensional demand of Gaussian Mixture Model (GMM) and the topology-preserving requirement. The proposed SOM-DAGMM employs a self-organizing map to extract features as a supplement with well-preserved input space topology for better network intrusion detection. The paper also studies the superiority of multi-scale topology over the single-scale one in improving the performance of DAGMM. The better performance of the SOM-DAGMM is empirically proven by extensive experiments involving six datasets. Experimental results show that single/multi-scale SOM-DAGMMs outperform the SOTA DAGMM on all tests and achieve up to 110.38% improvement in F1 score and with better stability.

Improved image reconstruction of 89Zr-immunoPET studies using a Bayesian penalized likelihood reconstruction algorithm

PurposeThe aim of this study was to evaluate the use of a Bayesian penalized likelihood reconstruction algorithm (Q.Clear) for 89Zr-immunoPET image reconstruction and its potential to improve image quality and reduce the administered activity of 89Zr-immunoPET tracers.MethodsEight 89Zr-immunoPET whole-body PET/CT scans from three 89Zr-immunoPET clinical trials were selected for analysis. On average, patients were imaged 6.3 days (range 5.0–8.0 days) after administration of 69 MBq (range 65–76 MBq) of [89Zr]Zr-DFO-daratumumab, [89Zr]Zr-DFO-pertuzumab, or [89Zr]Zr-DFO-trastuzumab. List-mode PET data was retrospectively reconstructed using Q.Clear with incremental β-values from 150 to 7200, as well as standard ordered-subset expectation maximization (OSEM) reconstruction (2-iterations, 16-subsets, a 6.4-mm Gaussian transaxial filter, “heavy” z-axis filtering and all manufacturers’ corrections active). Reduced activities were simulated by discarding 50% and 75% of original counts in each list mode stream. All reconstructed PET images were scored for image quality and lesion detectability using a 5-point scale. SUVmax for normal liver and sites of disease and liver signal-to-noise ratio were measured.ResultsQ.Clear reconstructions with β = 3600 provided the highest scores for image quality. Images reconstructed with β-values of 3600 or 5200 using only 50% or 25% of the original counts provided comparable or better image quality scores than standard OSEM reconstruction images using 100% of counts.ConclusionThe Bayesian penalized likelihood reconstruction algorithm Q.Clear improved the quality of 89Zr-immunoPET images. This could be used in future studies to improve image quality and/or decrease the administered activity of 89Zr-immunoPET tracers.

Envelope method with ignorable missing data.

Envelope method was recently proposed as a method to reduce the dimension of responses in multivariate regressions. However, when there exists missing data, the envelope method using the complete case observations may lead to biased and inefficient results. In this paper, we generalize the envelope estimation when the predictors and/or the responses are missing at random. Specifically, we incorporate the envelope structure in the expectation-maximization (EM) algorithm. As the parameters under the envelope method are not pointwise identifiable, the EM algorithm for the envelope method was not straightforward and requires a special decomposition. Our method is guaranteed to be more efficient, or at least as efficient as, the standard EM algorithm. Moreover, our method has the potential to outperform the full data MLE. We give asymptotic properties of our method under both normal and non-normal cases. The efficiency gain over the standard EM is confirmed in simulation studies and in an application to the Chronic Renal Insufficiency Cohort (CRIC) study.

Multifractal point processes and the spatial distribution of wildfires in French Mediterranean regions

We introduce a simple and wide class of multifractal spatial point patterns as Cox processes which intensity is multifractal, i.e., the class of Poisson processes with a stochastic intensity corresponding to a random multifractal measure. We then propose a maximum likelihood approach by means of a standard Expectation–Maximization procedure in order to estimate the distribution of these intensities at all scales. This provides, as validated on various numerical examples, a simple framework to estimate the scaling laws and therefore the multifractal properties for this class of spatial point processes. The wildfire distribution gathered in the Prométhée French Mediterranean wildfire database is investigated within this approach that notably allows us to compute the statistical moments associated with the spatial distribution of annual likelihood of fire event occurrence. We show that for each order q, these moments display a well defined scaling behavior with a non-linear spectrum of scaling exponents ζq. From our study, it thus appears that the spatial distribution of the wildfire ignition annual risk can be described by a non-trivial, multifractal singularity spectrum and that this risk cannot be reduced to providing a number of events per km2. Our analysis is confirmed by a direct spatial correlation estimation of the intensity logarithms whose the peculiar slowly decreasing shape corresponds to the hallmark of multifractal cascades. The multifractal features appear to be constant over time and similar over the three regions that are studied.

Comparison of Recent Acceleration Techniques for the EM Algorithm in One- and Two-Parameter Logistic IRT Models

The expectation–maximization (EM) algorithm is an important numerical method for maximum likelihood estimation in incomplete data problems. However, convergence of the EM algorithm can be slow, and for this reason, many EM acceleration techniques have been proposed. After a review of acceleration techniques in a unified notation with illustrations, three recently proposed EM acceleration techniques are compared in detail: quasi-Newton methods (QN), “squared” iterative methods (SQUAREM), and parabolic EM (PEM). These acceleration techniques are applied to marginal maximum likelihood estimation with the EM algorithm in one- and two-parameter logistic item response theory (IRT) models for binary data, and their performance is compared. QN and SQUAREM methods accelerate convergence of the EM algorithm for the two-parameter logistic model significantly in high-dimensional data problems. Compared to the standard EM, all three methods reduce the number of iterations, but increase the number of total marginal log-likelihood evaluations per iteration. Efficient approximations of the marginal log-likelihood are hence an important part of implementation.

Calibration of exponential Hawkes processes using a Modified Bionomic Algorithm

The aim of this research is to develop a fast and robust variant of the evolutionary heuristic Bionomic algorithm and assess its contribution to solving complex parametric estimation problems, in conjunction with other traditional optimization techniques. We introduce a modified version of the Bionomic Algorithm (MB), designed to efficiently compute the MLE of self-exciting exponential Hawkes processes with increasing dimensionality of the solution space. Performance tests, performed on simulated and historical S&P 500 financial data, show that the MB algorithm, with its solutions locally improved by either the standard Nelder Mead (NM) or Expectation Maximization (EM) algorithm, converges significantly faster and more frequently to a near-global solution than the NM or EM algorithms operating alone. These test results illustrate the robustness and computational efficiency of the MB algorithm, combined with traditional optimization methods, in the optimization of complex objective functions of high dimensionality.

High-dimensional count data clustering based on an exponential approximation to the multinomial Beta-Liouville distribution

In this paper, we propose a mixture model for high-dimensional count data clustering based on an exponential-family approximation of the Multinomial Beta-Liouville distribution, which we call EMBL. We deal simultaneously with the problems of fitting the model to observed data and selecting the number of components. The learning algorithm automatically selects the optimal number of components and avoids several drawbacks of the standard Expectation-Maximization algorithm, including the sensitivity to initialization and possible convergence to the boundary of the parameter space. We demonstrate the effectiveness and robustness of the proposed clustering approach through a set of extensive empirical experiments that involve challenging real-world applications. The results reveal that the novel proposed model strives to achieve higher accuracy compared to the state-of-the-art generative models for count data clustering. Furthermore, the superior performance of EMBL demonstrates its flexibility and ability to address the burstiness phenomenon successfully, as well as shows its computational efficiency, especially when dealing with sparse high-dimensional vectors.

Evaluation of a Bayesian penalized likelihood reconstruction algorithm for low-count clinical 18F-FDG PET/CT

BackgroundRecently, a Bayesian penalized likelihood (BPL) reconstruction algorithm was introduced for a commercial PET/CT with the potential to improve image quality. We compared the performance of this BPL algorithm with conventional reconstruction algorithms under realistic clinical conditions such as daily practiced at many European sites, i.e. low 18F-FDG dose and short acquisition times.ResultsTo study the performance of the BPL algorithm, regular clinical 18F-FDG whole body PET scans were made. In addition, two types of phantoms were scanned with 4-37 mm sized spheres filled with 18F-FDG at sphere-to-background ratios of 10-to-1, 4-to-1, and 2-to-1. Images were reconstructed using standard ordered-subset expectation maximization (OSEM), OSEM with point spread function (PSF), and the BPL algorithm using β-values of 450, 550 and 700. To quantify the image quality, the lesion detectability, activity recovery, and the coefficient of variation (COV) within a single bed position (BP) were determined. We found that when applying the BPL algorithm both smaller lesions in clinical studies as well as spheres in phantom studies can be detected more easily due to a higher SUV recovery, especially for higher contrast ratios. Under standard clinical scanning conditions, i.e. low number of counts, the COV is higher for the BPL (β=450) than the OSEM+PSF algorithm. Increase of the β-value to 550 or 700 results in a COV comparable to OSEM+PSF, however, at the cost of contrast, though still better than OSEM+PSF. At the edges of the axial field of view (FOV) where BPs overlap, COV can increase to levels at which bands become visible in clinical images, related to the lower local axial sensitivity of the PET/CT, which is due to the limited bed overlap of 23% such as advised by the manufacturer.ConclusionsThe BPL algorithm performs better than the standard OSEM+PSF algorithm on small lesion detectability, SUV recovery, and noise suppression. Increase of the percentage of bed overlap, time per BP, administered activity, or the β-value, all have a direct positive impact on image quality, though the latter with some loss of small lesion detectability. Thus, BPL algorithms are very interesting for improving image quality, especially in small lesion detectability.

Investigation of the optimal number of clusters by the adaptive EM algorithm

This paper considers the investigation of the optimal number of clusters for datasets that are modeled as the Gaussian mixture. For that purpose, the adaptive method that is based on the modified Expectation Maximization (EM) algorithm is developed. The modification is conducted within the hidden variable of the standard EM algorithm. Assuming that data are multivariate normally distributed, where each component of the Gaussian mixture corresponds to one cluster, the modification is provided by utilizing the fact that the Mahalanobis distance of samples follows a Chi-square distribution. Besides, the quantity measure is constructed in order to determine number of clusters. The proposed method is presented in several numerical examples.

PUlasso: High-Dimensional Variable Selection With Presence-Only Data

In various real-world problems, we are presented with classification problems with positive and unlabeled data, referred to as presence-only responses. In this article we study variable selection in the context of presence only responses where the number of features or covariates p is large. The combination of presence-only responses and high dimensionality presents both statistical and computational challenges. In this article, we develop the PUlasso algorithm for variable selection and classification with positive and unlabeled responses. Our algorithm involves using the majorization-minimization framework which is a generalization of the well-known expectation-maximization (EM) algorithm. In particular to make our algorithm scalable, we provide two computational speed-ups to the standard EM algorithm. We provide a theoretical guarantee where we first show that our algorithm converges to a stationary point, and then prove that any stationary point within a local neighborhood of the true parameter achieves the minimax optimal mean-squared error under both strict sparsity and group sparsity assumptions. We also demonstrate through simulations that our algorithm outperforms state-of-the-art algorithms in the moderate p settings in terms of classification performance. Finally, we demonstrate that our PUlasso algorithm performs well on a biochemistry example. Supplementary materials for this article are available online.

Variable Selection for Mixed Data Clustering: Application in Human Population Genomics

Model-based clustering of human population genomic data, composed of 1,318 individuals arisen from western Central Africa and 160,470 markers, is considered. This challenging analysis leads us to develop a new methodology for variable selection in clustering. To explain the differences between subpopulations and to increase the accuracy of the estimates, variable selection is done simultaneously to clustering. We proposed two approaches for selecting variables when clustering is managed by the latent class model (i.e., mixture considering independence within components). The first method simultaneously performs model selection and parameter inference. It optimizes the Bayesian Information Criterion with a modified version of the standard expectation–maximization algorithm. The second method performs model selection without requiring parameter inference by maximizing the Maximum Integrated Complete-data Likelihood criterion. Although the application considers categorical data, the proposed methods are introduced in the general context of mixed data (data composed of different types of features). As the first step, the interest of both proposed methods is shown on simulated and several benchmark real data. Then, we apply the clustering method to the human population genomic data which permits to detect the most discriminative genetic markers. The proposed method implemented in the R package VarSelLCM is available on CRAN.

Bayesian cylindrical data modeling using Abe–Ley mixtures

• A Bayesian method to estimate parameters of a cylindrical mixture distribution. • Application of cylindrical distributions to a new dataset with time-of-day as circular variable. • Estimation uncertainty is directly inferred in the proposed method without extra calculations. This paper proposes a Metropolis–Hastings algorithm based on Markov chain Monte Carlo sampling, to estimate the parameters of the Abe–Ley distribution, which is a recently proposed Weibull-Sine-Skewed-von Mises mixture model, for bivariate circular-linear data. Current literature estimates the parameters of these mixture models using the expectation-maximization method, but we will show that this exhibits a few shortcomings for the considered mixture model. First, standard expectation-maximization does not guarantee convergence to a global optimum, because the likelihood is multi-modal, which results from the high dimensionality of the mixture’s likelihood. Second, given that expectation-maximization provides point estimates of the parameters only, the uncertainties of the estimates (e.g., confidence intervals) are not directly available in these methods. Hence, extra calculations are needed to quantify such uncertainty. We propose a Metropolis–Hastings based algorithm that avoids both shortcomings of expectation-maximization. Indeed, Metropolis–Hastings provides an approximation to the complete (posterior) distribution, given that it samples from the joint posterior of the mixture parameters. This facilitates direct inference (e.g., about uncertainty, multi-modality) from the estimation. In developing the algorithm, we tackle various challenges including convergence speed, label switching and selecting the optimum number of mixture components. We then (i) verify the effectiveness of the proposed algorithm on sample datasets with known true parameters, and further (ii) validate our methodology on an environmental dataset (a traditional application domain of Abe–Ley mixtures where measurements are function of direction). Finally, we (iii) demonstrate the usefulness of our approach in an application domain where the circular measurement is periodic in time.

A Conditional Gradient Approach for Nonparametric Estimation of Mixtures of Choice Models

Mixture models are versatile tools that are used extensively in many fields, including operations, marketing, and econometrics. The main challenge in estimating mixture models is that the mixing distribution is often unknown and imposing apriori parametric assumptions can lead to model misspecification issues. In this paper, we propose a new methodology for nonparametric estimation of the mixing distribution of a mixture of logit models. We formulate the likelihood-based estimation problem as a constrained convex program and apply the conditional gradient (a.k.a. Frank-Wolfe) algorithm to solve this convex program. We show that our method iteratively generates the support of the mixing distribution and the mixing proportions. Theoretically, we establish sublinear convergence rate of our estimator and characterize the structure of the recovered mixing distribution. Empirically, we test our approach on real-world datasets. We show that it outperforms the standard expectation-maximization (EM) benchmark on speed (16x faster), in-sample fit (up to 24% reduction in the log-likelihood loss), and predictive (average 27% reduction in standard error metrics) and decision accuracies (extracts around 23% more revenue). On synthetic data, we show that our estimator is robust to different ground-truth mixing distributions and can also account for endogeneity.