Monte Carlo EM Research Articles

Parameter estimation for grouped data using EM and MCEM algorithms

Nowadays, the confidentiality of data and information is of great importance for many companies and organizations. For this reason, they may prefer not to release exact data, but instead to grant researchers access to approximate data. For example, rather than providing the exact measurements of their clients, they may only provide researchers with grouped data, that is, the number of clients falling in each of a set of non-overlapping measurement intervals. The challenge is to estimate the mean and variance structure of the hidden ungrouped data based on the observed grouped data. To tackle this problem, this work considers the exact observed data likelihood and applies the Expectation-Maximization (EM) and Monte Carlo EM (MCEM) algorithms for cases where the hidden data follow a univariate, bivariate, or multivariate normal distribution. Simulation studies are conducted to evaluate the performance of the proposed EM and MCEM algorithms. The well-known Galton data set is considered as an application example.

On robust probabilistic principal component analysis using multivariate t-distributions

Probabilistic principal component analysis (PPCA) is a probabilistic reformulation of principal component analysis (PCA), under the framework of a Gaussian latent variable model. To improve the robustness of PPCA, it has been proposed to change the underlying Gaussian distributions to multivariate t-distributions. Based on the representation of t-distribution as a scale mixture of Gaussian distributions, a hierarchical model is used for implementation. However, in the existing literature, the hierarchical model implemented does not yield the equivalent interpretation. In this paper, we present two sets of equivalent relationships between the high-level multivariate t-PPCA framework and the hierarchical model used for implementation. In doing so, we clarify a current misrepresentation in the literature, by specifying the correct correspondence. In addition, we discuss the performance of different multivariate t robust PPCA methods both in theory and simulation studies, and propose a new Monte Carlo expectation-maximization (MCEM) algorithm to implement one general type of such models.

Approximate Likelihood Inference in Generalized Linear Models with Censored Covariates

In many surveys and clinical trials, we obtain measurements on covariates or biomarkers that are left-censored due to the limit of detection. In such cases, it is necessary to correct for the left-censoring when studying covariate effects in regression models. The expectation-maximization (EM) algorithm is widely used for the likelihood inference in generalized linear models with censored covariates. The EM method, however, requires intensive computation involving high-dimensional integration with respect to the covariates when the dimension of the censored covariates is large. To reduce such computational difficulties, we propose and explore a Monte Carlo EM method based on the Metropolis algorithm. The finite-sample properties of the proposed estimators are studied using Monte Carlo simulations. An application is also provided using actual data obtained from a health and nutrition examination survey. Journal of Statistical Research 2021, Vol. 55, No. 2, pp. 359-375

Fast inference for robust nonlinear mixed-effects models

The interest for nonlinear mixed-effects models comes from application areas as pharmacokinetics, growth curves and HIV viral dynamics. However, the modeling procedure usually leads to many difficulties, as the inclusion of random effects, the estimation process and the model sensitivity to atypical or nonnormal data. The scale mixture of normal distributions include heavy-tailed models, as the Student-t, slash and contaminated normal distributions, and provide competitive alternatives to the usual models, enabling the obtention of robust estimates against outlying observations. Our proposal is to compare two estimation methods in nonlinear mixed-effects models where the random components follow a multivariate scale mixture of normal distributions. For this purpose, a Monte Carlo expectation-maximization algorithm (MCEM) and an efficient likelihood-based approximate method are developed. Results show that the approximate method is much faster and enables a fairly efficient likelihood maximization, although a slightly larger bias may be produced, especially for the fixed-effects parameters. A discussion on the robustness aspects of the proposed models are also provided. Two real nonlinear applications are discussed and a brief simulation study is presented.

Gaussian copula joint models for mixed longitudinal zero-inflated count and continuous responses

Gaussian copula joint models for mixed correlated longitudinal continuous and count responses with random effects are presented where the count responses have zero-inflated power series distribution. To account for associations between zero-inflated count and continuous responses, we use the Gaussian copula to indirectly specify their joint distributions. A full likelihood-based approach is applied to obtain IFM method to estimate marginal parameters marginally and share parameters jointly. In this method, we used the Monte Carlo EM algorithm to obtain the parameter estimates of Gaussian copula joint models. To illustrate the utility of the models, some simulation studies are performed. Finally, the proposed models are motivated by applying a medical data set. The data set is extracted from an observational study where the correlated responses are the continuous response of body mass index and the power series response of the number of joint damages.

Penalized Logistic Regression Analysis for Genetic Association Studies of Binary Phenotypes

Introduction: Increasingly, logistic regression methods for genetic association studies of binary phenotypes must be able to accommodate data sparsity, which arises from unbalanced case-control ratios and/or rare genetic variants. Sparseness leads to maximum likelihood estimators (MLEs) of log-OR parameters that are biased away from their null value of zero and tests with inflated type I errors. Different penalized likelihood methods have been developed to mitigate sparse data bias. We study penalized logistic regression using a class of log-F priors indexed by a shrinkage parameter m to shrink the biased MLE toward zero. Methods: We proposed a two-step approach to the analysis of a genetic association study: first, a set of variants that show evidence of association with the trait is used to estimate m; second, the estimated m is used for log-F-penalized logistic regression analyses of all variants using data augmentation with standard software. Our estimate of m is the maximizer of a marginal likelihood obtained by integrating the latent log-ORs out of the joint distribution of the parameters and observed data. We consider two approximate approaches to maximizing the marginal likelihood: (i) a Monte Carlo EM algorithm and (ii) a Laplace approximation to each integral, followed by derivative-free optimization of the approximation. Results: We evaluated the statistical properties of our proposed two-step method and compared its performance to other shrinkage methods by a simulation study. Our simulation studies suggest that the proposed log-F-penalized approach has lower bias and mean squared error than other methods considered. We also illustrated the approach on data from a study of genetic associations with “Super Senior” cases and middle-aged controls. Discussion/Conclusion: We have proposed a method for single rare variant analysis with binary phenotypes by logistic regression penalized by log-F priors. Our method has the advantage of being easily extended to correct for confounding due to population structure and genetic relatedness through a data augmentation approach.

The ARMA Point Process and its Estimation

An autoregressive–moving-average (ARMA) point process model is introduced, which combines self-exciting and shot-noise cluster mechanisms, both useful in a variety of applications. The process is analogous to the ARMA for integer-valued time series, sharing methodological and mathematical similarities. A maximum likelihood estimation procedure, based on MCEM (Monte Carlo Expectation Maximization), is derived and studied. This approach conveniently allows for: (i) trends in immigration/background intensity, (ii) multiple parametric specifications of memory functions and mark distributions, as well as (iii) cases where marks and immigrants are not observed. As such, the ARMA point process provides a flexible framework to disentangle cluster mechanisms in continuously observed count data.

Reducing subspace models for large-scale covariance regression.

We develop an envelope model for joint mean and covariance regression in the large p, small n setting. In contrast to existing envelope methods, which improve mean estimates by incorporating estimates of the covariance structure, we focus on identifying covariance heterogeneity by incorporating information about mean-level differences. We use a Monte Carlo EM algorithm to identify a low-dimensional subspace that explains differences in both means and covariances as a function of covariates, and then use MCMC to estimate the posterior uncertainty conditional on the inferred low-dimensional subspace. We demonstrate the utility of our model on a motivating application on the metabolomics of aging. We also provide R code that can be used to develop and test other generalizations of the response envelopemodel.

Censored Regression for Modelling Small Arms Trade Volumes and Its ‘Forensic’ Use for Exploring Unreported Trades

Abstract In this paper, we use a censored regression model to investigate data on the international trade of small arms and ammunition provided by the Norwegian Initiative on Small Arms Transfers. Taking a network-based view on the transfers, we do not only rely on exogenous covariates but also estimate endogenous network effects. We apply a spatial autocorrelation gravity model with multiple weight matrices. The likelihood is maximized employing the Monte Carlo expectation maximization algorithm. Our approach reveals strong and stable endogenous network effects. Furthermore, we find evidence for a substantial path dependence as well as a close connection between exports of civilian and military small arms. The model is then used in a ‘forensic’ manner to analyse latent network structures and thereby to identify countries with higher or lower tendency to export or import than reflected in the data. The approach is also validated using a simulation study.

Estimating Size and Number Density of Three-Dimensional Particles Using Truncated Cross-Sectional Data

Abstract The need for estimating three-dimensional (3D) information based on two-dimensional (2D) images has been increasing in numerous fields. It is essential in quality assessment, quality control, and process optimization. However, all the existing methods have not considered the data truncation issue, which is commonly faced in metrology. This paper proposes a new statistical approach to infer size distribution and volume number density (VND) of 3D particles based on 2D cross-sectional images with data truncation considered. In order to estimate the size distribution, a linkage is established between 3D particles and 2D observations with the existence of data truncation. Subsequently, this paper derives the likelihood function of 2D observations and an efficient Monte Carlo expectation-maximization algorithm is developed to estimate the parameters of size distribution. In addition, an explicit relationship between the 3D and 2D particle number densities is established and leveraged to estimate the VND and volume fraction. The effectiveness of the proposed method is demonstrated through both simulation study and real case studies in metal additive manufacturing and metal-matrix nanocomposites manufacturing.

Computationally efficient application of Sequential Monte Carlo expectation maximization to confined single particle tracking.

Single Particle Tracking (SPT) plays a crucial role in biophysics through its ability to reveal dynamic mechanisms and physical properties of biological macromolecules moving inside living cells. Such molecules are often subject to confinement and important information can be revealed by understanding the mobility of the molecules and the size of the domain they are restricted to. In previous work, we introduced a method known as Sequential Monte Carlo-Expectation Maximization (SMC-EM) to simultaneously estimate particle trajectories and model parameters. In this paper, we describe three modifications to SMC-EM aimed at improving its computationally efficiency and demonstrate it through analysis of simulated SPT data of a particle in a three dimensional confined environment. The first two modifications use approximation methods to reduce the complexity of the original motion and measurement models without significant loss of accuracy. The third modification replaces the previous SMC methods with a Gaussian particle filter combined with a backward simulation particle smoother, trading off some level of generality for improved computational performance. In addition, we take advantage of the improved efficiency to investigate the effect of data length on performance in localization and parameter estimation.

Copula-frailty models for recurrent event data based on Monte Carlo EM algorithm

Multi-type recurrent events are often encountered in medical applications when two or more different event types could repeatedly occur over an observation period. For example, patients may experience recurrences of multi-type nonmelanoma skin cancers in a clinical trial for skin cancer prevention. The aims in those applications are to characterize features of the marginal processes, evaluate covariate effects, and quantify both the within-subject recurrence dependence and the dependence among different event types. We use copula-frailty models to analyze correlated recurrent events of different types. Parameter estimation and inference are carried out by using a Monte Carlo expectation-maximization (MCEM) algorithm, which can handle a relatively large (i.e. three or more) number of event types. Performances of the proposed methods are evaluated via extensive simulation studies. The developed methods are used to model the recurrences of skin cancer with different types.

Inference over radiative transfer models using variational and expectation maximization methods

Earth observation from satellites offers the possibility to monitor our planet with unprecedented accuracy. Radiative transfer models (RTMs) encode the energy transfer through the atmosphere, and are used to model and understand the Earth system, as well as to estimate the parameters that describe the status of the Earth from satellite observations by inverse modeling. However, performing inference over such simulators is a challenging problem. RTMs are nonlinear, non-differentiable and computationally costly codes, which adds a high level of difficulty in inference. In this paper, we introduce two computational techniques to infer not only point estimates of biophysical parameters but also their joint distribution. One of them is based on a variational autoencoder approach and the second one is based on a Monte Carlo Expectation Maximization (MCEM) scheme. We compare and discuss benefits and drawbacks of each approach. We also provide numerical comparisons in synthetic simulations and the real PROSAIL model, a popular RTM that combines land vegetation leaf and canopy modeling. We analyze the performance of the two approaches for modeling and inferring the distribution of three key biophysical parameters for quantifying the terrestrial biosphere.

Estimation of multidimensional item response theory models with correlated latent variables using variational autoencoders

Artificial neural networks with a specific autoencoding structure are capable of estimating parameters for the multidimensional logistic 2-parameter (ML2P) model in item response theory (Curi et al. in International joint conference on neural networks (IJCNN), 2019), but with limitations, such as uncorrelated latent traits. In this work, we extend variational auto encoders (VAE) to estimate item parameters and correlated latent abilities, and directly compare the ML2P-VAE method to more traditional parameter estimation methods, such as Monte Carlo expectation-maximization. The incorporation of a non-identity covariance matrix in a VAE requires a novel VAE architecture, which can be utilized in applications outside of education. In addition, we show that the ML2P-VAE method is capable of estimating parameters for models with a large number of latent variables with low computational cost, where traditional methods are infeasible for data with high-dimensional latent traits.

Inference based on partly interval censored data from a two-parameter Rayleigh distribution

In this paper, the maximum likelihood and Bayesian estimation of the parameters of location-scale Rayleigh distribution with partly interval censored data is considered. For computing the maximum likelihood estimators with partly interval censored data, three methods are used, namely, Newton-Raphson, Expectation-Maximization and Monte-Carlo Expectation-Maximization algorithms. The standard errors of the estimates are computed using the observed information matrix. Also, two types of confidence intervals are constructed using the Wald method and the nonparametric percentile bootstrap confidence intervals. For computing the Bayes estimators, three methods viz Lindley's approximation, Tierney-Kadane approximation and importance sampling methods are used. Highest posterior density (HPD) credible intervals of the two parameters are constructed using importance sampling technique. Monte-Carlo simulation experiments are conducted to investigate the performance of the proposed methods. Finally, the methods are illustrated by using two real data sets, one is related with diabetic patients data set and the other is related to HIV infection data set.

Estimation and prediction of a generalized mixed-effects model with t-process for longitudinal correlated binary data

We propose a generalized mixed-effects model based on t-process for longitudinal correlated binary data. The correlations among repeated binary outcomes are defined by a latent t-process, which provides a new framework on modeling nonlinear random- effects. The covariance kernel of the process can adaptively capture the subject-specific variations while the heavy-tails of the t-process enable robust inferences. We develop an efficient estimation procedure based on Monte Carlo EM algorithm and a prediction approach through conditional inference. Numerical studies indicate that the estimation and prediction based on the proposed model is robust against outliers compared with Gaussian model. We use the renal anemia and meteorological data as illustrative examples.

A Monte Carlo Expectation Maximization Algorithm for Statistical Inference of Weibull Process with Left Censored Data

The Weibull process plays an important role in the failure analysis of repairable systems. In practice, there exists a situation that the data collected are incomplete. Some of the failure data are missing due to various reasons. Statistical inferences of a Weibull process with incomplete data using Monte Carlo expectation maximization algorithm is proposed. The estimation procedures are derived. A case study is performed to illustrate and compare the performances of this algorithm. It is observed that this method is effective and can simplify the estimation.

Estimating cell-type-specific DNA methylation effects in heterogeneous cellular populations.

Aim: To develop a method for estimating cell-specific effects in epigenomic association studies in the presence of cell type heterogeneity. Materials & methods: We utilized Monte Carlo Expectation-Maximization algorithm with Metropolis-Hastings sampler to reconstruct the 'missing' cell-specific methylations and to estimate their associations with phenotypes free of confounding by cell type proportions. Results: Simulations showed reliable performance of the method under various settings including when the cell type is rare. Application to a real dataset recapitulated the directly measured cell-specific methylation pattern in whole blood. Conclusion: This work provides a framework to identify important cell groups and account for cell type composition useful for studying the role of epigenetic changes in human traits and diseases.

On the Beta Prime Prior for Scale Parameters in High-Dimensional Bayesian Regression Models

We study high-dimensional Bayesian linear regression with a general beta prime distribution for the scale parameter. Under the assumption of sparsity, we show that appropriate selection of the hyperparameters in the beta prime prior leads to the (near) minimax posterior contraction rate when $p \gg n$. For finite samples, we propose a data-adaptive method for estimating the hyperparameters based on marginal maximum likelihood (MML). This enables our prior to adapt to both sparse and dense settings, and under our proposed empirical Bayes procedure, the MML estimates are never at risk of collapsing to zero. We derive efficient Monte Carlo EM and variational EM algorithms for implementing our model, which are available in the R package NormalBetaPrime. Simulations and analysis of a gene expression data set illustrate our model's self-adaptivity to varying levels of sparsity and signal strengths.

Simultaneous variable selection and estimation for joint models of longitudinal and failure time data with interval censoring.

This paper discusses variable selection in the context of joint analysis of longitudinal data and failure time data. A large literature has been developed for either variable selection or the joint analysis but there exists only limited literature for variable selection in the context of the joint analysis when failure time data are right censored. Corresponding to this, we will consider the situation where instead of right-censored data, one observes interval-censored failure time data, a more general and commonly occurring form of failure time data. For the problem, a class of penalized likelihood-based procedures will be developed for simultaneous variable selection and estimation of relevant covariate effects for both longitudinal and failure time variables of interest. In particular, a Monte Carlo EM (MCEM) algorithm is presented for the implementation of the proposed approach. The proposed method allows for the number of covariates to be diverging with the sample size and is shown to have the oracle property. An extensive simulation study is conducted to assess the finite sample performance of the proposed approach and indicates that it works well in practical situations. An application is alsoprovided.