An EM algorithm for estimation of the parameters of the geometric minification INAR model

Maximum likelihood estimation with missing outcomes: From simplicity to complexity.

In this paper, we propose an improved Expectation Maximization (EM) algorithm for hyperspectral image classification. As an excellent machine learning algorithm, EM is an iterative process for finding Maximum A Posteriori estimation (MAP) of parameters in Gaussian Mixture Models (GMMs). With the ability to deal with missing data, EM is considered excellent for solving the insufficient samples training problem of hyperspectral data classification. In the new algorithm, specially aimed at highly mixing hyperspectral data, endmember class separability metric is added into the convergence criteria of improved EM, which may yield better classification result than traditional EM. Three classification algorithms based on statistical probability were tested: the maximum likelihood method (ML), traditional EM, and improved EM. Experimental results on simulated data and real hyperspectral image demonstrate that improved EM can get higher classification accuracy in the case of a small number of training samples.

Mixture distribution is a very flexible distribution for modeling data. If the mixture distribution fitted to the data, the parameters are generally not known. This paper examines the estimation problem of the distribution mixture parameter. The method used to estimate the parameters of a mixture distribution is the maximum likelihood method. However, the maximum likelihood estimator can not be found analytically. To overcome these problems, the EM algorithm proposed. EM algorithm consists of two phases: expectation and maximization stage. EM algorithm converges to the maximum likelihood estimator. Performance of the EM algorithm is tested using simulated data. The results show that the EM algorithm can estimate the mixture distribution parameter. Further EM algorithm is implemented on real data encountered in everyday life.

Many diseases show dichotomous phenotypic variation but do not follow a simple Mendelian pattern of inheritance. Variances of these binary diseases are presumably controlled by multiple loci and environmental variants. A least-squares method has been developed for mapping such complex disease loci by treating the binary phenotypes (0 and 1) as if they were continuous. However, the least-squares method is not recommended because of its ad hoc nature. Maximum Likelihood (ML) and Bayesian methods have also been developed for binary disease mapping by incorporating the discrete nature of the phenotypic distribution. In the ML analysis, the likelihood function is usually maximized using some complicated maximization algorithms (e.g. the Newton-Raphson or the simplex algorithm). Under the threshold model of binary disease, we develop an Expectation Maximization (EM) algorithm to solve for the maximum likelihood estimates (MLEs). The new EM algorithm is developed by treating both the unobserved genotype and the disease liability as missing values. As a result, the EM iteration equations have the same form as the normal equation system in linear regression. The EM algorithm is further modified to take into account sexual dimorphism in the linkage maps. Applying the EM-implemented ML method to a four-way-cross mouse family, we detected two regions on the fourth chromosome that have evidence of QTLs controlling the segregation of fibrosarcoma, a form of connective tissue cancer. The two QTLs explain 50-60% of the variance in the disease liability. We also applied a Bayesian method previously developed (modified to take into account sex-specific maps) to this data set and detected one additional QTL on chromosome 13 that explains another 26% of the variance of the disease liability. All the QTLs detected primarily show dominance effects.

Felsenstein, J. (Department of Genetics SK-50, University of Washington, Seattle, Washington 98195). 1973. Maximum likelihood and minimum-steps methods for estimating evolutionary trees from data on discrete characters. Syst. Zool. 22:240-249.The general maximum likelihood approach to the statistical estimation of phylogenies is outlined, for data in which there are a number of discrete states for each character. The details of the maximum likelihood method will depend on the details of the probabilistic model of evolution assumed. There are a very large number of possible models of evolution. For a few of the simpler models, the calculation of the likelihood of an evolutionary tree is outlined. For these models, the maximum likelihood tree will be the same as the parsimonious (or minimum-steps) tree if the probability of change during the evolution of the group is assumed a priori to be very small. However, most sets of data require too many assumed state changes per character to be compatible with this assumption. Farris (1973) has argued that maximum likelihood and parsimony methods are identical under a much less restrictive set of assumptions. It is argued that the present methods are preferable to his, and a counterexample to his argument is presented. An algorithm which enables rapid calculation of the likelihood of a phylogeny is described. [Evolutionary trees: maximum likelihood.] The first systematic attempt to apply standard statistical inference procedures to the estimation of evolutionary trees was the work of Edwards and Cavalli-Sforza (1964; see also Cavalli-Sforza and Edwards, 1967). At about the same time, the parsimony' or minimum evolutionary steps method of Camin and Sokal (1965) gave a great impetus to the development of welldefined procedures for obtaining evolutionary trees. Edwards and Cavalli-Sforza concerned themselves with data from continuous variables such as gene frequencies and quantitative characters. The CaminSokal approach, on the other hand, was developed for characters which are recorded as a series of discrete states. Although some taxonomists have declared that the problem of guessing phylogenies should be viewed as a problem of statistical inference (Farris, 1967, 1968; Throckmorton, 1968), until recently there have been no attempts to explore the relationship between the statistical inference and minimum-steps approaches. Recently, Farris (1973) has presented a detailed argument that, under certain reasonable assumptions, the maximum-likelihood method of statistical inference appropriate to discrete-character data is precisely the parsimony method of Camin and Sokal. In this paper, I will examine the application of maximum likelihood methods to discrete characters, and will show that parsimony methods are not maximum likelihood methods under the assumptions made by Farris. They are maximum likelihood methods under considerably more restrictive assumptions about evolution. METHODS OF MAXIMUM LIKELIHOOD Suppose that we want to estimate the evolutionary tree, T, which is to be specified by the topological form of the tree and the times of branching. We are given a set of data, D, and a model of evolution, M, which incorporates not only the evolutionary processes, but also the processes of sampling by which we obtained the data. This model will usually be probabilistic, involving random events such as changes of the environment, occurrence of favorable

Objectives: This study investigated the efficiency of Multiple Imputation (MI) and Maximum Likelihood (ML) methods for estimating missing values. The study was set to use the findings to make recommendations for future studies about the impact of missing data imputation on the accuracy of Akaike Information Criterion (AIC) and Bayesian Information Criterion (BIC). Methods: The completedset (with no missing values) used in this study was collected in 2010/11 through the Income and Expenditure Survey (IES) and had 25328 observations. Missing data were generated by randomly deleting 10%, 20%, 30%, 40% and 50% of the values from the complete dataset. The missing values in each of the five datasets were imputed using MI and ML methods. Subsequently, absolute error values of AIC and BIC from multiple regression analysis were computed for each dataset. The study then compared the absolute errors for each missing value imputation method. Findings: The findings of the study revealed that AIC and BIC are more accurate when missing values are estimated by the Full Information Maximum Likelihood (FIML) of the ML algorithm, provided 10% of the data are missing. For all datasets, AIC and BIC were least accurate when missing values were imputed by Expectation Maximisation (EM) of the ML algorithm. The findings also showed that AIC and BIC are more accurate when the rate of MISSINGNESS gets large provided missing values were estimated using either the Fully Conditional Specification (FCS) or Markov Chain Monte Carlo (MCMC), MI algorithms. Application: When the rate of MISSINGNESS is small (at most 10%), FIML should be used to handle missing data if AIC and BIC are going to be used. Also both FCS and MCMC should be considered over EM algorithms when the rate of MISSINGNESS is high (at least 40% missing). Keywords: Maximum Likelihood Imputation, Multiple Imputation, AIC, BIC

Pharmacokinetic (PK) data often contain concentration measurements below the quantification limit (BQL). While specific values cannot be assigned to these observations, nevertheless these observed BQL data are informative and generally known to be lower than the lower limit of quantification (LLQ). Setting BQLs as missing data violates the usual missing at random (MAR) assumption applied to the statistical methods, and therefore leads to biased or less precise parameter estimation. By definition, these data lie within the interval [0, LLQ], and can be considered as censored observations. Statistical methods that handle censored data, such as maximum likelihood and Bayesian methods, are thus useful in the modelling of such data sets. The main aim of this work was to investigate the impact of the amount of BQL observations on the bias and precision of parameter estimates in population PK models (non-linear mixed effects models in general) under maximum likelihood method as implemented in SAS and NONMEM, and a Bayesian approach using Markov chain Monte Carlo (MCMC) as applied in WinBUGS. A second aim was to compare these different methods in dealing with BQL or censored data in a practical situation. The evaluation was illustrated by simulation based on a simple PK model, where a number of data sets were simulated from a one-compartment first-order elimination PK model. Several quantification limits were applied to each of the simulated data to generate data sets with certain amounts of BQL data. The average percentage of BQL ranged from 25% to 75%. Their influence on the bias and precision of all population PK model parameters such as clearance and volume distribution under each estimation approach was explored and compared.

The most popular estimation approach is the maximum likelihood (ML) method. In this chapter, the ML estimator is defined first, and important asymptotic properties of the ML estimator are formulated in Sect. 4.2. Trans- formations of estimators, not only ML estimators, are discussed in Sect. 4.3. To illustrate the ML approach, we consider the ML method in the linear exponential family (Sect. 4.4) and in univariate GLM (Sect. 4.5). A crucial assumption of ML estimation is the correct specification of the underlying statistical model. Therefore, we discuss the consequences of using the ML method in misspecified models in Sect. 4.6. Even if the model is misspecified, it is based on a likelihood, and the resulting estimator is therefore called a quasi maximum likelihood (QML) estimator (for an in-depth discussion, see White, 1982, 1994). The reader should note that QML estimation is different from quasi likelihood (QL) estimation. The latter approach is a generalization of the generalized linear model (McCullagh and Nelder, 1989; Wedderburn, 1974) and requires the correct specification of the first two moments.

Comparisons of Two Methods for Haplotype Reconstruction and Haplotype Frequency Estimation from Population Data

Latent trait models such as item response theory (IRT) hypothesize a functional relationship between an unobservable, or latent, variable and an observable outcome vari- able. In educational measurement, a discrete item response is usually the observable out- come variable, and the latent variable is associated with an examinee's trait level (e.g., skill, proficiency). The link between the two variables is called an item response function. This function, defined by a set of item parameters, models the probability of observing a given item response, conditional on a specific trait level. Typically in a measurement setting, neither the item parameters nor the trait levels are known, and so must be estimated from the pattern of observed item responses. Although a maximum likelihood approach can be taken in estimating these parameters, it usually cannot be employed directly. Instead, a method of marginal maximum likelihood (MML) is utilized, via the expectation-maximization (EM) algorithm. Alternating between an expectation (E) step and a maximization (M) step, the EM algorithm assures that the marginal log likelihood function will not decrease after each EM cycle, and will converge to a local maximum. Interestingly, the negative of this marginal log likelihood function is equal to the relative entropy, or Kullback-Leibler divergence, between the conditional distribution of the latent variables given the observable variables and the joint likelihood of the latent and observable variables. With an unconstrained optimization for the M-step proposed here, the EM algo- rithm as minimization of Kullback-Leibler divergence admits the convergence results due to Csiszar and Tusnady (Statistics & Decisions, 1:205-237, 1984), a consequence of the bi- nomial likelihood common to latent trait models with dichotomous response variables. For this unconstrained optimization, the EM algorithm converges to a global maximum of the marginal log likelihood function, yielding an information bound that permits a fixed point of reference against which models may be tested. A likelihood ratio test between marginal log likelihood functions obtained through constrained and unconstrained M-steps is pro- vided as a means for testing models against this bound. Empirical examples demonstrate the

It has been shown using simulated data that phase estimation of cross-track multibaseline synthetic aperture radar (SAR) interferometric data was most efficiently achieved through a maximum likelihood (ML) method. In this paper, we apply and assess the ML approach on real data, acquired with an experimental Ka-band multibaseline system. Compared to simulated data, dealing with real data implies that several calibration steps be carried out to ensure that the data fit the model. A processing chain was, therefore, designed, including steps responsible for compensating for imperfections observed in the data, such as beam elevation angle dependent phase errors or phase errors caused by imperfect motion compensation. The performance of the ML phase estimation was evaluated by comparing it to results based on a coarse-to-fine (C2F) algorithm, where information from the shorter baselines was used only to unwrap the phase from the longest available baseline. For this purpose, flat areas with high coherence and homogeneous texture were selected in the acquired data. The results show that with only four looks, the noise level was marginally better with the C2F approach and contained fewer outliers. However, with more looks, the ML method consistently delivered better results: noise variance with the C2F approach was slightly but steadily larger than the variance obtained with ML method.

Pseudo-stochastic EM for sub-Gaussian α-stable mixture models

In this paper, the Weibull extension distribution parameters are estimated under a progressive type-II censoring scheme with random removal. The parameters of the model are estimated using the maximum likelihood method, maximum product spacing, and Bayesian estimation methods. In classical estimation (maximum likelihood method and maximum product spacing), we did use the Newton–Raphson algorithm. The Bayesian estimation is done using the Metropolis–Hastings algorithm based on the square error loss function. The proposed estimation methods are compared using Monte Carlo simulations under a progressive type-II censoring scheme. An empirical study using a real data set of transformer insulation and a simulation study is performed to validate the introduced methods of inference. Based on the result of our study, it can be concluded that the Bayesian method outperforms the maximum likelihood and maximum product-spacing methods for estimating the Weibull extension parameters under a progressive type-II censoring scheme in both simulation and empirical studies.

In the field of statistical data mining, the Expectation Maximization (EM) algorithm is one of the most popular methods used for solving parameter estimation problems in the maximum likelihood (ML) framework. Compared to traditional methods such as steepest descent, conjugate gradient, or Newton-Raphson, which are often too complicated to use in solving these problems, EM has become a popular method because it takes advantage of some problem specific properties (Xu et al., 1996). The EM algorithm converges to the local maximum of the log-likelihood function under very general conditions (Demspter et al., 1977; Redner et al., 1984). Efficiently maximizing the likelihood by augmenting it with latent variables and guarantees of convergence are some of the important hallmarks of the EM algorithm. EM based methods have been applied successfully to solve a wide range of problems that arise in fields of pattern recognition, clustering, information retrieval, computer vision, bioinformatics (Reddy et al., 2006; Carson et al., 2002; Nigam et al., 2000), etc. Given an initial set of parameters, the EM algorithm can be implemented to compute parameter estimates that locally maximize the likelihood function of the data. In spite of its strong theoretical foundations, its wide applicability and important usage in solving some real-world problems, the standard EM algorithm suffers from certain fundamental drawbacks when used in practical settings. Some of the main difficulties of using the EM algorithm on a general log-likelihood surface are as follows (Reddy et al., 2008): • EM algorithm for mixture modeling converges to a local maximum of the log-likelihood function very quickly. • There are many other promising local optimal solutions in the close vicinity of the solutions obtained from the methods that provide good initial guesses of the solution. • Model selection criterion usually assumes that the global optimal solution of the log-likelihood function can be obtained. However, achieving this is computationally intractable. • Some regions in the search space do not contain any promising solutions. The promising and nonpromising regions co-exist and it becomes challenging to avoid wasting computational resources to search in non-promising regions. Of all the concerns mentioned above, the fact that most of the local maxima are not distributed uniformly makes it important to develop algorithms that not only help in avoiding some inefficient search over the lowlikelihood regions but also emphasize the importance of exploring promising subspaces more thoroughly (Zhang et al, 2004). This subspace search will also be useful for making the solution less sensitive to the initial set of parameters. In this chapter, we will discuss the theoretical aspects of the EM algorithm and demonstrate its use in obtaining the optimal estimates of the parameters for mixture models. We will also discuss some of the practical concerns of using the EM algorithm and present a few results on the performance of various algorithms that try to address these problems.

Centrifugal pumps are widely employed in the oil refinery industry due to their efficiency and effectiveness in fluid transfer applications. The reliability of pumps plays a pivotal role in ensuring uninterrupted plant productivity and safe operations. Analysis of failure history data shows that bearings have been identified as critical components in oil refinery pump groups. Analyzing historical failure data for such systems is a complex task due to censored data and missing information. This paper addresses the complexity of estimating the Weibull distribution parameters using the maximum likelihood method under these conditions. The likelihood equation lacks an explicit analytical solution, necessitating numerical methods for resolution. The proposed approach presented in this article leverages the expectation maximization (EM) algorithm for estimating the Weibull distribution parameters in a real-world case study of a complex engineering system. The results demonstrate the superior performance of the EM algorithm with censored data, showcasing its ability to overcome the limitations of traditional methods and provide more accurate estimates for reliability metrics. This highlights the importance of obtaining results through these methodologies in the analysis of reliability and in facilitating more informed decision making in complex systems.