Complete Data Likelihood Research Articles

Diagonal latent block model for binary data

This paper addresses the problem of co-clustering binary data in the latent block model framework with diagonal constraints for resulting data partitions. We consider the Bernoulli generative mixture model and present three new methods differing in the assumptions made about the degree of homogeneity of diagonal blocks. The proposed models are parsimonious and allow to take into account the structure of a data matrix when reorganizing it into homogeneous diagonal blocks. We derive algorithms for each of the presented models based on the classification expectation-maximization algorithm which maximizes the complete data likelihood. We show that our contribution can outperform other state-of-the-art (co)-clustering methods on synthetic sparse and non-sparse data. We also prove the efficiency of our approach in the context of document clustering, by using real-world benchmark data sets.

Exact ICL maximization in a non-stationary temporal extension of the stochastic block model for dynamic networks

The stochastic block model (SBM) is a flexible probabilistic tool that can be used to model interactions between clusters of nodes in a network. However, it does not account for interactions of time varying intensity between clusters. The extension of the SBM developed in this paper addresses this shortcoming through a temporal partition: assuming that interactions between nodes are recorded on fixed-length time intervals, the inference procedure associated with the model we propose allows us to cluster simultaneously the nodes of the network and the time intervals. The number of clusters of nodes and of time intervals, as well as the memberships to clusters, are obtained by maximizing an exact integrated complete-data likelihood, relying on a greedy search approach. Experiments on simulated and real data are carried out in order to assess the proposed methodology.

Capture–recapture abundance estimation using a semi-complete data likelihood approach

Capture-recapture data are often collected when abundance estimation is of interest. In the presence of unobserved individual heterogeneity, specified on a continuous scale for the capture probabilities, the likelihood is not generally available in closed form, but expressible only as an analytically intractable integral. Model-fitting algorithms to estimate abundance most notably include a numerical approximation for the likelihood or use of a Bayesian data augmentation technique considering the complete data likelihood. We consider a Bayesian hybrid approach, defining a "semi-complete" data likelihood, composed of the product of a complete data likelihood component for individuals seen at least once within the study and a marginal data likelihood component for the individuals not seen within the study, approximated using numerical integration. This approach combines the advantages of the two different approaches, with the semi-complete likelihood component specified as a single integral (over the dimension of the individual heterogeneity component). In addition, the models can be fitted within BUGS/JAGS (commonly used for the Bayesian complete data likelihood approach) but with significantly improved computational efficiency compared to the commonly used super-population data augmentation approaches (between about 10 and 77 times more efficient in the two examples we consider). The semi-complete likelihood approach is flexible and applicable to a range of models, including spatially explicit capture-recapture models. The model-fitting approach is applied to two different datasets corresponding to the closed population model $M_h$ for snowshoe hare data and a spatially explicit capture-recapture model applied to gibbon data.

Choosing the number of clusters in a finite mixture model using an exact integrated completed likelihood criterion

The integrated completed likelihood (ICL) criterion has proven to be a very popular approach in model-based clustering through automatically choosing the number of clusters in a mixture model. This approach effectively maximises the complete data likelihood, thereby including the allocation of observations to clusters in the model selection criterion. However for practical implementation one needs to introduce an approximation in order to estimate the ICL. Our contribution here is to illustrate that through the use of conjugate priors one can derive an exact expression for ICL and so avoiding any approximation. Moreover, we illustrate how one can find both the number of clusters and the best allocation of observations in one algorithmic framework. The performance of our algorithm is presented on several simulated and real examples.

Pitfalls of estimating the marginal likelihood using the modified harmonic mean

The modified harmonic mean is widely used for estimating the marginal likelihood. We investigate the empirical performance of two versions of this estimator: one based on the observed-data likelihood and the other on the complete-data likelihood. Through an empirical example using US and UK inflation, we show that the version based on the complete-data likelihood has a substantial bias and tends to select the wrong model, whereas the version based on the observed-data likelihood works well.

Model selection and clustering in stochastic block models based on the exact integrated complete data likelihood

The stochastic block model (SBM) is a mixture model for the clustering of nodes in networks. The SBM has now been employed for more than a decade to analyze very different types of networks in many scientific fields, including biology and the social sciences. Recently, an analytical expression based on the collapsing of the SBM parameters has been proposed, in combination with a sampling procedure that allows the clustering of the vertices and the estimation of the number of clusters to be performed simultaneously. Although the corresponding algorithm can technically accommodate up to 10 000 nodes and millions of edges, the Markov chain, however, tends to exhibit poor mixing properties, that is, low acceptance rates, for large networks. Therefore, the number of clusters tends to be highly overestimated, even for a very large number of samples. In this article, we rely on a similar expression, which we call the integrated complete data log likelihood, and propose a greedy inference algorithm that focuses on maximizing this exact quantity. This algorithm incurs a smaller computational cost than existing inference techniques for the SBM and can be employed to analyze large networks (several tens of thousands of nodes and millions of edges) with no convergence problems. Using toy datasets, the algorithm exhibits improvements over existing strategies, both in terms of clustering and model selection. An application to a network of blogs related to illustrations and comics is also provided.

Pitfalls of Estimating the Marginal Likelihood Using the Modified Harmonic Mean

The modified harmonic mean is widely used for estimating the marginal likelihood. We investigate the empirical performance of two versions of this estimator: one based on the observed-data likelihood and the other on the complete-data likelihood. Through an empirical example using US and UK inflation, we show that the version based on the complete-data likelihood has a substantial finite sample bias and tends to select the wrong model, whereas the version based on the observed-data likelihood works well.

Analysing Mark–Recapture–Recovery Data in the Presence of Missing Covariate Data Via Multiple Imputation

We consider mark–recapture–recovery data with additional individual time-varying continuous covariate data. For such data it is common to specify the model parameters, and in particular the survival probabilities, as a function of these covariates to incorporate individual heterogeneity. However, an issue arises in relation to missing covariate values, for (at least) the times when an individual is not observed, leading to an analytically intractable likelihood. We propose a two-step multiple imputation approach to obtain estimates of the demographic parameters. Firstly, a model is fitted to only the observed covariate values. Conditional on the fitted covariate model, multiple “complete” datasets are generated (i.e. all missing covariate values are imputed). Secondly, for each complete dataset, a closed form complete data likelihood can be maximised to obtain estimates of the model parameters which are subsequently combined to obtain an overall estimate of the parameters. Associated standard errors and 95 % confidence intervals are obtained using a non-parametric bootstrap. A simulation study is undertaken to assess the performance of the proposed two-step approach. We apply the method to data collected on a well-studied population of Soay sheep and compare the results with a Bayesian data augmentation approach. Supplementary materials accompanying this paper appear on-line.

Bayesian modelling of compositional heterogeneity in molecular phylogenetics.

In molecular phylogenetics, standard models of sequence evolution generally assume that sequence composition remains constant over evolutionary time. However, this assumption is violated in many datasets which show substantial heterogeneity in sequence composition across taxa. We propose a model which allows compositional heterogeneity across branches, and formulate the model in a Bayesian framework. Specifically, the root and each branch of the tree is associated with its own composition vector whilst a global matrix of exchangeability parameters applies everywhere on the tree. We encourage borrowing of strength between branches by developing two possible priors for the composition vectors: one in which information can be exchanged equally amongst all branches of the tree and another in which more information is exchanged between neighbouring branches than between distant branches. We also propose a Markov chain Monte Carlo (MCMC) algorithm for posterior inference which uses data augmentation of substitutional histories to yield a simple complete data likelihood function that factorises over branches and allows Gibbs updates for most parameters. Standard phylogenetic models are not informative about the root position. Therefore a significant advantage of the proposed model is that it allows inference about rooted trees. The position of the root is fundamental to the biological interpretation of trees, both for polarising trait evolution and for establishing the order of divergence among lineages. Furthermore, unlike some other related models from the literature, inference in the model we propose can be carried out through a simple MCMC scheme which does not require problematic dimension-changing moves. We investigate the performance of the model and priors in analyses of two alignments for which there is strong biological opinion about the tree topology and root position.

Mark‐resight abundance estimation under incomplete identification of marked individuals

Summary Often less expensive and less invasive than conventional mark–recapture, so‐called 'mark‐resight' methods are popular in the estimation of population abundance. These methods are most often applied when a subset of the population of interest is marked (naturally or artificially), and non‐invasive sighting data can be simultaneously collected for both marked and unmarked individuals. However, it can often be difficult to identify marked individuals with certainty during resighting surveys, and incomplete identification of marked individuals is potentially a major source of bias in mark‐resight abundance estimators. Previously proposed solutions are ad hoc and will tend to underperform unless marked individual identification rates are relatively high (>90%) or individual sighting heterogeneity is negligible. Based on a complete data likelihood, we present an approach that properly accounts for uncertainty in marked individual detection histories when incomplete identifications occur. The models allow for individual heterogeneity in detection, sampling with (e.g. Poisson) or without (e.g. Bernoulli) replacement, and an unknown number of marked individuals. Using a custom Markov chain Monte Carlo algorithm to facilitate Bayesian inference, we demonstrate these models using two example data sets and investigate their properties via simulation experiments. We estimate abundance for grassland sparrow populations in Pennsylvania, USA when sampling was conducted with replacement and the number of marked individuals was either known or unknown. To increase marked individual identification probabilities, extensive territory mapping was used to assign incomplete identifications to individuals based on location. Despite marked individual identification probabilities as low as 67% in the absence of this territorial mapping procedure, we generally found little return (or need) for this time‐consuming investment when using our proposed approach. We also estimate rookery abundance from Alaskan Steller sea lion counts when sampling was conducted without replacement, the number of marked individuals was unknown, and individual heterogeneity was suspected as non‐negligible. In terms of estimator performance, our simulation experiments and examples demonstrated advantages of our proposed approach over previous methods, particularly when marked individual identification probabilities are low and individual heterogeneity levels are high. Our methodology can also reduce field effort requirements for marked individual identification, thus, allowing potential investment into additional marking events or resighting surveys.

Estimating the Burr XII Parameters in Constant-Stress Partially Accelerated Life Tests Under Multiple Censored Data

In this article, we present the performance of the maximum likelihood estimates of the Burr XII parameters for constant-stress partially accelerated life tests under multiple censored data. Two maximum likelihood estimation methods are considered. One method is based on observed-data likelihood function and the maximum likelihood estimates are obtained by using the quasi-Newton algorithm. The other method is based on complete-data likelihood function and the maximum likelihood estimates are derived by using the expectation-maximization (EM) algorithm. The variance–covariance matrices are derived to construct the confidence intervals of the parameters. The performance of these two algorithms is compared with each other by a simulation study. The simulation results show that the maximum likelihood estimation via the EM algorithm outperforms the quasi-Newton algorithm in terms of the absolute relative bias, the bias, the root mean square error and the coverage rate. Finally, a numerical example is given to illustrate the performance of the proposed methods.

Improved Convolutive and Under-Determined Blind Audio Source Separation with MRF Smoothing

Convolutive and under-determined blind audio source separation from noisy recordings is a challenging problem. Several computational strategies have been proposed to address this problem. This study is concerned with several modifications to the expectation-minimization-based algorithm, which iteratively estimates the mixing and source parameters. This strategy assumes that any entry in each source spectrogram is modeled using superimposed Gaussian components, which are mutually and individually independent across frequency and time bins. In our approach, we resolve this issue by considering a locally smooth temporal and frequency structure in the power source spectrograms. Local smoothness is enforced by incorporating a Gibbs prior in the complete data likelihood function, which models the interactions between neighboring spectrogram bins using a Markov random field. Simulations using audio files derived from stereo audio source separation evaluation campaign 2008 demonstrate high efficiency with the proposed improvement.

Variational Bayesian inference and complexity control for stochastic block models

It is now widely accepted that knowledge can be acquired from networks by clustering their vertices according to the connection profiles. Many methods have been proposed and in this paper we concentrate on the Stochastic Block Model (SBM). The clustering of vertices and the estimation of SBM model parameters have been subject to previous work, and numerous inference strategies such as variational expectation maximization (EM) and classification EM have been proposed. However, SBM still suffers from a lack of criteria to estimate the number of components in the mixture. To our knowledge, only one model-based criterion, Integrated Complete-data Likelihood (ICL), has been derived for SBM in the literature. It relies on an asymptotic approximation of the integrated complete-data likelihood and recent studies have shown that it tends to be too conservative in the case of small networks. To tackle this issue, we propose a new criterion that we call Integrated Likelihood Variational Bayes (ILvb), based on a non-asymptotic approximation of the marginal likelihood. We describe how the criterion can be computed through a variational Bayes EM algorithm.

Full Open Population Capture–Recapture Models With Individual Covariates

Traditional analyses of capture–recapture data are based on likelihood functions that explicitly integrate out all missing data. We use a complete data likelihood (CDL) to show how a wide range of capture–recapture models can be easily fitted using readily available software JAGS/BUGS even when there are individual-specific time-varying covariates. The models we describe extend those that condition on first capture to include abundance parameters, or parameters related to abundance, such as population size, birth rates or lifetime. The use of a CDL means that any missing data, including uncertain individual covariates, can be included in models without the need for customized likelihood functions. This approach also facilitates modeling processes of demographic interest rather than the complexities caused by non-ignorable missing data. We illustrate using two examples, (i) open population modeling in the presence of a censored time-varying individual covariate in a full robust design, and (ii) full open population multi-state modeling in the presence of a partially observed categorical variable. Supplemental materials for this article are available online.

Exact and Monte Carlo calculations of integrated likelihoods for the latent class model

The latent class model or multivariate multinomial mixture is a powerful approach for clustering categorical data. It uses a conditional independence assumption given the latent class to which a statistical unit is belonging. In this paper, we exploit the fact that a fully Bayesian analysis with Jeffreys non-informative prior distributions does not involve technical difficulty to propose an exact expression of the integrated complete-data likelihood, which is known as being a meaningful model selection criterion in a clustering perspective. Similarly, a Monte Carlo approximation of the integrated observed-data likelihood can be obtained in two steps: an exact integration over the parameters is followed by an approximation of the sum over all possible partitions through an importance sampling strategy. Then, the exact and the approximate criteria experimentally compete, respectively, with their standard asymptotic BIC approximations for choosing the number of mixture components. Numerical experiments on simulated data and a biological example highlight that asymptotic criteria are usually dramatically more conservative than the non-asymptotic presented criteria, not only for moderate sample sizes as expected but also for quite large sample sizes. This research highlights that asymptotic standard criteria could often fail to select some interesting structures present in the data.

Comments on: A review on empirical likelihood methods for regression

I congratulate Chen and Van Keilegom for writing this extensive and useful review on empirical likelihood methods in many contexts of regression. I am also very grateful to the editors of Test for the opportunity to take part in this discussion. This invited paper starts with parametric regression, where the main concepts and advantages of empirical likelihood are clearly shown. Later on, nonparametric and semiparametric models, missing and censored data, and a goodness-of-fit test are treated with the most recent and relevant proposals in each framework. For this reason, this review is very useful both as an introduction to the empirical likelihood methods for regression and as an updating on recent developments. My first specific comment is about the possibility of using empirical likelihood to obtain confidence bands in nonparametric regression. I did not find this in the literature, so I wonder whether it has been done, and if not, how it could be done. In parametric regression with censored response (Sect. 6.1), this review presents a very interesting option between a censored data likelihood and a complete data likelihood. The main details of Zhou and Li (2008) are given, where a censored data likelihood is used for mean regression. The proposal by Qin and Tsao (2003) is also presented, where a complete data likelihood is used in the context of median regression. Finally, the advantage of censored data likelihood is stressed, in the sense that it does not require the estimation of unknown functions. This is very appealing, so I looked for other frameworks where a censored data likelihood could be applied. I think it is the case of Stute (1999), where a weighted

Parameter estimation in continuous time Markov switching models: a semi-continuous Markov chain Monte Carlo approach

We want to estimate the parameters governing a continuous time Markov switching model given observations at discrete times only. For parameter estimation in a setting with continuous time and a latent state process, using MCMC methods two approaches are common: Using time-discretization and augmenting the unknowns with the (then discrete) state process, or working in continuous time and augmenting with the full state process. In this paper, we combine useful aspects of both approaches. On the one hand, we are inspired by the discretization, where filtering for the state process is possible, on the other hand, we catch attractive features of the continuous time method, like exact estimation (i.e. no discretization error) and direct estimation of the generator matrix rather than the transition matrix. This is achieved by taking not the whole state process for data augmentation but only the states at observation times. Using results on the distribution of occupation times in Markov processes, it is possible to compute the complete data likelihood exactly. We obtain a sampler that works more robustly and more accurately especially for fast switching in the state process.

Data augmentation, frequentist estimation, and the Bayesian analysis of multinomial logit models

This article describes a convenient method of selecting Metropolis– Hastings proposal distributions for multinomial logit models. There are two key ideas involved. The first is that multinomial logit models have a latent variable representation similar to that exploited by Albert and Chib (J Am Stat Assoc 88:669–679, 1993) for probit regression. Augmenting the latent variables replaces the multinomial logit likelihood function with the complete data likelihood for a linear model with extreme value errors. While no conjugate prior is available for this model, a least squares estimate of the parameters is easily obtained. The asymptotic sampling distribution of the least squares estimate is Gaussian with known variance. The second key idea in this paper is to generate a Metropolis–Hastings proposal distribution by conditioning on the estimator instead of the full data set. The resulting sampler has many of the benefits of so-called tailored or approximation Metropolis–Hastings samplers. However, because the proposal distributions are available in closed form they can be implemented without numerical methods for exploring the posterior distribution. The algorithm converges geometrically ergodically, its computational burden is minor, and it requires minimal user input. Improvements to the sampler’s mixing rate are investigated. The algorithm is also applied to partial credit models describing ordinal item response data from the 1998 National Assessment of Educational Progress. Its application to hierarchical models and Poisson regression are briefly discussed.

Bayesian analysis of mark‐recapture data with travel time‐dependent survival probabilities

AbstractThe authors extend the classical Cormack‐Jolly‐Seber mark‐recapture model to account for both temporal and spatial movement through a series of markers (e.g., dams). Survival rates are modeled as a function of (possibly) unobserved travel times. Because of the complex nature of the likelihood, they use a Bayesian approach based on the complete data likelihood, and integrate the posterior through Markov chain Monte Carlo methods. They test the model through simulations and apply it also to actual salmon data arising from the Columbia river system. The methodology was developed for use by the Pacific Ocean Shelf Tracking (POST) project.

Flexible hierarchical mark-recapture modeling for open populations using WinBUGS

Hierarchical mark-recapture models offer three advantages over classical mark-recapture models: (i) they allow expression of complicated models in terms of simple components; (ii) they provide a convenient way of modeling missing data and latent variables in a way that allows expression of relationships involving latent variables in the model; (iii) they provide a convenient way of introducing parsimony into models involving many nuisance parameters. Expressing models using the complete data likelihood we show how many of the standard mark-recapture models for open populations can be readily fitted using the software WinBUGS. We include examples that illustrate fitting the Cormack–Jolly–Seber model, multi-state and multi-event models, models including auxiliary data, and models including density dependence.