Marginal Likelihood Estimation Research Articles

Tuning complexity in regularized kernel-based regression and linear system identification: The robustness of the marginal likelihood estimator

Kernel-based regularization approaches have been successfully applied in the last years for regression purposes. Recently, these machine learning techniques have been also introduced in linear system identification, by interpreting impulse response estimation as a function learning problem. The adopted estimator solves a regularized least squares problem which admits also a Bayesian interpretation where the impulse response is modeled as a zero-mean Gaussian vector. A possible choice for the covariance is the so called stable spline kernel. It includes information on smoothness and exponential stability, containing just two unknown parameters which can be tuned via marginal likelihood (ML) optimization. Experimental evidence has shown that this new approach may outperform traditional system identification approaches, such as PEM and subspace techniques.The aim of this work is to provide new insights on the stable spline estimator equipped with ML estimation of hyperparameters. To this purpose, we study the mean squared error properties of the ML procedure for hyperparameter estimation; in doing so we shall not assume the correctness of the Bayesian priors. Then, we derive the notion of excess degrees of freedom. This notion measures the additional complexity to be assigned to an estimator which is also required to determine hyperparameters from data. The conclusion of our investigation is that much of criticisms reported in the literature to robustness of ML is not well founded. On the contrary, in many situations ML can well balance data fit and excess degrees of freedom. Hence, it turns out an important tool for tuning model complexity in linear system identification also when undermodeling affects the kernel-based description of the impulse response.

Diversification of North American natricine snakes

The phylogeny of thamnophiine snakes has not been fully resolved, largely because previous phylogenetic estimates have been based on incomplete taxon sampling or relied solely on mitochondrial sequence data. To address this deficiency, we collected data from multiple autosomal loci collected from 50 taxa before estimating the most robust phylogeny of Thamnophiini to date. Our findings clarify the relationships of taxa not previously included in molecular analyses and also lend evidence to previously recommended taxonomic revisions. Differences in topological estimates between competing models of evolution were minimal and not strongly supported; however, a multispecies coalescent model of evolution was highly favoured over a concatenated model based on marginal likelihood estimates. Additionally, we estimated the timing of divergence among the three major lineages to have occurred during the Miocene period (approximately 11–14 Mya), followed by a decline in speciation rates in all major lineages.

Maximum Marginal Likelihood Estimation of a Monotonic Polynomial Generalized Partial Credit Model with Applications to Multiple Group Analysis.

We present a semi-parametric approach to estimating item response functions (IRF) useful when the true IRF does not strictly follow commonly used functions. Our approach replaces the linear predictor of the generalized partial credit model with a monotonic polynomial. The model includes the regular generalized partial credit model at the lowest order polynomial. Our approach extends Liang's (A semi-parametric approach to estimate IRFs, Unpublished doctoral dissertation, 2007) method for dichotomous item responses to the case of polytomous data. Furthermore, item parameter estimation is implemented with maximum marginal likelihood using the Bock-Aitkin EM algorithm, thereby facilitating multiple group analyses useful in operational settings. Our approach is demonstrated on both educational and psychological data. We present simulation results comparing our approach to more standard IRF estimation approaches and other non-parametric and semi-parametric alternatives.

Estimation of Contextual Effects Through Nonlinear Multilevel Latent Variable Modeling With a Metropolis–Hastings Robbins–Monro Algorithm

The main purpose of this study is to improve estimation efficiency in obtaining maximum marginal likelihood estimates of contextual effects in the framework of nonlinear multilevel latent variable model by adopting the Metropolis–Hastings Robbins–Monro algorithm (MH-RM). Results indicate that the MH-RM algorithm can produce estimates and standard errors efficiently. Simulations, with various sampling and measurement structure conditions, were conducted to obtain information about the performance of nonlinear multilevel latent variable modeling compared to traditional hierarchical linear modeling. Results suggest that nonlinear multilevel latent variable modeling can more properly estimate and detect contextual effects than the traditional approach. As an empirical illustration, data from the Programme for International Student Assessment were analyzed.

Compressive sensing by learning a Gaussian mixture model from measurements.

Compressive sensing of signals drawn from a Gaussian mixture model (GMM) admits closed-form minimum mean squared error reconstruction from incomplete linear measurements. An accurate GMM signal model is usually not available a priori, because it is difficult to obtain training signals that match the statistics of the signals being sensed. We propose to solve that problem by learning the signal model in situ, based directly on the compressive measurements of the signals, without resorting to other signals to train a model. A key feature of our method is that the signals being sensed are treated as random variables and are integrated out in the likelihood. We derive a maximum marginal likelihood estimator (MMLE) that maximizes the likelihood of the GMM of the underlying signals given only their linear compressive measurements. We extend the MMLE to a GMM with dominantly low-rank covariance matrices, to gain computational speedup. We report extensive experimental results on image inpainting, compressive sensing of high-speed video, and compressive hyperspectral imaging (the latter two based on real compressive cameras). The results demonstrate that the proposed methods outperform state-of-the-art methods by significant margins.

Recursive Pathways to Marginal Likelihood Estimation with Prior-Sensitivity Analysis

We investigate the utility to computational Bayesian analyses of a particular family of recursive marginal likelihood estimators characterized by the (equivalent) algorithms known as "biased sampling" or "reverse logistic regression" in the statistics literature and "the density of states" in physics. Through a pair of numerical examples (including mixture modeling of the well-known galaxy data set) we highlight the remarkable diversity of sampling schemes amenable to such recursive normalization, as well as the notable efficiency of the resulting pseudo-mixture distributions for gauging prior sensitivity in the Bayesian model selection context. Our key theoretical contributions are to introduce a novel heuristic ("thermodynamic integration via importance sampling") for qualifying the role of the bridging sequence in this procedure and to reveal various connections between these recursive estimators and the nested sampling technique.

General partially linear additive transformation model with right-censored data

We propose a class of general partially linear additive transformation models (GPLATM) with right-censored survival data in this work. The class of models are flexible enough to cover many commonly used parametric and nonparametric survival analysis models as its special cases. Based on the B spline interpolation technique, we estimate the unknown regression parameters and functions by the maximum marginal likelihood estimation method. One important feature of the estimation procedure is that it does not need the baseline and censoring cumulative density distributions. Some numerical studies illustrate that this procedure can work very well for the moderate sample size.

On the use of marginal posteriors in marginal likelihood estimation via importance sampling

The efficiency of a marginal likelihood estimator where the product of the marginal posterior distributions is used as an importance sampling function is investigated. The approach is generally applicable to multi-block parameter vector settings, does not require additional Markov Chain Monte Carlo (MCMC) sampling and is not dependent on the type of MCMC scheme used to sample from the posterior. The proposed approach is applied to normal regression models, finite normal mixtures and longitudinal Poisson models, and leads to accurate marginal likelihood estimates.

The Data-augmentation Techniques in Item Response Modeling: Current Approaches and New Developments

在心理与教育测量中，项目反应理论（Item Response Theory，IRT）模型的参数估计方法是理论研究与实践应用的基本工具。最近，由于IRT模型的不断扩展与EM（expectation-maximization）算法自身的固有问题，参数估计方法的改进与发展显得尤为重要。这里介绍了IRT模型中边际极大似然估计的发展，提出了它的阶段性特征，即联合极大似然估计阶段、确定性潜在心理特质“填补”阶段、随机潜在心理特质“填补”阶段，重点阐述了它的潜在心理特质“填补”（dataaugmentation）思想。EM算法与Metropolis-Hastings Robbins—Monro（MH-RM）算法作为不同的潜在心理特质“填补”方法，都是边际极大似然估计的思想跨越。目前，潜在心理特质“填补”的参数估计方法仍在不断发展与完善。

Modeling the Propensity to Use a Sustainable Mode in the Context of a Program of Voluntary Change in Travel Behavior

This paper proposes a modeling approach for evaluating the effect of a personalized travel plan on a sustainable mode choice. A panel binary probit was estimated by using the approach of composite marginal-likelihood estimation. The formulation modeled the choice of using a light rail service (versus that of not using it) by means of daily individual panel observations collected in the context of a program of voluntary travel behavior change (VTBC) before and after the provision of a personalized travel plan. In this regard, a VTBC program was a policy measure that used communication and information to encourage individuals to use more sustainable travel modes. In this study, the VTBC program was implemented by providing car users with personalized information about how to introduce the light rail service into their travel patterns.

Species Delimitation Using Bayes Factors: Simulations and Application to the Sceloporus scalaris Species Group (Squamata: Phrynosomatidae)

Current molecular methods of species delimitation are limited by the types of species delimitation models and scenarios that can be tested. Bayes factors allow for more flexibility in testing non-nested species delimitation models and hypotheses of individual assignment to alternative lineages. Here, we examined the efficacy of Bayes factors in delimiting species through simulations and empirical data from the Sceloporus scalaris species group. Marginal-likelihood scores of competing species delimitation models, from which Bayes factor values were compared, were estimated with four different methods: harmonic mean estimation (HME), smoothed harmonic mean estimation (sHME), path-sampling/thermodynamic integration (PS), and stepping-stone (SS) analysis. We also performed model selection using a posterior simulation-based analog of the Akaike information criterion through Markov chain Monte Carlo analysis (AICM). Bayes factor species delimitation results from the empirical data were then compared with results from the reversible-jump MCMC (rjMCMC) coalescent-based species delimitation method Bayesian Phylogenetics and Phylogeography (BP&P). Simulation results show that HME and sHME perform poorly compared with PS and SS marginal-likelihood estimators when identifying the true species delimitation model. Furthermore, Bayes factor delimitation (BFD) of species showed improved performance when species limits are tested by reassigning individuals between species, as opposed to either lumping or splitting lineages. In the empirical data, BFD through PS and SS analyses, as well as the rjMCMC method, each provide support for the recognition of all scalaris group taxa as independent evolutionary lineages. Bayes factor species delimitation and BP&P also support the recognition of three previously undescribed lineages. In both simulated and empirical data sets, harmonic and smoothed harmonic mean marginal-likelihood estimators provided much higher marginal-likelihood estimates than PS and SS estimators. The AICM displayed poor repeatability in both simulated and empirical data sets, and produced inconsistent model rankings across replicate runs with the empirical data. Our results suggest that species delimitation through the use of Bayes factors with marginal-likelihood estimates via PS or SS analyses provide a useful and complementary alternative to existing species delimitation methods.

Penalized marginal likelihood estimation of finite mixtures of Archimedean copulas

This paper proposes finite mixtures of different Archimedean copula families as a flexible tool for modelling the dependence structure in multivariate data. A novel approach to estimating the parameters in this mixture model is presented by maximizing the penalized marginal likelihood via iterative quadratic programming. The motivation for the penalized marginal likelihood stems from an underlying Bayesian model that imposes a prior distribution on the parameter of each Archimedean copula family. An approximative marginal likelihood is obtained by a classical quadrature discretization of the integral w.r.t. each family-specific prior distribution, thus yielding a finite mixture model. Family-specific smoothness penalties are added and the penalized marginal likelihood is maximized using an iterative quadratic programming routine. For comparison purposes, we also present a fully Bayesian approach via simulation-based posterior computation. The performance of the novel estimation approach is evaluated by simulations and two examples involving the modelling of the interdependence of exchange rates and of wind speed measurements, respectively. For these examples, penalized marginal likelihood estimates are compared to the corresponding Bayesian estimates.

Estimation of a Ramsay-Curve Item Response Theory Model by the Metropolis–Hastings Robbins–Monro Algorithm

In Ramsay curve item response theory (RC-IRT) modeling, the shape of the latent trait distribution is estimated simultaneously with the item parameters. In its original implementation, RC-IRT is estimated via Bock and Aitkin’s EM algorithm, which yields maximum marginal likelihood estimates. This method, however, does not produce the parameter covariance matrix as an automatic byproduct on convergence. In turn, researchers are limited in when they can employ RC-IRT, as the covariance matrix is needed for many statistical inference procedures. The present research remedies this problem by estimating the RC-IRT model parameters by the Metropolis–Hastings Robbins–Monro (MH-RM) algorithm. An attractive feature of MH-RM is that the structure of the algorithm makes estimation of the covariance matrix convenient. Additionally, MH-RM is ideally suited for multidimensional IRT, whereas EM is limited by the “curse of dimensionality.” Based on the current research, when RC-IRT or similar IRT models are eventually generalized to include multiple latent dimensions, MH-RM would appear to be the logical choice for estimation.

Partial marginal likelihood estimation for general transformation models

We consider a large class of transformation models introduced by Gu et al. (2005) [14]. They proposed an estimation procedure for calculating the maximum partial marginal likelihood estimator (MPMLE) of regression parameters. A big advantage of MPMLE is that it avoids estimating two infinitely dimensional nuisance parameters: baseline and censoring survival functions. And they showed the validity of MPMLE through extensive simulations. In this paper, we establish the asymptotic properties of MPMLE in the general transformation models for either right or left censored data. The difficulty in establishing these asymptotic results comes from the fact that the score function derived from the partial marginal likelihood does not have ordinary independence or martingale structure. We develop a novel discretization method to resolve the difficulty. The estimation procedure is further examined using simulation studies and the analysis of the ACTG019 data.

Bayesian evolutionary model testing in the phylogenomics era: matching model complexity with computational efficiency

The advent of new sequencing technologies has led to increasing amounts of data being available to perform phylogenetic analyses, with genomic data giving rise to the field of phylogenomics. High-performance computing is becoming an indispensable research tool to fit complex evolutionary models, which take into account specific genomic properties, to large datasets. Here, we perform an extensive Bayesian phylogenetic model selection study, comparing codon and nucleotide substitution models, including codon position partitioning for nucleotide data as well gene-specific substitution models for both data types. For the best fitting partitioned models, we also compare independent partitioning with standard diffuse prior specification to conditional partitioning via hierarchical prior specification. To compare the different models, we use state-of-the-art marginal likelihood estimation techniques, including path sampling and stepping-stone sampling. We show that a full codon model best describes the features of a whole mitochondrial genome dataset, consisting of 12 protein-coding genes, but only when each gene is allowed to evolve under a separate codon model. However, when using hierarchical prior specification for the partition-specific parameters instead of independent diffuse priors, codon position partitioned nucleotide models can still outperform standard codon models. We demonstrate the feasibility of fitting such a combination of complex models using the BEAGLE library for BEAST in combination with recent graphics cards. We argue that development and use of such models needs to be accompanied by state-of-the-art marginal likelihood estimators because the more traditional and computationally less demanding estimators do not offer adequate accuracy.

The composite marginal likelihood (CML) estimation of panel ordered-response models

In the context of panel ordered-response structures, the current paper compares the performance of the maximum-simulated likelihood (MSL) inference approach and the composite marginal likelihood (CML) inference approach. The panel structures considered include the pure random coefficients (RC) model with no autoregressive error component, as well as the more general case of random coefficients combined with an autoregressive error component. The ability of the MSL and CML approaches to recover the true parameters is examined using simulated datasets. The results indicate that the performances of the MSL approach (with 150 scrambled and randomized Halton draws) and the simulation-free CML approach are of about the same order in all panel structures in terms of the absolute percentage bias (APB) of the parameters and econometric efficiency. However, the simulation-free CML approach exhibits no convergence problems of the type that affect the MSL approach. At the same time, the CML approach is about 5–12 times faster than the MSL approach for the simple random coefficients panel structure, and about 100 times faster than the MSL approach when an autoregressive error component is added. As the number of random coefficients increases, or if higher order autoregressive error structures are considered, one can expect even higher computational efficiency factors for the CML over the MSL approach. These results are promising for the use of the CML method for the quick, accurate, and practical estimation of panel ordered-response models with flexible and rich stochastic specifications.

Marginal likelihood estimation from the Metropolis output: tips and tricks for efficient implementation in generalized linear latent variable models

The marginal likelihood can be notoriously difficult to compute, and particularly so in high-dimensional problems. Chib and Jeliazkov employed the local reversibility of the Metropolis–Hastings algorithm to construct an estimator in models where full conditional densities are not available analytically. The estimator is free of distributional assumptions and is directly linked to the simulation algorithm. However, it generally requires a sequence of reduced Markov chain Monte Carlo runs which makes the method computationally demanding especially in cases when the parameter space is large. In this article, we study the implementation of this estimator on latent variable models which embed independence of the responses to the observables given the latent variables (conditional or local independence). This property is employed in the construction of a multi-block Metropolis-within-Gibbs algorithm that allows to compute the estimator in a single run, regardless of the dimensionality of the parameter space. The counterpart one-block algorithm is also considered here, by pointing out the difference between the two approaches. The paper closes with the illustration of the estimator in simulated and real-life data sets.

Make the most of your samples: Bayes factor estimators for high-dimensional models of sequence evolution

BackgroundAccurate model comparison requires extensive computation times, especially for parameter-rich models of sequence evolution. In the Bayesian framework, model selection is typically performed through the evaluation of a Bayes factor, the ratio of two marginal likelihoods (one for each model). Recently introduced techniques to estimate (log) marginal likelihoods, such as path sampling and stepping-stone sampling, offer increased accuracy over the traditional harmonic mean estimator at an increased computational cost. Most often, each model’s marginal likelihood will be estimated individually, which leads the resulting Bayes factor to suffer from errors associated with each of these independent estimation processes.ResultsWe here assess the original ‘model-switch’ path sampling approach for direct Bayes factor estimation in phylogenetics, as well as an extension that uses more samples, to construct a direct path between two competing models, thereby eliminating the need to calculate each model’s marginal likelihood independently. Further, we provide a competing Bayes factor estimator using an adaptation of the recently introduced stepping-stone sampling algorithm and set out to determine appropriate settings for accurately calculating such Bayes factors, with context-dependent evolutionary models as an example. While we show that modest efforts are required to roughly identify the increase in model fit, only drastically increased computation times ensure the accuracy needed to detect more subtle details of the evolutionary process.ConclusionsWe show that our adaptation of stepping-stone sampling for direct Bayes factor calculation outperforms the original path sampling approach as well as an extension that exploits more samples. Our proposed approach for Bayes factor estimation also has preferable statistical properties over the use of individual marginal likelihood estimates for both models under comparison. Assuming a sigmoid function to determine the path between two competing models, we provide evidence that a single well-chosen sigmoid shape value requires less computational efforts in order to approximate the true value of the (log) Bayes factor compared to the original approach. We show that the (log) Bayes factors calculated using path sampling and stepping-stone sampling differ drastically from those estimated using either of the harmonic mean estimators, supporting earlier claims that the latter systematically overestimate the performance of high-dimensional models, which we show can lead to erroneous conclusions. Based on our results, we argue that highly accurate estimation of differences in model fit for high-dimensional models requires much more computational effort than suggested in recent studies on marginal likelihood estimation.

Flexible marginalized models for bivariate longitudinal ordinal data

Random effects models are commonly used to analyze longitudinal categorical data. Marginalized random effects models are a class of models that permit direct estimation of marginal mean parameters and characterize serial correlation for longitudinal categorical data via random effects (Heagerty, 1999). Marginally specified logistic-normal models for longitudinal binary data. Biometrics 55, 688-698; Lee and Daniels, 2008. Marginalized models for longitudinal ordinal data with application to quality of life studies. Statistics in Medicine 27, 4359-4380). In this paper, we propose a Kronecker product (KP) covariance structure to capture the correlation between processes at a given time and the correlation within a process over time (serial correlation) for bivariate longitudinal ordinal data. For the latter, we consider a more general class of models than standard (first-order) autoregressive correlation models, by re-parameterizing the correlation matrix using partial autocorrelations (Daniels and Pourahmadi, 2009). Modeling covariance matrices via partial autocorrelations. Journal of Multivariate Analysis 100, 2352-2363). We assess the reasonableness of the KP structure with a score test. A maximum marginal likelihood estimation method is proposed utilizing a quasi-Newton algorithm with quasi-Monte Carlo integration of the random effects. We examine the effects of demographic factors on metabolic syndrome and C-reactive protein using the proposed models.

Maximum Marginal Likelihood Estimation for Nonnegative Dictionary Learning in the Gamma-Poisson Model

In this paper we describe an alternative to standard nonnegative matrix factorization (NMF) for nonnegative dictionary learning, i.e., the task of learning a dictionary with nonnegative values from nonnegative data, under the assumption of nonnegative expansion coefficients. A popular cost function used for NMF is the Kullback-Leibler divergence, which underlies a Poisson observation model. NMF can thus be considered as maximization of the joint likelihood of the dictionary and the expansion coefficients. This approach lacks optimality because the number of parameters (which include the expansion coefficients) grows with the number of observations. In this paper we describe variational Bayes and Monte-Carlo EM algorithms for optimization of the marginal likelihood, i.e., the likelihood of the dictionary where the expansion coefficients have been integrated out (given a Gamma prior). We compare the output of both maximum joint likelihood estimation (i.e., standard NMF) and maximum marginal likelihood estimation (MMLE) on real and synthetical datasets. In particular we present face reconstruction results on CBCL dataset and text retrieval results over the musiXmatch dataset, a collection of word counts in song lyrics. The MMLE approach is shown to prevent overfitting by automatically pruning out irrelevant dictionary columns, i.e., embedding automatic model order selection.