Integrated Likelihood Function Research Articles

Integrated deviance information criterion for spatial autoregressive models with heteroskedasticity

In this study, we introduce the integrated deviance information criterion (DIC) for nested and non-nested model selection problems in heteroskedastic spatial autoregressive models. In a Bayesian estimation setting, we assume that the idiosyncratic error terms of our spatial autoregressive model have a scale mixture of normal distributions, where the scale mixture variables are latent variables that induce heteroskedasticity. We first derive the integrated likelihood function by analytically integrating out the scale mixture variables from the complete-data likelihood function. We then use the integrated likelihood function to formulate the integrated DIC measure. We investigate the finite sample performance of the integrated DIC in selecting the true model in a simulation study. The simulation results show that the integrated DIC performs satisfactorily and can be useful for selecting the correct model in specification search exercises. Finally, in a spatially augmented economic growth model, we use the integrated DIC to choose the spatial weights matrix that leads to better predictive accuracy.

Bayesian model calibration for block copolymer self-assembly: Likelihood-free inference and expected information gain computation via measure transport

We consider the Bayesian calibration of models describing the phenomenon of block copolymer (BCP) self-assembly, which is to infer model parameters and their uncertainty from noisy image data produced by microscopy or X-ray scattering characterization of BCP structures. To account for the long-range disorder in BCP structures, we introduce auxiliary variables in the model to represent this aleatory uncertainty. These variables, however, result in an integrated likelihood function for high-dimensional image data that is generally intractable to evaluate. We tackle this Bayesian inference problem using a likelihood-free inference (LFI) approach based on measure transport together with the construction of summary statistics for the image data. We show that expected information gains (EIG) from observed data about the model parameters can be computed with no significant additional cost. Lastly, we present a numerical case study using the Ohta–Kawasaki model for diblock copolymer thin film self-assembly and top-down microscopy characterization. We introduce several domain-specific energy and Fourier-based summary statistics for calibration and quantify their informativeness using EIG. The effect of various data corruptions, summary statistics, and experimental designs on the calibration results are studied using the proposed LFI method.

Information criteria for matrix exponential spatial specifications

In this study, we suggest using information criteria for nested and non-nested model selection problems for the matrix exponential spatial specifications (MESS) under both homoskedasticity and heteroskedasticity. To this end, we consider the deviance information criterion, the Akaike information criterion and the Bayesian information criterion in a Bayesian setting. In the heteroskedastic case, we assume that the error terms have a scale mixture of normal distributions, where the scale mixture variables are latent variables that lead to different distributions. We demonstrate how the integrated likelihood function can be obtained analytically by integrating out the scale mixture variables from the complete-data likelihood function, and how this integrated likelihood function can be used to formulate the information criteria. We investigate the finite sample performance of these criteria in selecting the true model in a simulation study. The results show that these criteria perform satisfactorily and can be useful for selecting the correct model in specification search exercises. Finally, we apply the proposed information criteria to a spatially augmented growth model and a carbon emission model to show their usefulness for both nested and non-nested model selection problems.

Bayesian model selection for generalized linear mixed models.

We propose a Bayesian model selection approach for generalized linear mixed models (GLMMs). We consider covariance structures for the random effects that are widely used in areas such as longitudinal studies, genome-wide association studies, and spatial statistics. Since the random effects cannot be integrated out of GLMMs analytically, we approximate the integrated likelihood function using a pseudo-likelihood approach. Our Bayesian approach assumes a flat prior for the fixed effects and includes both approximate reference prior and half-Cauchy prior choices for the variances of random effects. Since the flat prior on the fixed effects is improper, we develop a fractional Bayes factor approach to obtain posterior probabilities of the several competing models. Simulation studies with Poisson GLMMs with spatial random effects and overdispersion random effects show that our approach performs favorably when compared to widely used competing Bayesian methods including deviance information criterion and Watanabe-Akaike information criterion. We illustrate the usefulness and flexibility of our approach with three case studies including a Poisson longitudinal model, a Poisson spatial model, and a logistic mixed model. Our proposed approach is implemented in the R package GLMMselect that is available onCRAN.

Bayesian estimation of stochastic tail index from high-frequency financial data

Tails of the return distribution of an asset are informative about the (financial) risk behavior of that asset. Stochastic tail index (STI) models are designed to quantify riskiness by estimating a time-varying tail index from the distribution of extreme values using high-frequency financial data. In this paper, we propose a computationally efficient Bayesian estimation method for STI models based on the recent advances in band and sparse matrix algorithms. We then show how the deviance information criterion (DIC) can be calculated from the integrated likelihood function for model comparison exercises. In a Monte Carlo study, we investigate the finite sample properties of the Bayesian estimator as well as the performance of two DIC measures. Our results show that the Bayesian estimator performs sufficiently well and the DIC measures based on the integrated likelihood function are useful for model selection exercises. In an empirical application, we illustrate calculation of the tail index using high-frequency data on IBM stock returns. Our estimation results indicate that the daily tail index of the return distribution of IBM stock has a time-varying feature: It tends to decline when there are large losses, whereas it tends to increase when there are small losses.

A new integrated likelihood for estimating population size in dependent dual‐record system

AbstractEfficient estimation of the population size from dependent dual‐record system (DRS) remains a statistical challenge in the capture‐recapture type experiment. Owing to the non‐identifiability of the suitable time‐behavioural response variation model (denoted as Mtb) under DRS, few methods are developed in the Bayesian paradigm based on informative priors. Our contribution in this article is to develop a new integrated likelihood function from model Mtb motivated by a novel approach developed by Severini (2007). A suitable weight function on the nuisance parameter is derived with the knowledge of the direction of behavioural dependency. A pseudo‐likelihood function is constructed so that the resulting estimator possess some desirable properties including negligible prior (or weight) sensitiveness. Extensive simulations show the superior performance of our proposed method to that of the existing Bayesian methods. Moreover, the proposed estimator is easy to implement from the computational perspective. Applications to two real data sets are presented. The Canadian Journal of Statistics 46: 577–592; 2018 © 2018 Société statistique du Canada

Integrated likelihoods for functions of a parameter

Non‐Bayesian inference regarding a parameter of interest in the presence of a nuisance parameter may be based on the integrated likelihood function, in which the nuisance parameter is eliminated by averaging the likelihood function with respect to a weight function for the nuisance parameter. Recent research has shown that integrated likelihood methods work particularly well when the model is reparameterized in terms of a nuisance parameter chosen to be “unrelated” to the parameter of interest and the corresponding weight function for this nuisance parameter is chosen so that it does not depend on the parameter of interest. One choice for such a nuisance parameter is the zero score expectation (ZSE) parameter. The purpose of this note is to extend the definition of the ZSE parameter to the case in which the model has a parameter vector θ, with the parameter of interest of the model taken to be a function of θ; that is, the definition of the ZSE parameter is extended to the case in which there is not an explicit nuisance parameter for the model. The resulting integrated likelihood function has the same desirable properties as integrated likelihoods based on the ZSE parameter in models with an explicit nuisance parameter.

Analysis of Deviance for Hypothesis Testing in Generalized Partially Linear Models

In this study, we develop nonparametric analysis of deviance tools for generalized partially linear models based on local polynomial fitting. Assuming a canonical link, we propose expressions for both local and global analysis of deviance, which admit an additivity property that reduces to analysis of variance decompositions in the Gaussian case. Chi-square tests based on integrated likelihood functions are proposed to formally test whether the nonparametric term is significant. Simulation results are shown to illustrate the proposed chi-square tests and to compare them with an existing procedure based on penalized splines. The methodology is applied to German Bundesbank Federal Reserve data.

Analysis of Deviance in Generalized Partial Linear Models

We develop analysis of deviance tools for generalized partial linear models based on local polynomial fitting. Assuming a canonical link, we propose expressions for both local and global analysis of deviance, which admit an additivity property that reduces to ANOVA decompositions in the Gaussian case. Chi-square tests based on integrated likelihood functions are proposed to formally test whether the non-parametric term is significant. Simulation results are shown to illustrate the proposed chi-square tests. The methodology is applied to German Bundesbank Federal Reserve data.

Integrated likelihood computation methods

Suppose a model has parameter $$\theta =(\psi , \lambda )$$ź=(ź,ź), where $$\psi $$ź is the parameter of interest and $$\lambda $$ź is a nuisance parameter. The integrated likelihood method eliminates $$\lambda $$ź from the likelihood function $$L(\psi , \lambda )$$L(ź,ź) by integrating with respect to a weight function $$\pi (\lambda | \psi )$$ź(ź|ź). The resulting integrated likelihood function $$\bar{L}(\psi )$$L¯(ź) can be used for inference for $$\psi $$ź. However, the analytical form for the integrated likelihood is not always available. This paper discusses 12 different approaches to computing the integrated likelihood. Some methods were originally developed for other computation purposes and they are modified to fit in the integrated likelihood framework. Methods considered include direct numerical integration methods such as Monte Carlo integration method, importance sampling, Laplace method; marginal likelihood computation methods; and methods for computing the marginal posterior density. Simulation studies and real data example are presented to evaluate and compare these methods empirically.

Reconstructing time series and their uncertainty from observations with universal noise

AbstractA new approach is presented for the reconstruction of time series and other (y,x) functions from observables with any type of stochastic noise. In particular, noise may exist in both dependent and independent variables, i.e., y and x, or t, and may even be correlated between these variables. This situation occurs in many areas of the geosciences when the independent time variable is itself the result of a measurement process, such as in paleo– sea level estimation. Uncertainty in the recovered time series is quantified in probabilistic terms using Bayesian changepoint modeling. The main contribution of the paper is the derivation of a new form of integrated likelihood function which can measure the data fit for a curve to (y,t) observables contaminated by any type of random noise. Closed form expressions are found for the special case of correlated Gaussian data noise and curves built from the sum of piecewise linear polynomials. The technique is illustrated by estimating relative sea level variations, over the last five glacial cycles, from a data set of 1928 δ18O measurements. Comparisons are also made with other techniques including those that assume an error free “independent” variable. Experiments illustrate several benefits of accounting for timing errors. These include allowing rigorous uncertainty information of both time‐dependent signals and their gradients. Derivatives of the integrated likelihood function are also given, which allow implementation of likelihood maximization. The new likelihood function better reflects real errors in data and can improve recovery of the estimated time series.

Likelihood-based inference for population size in a capture–recapture experiment with varying probabilities from occasion to occasion

The estimation of the size of a population is, in general, performed using capture–recapture experiments. In this paper, we consider a closed population capture–recapture model in which individuals are captured independently and with the same probability in each sampling occasion, but the probabilities may vary from occasion to occasion. The unknown number of individuals is the parameter of interest, while the capture probabilities are the nuisance ones. Four likelihood functions free of nuisance parameters, namely the profile, conditional, uniform and Jeffrey’s integrated likelihood functions are derived and procedures for point and interval estimation are discussed. The estimation of population size is illustrated on a real dataset. The frequentist properties of the estimators are evaluated by means of a simulation study. The Jeffrey’s integrated likelihood achieved the best performance over all considered estimators for both point and interval estimation, particularly in situations with little information with small number of elements, small capture probabilities and small number of capture occasions.

A unified framework for multilevel uncertainty quantification in Bayesian inverse problems

In this paper a unified probabilistic framework for solving inverse problems in the presence of epistemic and aleatory uncertainty is presented. The aim is to establish a flexible theory that facilitates Bayesian data analysis in experimental scenarios as they are commonly met in engineering practice. Problems are addressed where learning about unobservable inputs of a forward model, e.g. reducing the epistemic uncertainty of fixed yet unknown parameters and/or quantifying the aleatory uncertainty of variable inputs, is based on processing response measurements. Approaches to Bayesian inversion, hierarchical modeling and uncertainty quantification are combined into a generic framework that eventually allows to interpret and accomplish this task as multilevel model calibration. A joint problem formulation, where quantities that are not of particular interest are marginalized out from a joint posterior distribution, or an intrinsically marginal formulation, which is based on an integrated likelihood function, can be chosen according to the inferential objective and computational convenience. Fully Bayesian probabilistic inversion, i.e. the inference the variability of unobservable model inputs across a number of experiments, is derived as a special case of multilevel inversion. Borrowing strength, i.e. the optimal estimation of experiment-specific unknown forward model inputs, is introduced as a means for combining information in inverse problems. Two related statistical models for situations involving finite or zero model/measurement error are devised. Multilevel-specific obstacles to Bayesian posterior computation via Markov chain Monte Carlo are discussed. The inferential machinery of Bayesian multilevel model calibration and its underlying flow of information are studied on the basis of a system from the domain of civil engineering. A population of identically manufactured structural elements serves as an exemplary system for examining different experimental settings from the standpoint of uncertainty quantification and reduction. In a series of tests the material variability throughout the ensemble of specimens, the entirety of specimen-specific material properties and the measurement error level are inferred under various uncertainties in the problem setup.

Integrated likelihoods in models with stratum nuisance parameters

Frequentist inference about a parameter of interest in presence of a nuisance parameter can be based on an integrated likelihood function. We analyze the behaviour of inferential quantities based on such a pseudo-likelihood in a two-index-asymptotics framework, in which both sample size and dimension of the nuisance parameter may diverge to infinity. We show that a properly chosen integrated likelihood largely outperforms standard likelihood methods, such as those based on the profile likelihood. These results are confirmed by simulation studies, in which comparisons with modified profile likelihood are also considered.

On the integrated maximum likelihood estimators for a closed population capture–recapture model with unequal capture probabilities

Nuisance parameter elimination is a central problem in capture–recapture modelling. In this paper, we consider a closed population capture–recapture model which assumes the capture probabilities varies only with the sampling occasions. In this model, the capture probabilities are regarded as nuisance parameters and the unknown number of individuals is the parameter of interest. In order to eliminate the nuisance parameters, the likelihood function is integrated with respect to a weight function (uniform and Jeffrey's) of the nuisance parameters resulting in an integrated likelihood function depending only on the population size. For these integrated likelihood functions, analytical expressions for the maximum likelihood estimates are obtained and it is proved that they are always finite and unique. Variance estimates of the proposed estimators are obtained via a parametric bootstrap resampling procedure. The proposed methods are illustrated on a real data set and their frequentist properties are assessed by means of a simulation study.

Integrated likelihood inference in semiparametric regression models

Consider a linear semiparametric regression model with normal errors in which the mean function depends on two parameters, a p-dimensional regression parameter, which is the parameter of interest, and an unknown function, which is a nuisance parameter. We consider estimation of the parameter of interest using an integrated likelihood function, in which the nuisance parameter is eliminated from the likelihood function by averaging with respect to some distribution. Here we take this distribution to be a Gaussian process with a given covariance function, which may depend on additional parameters. Likelihood inference based on the resulting integrated likelihood is considered and the properties of the score statistic based on the integrated likelihood, the maximum integrated likelihood estimator, and the integrated likelihood ratio statistic are presented. The methodology is illustrated on two examples.

Maximum integrated likelihood estimator of the interest parameter when the nuisance parameter is location or scale

The problem of estimation of an interest parameter in the presence of a nuisance parameter, which is either location or scale, is studied. Two estimators are considered: the usual maximum likelihood estimator and the estimator based on maximization of the integrated likelihood function. The estimators are compared, asymptotically, with respect to the bias and with respect to the mean squared error. The examples are given.

Analysis of Deviance in Generalized Partial Linear Models

We develop analysis of deviance tools for generalized partial linear models based on local polynomial tting. Assuming a canonical link, we propose expressions for both local and global analysis of deviance, which admit an additivity property that reduces to ANOVA decompositions in the Gaussian case. Chi-square tests based on integrated likelihood functions are proposed to formally test whether the nonparametric term is signicant. Simulation results are shown to illustrate the proposed chi-square tests. The methodology is applied to German Bundesbank Federal Reserve data.

Decision Theory Applied to a Linear Panel Data Model

This paper applies some general concepts in decision theory to a linear panel data model. A simple version of the model is an autoregression with a separate intercept for each unit in the cross section, with errors that are independent and identically distributed with a normal distribution. There is a parameter of interest γ and a nuisance parameter τ ,a N × K matrix, where N is the cross-section sample size. The focus is on dealing with the incidental parameters problem created by a potentially high-dimension nuisance parameter. We adopt a “fixed-effects” approach that seeks to protect against any sequence of incidental parameters. We transform τ to (δ� ρ� ω) ,w hereδ is a J × K matrix of coefficients from the least-squares projection of τ on a N × J matrix x of strictly exogenous variables, ρ is a K × K symmetric, positive semidefinite matrix obtained from the residual sums of squares and cross-products in the projection of τ on x, and ω is a (N −J)× K matrix whose columns are orthogonal and have unit length. The model is invariant under the actions of a group on the sample space and the parameter space, and we find a maximal invariant statistic. The distribution of the maximal invariant statistic does not depend upon ω. There is a unique invariant distribution for ω .W e use this invariant distribution as a prior distribution to obtain an integrated likelihood function. It depends upon the observation only through the maximal invariant statistic. We use the maximal invariant statistic to construct a marginal likelihood function, so we can eliminate ω by integration with respect to the invariant prior distribution or by working with the marginal likelihood function. The two approaches coincide. Decision rules based on the invariant distribution for ω have a minimax property. Given a loss function that does not depend upon ω and given a prior distribution for (γ� δ� ρ) , we show how to minimize the average—with respect to the prior distribution for (γ� δ� ρ) —of the maximum risk, where the maximum is with respect to ω. There is a family of prior distributions for (δ� ρ) that leads to a simple closed form for the integrated likelihood function. This integrated likelihood function coincides with the likelihood function for a normal, correlated random-effects model. Under random sampling, the corresponding quasi maximum likelihood estimator is consistent for γ as N →∞ , with a standard limiting distribution. The limit results do not require normality or homoskedasticity (conditional on x) assumptions.

Integrated likelihood functions for non-Bayesian inference

Consider a model with parameter 0 = (ψ, λ), where ψ is the parameter of interest, and let L(ψ, λ) denote the likelihood function. One approach to likelihood inference for ψ is to use an integrated likelihood function, in which λ is eliminated from L(ψ, λ) by integrating with respect to a density function π(λ|ψ). The goal of this paper is to consider the problem of selecting π (λ |ψ) so that the resulting integrated likelihood function is useful for non-Bayesian likelihood inference. The desirable properties of an integrated likelihood function are analyzed and these suggest that π(λ|ψ) should be chosen by finding a nuisance parameter Φ that is unrelated to ψ and then taking the prior density for Φ to be independent of ψ. Such an unrelated parameter is constructed and the resulting integrated likelihood is shown to be closely related to the modified profile likelihood.