Approximate Likelihood Function Research Articles

Variational Bayes with synthetic likelihood

Synthetic likelihood is an attractive approach to likelihood-free inference when an approximately Gaussian summary statistic for the data, informative for inference about the parameters, is available. The synthetic likelihood method derives an approximate likelihood function from a plug-in normal density estimate for the summary statistic, with plug-in mean and covariance matrix obtained by Monte Carlo simulation from the model. In this article, we develop alternatives to Markov chain Monte Carlo implementations of Bayesian synthetic likelihoods with reduced computational overheads. Our approach uses stochastic gradient variational inference methods for posterior approximation in the synthetic likelihood context, employing unbiased estimates of the log likelihood. We compare the new method with a related likelihood free variational inference technique in the literature, while at the same time improving the implementation of that approach in a number of ways. These new algorithms are feasible to implement in situations which are challenging for conventional approximate Bayesian computation (ABC) methods, in terms of the dimensionality of the parameter and summary statistic.

Semi-parametric Dynamic Models for Longitudinal Ordinal Categorical Data

The over all regression function in a semi-parametric model involves a partly specified regression function in some primary covariates and a non-parametric function in some other secondary covariates. This type of semi-parametric models in a longitudinal setup has recently been discussed extensively both for repeated Poisson and negative binomial count data. However, when it is appropriate to interpret the longitudinal binary responses through a binary dynamic logits model, the inferences for semi-parametric Poisson and negative binomial models cannot be applied to such binary models as these models unlike the count data models produce recursive means and variances containing the dynamic dependence or correlation parameters. In this paper, we consider a general multinomial dynamic logits model in a semi-parametric setup first to analyze nominal categorical data in a semi-parametric longitudinal setup, and then modify this model to analyze ordinal categorical data. The ordinal responses are fitted by using a cumulative semi-parametric multinomial dynamic logits model. For the benefits of practitioners, a step by step estimation approach is developed for the non-parametric function, and for both regression and dynamic dependence parameters. In summary, a kernel-based semi-parametric weighted likelihood approach is used for the estimation of the non-parametric function. This weighted likelihood estimate for the non-parametric function is shown to be consistent. The regression and dynamic dependence parameters of the model are estimated by maximizing an approximate semi-parametric likelihood function for the parameters, which is constructed by replacing the non-parametric function with its consistent estimate. Asymptotic properties including the proofs for the consistency of the likelihood estimators of the regression and dynamic dependence parameters are discussed.

The Correct Regularity Condition and Interpretation of Asymmetry in EGARCH

In the class of univariate conditional volatility models, the three most popular are the generalized autoregressive conditional heteroskedasticity (GARCH) model of Engle (1982) and Bollerslev (1986), the GJR (or threshold GARCH) model of Glosten, Jagannathan and Runkle (1992), and the exponential GARCH (or EGARCH) model of Nelson (1990, 1991). For purposes of deriving the mathematical regularity properties, including invertibility, to determine the likelihood function for estimation, and the statistical conditions to establish asymptotic properties, it is convenient to understand the stochastic properties underlying the three univariate models. The random coefficient autoregressive process was used to obtain GARCH by Tsay (1987), an extension of which was used by McAleer (2004) to obtain GJR. A random coefficient complex nonlinear moving average process was used by McAleer and Hafner (2014) to obtain EGARCH. These models can be used to capture asymmetry, which denotes the different effects on conditional volatility of positive and negative effects of equal magnitude, and possibly also leverage, which is the negative correlation between returns shocks and subsequent shocks to volatility (see Black 1979). McAleer (2014) showed that asymmetry was possible for GJR, but not leverage. McAleer and Hafner showed that leverage was not possible for EGARCH. Surprisingly, the conditions for asymmetry in EGARCH seem to have been ignored in the literature, or have concentrated on the incorrect conditions, with no clear explanation, and hence with associated misleading interpretations. The purpose of the paper is to derive the regularity condition for asymmetry in EGARCH to provide the correct interpretation. It is shown that, in practice, EGARCH always displays asymmetry, though not leverage.

Monte Carlo approximation of likelihood function in spatial GLMMs through an empirical Bayes method

ABSTRACTIn spatial generalized linear mixed models (SGLMMs), statistical inference encounters problems, since random effects in the model imply high-dimensional integrals to calculate the marginal likelihood function. In this article, we temporarily treat parameters as random variables and express the marginal likelihood function as a posterior expectation. Hence, the marginal likelihood function is approximated using the obtained samples from the posterior density of the latent variables and parameters given the data. However, in this setting, misspecification of prior distribution of correlation function parameter and problems associated with convergence of Markov chain Monte Carlo (MCMC) methods could have an unpleasant influence on the likelihood approximation. To avoid these challenges, we utilize an empirical Bayes approach to estimate prior hyperparameters. We also use a computationally efficient hybrid algorithm by combining inverse Bayes formula (IBF) and Gibbs sampler procedures. A simulation study is conducted to assess the performance of our method. Finally, we illustrate the method applying a dataset of standard penetration test of soil in an area in south of Iran.

A new approach to regression analysis of censored competing-risks data.

An approximate likelihood approach is developed for regression analysis of censored competing-risks data. This approach models directly the cumulative incidence function, instead of the cause-specific hazard function, in terms of explanatory covariates under a proportional subdistribution hazards assumption. It uses a self-consistent iterative procedure to maximize an approximate semiparametric likelihood function, leading to an asymptotically normal and efficient estimator of the vector of regression parameters. Simulation studies demonstrate its advantages over previous methods.

Likelihood-Based Parameter Estimation for High-Dimensional Phylogenetic Comparative Models: Overcoming the Limitations of "Distance-Based" Methods.

Recently, a suite of distance-based multivariate phylogenetic comparative methods has been proposed for studying the evolution of high-dimensional traits, such as morphometric coordinates, gene expression data, and function-valued traits. These methods allow for the statistical comparison of evolutionary rates, assessment of phylogenetic signal, and tests of correlated high-dimensional trait evolution. Simulations reveal that distance-based comparative methods exhibit low statistical power and high Type I error under various evolutionary scenarios. Distance-based methods are also limited to relatively simple model specification (e.g., Brownian motion evolution) due to the lack of a likelihood function for parameter estimation. Here I propose an alternative method for studying high-dimensional trait evolution which overcomes some of the statistical limitations associated with distance-based methods. This framework, based on parametric bootstrapping and maximum pseudolikelihood parameter estimation, opens up the ability to estimate alternative evolutionary models, combine multiple evolutionary hypotheses, and potentially allow missing data and within-species variation. Simulations reveal that pairwise composite likelihood methods demonstrate appropriate Type I error and high statistical power, thus providing a robust framework for studying high-dimensional trait evolution. These methods are implemented in the R package phylocurve [Covariance; distance; evolutionary rate; function-valued trait; high-dimensional; morphometric; multivariate; pairwise composite likelihood; phylogenetic comparative method; phylogenetic generalized least squares; phylogenetic signal.].

Nonparametric Feature Matching Based Conditional Random Fields for Gesture Recognition from Multi-Modal Video.

We present a new gesture recognition method that is based on the conditional random field (CRF) model using multiple feature matching. Our approach solves the labeling problem, determining gesture categories and their temporal ranges at the same time. A generative probabilistic model is formalized and probability densities are nonparametrically estimated by matching input features with a training dataset. In addition to the conventional skeletal joint-based features, the appearance information near the active hand in an RGB image is exploited to capture the detailed motion of fingers. The estimated likelihood function is then used as the unary term for our CRF model. The smoothness term is also incorporated to enforce the temporal coherence of our solution. Frame-wise recognition results can then be obtained by applying an efficient dynamic programming technique. To estimate the parameters of the proposed CRF model, we incorporate the structured support vector machine (SSVM) framework that can perform efficient structured learning by using large-scale datasets. Experimental results demonstrate that our method provides effective gesture recognition results for challenging real gesture datasets. By scoring 0.8563 in the mean Jaccard index, our method has obtained the state-of-the-art results for the gesture recognition track of the 2014 ChaLearn Looking at People (LAP) Challenge.

An approximate likelihood function of spatial correlation parameters

Even under assumption of normality, likelihood-based inferences are often difficult for large and irregularly spaced spatial datasets. Exact calculations of the likelihood for a Gaussian spatial process observed in n locations require O(n3) operations. Instead of Whittle’s approximation to the Gaussian log likelihood for large spatial datasets, this paper introduces an approximated likelihood function of spatial parameters based on the correlogram, which involves no calculation of determinants and is computationally feasible. The proposed likelihood approximation method for spatial parameter is applied to the estimation of the spatial structure of changes in the average summer temperature based on 30 years of data by using an regional climate model (RCM) with a particular global climate model (GCM) boundary condition. The results verify the benefits and the performance of the proposed method.

Climate index weighting of ensemble streamflow forecasts using a simple Bayesian approach

Climate state can be an important predictor of future hydrologic conditions. In ensemble streamflow forecasting, where historical weather inputs or streamflow observations are used to generate the ensemble, climate index weighting is one way to represent the influence of climate state. Using a climate index, each forecast variable member of the ensemble is selectively weighted to reflect the climate state at the time of the forecast. A new approach to climate index weighting of ensemble forecasts is presented. The method is based on a sampling-resampling approach for Bayesian updating. The original hydrologic ensemble members define a sample drawn from the prior distribution; the relationship between the climate index and the ensemble member forecast variable is used to estimate a likelihood function. Given an observation of the climate index at the time of the forecast, the estimated likelihood function is then used to assign weights to each ensemble member. The weights define the probability of each ensemble member outcome given the observed climate index. The weighted ensemble forecast is then used to estimate the posterior distribution of the forecast variable conditioned on the climate index. The Bayesian climate index weighting approach is easy to apply to hydrologic ensemble forecasts; its parameters do not require calibration with hindcasts, and it adapts to the strength of the relation between climate and the forecast variable, defaulting to equal weighting of ensemble members when no relationship exists. A hydrologic forecasting application illustrates the approach and contrasts it with traditional climate index weighting approaches. This article is protected by copyright. All rights reserved.

Efficient maximum approximated likelihood inference for Tukey’s [formula omitted]-and-[formula omitted] distribution

Tukey’s g-and-h distribution has been a powerful tool for data exploration and modeling since its introduction. However, two long standing challenges associated with this distribution family have remained unsolved until this day: how to find an optimal estimation procedure and how to make valid statistical inference on unknown parameters. To overcome these two challenges, a computationally efficient estimation procedure based on maximizing an approximated likelihood function of Tukey’s g-and-h distribution is proposed and is shown to have the same estimation efficiency as the maximum likelihood estimator under mild conditions. The asymptotic distribution of the proposed estimator is derived and a series of approximated likelihood ratio test statistics are developed to conduct hypothesis tests involving two shape parameters of Tukey’s g-and-h distribution. Simulation examples and an analysis of air pollution data are used to demonstrate the effectiveness of the proposed estimation and testing procedures.

Estimation of fixed effects panel regression models with separable and nonseparable space–time filters

This paper considers a quasi-maximum likelihood estimation for a linear panel data model with time and individual fixed effects, where the disturbances have dynamic and spatial correlations which might be spatially stable or unstable. We first consider both separable and nonseparable space–time filters for the stable model. The separable space–time filter is subject to a parametric restriction which results in relative computational simplicity. In contrast to the spatial econometrics literature, we expose economic restrictions imposed by the separable space–time filter model and explore computational tractability of the nonseparable filter model. Throughout this paper, the effect of initial observations is taken into account, which results in an exact likelihood function for estimation. This is important when the span of time periods is short. We then investigate spatial unstable cases, where we propose to apply a “spatial differencing” to all variables in the regression equation as a data transformation, which may eliminate unstable or explosive spatial components in order to achieve a robust estimator. For estimates of the parameters in both the regression part and the disturbance process, they are nT-consistent and asymptotically centered normal regardless of whether T is large or not and whether the process is stable or not.

The importance of hydrological uncertainty assessment methods in climate change impact studies

Abstract. Climate change impact assessments have become more and more popular in hydrology since the middle 1980s with a recent boost after the publication of the IPCC AR4 report. From hundreds of impact studies a quasi-standard methodology has emerged, to a large extent shaped by the growing public demand for predicting how water resources management or flood protection should change in the coming decades. The "standard" workflow relies on a model cascade from global circulation model (GCM) predictions for selected IPCC scenarios to future catchment hydrology. Uncertainty is present at each level and propagates through the model cascade. There is an emerging consensus between many studies on the relative importance of the different uncertainty sources. The prevailing perception is that GCM uncertainty dominates hydrological impact studies. Our hypothesis was that the relative importance of climatic and hydrologic uncertainty is (among other factors) heavily influenced by the uncertainty assessment method. To test this we carried out a climate change impact assessment and estimated the relative importance of the uncertainty sources. The study was performed on two small catchments in the Swiss Plateau with a lumped conceptual rainfall runoff model. In the climatic part we applied the standard ensemble approach to quantify uncertainty but in hydrology we used formal Bayesian uncertainty assessment with two different likelihood functions. One was a time series error model that was able to deal with the complicated statistical properties of hydrological model residuals. The second was an approximate likelihood function for the flow quantiles. The results showed that the expected climatic impact on flow quantiles was small compared to prediction uncertainty. The choice of uncertainty assessment method actually determined what sources of uncertainty could be identified at all. This demonstrated that one could arrive at rather different conclusions about the causes behind predictive uncertainty for the same hydrological model and calibration data when considering different objective functions for calibration.

Maximum-likelihood estimates of the frequency of signals of laser Doppler anemometers

The likelihood function for estimation of parameters of laser Doppler anemometers is derived and ways for estimation of the most informative parameter, specifically, the Doppler frequency (velocity) are demonstrated. For the most widespread anemometer designs, the results of a model experiment, which characterize the quality of the estimates, are reported.

A FREQUENCY DOMAIN APPROACH FOR THE ESTIMATION OF PARAMETERS OF SPATIO‐TEMPORAL STATIONARY RANDOM PROCESSES

A frequency domain methodology is proposed for estimating parameters of covariance functions of stationary spatio‐temporal processes. Finite Fourier transforms of the processes are defined at each location. Based on the joint distribution of these complex valued random variables, an approximate likelihood function is constructed. The sampling properties of the estimators are investigated. It is observed that the expectation of these transforms can be considered to be a frequency domain analogue of the classical variogram. We call this measure frequency variogram. The method is applied to simulated data and also to Pacific wind speed data considered earlier by Cressie and Huang (1999). The proposed method does not depend on the distributional assumptions about the process.

Parameter Estimation of Population Pharmacokinetic Models with Stochastic Differential Equations: Implementation of an Estimation Algorithm

Population pharmacokinetic (PPK) models play a pivotal role in quantitative pharmacology study, which are classically analyzed by nonlinear mixed-effects models based on ordinary differential equations. This paper describes the implementation of SDEs in population pharmacokinetic models, where parameters are estimated by a novel approximation of likelihood function. This approximation is constructed by combining the MCMC method used in nonlinear mixed-effects modeling with the extended Kalman filter used in SDE models. The analysis and simulation results show that the performance of the approximation of likelihood function for mixed-effects SDEs model and analysis of population pharmacokinetic data is reliable. The results suggest that the proposed method is feasible for the analysis of population pharmacokinetic data.

Kernel Density-Based Linear Regression Estimate

For linear regression models with non normally distributed errors, the least squares estimate (LSE) will lose some efficiency compared to the maximum likelihood estimate (MLE). In this article, we propose a kernel density-based regression estimate (KDRE) that is adaptive to the unknown error distribution. The key idea is to approximate the likelihood function by using a nonparametric kernel density estimate of the error density based on some initial parameter estimate. The proposed estimate is shown to be asymptotically as efficient as the oracle MLE which assumes the error density were known. In addition, we propose an EM type algorithm to maximize the estimated likelihood function and show that the KDRE can be considered as an iterated weighted least squares estimate, which provides us some insights on the adaptiveness of KDRE to the unknown error distribution. Our Monte Carlo simulation studies show that, while comparable to the traditional LSE for normal errors, the proposed estimation procedure can have substantial efficiency gain for non normal errors. Moreover, the efficiency gain can be achieved even for a small sample size.

Bayesian Identification of Seasonal Multivariate Autoregressive Processes

The main objective of this article is to develop an approximate Bayesian procedure to identify the orders of seasonal multivariate autoregressive processes. Using either a matrix normal-Wishart prior density or a non informative prior, which is combined with an approximate conditional likelihood function, the foundation of the proposed technique is to derive the joint posterior mass function of the model orders in a convenient form. Then one may easily evaluate the joint posterior probabilities of all possible values of the model orders and choose the orders at which the joint posterior mass function attains its maximum to be the identified orders. A simulation study, with different prior mass functions, is carried out to determine the effectiveness of the proposed technique in handling the identification problem of the seasonal bivariate autoregressive processes.

AN EFFICIENT APPROXIMATION TO THE LIKELIHOOD FOR GRAVITATIONAL WAVE STOCHASTIC BACKGROUND DETECTION USING PULSAR TIMING DATA

Direct detection of gravitational waves by pulsar timing arrays will become feasible over the next few years. In the low frequency regime ($10^{-7}$ Hz -- $10^{-9}$ Hz), we expect that a superposition of gravitational waves from many sources will manifest itself as an isotropic stochastic gravitational wave background. Currently, a number of techniques exist to detect such a signal; however, many detection methods are computationally challenging. Here we introduce an approximation to the full likelihood function for a pulsar timing array that results in computational savings proportional to the square of the number of pulsars in the array. Through a series of simulations we show that the approximate likelihood function reproduces results obtained from the full likelihood function. We further show, both analytically and through simulations, that, on average, this approximate likelihood function gives unbiased parameter estimates for astrophysically realistic stochastic background amplitudes.

An Efficient Hierarchical Generalized Linear Mixed Model for Mapping QTL of Ordinal Traits in Crop Cultivars

Many important phenotypic traits in plants are ordinal. However, relatively little is known about the methodologies for ordinal trait association studies. In this study, we proposed a hierarchical generalized linear mixed model for mapping quantitative trait locus (QTL) of ordinal traits in crop cultivars. In this model, all the main-effect QTL and QTL-by-environment interaction were treated as random, while population mean, environmental effect and population structure were fixed. In the estimation of parameters, the pseudo data normal approximation of likelihood function and empirical Bayes approach were adopted. A series of Monte Carlo simulation experiments were performed to confirm the reliability of new method. The result showed that new method works well with satisfactory statistical power and precision. The new method was also adopted to dissect the genetic basis of soybean alkaline-salt tolerance in 257 soybean cultivars obtained, by stratified random sampling, from 6 geographic ecotypes in China. As a result, 6 main-effect QTL and 3 QTL-by-environment interactions were identified.

First difference maximum likelihood and dynamic panel estimation

First difference maximum likelihood (FDML) seems an attractive estimation methodology in dynamic panel data modeling because differencing eliminates fixed effects and, in the case of a unit root, differencing transforms the data to stationarity, thereby addressing both incidental parameter problems and the possible effects of nonstationarity. This paper draws attention to certain pathologies that arise in the use of FDML that have gone unnoticed in the literature and that affect both finite sample performance and asymptotics. FDML uses the Gaussian likelihood function for first differenced data and parameter estimation is based on the whole domain over which the log-likelihood is defined. However, extending the domain of the likelihood beyond the stationary region has certain consequences that have a major effect on finite sample and asymptotic performance. First, the extended likelihood is not the true likelihood even in the Gaussian case and it has a finite upper bound of definition. Second, it is often bimodal, and one of its peaks can be so peculiar that numerical maximization of the extended likelihood frequently fails to locate the global maximum. As a result of these pathologies, the FDML estimator is a restricted estimator, numerical implementation is not straightforward and asymptotics are hard to derive in cases where the peculiarity occurs with non-negligible probabilities. The peculiarities in the likelihood are found to be particularly marked in time series with a unit root. In this case, the asymptotic distribution of the FDMLE has bounded support and its density is infinite at the upper bound when the time series sample size T→∞. As the panel width n→∞ the pathology is removed and the limit theory is normal. This result applies even for T fixed and we present an expression for the asymptotic distribution which does not depend on the time dimension. We also show how this limit theory depends on the form of the extended likelihood.