Approximate Marginal Likelihood Research Articles

Conditional Deep Gaussian Processes: Multi-Fidelity Kernel Learning.

Deep Gaussian Processes (DGPs) were proposed as an expressive Bayesian model capable of a mathematically grounded estimation of uncertainty. The expressivity of DPGs results from not only the compositional character but the distribution propagation within the hierarchy. Recently, it was pointed out that the hierarchical structure of DGP well suited modeling the multi-fidelity regression, in which one is provided sparse observations with high precision and plenty of low fidelity observations. We propose the conditional DGP model in which the latent GPs are directly supported by the fixed lower fidelity data. Then the moment matching method is applied to approximate the marginal prior of conditional DGP with a GP. The obtained effective kernels are implicit functions of the lower-fidelity data, manifesting the expressivity contributed by distribution propagation within the hierarchy. The hyperparameters are learned via optimizing the approximate marginal likelihood. Experiments with synthetic and high dimensional data show comparable performance against other multi-fidelity regression methods, variational inference, and multi-output GP. We conclude that, with the low fidelity data and the hierarchical DGP structure, the effective kernel encodes the inductive bias for true function allowing the compositional freedom.

Conditional Deep Gaussian Processes: Empirical Bayes Hyperdata Learning.

It is desirable to combine the expressive power of deep learning with Gaussian Process (GP) in one expressive Bayesian learning model. Deep kernel learning showed success as a deep network used for feature extraction. Then, a GP was used as the function model. Recently, it was suggested that, albeit training with marginal likelihood, the deterministic nature of a feature extractor might lead to overfitting, and replacement with a Bayesian network seemed to cure it. Here, we propose the conditional deep Gaussian process (DGP) in which the intermediate GPs in hierarchical composition are supported by the hyperdata and the exposed GP remains zero mean. Motivated by the inducing points in sparse GP, the hyperdata also play the role of function supports, but are hyperparameters rather than random variables. It follows our previous moment matching approach to approximate the marginal prior for conditional DGP with a GP carrying an effective kernel. Thus, as in empirical Bayes, the hyperdata are learned by optimizing the approximate marginal likelihood which implicitly depends on the hyperdata via the kernel. We show the equivalence with the deep kernel learning in the limit of dense hyperdata in latent space. However, the conditional DGP and the corresponding approximate inference enjoy the benefit of being more Bayesian than deep kernel learning. Preliminary extrapolation results demonstrate expressive power from the depth of hierarchy by exploiting the exact covariance and hyperdata learning, in comparison with GP kernel composition, DGP variational inference and deep kernel learning. We also address the non-Gaussian aspect of our model as well as way of upgrading to a full Bayes inference.

Multi-Dimensional Penalized Hazard Model with Continuous Covariates: Applications for Studying Trends and Social Inequalities in Cancer Survival

SummaryDescribing the dynamics of patient mortality hazard is a major concern for cancer epidemiologists. In addition to time and age, other continuous covariates have often to be included in the model. For example, survival trend analyses and socio-economic studies deal respectively with the year of diagnosis and a deprivation index. Taking advantage of a recent theoretical framework for general smooth models, the paper proposes a penalized approach to hazard and excess hazard models in time-to-event analyses. The baseline hazard and the functional forms of the covariates were specified by using penalized natural cubic regression splines with associated quadratic penalties. Interactions between continuous covariates and time-dependent effects were dealt with by forming a tensor product smooth. The smoothing parameters were estimated by optimizing either the Laplace approximate marginal likelihood criterion or the likelihood cross-validation criterion. The regression parameters were estimated by direct maximization of the penalized likelihood of the survival model, which avoids data augmentation and the Poisson likelihood approach. The implementation proposed was evaluated on simulations and applied to real data. It was found to be numerically stable, efficient and useful for choosing the appropriate degree of complexity in overall survival and net survival contexts; moreover, it simplified the model building process.

Smoothing Parameter and Model Selection for General Smooth Models

ABSTRACTThis article discusses a general framework for smoothing parameter estimation for models with regular likelihoods constructed in terms of unknown smooth functions of covariates. Gaussian random effects and parametric terms may also be present. By construction the method is numerically stable and convergent, and enables smoothing parameter uncertainty to be quantified. The latter enables us to fix a well known problem with AIC for such models, thereby improving the range of model selection tools available. The smooth functions are represented by reduced rank spline like smoothers, with associated quadratic penalties measuring function smoothness. Model estimation is by penalized likelihood maximization, where the smoothing parameters controlling the extent of penalization are estimated by Laplace approximate marginal likelihood. The methods cover, for example, generalized additive models for nonexponential family responses (e.g., beta, ordered categorical, scaled t distribution, negative binomial and Tweedie distributions), generalized additive models for location scale and shape (e.g., two stage zero inflation models, and Gaussian location-scale models), Cox proportional hazards models and multivariate additive models. The framework reduces the implementation of new model classes to the coding of some standard derivatives of the log-likelihood. Supplementary materials for this article are available online.

Robust Gaussian process modeling using EM algorithm

Gaussian process (GP) regression is a fully probabilistic method for performing non-linear regression. In a Bayesian framework, regression models can be made robust by using heavy-tailed distributions instead of using normal distribution for modeling noise. This work focuses on estimation of parameters for robust GP regression. In literature, these are learned by maximizing the approximate marginal likelihood of data. However, gradient-based optimization algorithms which are used for this purpose can be unstable or may require tuning. In this work, an EM algorithm based approach is derived and implemented to infer the parameters. The pros and cons of the two approaches are analyzed. The advantage of EM algorithm lies in its ease of implementation and theoretical guarantees of numerical stability and convergence while its prediction performance is still comparable to gradient-based approaches. In some cases EM algorithm may be slow to converge. To circumvent this issue a faster EM based approach known as Expectation Conjugate Gradient (ECG) is implemented on robust GP regression. Finally, the proposed EM approach to robust GP regression is validated using an industrial data set.

Counting People With Low-Level Features and Bayesian Regression

An approach to the problem of estimating the size of inhomogeneous crowds, which are composed of pedestrians that travel in different directions, without using explicit object segmentation or tracking is proposed. Instead, the crowd is segmented into components of homogeneous motion, using the mixture of dynamic-texture motion model. A set of holistic low-level features is extracted from each segmented region, and a function that maps features into estimates of the number of people per segment is learned with Bayesian regression. Two Bayesian regression models are examined. The first is a combination of Gaussian process regression with a compound kernel, which accounts for both the global and local trends of the count mapping but is limited by the real-valued outputs that do not match the discrete counts. We address this limitation with a second model, which is based on a Bayesian treatment of Poisson regression that introduces a prior distribution on the linear weights of the model. Since exact inference is analytically intractable, a closed-form approximation is derived that is computationally efficient and kernelizable, enabling the representation of nonlinear functions. An approximate marginal likelihood is also derived for kernel hyperparameter learning. The two regression-based crowd counting methods are evaluated on a large pedestrian data set, containing very distinct camera views, pedestrian traffic, and outliers, such as bikes or skateboarders. Experimental results show that regression-based counts are accurate regardless of the crowd size, outperforming the count estimates produced by state-of-the-art pedestrian detectors. Results on 2 h of video demonstrate the efficiency and robustness of the regression-based crowd size estimation over long periods of time.

Semiparametric inference in mixture models with predictive recursion marginal likelihood

Predictive recursion is an accurate and computationally efficient algorithm for nonparametric estimation of mixing densities in mixture models. In semiparametric mixture models, however, the algorithm fails to account for any uncertainty in the additional unknown structural parameter. As an alternative to existing profile likelihood methods, we treat predictive recursion as a filter approximation to fitting a fully Bayes model, whereby an approximate marginal likelihood of the structural parameter emerges and can be used for inference. We call this the predictive recursion marginal likelihood. Convergence properties of predictive recursion under model mis-specification also lead to an attractive construction of this new procedure. We show pointwise convergence of a normalized version of this marginal likelihood function. Simulations compare the performance of this new marginal likelihood approach that of existing profile likelihood methods as well as Dirichlet process mixtures in density estimation. Mixed-effects models and an empirical Bayes multiple testing application in time series analysis are also considered.

Fast Stable Restricted Maximum Likelihood and Marginal Likelihood Estimation of Semiparametric Generalized Linear Models

Summary Recent work by Reiss and Ogden provides a theoretical basis for sometimes preferring restricted maximum likelihood (REML) to generalized cross-validation (GCV) for smoothing parameter selection in semiparametric regression. However, existing REML or marginal likelihood (ML) based methods for semiparametric generalized linear models (GLMs) use iterative REML or ML estimation of the smoothing parameters of working linear approximations to the GLM. Such indirect schemes need not converge and fail to do so in a non-negligible proportion of practical analyses. By contrast, very reliable prediction error criteria smoothing parameter selection methods are available, based on direct optimization of GCV, or related criteria, for the GLM itself. Since such methods directly optimize properly defined functions of the smoothing parameters, they have much more reliable convergence properties. The paper develops the first such method for REML or ML estimation of smoothing parameters. A Laplace approximation is used to obtain an approximate REML or ML for any GLM, which is suitable for efficient direct optimization. This REML or ML criterion requires that Newton–Raphson iteration, rather than Fisher scoring, be used for GLM fitting, and a computationally stable approach to this is proposed. The REML or ML criterion itself is optimized by a Newton method, with the derivatives required obtained by a mixture of implicit differentiation and direct methods. The method will cope with numerical rank deficiency in the fitted model and in fact provides a slight improvement in numerical robustness on the earlier method of Wood for prediction error criteria based smoothness selection. Simulation results suggest that the new REML and ML methods offer some improvement in mean-square error performance relative to GCV or Akaike’s information criterion in most cases, without the small number of severe undersmoothing failures to which Akaike’s information criterion and GCV are prone. This is achieved at the same computational cost as GCV or Akaike’s information criterion. The new approach also eliminates the convergence failures of previous REML- or ML-based approaches for penalized GLMs and usually has lower computational cost than these alternatives. Example applications are presented in adaptive smoothing, scalar on function regression and generalized additive model selection.

Statistical Inference for the Multidimensional Mixed Rasch Model

Inference in generalized linear mixed models with multivariate random effects is often made cumbersome by the high-dimensional intractable integrals involved in the marginal likelihood. This article presents an inferential methodology based on the GEE approach. This method involves the approximations of the marginal likelihood and joint moments of the variables. It is also proposed an approximate Akaike and Bayesian information criterions based on the approximate marginal likelihood using the estimation of the parameters by the GEE approach. The different results are illustrated with a simulation study and with an analysis of real data from health-related quality of life.

Mixed model-based inference in geoadditive hazard regression for interval-censored survival times

Mixed model-based estimation of additive or geoadditive regression models has become popular throughout recent years. It provides a unified and modular framework that facilitates joint estimation of nonparametric covariate effects and the corresponding smoothing parameters. Therefore, extensions of mixed model-based inference to a Cox-type regression model for the hazard rate are considered, allowing for a combination of general censoring schemes for the survival times and a flexible, geoadditive predictor. In particular, the proposed methodology allows for arbitrary combinations of right, left, and interval censoring as well as left truncation. The geoadditive predictor comprises time-varying effects, nonparametric effects of continuous covariates, spatial effects, and potentially a number of extensions such as cluster-specific frailties or interaction surfaces. In addition, all covariates are allowed to be piecewise constant time-varying. Nonlinear and time-varying effects as well as the baseline hazard rate are modeled by penalized splines. Spatial effects can be included based on either Markov random fields or stationary Gaussian random fields. Estimation is based on a reparametrization of the model as a variance component mixed model. The variance parameters, corresponding to inverse smoothing parameters, can then be determined using an approximate marginal likelihood approach. An analysis on childhood mortality in Nigeria serves as an application, where the interval censoring framework additionally allows to deal with the problem of heaped survival times. The effect of ignoring the impact of interval-censored observations is investigated in a simulation study.

Exponential Linear Models: A Two-Pass Procedure for Saddlepoint Approximation

SUMMARY For an exponential linear model, the saddlepoint method gives accurate approximations for the density of the minimal sufficient statistic or maximum likelihood estimate, and for the corresponding distribution functions. In this paper we describe a simple numerical procedure that constructs such approximations for a real parameter in an exponential linear model, using only a two-pass calculation on the observed likelihood function for the original data. Simple examples of the numerical procedure are discussed, but we take the general accuracy of the saddlepoint procedure as given. An immediate application of this is to exponential family models, where inference for a component of the canonical parameter is to be based on the conditional density of the corresponding component of the sufficient statistic, given the remaining components. This conditional density is also of exponential family form, but its functional form and cumulant-generating function may not be accessible. The procedure is applied to the corresponding likelihood, approximated as the full likelihood divided by an approximate marginal likelihood obtained from Barndorff-Nielsen's formula. A double saddlepoint approximation provides another means of bypassing this difficulty. The computational procedure is also examined as a numerical procedure for obtaining the saddlepoint approximation to the Fourier inversion of a characteristic function. As such it is a two-pass calculation on a table of the cumulant-generating function.