Marginal Inference Research Articles

Bias due to Berkson error: issues when using predicted values in place of observed covariates.

Studies often want to test for the association between an unmeasured covariate and an outcome. In the absence of a measurement, the study may substitute values generated from a prediction model. Justification for such methods can be found by noting that, with standard assumptions, this is equivalent to fitting a regression model for an outcome variable when at least one covariate is measured with Berkson error. Under this setting, it is known that consistent or nearly consistent inference can be obtained under many linear and nonlinear outcome models. In this article, we focus on the linear regression outcome model and show that this consistency property does not hold when there is unmeasured confounding in the outcome model, in which case the marginal inference based on a covariate measured with Berkson error differs from the same inference based on observed covariates. Since unmeasured confounding is ubiquitous in applications, this severely limits the practical use of such measurements, and, in particular, the substitution of predicted values for observed covariates. These issues are illustrated using data from the National Health and Nutrition Examination Survey to study the joint association of total percent body fat and body mass index with HbA1c. It is shown that using predicted total percent body fat in place of observed percent body fat yields inferences which often differ significantly, in some cases suggesting opposite relationships among covariates.

Anchored Bayesian Gaussian mixture models

Finite mixtures are a flexible modeling tool for irregularly shaped densities and samples from heterogeneous populations. When modeling with mixtures using an exchangeable prior on the component features, the component labels are arbitrary and are indistinguishable in posterior analysis. This makes it impossible to attribute any meaningful interpretation to the marginal posterior distributions of the component features. We propose a model in which a small number of observations are assumed to arise from some of the labeled component densities. The resulting model is not exchangeable, allowing inference on the component features without post-processing. Our method assigns meaning to the component labels at the modeling stage and can be justified as a data-dependent informative prior on the labelings. We show that our method produces interpretable results, often (but not always) similar to those resulting from relabeling algorithms, with the added benefit that the marginal inferences originate directly from a well specified probability model rather than a post hoc manipulation. We provide asymptotic results leading to practical guidelines for model selection that are motivated by maximizing prior information about the class labels and demonstrate our method on real and simulated data.

Permutation inference methods for multivariate meta-analysis.

Multivariate meta-analysis is gaining prominence in evidence synthesis research because it enables simultaneous synthesis of multiple correlated outcome data, and random-effects models have generally been used for addressing between-studies heterogeneities. However, coverage probabilities of confidence regions or intervals for standard inference methods for random-effects models (eg, restricted maximum likelihood estimation) cannot retain their nominal confidence levels in general, especially when the number of synthesized studies is small because their validities depend on large sample approximations. In this article, we provide permutation-based inference methods that enable exact joint inferences for average outcome measures without large sample approximations. We also provide accurate marginal inference methods under general settings of multivariate meta-analyses. We propose effective approaches for permutation inferences using optimal weighting based on the efficient score statistic. The effectiveness of the proposed methods is illustrated via applications to bivariate meta-analyses of diagnostic accuracy studies for airway eosinophilia in asthma and a network meta-analysis for antihypertensive drugs on incident diabetes, as well as through simulation experiments. In numerical evaluations performed via simulations, our methods generally provided accurate confidence regions or intervals under a broad range of settings, whereas the current standard inference methods exhibited serious undercoverage properties.

Marginal Inference in Continuous Markov Random Fields Using Mixtures

Exact marginal inference in continuous graphical models is computationally challenging outside of a few special cases. Existing work on approximate inference has focused on approximately computing the messages as part of the loopy belief propagation algorithm either via sampling methods or moment matching relaxations. In this work, we present an alternative family of approximations that, instead of approximating the messages, approximates the beliefs in the continuous Bethe free energy using mixture distributions. We show that these types of approximations can be combined with numerical quadrature to yield algorithms with both theoretical guarantees on the quality of the approximation and significantly better practical performance in a variety of applications that are challenging for current state-of-the-art methods.

Deep Convolutional Sum-Product Networks

We give conditions under which convolutional neural networks (CNNs) define valid sum-product networks (SPNs). One subclass, called convolutional SPNs (CSPNs), can be implemented using tensors, but also can suffer from being too shallow. Fortunately, tensors can be augmented while maintaining valid SPNs. This yields a larger subclass of CNNs, which we call deep convolutional SPNs (DCSPNs), where the convolutional and sum-pooling layers form rich directed acyclic graph structures. One salient feature of DCSPNs is that they are a rigorous probabilistic model. As such, they can exploit multiple kinds of probabilistic reasoning, including marginal inference and most probable explanation (MPE) inference. This allows an alternative method for learning DCSPNs using vectorized differentiable MPE, which plays a similar role to the generator in generative adversarial networks (GANs). Image sampling is yet another application demonstrating the robustness of DCSPNs. Our preliminary results on image sampling are encouraging, since the DCSPN sampled images exhibit variability. Experiments on image completion show that DCSPNs significantly outperform competing methods by achieving several state-of-the-art mean squared error (MSE) scores in both left-completion and bottom-completion in benchmark datasets.

A semiparametric marginalized zero-inflated model for analyzing healthcare utilization panel data with missingness

ABSTRACT Zero-inflated count outcomes arise quite often in research and practice. Parametric models such as the zero-inflated Poisson and zero-inflated negative binomial are widely used to model such responses. However, interpretations of those models focus on the at-risk subpopulation of a two-component population mixture and fail to provide direct inference about marginal effects for the overall population. Recently, new approaches have been proposed to facilitate such marginal inferences for count responses with excess zeros. However, they are likelihood based and impose strong assumptions on data distributions. In this paper, we propose a new distribution-free, or semiparametric, alternative to provide robust inference for marginal effects when population mixtures are defined by zero-inflated count outcomes. The proposed method also applies to longitudinal studies with missing data following the general missing at random mechanism. The proposed approach is illustrated with both simulated and real study data.

Marginal analysis of ordinal clustered longitudinal data with informative cluster size.

The issue of informative cluster size (ICS) often arises in the analysis of dental data. ICS describes a situation where the outcome of interest is related to cluster size. Much of the work on modeling marginal inference in longitudinal studies with potential ICS has focused on continuous outcomes. However, periodontal disease outcomes, including clinical attachment loss, are often assessed using ordinal scoring systems. In addition, participants may lose teeth over the course of the study due to advancing disease status. Here we develop longitudinal cluster-weighted generalized estimating equations (CWGEE) to model the association of ordinal clustered longitudinal outcomes with participant-level health-related covariates, including metabolic syndrome and smoking status, and potentially decreasing cluster size due to tooth-loss, by fitting a proportional odds logistic regression model. The within-teeth correlation coefficient over time is estimated using the two-stage quasi-least squares method. The motivation for our work stems from the Department of Veterans Affairs Dental Longitudinal Study in which participants regularly received general and oral health examinations. In an extensive simulation study, we compare results obtained from CWGEE with various working correlation structures to those obtained from conventional GEE which does not account for ICS. Our proposed method yields results with very low bias and excellent coverage probability in contrast to a conventional generalized estimating equations approach.

Projective Latent Dependency Forest Models

Latent dependence forest models (LDFM) are a new type of probabilistic models with the advantage of not requiring the difficult procedure of structure learning in model learning. However, normalized joint probability computation and marginal inference are intractable for LDFM. In this paper, we proposed projective LDFMs (PLDFMs), a variant of LDFM, for which joint and marginal probabilities become tractable (cubic time with respect to the number of random variables) to compute while learning remains easy. We show that PLDFMs can be seen as a special case of sum-product networks (SPNs). We then propose sum-product projective dependence networks, a combination of PLDFMs and SPNs that scales up to a large number of random variables. Our extensive experiments on 29 datasets show that our models achieve competitive results with other probabilistic models.

Approximate composite marginal likelihood inference in spatial generalized linear mixed models

ABSTRACTNon-Gaussian spatial responses are usually modeled using spatial generalized linear mixed model with spatial random effects. The likelihood function of this model cannot usually be given in a closed form, thus the maximum likelihood approach is very challenging. There are numerical ways to maximize the likelihood function, such as Monte Carlo Expectation Maximization and Quadrature Pairwise Expectation Maximization algorithms. They can be applied but may in such cases be computationally very slow or even prohibitive. Gauss–Hermite quadrature approximation only suitable for low-dimensional latent variables and its accuracy depends on the number of quadrature points. Here, we propose a new approximate pairwise maximum likelihood method to the inference of the spatial generalized linear mixed model. This approximate method is fast and deterministic, using no sampling-based strategies. The performance of the proposed method is illustrated through two simulation examples and practical aspects are investigated through a case study on a rainfall data set.

Robustifying sum-product networks

Sum-product networks are a relatively new and increasingly popular family of probabilistic graphical models that allow for marginal inference with polynomial effort. They have been shown to achieve state-of-the-art performance in several tasks involving density estimation. Sum-product networks are typically learned from data; as such, inferences produced with them are prone to be unreliable and overconfident when data is scarce. In this work, we develop the credal sum-product networks, a generalization of sum-product networks that uses set-valued parameters. We present algorithms and complexity results for common inference tasks with this class of models. We also present an approach for assessing the reliability of classifications made with sum-product networks. We apply this approach on benchmark classification tasks as well as a new application in predicting the age of stars. Our experiments show that the use of credal sum-product networks allow us to distinguish between reliable and unreliable classifications with higher accuracy than standard approaches based on (precise) probability values.

A collection of marginalized two-part random-effects models for analyzing medical expenditure panel data: Impact of the New Cooperative Medical Scheme on healthcare expenditures in China

Marginalized two-part random-effects generalized Gamma models have been proposed for analyzing medical expenditure panel data with excessive zeros. While these models provide marginal inference on expected healthcare expenditures, the usual unilateral specification of heteroscedastic variance on one of the two shape parameters for the generalized Gamma distribution in these models fails to encompass important special cases within the generalized gamma modeling framework. In this article, we construct marginalized two-part random-effects models that employ the log-normal, log-skew-normal, generalized Gamma, Weibull, Gamma, and inverse Gamma distributions to delineate the spectrum of nonzero healthcare expenditures in the second part of the models. These marginalized models supply additional choices for analyzing healthcare expenditure panel data with excessive zeros. We review the concepts of marginal effect and incremental effect, and summarize how these effects are estimated. For studies whose primary goal is to make inference on marginal effect or incremental effect of an independent variable with respect to healthcare expenditures, we advocate empirical mean square error criterion and information criteria to choose among candidate models. Then, we use the proposed models in an empirical analysis to examine the impact of the New Cooperative Medical Scheme on healthcare expenditures among older adults in rural China.

Are marginalized two-part models superior to non-marginalized two-part models for count data with excess zeroes? Estimation of marginal effects, model misspecification, and model selection

The marginalized two-part models, including the marginalized zero-inflated Poisson and negative binomial models, have been proposed in the literature for modelling cross-sectional healthcare utilization count data with excess zeroes and overdispersion. The motivation for these proposals was to directly capture the overall marginal effects and to avoid post-modelling effect calculations that are needed for the non-marginalized conventional two-part models. However, are marginalized two-part models superior to non-marginalized two-part models because of their structural property? Is it true that the marginalized two-part models can provide direct marginal inference? This article aims to answer these questions through a comprehensive investigation. We first summarize the existing non-marginalized and marginalized two-part models and then develop marginalized hurdle Poisson and negative binomial models for cross-sectional count data with abundant zero counts. Our interest in the investigation lies particularly in the (average) marginal effect and (average) incremental effect and the comparison of these effects. The estimators of these effects are presented, and variance estimators are derived by using delta methods and Taylor series approximations. Though the marginalized models attract attention because of the alleged convenience of direct marginal inference, we provide evidence for the impact of model misspecification of the marginalized models over the conventional models, and provide evidence for the importance of goodness-of-fit evaluation and model selection in differentiating between the marginalized and non-marginalized models. An empirical analysis of the German Socioeconomic Panel data is presented.

Maximum A Posteriori Inference in Sum-Product Networks

Sum-product networks (SPNs) are a class of probabilistic graphical models that allow tractable marginal inference. However, the maximum a posteriori (MAP) inference in SPNs is NP-hard. We investigate MAP inference in SPNs from both theoretical and algorithmic perspectives. For the theoretical part, we reduce general MAP inference to its special case without evidence and hidden variables; we also show that it is NP-hard to approximate the MAP problem to 2nε for fixed 0 ≤ ε < 1, where n is the input size. For the algorithmic part, we first present an exact MAP solver that runs reasonably fast and could handle SPNs with up to 1k variables and 150k arcs in our experiments. We then present a new approximate MAP solver with a good balance between speed and accuracy, and our comprehensive experiments on real-world datasets show that it has better overall performance than existing approximate solvers.

Unification of MAP Estimation and Marginal Inference in Recurrent Neural Networks.

Numerous experimental data show that human brain can represent probability distributions and perform Bayesian inference. However, it remains unclear how the brain implements probabilistic inference in the form of neural circuits. Several models have been proposed that aim at explaining how the network of neurons carry out maximum a posterior inference (MAP) estimation and marginal inference, but they are all task specific in that they treat MAP estimation and marginal inference separately. In this brief, we propose that human brain could implement MAP estimation and marginal inference in the same network of neurons. We illustrate our result in hidden Markov models and prove that a recurrent neural network (RNN) implementation of belief propagation can be tuned to perform approximate Bayesian inference (to provide posterior or conditional distribution over the latent causes of observations) or identify the MAP or peak of the joint distribution. The key tuning parameter is a temperature parameter that controls the precision of probability distributions that are optimized. Theoretical analyses and experimental results demonstrate that RNNs can carry out near-optimal MAP estimation and marginal inference.

Direct and flexible marginal inference for semicontinuous data: Response.

Direct and flexible marginal inference for semicontinuous data: Response.

Efficient Message Passing Methods With Fully Connected Models for Early Vision.

Fully connected Markov random fields and conditional random fields have recently been shown to be advantageous in many early vision tasks being formulated as multi-labeling problems, such as stereo matching and image segmentation. The maximum posterior marginal (MPM) inference method in solving fully connected models uses a hybrid framework of mean-field (MF) method and a filtering like approach, and yields excellent results. In this paper, we extend this framework in several aspects. First, we provide an alternative inference method employing fractional belief propagation based method instead of MF. Second, we reformulate the MPM problem into a maximum a posterior (MAP) problem and provide efficient algorithms for solving this. Third, we extend the fully connected model into a multi-resolution approach. Finally, we propose an integral image based approach which makes it possible for efficiently integrating the local linear regression technique into this framework. Comparisons are carried out among different algorithms and different formulations to find the best combination. We demonstrate that the use of our multi-resolution approach with MAP formulation substantially outperforms the ordinary MF-based inference scheme.

Estimation and Inference of Regression Discontinuity Design with Ordered or Discrete Duration Outcomes

We consider the regression discontinuity (RD) design with the duration outcome which has discrete support. The parameters of policy interest are treatment effects on unconditional (duration effect) and conditional (hazard effect) exiting probabilities for each discrete level. We find that a flexible separability structure of the underlying continuous-time duration process can be exploited to substantially improve the quality of the fully nonparametric estimator. We propose global sieve-based estimators, and associated marginal and simultaneous inference. Simultaneous inference over discrete levels is nonstandard since the asymptotic variance matrix is singular with unknown rank. The peculiarity is delivered by the nature of the RD estimand, and we provide solutions. Random censoring and competing risks can also be allowed in our framework. The standard practice of applying local linear estimators to a sequence of binary outcomes is in general unsatisfactory, which motivates our semi-nonparametric approach. First, it provides poor hazard estimates near the end of the observation period due to small sizes of risk sets (in the neighborhood of the cutoff). Second, it fits each probability separately and thus does not support joint inference. The estimation and inference methods we advocate in this paper are computationally easy and fast to implement, which is illustrated by numerical examples.

Anytime Best+Depth-First Search for Bounding Marginal MAP

We introduce new anytime search algorithms that combine best-first with depth-first search into hybrid schemes for Marginal MAP inference in graphical models. The main goal is to facilitate the generation of upper bounds (via the best-first part) alongside the lower bounds of solutions (via the depth-first part) in an anytime fashion. We compare against two of the best current state-of-the-art schemes and show that our best+depth search scheme produces higher quality solutions faster while also producing a bound on their accuracy, which can be used to measure solution quality during search. An extensive empirical evaluation demonstrates the effectiveness of our new methods which enjoy the strength of best-first (optimality of search) and of depth-first (memory robustness), leading to solutions for difficult instances where previous solvers were unable to find even a single solution.

Estimating marginal and incremental effects in the analysis of medical expenditure panel data using marginalized two-part random-effects generalized Gamma models: Evidence from China healthcare cost data.

Conditional two-part random-effects models have been proposed for the analysis of healthcare cost panel data that contain both zero costs from the non-users of healthcare facilities and positive costs from the users. These models have been extended to accommodate more flexible data structures when using the generalized Gamma distribution to model the positive healthcare expenditures. However, a major drawback with the extended model, which is inherited from the conditional models, is that it is fairly difficult to make direct marginal inference with respect to overall healthcare costs that includes both zeros and non-zeros, or even on positive healthcare costs. In this article, we first propose two types of marginalized two-part random-effects generalized Gamma models (m2RGGMs): Type I m2RGGMs for the inference on positive healthcare costs and Type II m2RGGMs for the inference on overall healthcare costs. Then, the concepts of marginal effect and incremental effect of a covariate on overall and positive healthcare costs are introduced, and estimation of these effects is carefully discussed. Especially, we derive the variance estimates of these effects by following the delta methods and Taylor series approximations for the purpose of making marginal inference. Parameter estimates of Type I and Type II m2RGGMs are obtained through maximum likelihood estimation. An empirical analysis of longitudinal healthcare costs collected in the China Health and Nutrition Survey is conducted using the proposed methodologies.

Stan: A Probabilistic Programming Language.

Stan is a probabilistic programming language for specifying statistical models. A Stan program imperatively defines a log probability function over parameters conditioned on specified data and constants. As of version 2.14.0, Stan provides full Bayesian inference for continuous-variable models through Markov chain Monte Carlo methods such as the No-U-Turn sampler, an adaptive form of Hamiltonian Monte Carlo sampling. Penalized maximum likelihood estimates are calculated using optimization methods such as the limited memory Broyden-Fletcher-Goldfarb-Shanno algorithm. Stan is also a platform for computing log densities and their gradients and Hessians, which can be used in alternative algorithms such as variational Bayes, expectation propagation, and marginal inference using approximate integration. To this end, Stan is set up so that the densities, gradients, and Hessians, along with intermediate quantities of the algorithm such as acceptance probabilities, are easily accessible. Stan can be called from the command line using the cmdstan package, through R using the rstan package, and through Python using the pystan package. All three interfaces support sampling and optimization-based inference with diagnostics and posterior analysis. rstan and pystan also provide access to log probabilities, gradients, Hessians, parameter transforms, and specialized plotting.