Marginal Inference Research Articles

Marginal inference for hierarchical generalized linear mixed models with patterned covariance matrices using the Laplace approximation

AbstractWe develop hierarchical models and methods in a fully parametric approach to generalized linear mixed models for any patterned covariance matrix. The Laplace approximation is used to marginally estimate covariance parameters by integrating over all fixed and latent random effects. The Laplace approximation relies on Newton–Raphson updates, which also leads to predictions for the latent random effects. We develop methodology for complete marginal inference, from estimating covariance parameters and fixed effects to making predictions for unobserved data. The marginal likelihood is developed for six distributions that are often used for binary, count, and positive continuous data, and our framework is easily extended to other distributions. We compare our methods to fully Bayesian methods, automatic differentiation, and integrated nested Laplace approximations (INLA) for bias, mean‐squared (prediction) error, and interval coverage, and all methods yield very similar results. However, our methods are much faster than Bayesian methods, and more general than INLA. Examples with binary and proportional data, count data, and positive‐continuous data are used to illustrate all six distributions with a variety of patterned covariance structures that include spatial models (both geostatistical and areal models), time series models, and mixtures with typical random intercepts based on grouping.

Neural Network Approximators for Marginal MAP in Probabilistic Circuits

Probabilistic circuits (PCs) such as sum-product networks efficiently represent large multi-variate probability distributions. They are preferred in practice over other probabilistic representations, such as Bayesian and Markov networks, because PCs can solve marginal inference (MAR) tasks in time that scales linearly in the size of the network. Unfortunately, the most probable explanation (MPE) task and its generalization, the marginal maximum-a-posteriori (MMAP) inference task remain NP-hard in these models. Inspired by the recent work on using neural networks for generating near-optimal solutions to optimization problems such as integer linear programming, we propose an approach that uses neural networks to approximate MMAP inference in PCs. The key idea in our approach is to approximate the cost of an assignment to the query variables using a continuous multilinear function and then use the latter as a loss function. The two main benefits of our new method are that it is self-supervised, and after the neural network is learned, it requires only linear time to output a solution. We evaluate our new approach on several benchmark datasets and show that it outperforms three competing linear time approximations: max-product inference, max-marginal inference, and sequential estimation, which are used in practice to solve MMAP tasks in PCs.

Marginal clustered multistate models for longitudinal progressive processes with informative cluster size

AbstractInformative cluster size (ICS) is a phenomenon where cluster size is related to the outcome. While multistate models can be applied to characterize the unit‐level transition process for clustered interval‐censored data, there is a research gap addressing ICS within this framework. We propose two extensions of multistate model that account for ICS to make marginal inference: one by incorporating within‐cluster resampling and another by constructing cluster‐weighted score functions. We evaluate the performances of the proposed methods through simulation studies and apply them to the Veterans Affairs Dental Longitudinal Study (VADLS) to understand the effect of risk factors on periodontal disease progression. ICS occurs frequently in dental data, particularly in the study of periodontal disease, as people with fewer teeth due to the disease are more susceptible to disease progression. According to the simulation results, the mean estimates of the parameters obtained from the proposed methods are close to the true values, but methods that ignore ICS can lead to substantial bias. Our proposed methods for clustered multistate model are able to appropriately take ICS into account when making marginal inference of a typical unit from a randomly sampled cluster.

Partial conditioning for inference of many-normal-means with Hölder constraints

Inferential models have been proposed for valid and efficient prior-free probabilistic inference. As it gradually gained popularity, this theory is subject to further developments for practically challenging problems. This paper considers the many-normal-means problem with the means constrained to be in the neighborhood of each other, formally represented by a Hölder space. A new method, called partial conditioning, is proposed to generate valid and efficient marginal inference about the individual means. It is shown that the method outperforms both a fiducial-counterpart in terms of validity and a conservative-counterpart in terms of efficiency. We conclude the paper by remarking that a general theory of partial conditioning for inferential models deserves future development.

Swift Markov Logic for Probabilistic Reasoning on Knowledge Graphs

AbstractWe provide a framework for probabilistic reasoning in Vadalog-based Knowledge Graphs (KGs), satisfying the requirements of ontological reasoning: full recursion, powerful existential quantification, expression of inductive definitions. Vadalog is a Knowledge Representation and Reasoning (KRR) language based on Warded Datalog+/–, a logical core language of existential rules, with a good balance between computational complexity and expressive power. Handling uncertainty is essential for reasoning with KGs. Yet Vadalog and Warded Datalog+/– are not covered by the existing probabilistic logic programming and statistical relational learning approaches for several reasons, including insufficient support for recursion with existential quantification and the impossibility to express inductive definitions. In this work, we introduce Soft Vadalog, a probabilistic extension to Vadalog, satisfying these desiderata. A Soft Vadalog program induces what we call a Probabilistic Knowledge Graph (PKG), which consists of a probability distribution on a network of chase instances, structures obtained by grounding the rules over a database using the chase procedure. We exploit PKGs for probabilistic marginal inference. We discuss the theory and present MCMC-chase, a Monte Carlo method to use Soft Vadalog in practice. We apply our framework to solve data management and industrial problems and experimentally evaluate it in the Vadalog system.

Columbian sharp‐tailed grouse female and nest survival in northwestern Colorado

AbstractWildlife management and conservation can be challenging when the demography of a focal species is unknown or limited given that fecundity and adult survival influence population growth. The Columbian sharp‐tailed grouse (Tympanuchus phasianellus columbianus) have been reduced to ≤10% of their former range since the early 1900s. We conducted a 3‐year study (2015–2017) across 4 study sites in northwestern Colorado, USA, to evaluate female hazard and nest survival. We trapped and marked 270 female sharp‐tailed grouse and identified 275 nests for our hazard and survival analyses. Females during the breeding stage of the reproductive season experienced more hazard compared to the nesting and the early and late post‐nesting stages for females without broods. Females experienced a higher degree of hazard during the breeding stage and mortality risk was >3 times higher than the nesting stage, >7 times higher than early post‐nesting (EPN)‐no brood stage, and >5 times higher than the late post‐nesting (LPN)‐unsuccessful stage. Two reproductive season stages (LPN‐successful and EPN‐brood) provided marginal inference. Nest incubation initiation date and nest age best described nest daily survival. Females that initiated incubation of initial nests earlier in the season experienced lower nest daily survival than later in the incubation season. Because female Columbian sharp‐tailed grouse hazard varied among different reproductive season stages, we recommend that wildlife managers develop management actions that reduce hazard during the specific reproductive season stages (i.e., the breeding season). For Columbian sharp‐tailed grouse in Colorado, we recommend that Colorado Parks and Wildlife collaborate with federal farm program agencies to implement a no‐tillage restriction from 1 May through 30 June for active agricultural fields within 2 km of active Columbian sharp‐tailed grouse leks to enhance nest survival.

Region‐based estimation of the partition functions for hybrid Bayesian network models

The partition function Ƶ ${\boldsymbol{Ƶ}}$ is a normalization constant for normalizing all the distributions in probabilistic inference. Ƶ ${\boldsymbol{Ƶ}}$ is closely related to the log probability of evidence ( log p ( e ) $\mathrm{log}\unicode{x0200A}p(e)$ ) for Bayesian networks (BNs), which plays an important role in many applications, such as parameter learning, classification, and clustering. However, evaluating Ƶ ${\boldsymbol{Ƶ}}$ and log p ( e ) $\mathrm{log}\unicode{x0200A}p(e)$ for BNs containing both discrete and continuous variables (known as hybrid Bayesian networks [HBNs]) is generally difficult for analytical and simulation-based solutions. This paper describes the work addressing the problem of estimating the partition function for HBNs when exact methods become inefficient or numerically unstable. We use dynamic discretization to provide a discretized version of the model and then perform region-based free energy optimization to obtain the log p ( e ) $\mathrm{log}\unicode{x0200A}p(e)$ for the HBN efficiently, which is called the DDJT-region partition function (RPR) algorithm. It is novel since the region-based methods have previously only been applied to discrete BNs. We show that the log p ( e ) $\mathrm{log}\unicode{x0200A}p(e)$ obtained by DDJT-RPR is exact for discretized models and with space complexity equal to that in marginal inference, thus generally applicable to a wide range of applications. To demonstrate these properties, we combine the DDJT-RPR algorithm with an improved expectation-maximization algorithm to learn Gaussian mixture models (GMMs), for applications requiring data only and with prior knowledge, which performed more robustly than conventional GMM learning algorithms.

On Cost-Efficient Learning of Data Dependency

In this paper, we consider the problem of learning a tree graph structure that represents the statistical data dependency among nodes for a set of data samples generated by nodes, which provides the basic structure to perform a probabilistic inference task. Inference in the data graph includes marginal inference and maximum a posteriori (MAP) estimation, and belief propagation (BP) is a commonly used algorithm to compute the marginal distribution of nodes via message-passing, incurring non-negligible amount of communication cost. We inevitably have the trade-off between the inference accuracy and the message-passing cost because the learned structure of data dependency and physical connectivity graph are often highly different. In this paper, we formalize this trade-off in an optimization problem which outputs the data dependency graph that jointly considers learning accuracy and message-passing costs. We focus on two popular implementations of BP, <monospace xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">ASYNC-BP</monospace> and <monospace xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">SYNC-BP</monospace> , which have different message-passing mechanisms and cost structures. In <monospace xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">ASYNC-BP</monospace> , we propose a polynomial-time learning algorithm that is optimal, motivated by finding a maximum weight spanning tree of a complete graph. In <monospace xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">SYNC-BP</monospace> , we prove the NP-hardness of the problem and propose a greedy heuristic. For both BP implementations, we quantify how the error probability that the learned cost-efficient data graph differs from the ideal one decays as the number of data samples grows, using the large deviation principle, which provides a guideline on how many samples are necessary to obtain a certain trade-off. We validate our theoretical findings through extensive simulations, which confirms that it has a good match.

Accounting for drop-out using inverse probability censoring weights in longitudinal clustered data with informative cluster size

Periodontal disease is a serious gum infection impacting half of the U.S. adult population that may lead to loss of teeth. Using standard marginal models to study the association between patient-level predictors and tooth-level outcomes can lead to biased estimates because the independence assumption between the outcome (periodontal disease) and cluster size (number of teeth per patient) is violated. Specifically, the baseline number of teeth of a patient is informative. In this setting a cluster-weighted generalized estimating equations (CWGEE) approach can be used to obtain unbiased marginal inference from data with informative cluster size (ICS). However, in many longitudinal studies of dental health, including the Veterans Affairs Dental Longitudinal Study, the rate of tooth-loss or tooth drop-out over time is also informative, creating a missing at random data mechanism. Here, we propose a novel modeling approach that incorporates the technique of inverse probability censoring weights into CWGEE with binary outcomes to account for ICS and informative drop-out over time. In an extensive simulation study we demonstrate that results obtained from our proposed method yield lower bias and excellent coverage probability, compared to those obtained from traditional methods which do not account for ICS or drop-out.

Marginal and Conditional Multiple Inference for Linear Mixed Model Predictors

In spite of its high practical relevance, cluster specific multiple inference for linear mixed model predictors has hardly been addressed so far. While marginal inference for population parameters is well understood, conditional inference for the cluster specific predictors is more intricate. This work introduces a general framework for multiple inference in linear mixed models for cluster specific predictors. Consistent confidence sets for multiple inference are constructed under both, the marginal and the conditional law. Furthermore, it is shown that, remarkably, corresponding multiple marginal confidence sets are also asymptotically valid for conditional inference. Those lend themselves for testing linear hypotheses using standard quantiles without the need of resampling techniques. All findings are validated in simulations and illustrated along a study on Covid-19 mortality in the U.S. state prisons. Supplementary materials for this article are available online.

Marginalized Two-Part Joint Modeling of Longitudinal Semi-Continuous Responses and Survival Data: With Application to Medical Costs

Non-negative continuous outcomes with a substantial number of zero values and incomplete longitudinal follow-up are quite common in medical costs data. It is thus critical to incorporate the potential dependence of survival status and longitudinal medical costs in joint modeling, where censorship is death-related. Despite the wide use of conventional two-part joint models (CTJMs) to capture zero-inflation, they are limited to conditional interpretations of the regression coefficients in the model’s continuous part. In this paper, we propose a marginalized two-part joint model (MTJM) to jointly analyze semi-continuous longitudinal costs data and survival data. We compare it to the conventional two-part joint model (CTJM) for handling marginal inferences about covariate effects on average costs. We conducted a series of simulation studies to evaluate the superior performance of the proposed MTJM over the CTJM. To illustrate the applicability of the MTJM, we applied the model to a set of real electronic health record (EHR) data recently collected in Iran. We found that the MTJM yielded a smaller standard error, root-mean-square error of estimates, and AIC value, with unbiased parameter estimates. With this MTJM, we identified a significant positive correlation between costs and survival, which was consistent with the simulation results.

Exact Lifted Inference with Distinct Soft Evidence on Every Object

The presence of non-symmetric evidence has been a barrier for the application of lifted inference since the evidence destroys the symmetry of the first-order probabilistic model. In the extreme case, if distinct soft evidence is obtained about each individual object in the domain then, often, all current exact lifted inference methods reduce to traditional inference at the ground level. However, it is of interest to ask whether the symmetry of the model itself before evidence is obtained can be exploited. We present new results showing that this is, in fact, possible. In particular, we show that both exact maximum a posteriori (MAP) and marginal inference can be lifted for the case of distinct soft evidence on a unary Markov Logic predicate. Our methods result in efficient procedures for MAP and marginal inference for a class of graphical models previously thought to be intractable.

Estimating Spatial Econometrics Models with Integrated Nested Laplace Approximation

The integrated nested Laplace approximation (INLA) provides a fast and effective method for marginal inference in Bayesian hierarchical models. This methodology has been implemented in the R-INLA package which permits INLA to be used from within R statistical software. Although INLA is implemented as a general methodology, its use in practice is limited to the models implemented in the R-INLA package. Spatial autoregressive models are widely used in spatial econometrics but have until now been lacking from the R-INLA package. In this paper, we describe the implementation and application of a new class of latent models in INLA made available through R-INLA. This new latent class implements a standard spatial lag model. The implementation of this latent model in R-INLA also means that all the other features of INLA can be used for model fitting, model selection and inference in spatial econometrics, as will be shown in this paper. Finally, we will illustrate the use of this new latent model and its applications with two data sets based on Gaussian and binary outcomes.

Risk prediction in multicentre studies when there is confounding by cluster or informative cluster size

BackgroundClustered data arise in research when patients are clustered within larger units. Generalised Estimating Equations (GEE) and Generalised Linear Models (GLMM) can be used to provide marginal and cluster-specific inference and predictions, respectively.MethodsConfounding by Cluster (CBC) and Informative cluster size (ICS) are two complications that may arise when modelling clustered data. CBC can arise when the distribution of a predictor variable (termed ‘exposure’), varies between clusters causing confounding of the exposure-outcome relationship. ICS means that the cluster size conditional on covariates is not independent of the outcome. In both situations, standard GEE and GLMM may provide biased or misleading inference, and modifications have been proposed. However, both CBC and ICS are routinely overlooked in the context of risk prediction, and their impact on the predictive ability of the models has been little explored. We study the effect of CBC and ICS on the predictive ability of risk models for binary outcomes when GEE and GLMM are used. We examine whether two simple approaches to handle CBC and ICS, which involve adjusting for the cluster mean of the exposure and the cluster size, respectively, can improve the accuracy of predictions.ResultsBoth CBC and ICS can be viewed as violations of the assumptions in the standard GLMM; the random effects are correlated with exposure for CBC and cluster size for ICS. Based on these principles, we simulated data subject to CBC/ICS. The simulation studies suggested that the predictive ability of models derived from using standard GLMM and GEE ignoring CBC/ICS was affected. Marginal predictions were found to be mis-calibrated. Adjusting for the cluster-mean of the exposure or the cluster size improved calibration, discrimination and the overall predictive accuracy of marginal predictions, by explaining part of the between cluster variability. The presence of CBC/ICS did not affect the accuracy of conditional predictions. We illustrate these concepts using real data from a multicentre study with potential CBC.ConclusionIgnoring CBC and ICS when developing prediction models for clustered data can affect the accuracy of marginal predictions. Adjusting for the cluster mean of the exposure or the cluster size can improve the predictive accuracy of marginal predictions.

Multi-Marginal Optimal Transport and Probabilistic Graphical Models

We study multi-marginal optimal transport problems from a probabilistic graphical model perspective. We point out an elegant connection between the two when the underlying cost for optimal transport allows a graph structure. In particular, an entropy regularized multi-marginal optimal transport is equivalent to a Bayesian marginal inference problem for probabilistic graphical models with the additional requirement that some of the marginal distributions are specified. This relation on the one hand extends the optimal transport as well as the probabilistic graphical model theories, and on the other hand leads to fast algorithms for multi-marginal optimal transport by leveraging the well-developed algorithms in Bayesian inference. Several numerical examples are provided to highlight the results.

Marginal posterior distributions for regression parameters in the Cox model using Dirichlet and gamma process priors

A Bayesian treatment of the proportional hazard (PH) model is revisited using Dirichlet process and gamma process priors for the baseline survival and cumulative hazard functions respectively. Such priors, due to their discrete support, conflict with the absolutely continuous nature of survival responses expressed through the hazard function structure that defines the PH model. We resolve this conflict through the use of a proposed ε-grid likelihood approach which we apply to the PH model to consider marginal inference for the regression parameter β using expansions as ε↓0. Using Dirichlet process priors, the ε-grid likelihood approach leads to a new explicit marginal posterior density expansion for β which accommodates the most general setting with arbitrary ties and right censoring. In the previously considered case of gamma process priors, the approach extends the results of Kalbfleisch (1978) to deal with arbitrary arrangements of ties and also provides a rigorous justification for his posterior expressions. As in Kalbfleisch (1978), we show that the leading terms of both expansions approximate Cox’s partial likelihood when there are no ties and the process priors are diffuse. The ε-grid likelihood approach is similar in concept to the grouped data likelihood approach of Sinha et al. (2003) but differs in the likelihood approximation. Whereas our expansions are Poincaré expansions (with relative error O(ε) as ε↓0), those of Sinha et al. (2003) are stochastic expansions and not proper Poincaré expansion (as we show), so that they lack relative error O(ε). Using a flat improper prior for β, the two marginal posterior expressions for β are shown to be integrable for general arrangements of ties and censoring under weak conditions on the design and failures.

Marginal modeling of cluster-period means and intraclass correlations in stepped wedge designs with binary outcomes.

SummaryStepped wedge cluster randomized trials (SW-CRTs) with binary outcomes are increasingly used in prevention and implementation studies. Marginal models represent a flexible tool for analyzing SW-CRTs with population-averaged interpretations, but the joint estimation of the mean and intraclass correlation coefficients (ICCs) can be computationally intensive due to large cluster-period sizes. Motivated by the need for marginal inference in SW-CRTs, we propose a simple and efficient estimating equations approach to analyze cluster-period means. We show that the quasi-score for the marginal mean defined from individual-level observations can be reformulated as the quasi-score for the same marginal mean defined from the cluster-period means. An additional mapping of the individual-level ICCs into correlations for the cluster-period means further provides a rigorous justification for the cluster-period approach. The proposed approach addresses a long-recognized computational burden associated with estimating equations defined based on individual-level observations, and enables fast point and interval estimation of the intervention effect and correlations. We further propose matrix-adjusted estimating equations to improve the finite-sample inference for ICCs. By providing a valid approach to estimate ICCs within the class of generalized linear models for correlated binary outcomes, this article operationalizes key recommendations from the CONSORT extension to SW-CRTs, including the reporting of ICCs.

Estimation of Covid-19 prevalence from serology tests: A partial identification approach

We propose a partial identification method for estimating disease prevalence from serology studies. Our data are results from antibody tests in some population sample, where the test parameters, such as the true/false positive rates, are unknown. Our method scans the entire parameter space, and rejects parameter values using the joint data density as the test statistic. The proposed method is conservative for marginal inference, in general, but its key advantage over more standard approaches is that it is valid in finite samples even when the underlying model is not point identified. Moreover, our method requires only independence of serology test results, and does not rely on asymptotic arguments, normality assumptions, or other approximations. We use recent Covid-19 serology studies in the US, and show that the parameter confidence set is generally wide, and cannot support definite conclusions. Specifically, recent serology studies from California suggest a prevalence anywhere in the range 0%-2% (at the time of study), and are therefore inconclusive. However, this range could be narrowed down to 0.7%–1.5% if the actual false positive rate of the antibody test was indeed near its empirical estimate (∼0.5%). In another study from New York state, Covid-19 prevalence is confidently estimated in the range 13%–17% in mid-April of 2020, which also suggests significant geographic variation in Covid-19 exposure across the US. Combining all datasets yields a 5%–8% prevalence range. Our results overall suggest that serology testing on a massive scale can give crucial information for future policy design, even when such tests are imperfect and their parameters unknown.

Bayesian Spatial Clustering of Extremal Behavior for Hydrological Variables

To address the need for efficient inference for a range of hydrological extreme value problems, spatial pooling of information is the standard approach for marginal tail estimation. We propose the first extreme value spatial clustering methods which account for both the similarity of the marginal tails and the spatial dependence structure of the data to determine the appropriate level of pooling. Spatial dependence is incorporated in two ways: to determine the cluster selection and to account for dependence of the data over sites within a cluster when making the marginal inference. We introduce a statistical model for the pairwise extremal dependence which incorporates distance between sites, and accommodates our belief that sites within the same cluster tend to exhibit a higher degree of dependence than sites in different clusters. By combining the models for the marginal tails and the dependence structure, we obtain a composite likelihood for the joint spatial distribution. We use a Bayesian framework which learns about both the number of clusters and their spatial structure, and that enables the inference of site-specific marginal distributions of extremes to incorporate uncertainty in the clustering allocation. The approach is illustrated using simulations, the analysis of daily precipitation levels in Norway and daily river flow levels in the UK. Code and data for the simulation study and river flow example are available in the online supplementary materials.

Bayesian analysis of Turkish Income and Living Conditions data, using clustered longitudinal ordinal modelling with Bridge distributed random effects

This article is motivated by the panel surveys, called Statistics on Income and Living Conditions (SILC), conducted annually on (randomly selected) country representative households to monitor EU 2020 aims on poverty reduction. We particularly consider the surveys conducted in Turkey within the scope of integration to the EU. Our main interests are on health aspects of economic and living conditions. The outcome is self-reported health that is clustered longitudinal ordinal, since repeated measures of it are nested within individuals and individuals are nested within families. Economic and living conditions have been measured through a number of individual- and family-level explanatory variables. The questions of interest are on the marginal relationships between the outcome and covariates that we address using a polytomous logistic regression with Bridge distributed random effects. This choice of distribution allows us to directly obtain marginal inferences in the presence of random effects. Widely used Normal distribution is also considered as the random effects distribution. Samples from the joint posterior densities of parameters and random effects are drawn using Markov Chain Monte Carlo. Interesting findings from the public health point of view are that differences were found between the subgroups of employment status, income level and panel year in terms of odds of reporting better health.