Conditional Independence Structure Research Articles

Jewel: A Novel Method for Joint Estimation of Gaussian Graphical Models

In this paper, we consider the problem of estimating multiple Gaussian Graphical Models from high-dimensional datasets. We assume that these datasets are sampled from different distributions with the same conditional independence structure, but not the same precision matrix. We propose jewel, a joint data estimation method that uses a node-wise penalized regression approach. In particular, jewel uses a group Lasso penalty to simultaneously guarantee the resulting adjacency matrix’s symmetry and the graphs’ joint learning. We solve the minimization problem using the group descend algorithm and propose two procedures for estimating the regularization parameter. Furthermore, we establish the estimator’s consistency property. Finally, we illustrate our estimator’s performance through simulated and real data examples on gene regulatory networks.

Characterisation of conditional independence structures for polymatroids using vanishing sets

Characterisation of conditional independence structures for polymatroids using vanishing sets

Discovering Causal Models with Optimization: Confounders, Cycles, and Feature Selection

We propose a new method for learning causal structures from observational data, a process known as causal discovery. Our method takes as input observational data over a set of variables and returns a graph in which causal relations are specified by directed edges. We consider a highly general search space that accommodates latent confounders and feedback cycles, which few extant methods do. We formulate the discovery problem as an integer program, and propose a solution technique that leverages the conditional independence structure in the data to identify promising edges for inclusion in the output graph. In the large-sample limit, our method recovers a graph that is equivalent to the true data-generating graph. Computationally, our method is competitive with the state-of-the-art, and can solve in minutes instances that are intractable for alternative causal discovery methods. We demonstrate our approach by showing how it can be used to examine the validity of instrumental variables, which are widely used for causal inference. In particular, we analyze US Census data from the seminal paper on the returns to education by Angrist and Krueger (1991), and find that the causal structures uncovered by our method are consistent with the literature.

A loss‐based prior for Gaussian graphical models

SummaryGaussian graphical models play an important role in various areas such as genetics, finance, statistical physics and others. They are a powerful modelling tool, which allows one to describe the relationships among the variables of interest. From the Bayesian perspective, there are two sources of randomness: one is related to the multivariate distribution and the quantities that may parametrise the model, and the other has to do with the underlying graph, G, equivalent to describing the conditional independence structure of the model under consideration. In this paper, we propose a prior on G based on two loss components. One considers the loss in information one would incur in selecting the wrong graph, while the second penalises for large number of edges, favouring sparsity. We illustrate the prior on simulated data and on real datasets, and compare the results with other priors on G used in the literature. Moreover, we present a default choice of the prior as well as discuss how it can be calibrated so as to reflect available prior information.

Bayesian Network based modeling of regional rainfall from multiple local meteorological drivers

This study aims to establish the conditional independence structure between regional monthly rainfall and several local meteorological drivers (probable predictors) to develop a parsimonious prediction model in the framework of Bayesian Networks (BN). The chosen study area is the Gangetic West Bengal (GWB) region, India and the chosen meteorological drivers at the surface and different pressure levels are the 12 probable predictors, viz. air temperature, total precipitable water, relative humidity, geo-potential height, zonal wind, meridional wind, omega and soil moisture, each up to a lead time of 6 months, making a total of 72 inputs. Within the BN architecture, two heuristic algorithms namely Hill-Climb (HC) and Max-Min Hill Climb (MMHC) were used to establish that a large part (89% for HC and 97% for MMHC) of the input information was redundant, given the potential predictors that are selected via these algorithms. The potential predictors revealed via these algorithms include geo-potential height at a pressure of 850 mb, total precipitable water, and soil moisture, each at a lead time of 2 months. The selected predictors were further used to develop a prediction model that results in good agreement with the observations. Additionally, two widely used machine learning techniques namely, Artificial Neural Networks (ANN) and Support Vector Regression (SVR) were also used to develop standalone models wherein all the 72 probable predictors are used for prediction, and also in conjunction with BN (i.e., BN-ANN and BN-SVR), wherein only the drivers selected via BN are used as predictors. The results indicate that the parsimonious models perform better or equally well as the higher dimensional models, with refined Index of agreement values: MDBN = 0.824, MDBN-ANN = 0.835 and MDBN-SVR = 0.841, MDANN = 0.813, MDSVR = 0.823, and Normalized Mean Square Error values: NMSEBN = 0.242, NMSEBN-ANN = 0.226 and NMSEBN-SVR = 0.234, NMSEANN = 0.226, NMSESVR = 0.238 for validation period (averaged values across three folds). The model predictions are further used to categorize monthly rainfall into dry/ intermediate/ wet in terms of Standardized Precipitation Anomaly Index (SPAI). All the approaches show consistent performance, with the ‘dry’ cases being captured slightly better than ‘wet’ cases.

Regularized Within-Class Precision Matrix Based PLDA in Text-Dependent Speaker Verification

In the field of speaker verification, probabilistic linear discriminant analysis (PLDA) is the dominant method for back-end scoring. To estimate the PLDA model, the between-class covariance and within-class precision matrices must be estimated from samples. However, the empirical covariance/precision estimated from samples has estimation errors due to the limited number of samples available. In this paper, we propose a method to improve the conventional PLDA by estimating the PLDA model using the regularized within-class precision matrix. We use graphical least absolute shrinking and selection operator (GLASSO) for the regularization. The GLASSO regularization decreases the estimation errors in the empirical precision matrix by making the precision matrix sparse, which corresponds to the reflection of the conditional independence structure. The experimental results on text-dependent speaker verification reveal that the proposed method reduce the relative equal error rate by up to 23% compared with the conventional PLDA.

Within-person networks of clinical features of social anxiety disorder during cognitive and interpersonal therapy

Analysis of longitudinal within-person networks over the course of therapy allows an identification of possible targets of treatment. This study examined within-person networks of clinical features in social anxiety disorder (SAD) patients during cognitive (CT) and interpersonal (IPT) therapy. Patients (n = 80) were randomized to either CT or IPT in a 10 week residential program. They completed a measure of clinical features two times a week. The 60 (75 %) patients who had completed at least 18 measurements were included in the analyses. The multilevel vector autoregressive (mlVAR) model was used to analyze the data, producing a temporal, contemporaneous, and between-person network. In the temporal network and as expected, more homework compliance in a half-week predicted less social anxiety and less social avoidance the subsequent half-week. Also better social function predicted less social avoidance, whereas more social anxiety predicted more self-focus. Unexpectedly, less social avoidance predicted more self-focus and less self-focus predicted less social function and social joy. In the contemporaneous network, self-focus, anxiety and avoidance displayed a conditional independence structure. The estimated temporal network suggests that homework compliance and social function are promising targets of treatment.

Identification of potential causal variables for statistical downscaling models: effectiveness of graphical modeling approach

Selection of potential causal variables (PCVs) from a pool of many possibly associated variables is a critical issue since it can significantly affect the performance of any statistical downscaling model. Generally, the variable to be downscaled is associated with many other hydrologic and climatic (aka hydroclimatic) variables. Most of the existing approaches, such as correlation analysis (CA), partial correlation analysis (PaCA), and stepwise regression analysis (SRA), rely mostly on the mutual association for the selection of PCVs. However, none of these approaches investigate the detailed dependence structure that may be helpful in eliminating the unwanted information and efficiently selecting the PCVs for downscaling the target variable. In this study, the effectiveness of graphical modeling (GM) approach is explored for the selection of the PCVs as GM can effectively identify the detailed conditional independence structure among all the associated variables. For demonstration, downscaling of monthly precipitation is undertaken using the PCVs, identified by CA, PaCA, SRA, and the proposed GM approach. Two different downscaling models, namely statistical downscaling model (SDSM) and support vector regression (SVR)–based downscaling model, are utilized. The results show that the PCVs identified through the proposed GM approach provides consistent as well as robust performance, across different regions and seasons, due to its ability to capture the complete conditional indepedence structure among the variables. The downscaled monthly precipitation obtained using the proposed approach is better matching with the observed data in terms of the mean, variance as well as the probability distribution. Overall, this study recommends the GM approach for the identification of the PCVs for the downscaling models.

Bayesian Structure Learning in Multilayered Genomic Networks

Integrative network modeling of data arising from multiple genomic platforms provides insight into the holistic picture of the interactive system, as well as the flow of information across many disease domains including cancer. The basic data structure consists of a sequence of hierarchically ordered datasets for each individual subject, which facilitates integration of diverse inputs, such as genomic, transcriptomic, and proteomic data. A primary analytical task in such contexts is to model the layered architecture of networks where the vertices can be naturally partitioned into ordered layers, dictated by multiple platforms, and exhibit both undirected and directed relationships. We propose a multilayered Gaussian graphical model (mlGGM) to investigate conditional independence structures in such multilevel genomic networks in human cancers. We implement a Bayesian node-wise selection (BANS) approach based on variable selection techniques that coherently accounts for the multiple types of dependencies in mlGGM; this flexible strategy exploits edge-specific prior knowledge and selects sparse and interpretable models. Through simulated data generated under various scenarios, we demonstrate that BANS outperforms other existing multivariate regression-based methodologies. Our integrative genomic network analysis for key signaling pathways across multiple cancer types highlights commonalities and differences of p53 integrative networks and epigenetic effects of BRCA2 on p53 and its interaction with T68 phosphorylated CHK2, that may have translational utilities of finding biomarkers and therapeutic targets. Supplementary materials for this article, including a standardized description of the materials available for reproducing the work, are available as an online supplement.

Modules or Mean-Fields?

The segregation of neural processing into distinct streams has been interpreted by some as evidence in favour of a modular view of brain function. This implies a set of specialised ‘modules’, each of which performs a specific kind of computation in isolation of other brain systems, before sharing the result of this operation with other modules. In light of a modern understanding of stochastic non-equilibrium systems, like the brain, a simpler and more parsimonious explanation presents itself. Formulating the evolution of a non-equilibrium steady state system in terms of its density dynamics reveals that such systems appear on average to perform a gradient ascent on their steady state density. If this steady state implies a sufficiently sparse conditional independency structure, this endorses a mean-field dynamical formulation. This decomposes the density over all states in a system into the product of marginal probabilities for those states. This factorisation lends the system a modular appearance, in the sense that we can interpret the dynamics of each factor independently. However, the argument here is that it is factorisation, as opposed to modularisation, that gives rise to the functional anatomy of the brain or, indeed, any sentient system. In the following, we briefly overview mean-field theory and its applications to stochastic dynamical systems. We then unpack the consequences of this factorisation through simple numerical simulations and highlight the implications for neuronal message passing and the computational architecture of sentience.

Conditionally structured variational Gaussian approximation with importance weights

We develop flexible methods of deriving variational inference for models with complex latent variable structure. By splitting the variables in these models into “global” parameters and “local” latent variables, we define a class of variational approximations that exploit this partitioning and go beyond Gaussian variational approximation. This approximation is motivated by the fact that in many hierarchical models, there are global variance parameters which determine the scale of local latent variables in their posterior conditional on the global parameters. We also consider parsimonious parametrizations by using conditional independence structure and improved estimation of the log marginal likelihood and variational density using importance weights. These methods are shown to improve significantly on Gaussian variational approximation methods for a similar computational cost. Application of the methodology is illustrated using generalized linear mixed models and state space models.

Coherent combination of probabilistic outputs for group decision making: an algebraic approach

Current decision support systems address domains that are heterogeneous in nature and becoming progressively larger. Such systems often require the input of expert judgement about a variety of different fields and an intensive computational power to produce the scores necessary to rank the available policies. Recently, integrating decision support systems have been introduced to enable a formal Bayesian multi-agent decision analysis to be distributed and consequently efficient. In such systems, where different panels of experts independently oversee disjoint but correlated vectors of variables, each expert group needs to deliver only certain summaries of the variables under their jurisdiction, derived from a conditional independence structure common to all panels, to properly derive an overall score for the available policies. Here we present an algebraic approach that makes this methodology feasible for a wide range of modelling contexts and that enables us to identify the summaries needed for such a combination of judgements. We are also able to demonstrate that coherence, in a sense we formalize here, is still guaranteed when panels only share a partial specification of their model with other panel members. We illustrate this algebraic approach by applying it to a specific class of Bayesian networks and demonstrate how we can use it to derive closed form formulae for the computations of the joint moments of variables that determine the score of different policies.

Metropolized Knockoff Sampling

Model-X knockoffs is a wrapper that transforms essentially any feature importance measure into a variable selection algorithm, which discovers true effects while rigorously controlling the expected fraction of false positives. A frequently discussed challenge to apply this method is to construct knockoff variables, which are synthetic variables obeying a crucial exchangeability property with the explanatory variables under study. This article introduces techniques for knockoff generation in great generality: we provide a sequential characterization of all possible knockoff distributions, which leads to a Metropolis–Hastings formulation of an exact knockoff sampler. We further show how to use conditional independence structure to speed up computations. Combining these two threads, we introduce an explicit set of sequential algorithms and empirically demonstrate their effectiveness. Our theoretical analysis proves that our algorithms achieve near-optimal computational complexity in certain cases. The techniques we develop are sufficiently rich to enable knockoff sampling in challenging models including cases where the covariates are continuous and heavy-tailed, and follow a graphical model such as the Ising model. Supplementary materials for this article are available online.

A Proximal Point Dual Newton Algorithm for Solving Group Graphical Lasso Problems

Undirected graphical models have been especially popular for learning the conditional independence structure among a large number of variables where the observations are drawn independently and identically from the same distribution. However, many modern statistical problems would involve categorical data or time-varying data, which might follow different but related underlying distributions. In order to learn a collection of related graphical models simultaneously, various joint graphical models inducing sparsity in graphs and similarity across graphs have been proposed. In this paper, we aim to propose an implementable proximal point dual Newton algorithm (PPDNA) for solving the group graphical Lasso model, which encourages a shared pattern of sparsity across graphs. Though the group graphical Lasso regularizer is non-polyhedral, the asymptotic superlinear convergence of our proposed method PPDNA can be obtained by leveraging on the local Lipschitz continuity of the Karush-Kuhn-Tucker solution mapping associated with the group graphical Lasso model. A variety of numerical experiments on real data sets illustrates that the PPDNA for solving the group graphical Lasso model can be highly efficient and robust.

Compatible priors for model selection of high-dimensional Gaussian DAGs

Graphical models represent a powerful framework to incorporate conditional independence structure for the statistical analysis of high-dimensional data. In this paper we focus on Directed Acyclic Graphs (DAGs). In the Gaussian setting, a prior recently introduced for the parameters associated to the (modified) Cholesky decomposition of the precision matrix is the DAG-Wishart. The flexibility introduced through a rich choice of shape hyperparameters coupled with conjugacy are two desirable assets of this prior which are especially welcome for estimation and prediction. In this paper we look at the DAG-Wishart prior from the perspective of model selection, with special reference to its consistency properties in high dimensional settings. We show that Bayes factor consistency only holds when comparing two DAGs which do not belong to the same Markov equivalence class, equivalently they encode distinct conditional independencies; a similar result holds for posterior ratio consistency. We also prove that DAG-Wishart distributions with arbitrarily chosen hyperparameters will lead to incompatible priors for model selection, because they assign different marginal likelihoods to Markov equivalent graphs. To overcome this difficulty, we propose a constructive method to specify DAG-Wishart priors whose suitably constrained shape hyperparameters ensure compatibility for DAG model selection.

Combining Opinions for Use in Bayesian Networks: A Measurement Error Approach

SummaryBayesian networks (BNs) are graphical probabilistic models used for reasoning under uncertainty. These models are becoming increasingly popular in a range of fields including engineering, ecology, computational biology, medical diagnosis and forensics. In most of these cases, the BNs are quantified using information from experts or from users' opinions. While this quantification is straightforward for one expert, there is still debate about how to represent opinions from multiple experts in a BN. This paper proposes the use of a measurement error model to achieve this. The proposed model addresses the issues associated with current methods of combining opinions such as the absence of a coherent probability model, the loss of the conditional independence structure of the BN and the provision of only a point estimate for the consensus. The proposed model is applied to a subnetwork (the three final nodes) of a larger BN about wayfinding in airports. It is shown that the approach performs well than do existing methods of combining opinions.

Post-Processing Posteriors Over Precision Matrices to Produce Sparse Graph Estimates

A variety of computationally efficient Bayesian models for the covariance matrix of a multivariate Gaussian distribution are available. However, all produce a relatively dense estimate of the precision matrix, and are therefore unsatisfactory when one wishes to use the precision matrix to consider the conditional independence structure of the data. This paper considers the posterior predictive distribution of model fit for these covariance models. We then undertake post-processing of the Bayes point estimate for the precision matrix to produce a sparse model whose expected fit lies within the upper 95% of the posterior predictive distribution of fit. The impact of the method for selecting the zero elements of the precision matrix is evaluated. Good results were obtained using models that encouraged a sparse posterior (G-Wishart, Bayesian adaptive graphical lasso) and selection using credible intervals. We also find that this approach is easily extended to the problem of finding a sparse set of elements that differ across a set of precision matrices, a natural summary when a common set of variables is observed under multiple conditions. We illustrate our findings with moderate dimensional data examples from finance and metabolomics.

Motor unit number estimation via sequential Monte Carlo

A change in the number of motor units that operate a particular muscle is an important indicator for the progress of a neuromuscular disease and the efficacy of a therapy. Inference for realistic statistical models of the typical data produced when testing muscle function is difficult, and estimating the number of motor units is an ongoing statistical challenge. We consider a set of models for the data, each with a different number of working motor units, and present a novel method for Bayesian inference based on sequential Monte Carlo. This provides estimates of the marginal likelihood and, hence, a posterior probability for each model. Implementing this approach in practice requires a sequential Monte Carlo method that has excellent computational and Monte Carlo properties. We achieve this by benefiting from the model’s conditional independence structure, where, given knowledge of which motor units fired as a result of a particular stimulus, parameters that specify the size of each unit’s response are independent of the parameters defining the probability that a unit will respond at all. The scalability of our methodology relies on the natural conjugacy structure that we create for the former and an enforced, approximate, conjugate structure for the latter. A simulation study demonstrates the accuracy of our method, and inferences are consistent across two different datasets arising from the same rat tibial muscle.

Minimal Characterization of Shannon-Type Inequalities Under Functional Dependence and Full Conditional Independence Structures

The minimal set of Shannon-type inequalities (or elemental inequalities) plays a central role in efficiently determining whether a given inequality is in fact Shannon-type or not and in computing the linear programming bound for network coding capacity. In many cases, random variables under consideration are subject to additional constraints, such as functional dependence and conditional independence constraints. For example, functional dependence constraints are common in many communication problems due to deterministic encoding and decoding constraints. In other situations, the variables involved may form a Markov chain or in general a Markov random field, leading to conditional independence constraint. Subject to additional constraints, the challenge is how to identify the non-redundant inequalities. While one can always numerically determine the non-redundant inequalities (subject to additional linear equality constraints), it will be instrumental and also important if the non-redundant inequalities can be listed explicitly. In this paper, we show that this is achievable under the functional dependence and full conditional independence constraints.

Fast Bayesian inference in large Gaussian graphical models.

Despite major methodological developments, Bayesian inference in Gaussian graphical models remains challenging in high dimension due to the tremendous size of the model space. This article proposes a method to infer the marginal and conditional independence structures between variables by multiple testing, which bypasses the exploration of the model space. Specifically, we introduce closed‐form Bayes factors under the Gaussian conjugate model to evaluate the null hypotheses of marginal and conditional independence between variables. Their computation for all pairs of variables is shown to be extremely efficient, thereby allowing us to address large problems with thousands of nodes as required by modern applications. Moreover, we derive exact tail probabilities from the null distributions of the Bayes factors. These allow the use of any multiplicity correction procedure to control error rates for incorrect edge inclusion. We demonstrate the proposed approach on various simulated examples as well as on a large gene expression data set from The Cancer Genome Atlas.