Conditional Independence Constraints Research Articles

Scores for Learning Discrete Causal Graphs with Unobserved Confounders

Structural learning is arguably one of the most challenging and pervasive tasks found throughout the data sciences. There exists a growing literature that studies structural learning in non-parametric settings where conditional independence constraints are taken to define the equivalence class. In the presence of unobserved confounders, it is understood that non-conditional independence constraints are imposed over the observational distribution, including certain equalities and inequalities between functionals of the joint distribution. In this paper, we develop structural learning methods that leverage additional constraints beyond conditional independences. Specifically, we first introduce a score for arbitrary graphs combining Watanabe's asymptotic expansion of the marginal likelihood and new bounds over the cardinality of the exogenous variables. Second, we show that the new score has desirable properties in terms of expressiveness and computability. In terms of expressiveness, we prove that the score captures distinct constraints imprinted in the data, including Verma's and inequalities'. In terms of computability, we show properties of score equivalence and decomposability, which allows, in principle, to break the problem of structural learning in smaller and more manageable pieces. Third, we implement this score using an MCMC sampling algorithm and test its properties in several simulation scenarios.

Undirected Structural Markov Property for Bayesian Model Determination

This paper generalizes the structural Markov properties for undirected decomposable graphs to arbitrary ones. This helps us to exploit the conditional independence properties of joint prior laws to analyze and compare multiple graphical structures, while being able to take advantage of the common conditional independence constraints. This work provides a theoretical support for full Bayesian posterior updating about the structure of a graph using data from a certain distribution. We further investigate the ratio of graph law so as to simplify the acceptance probability of the Metropolis–Hastings sampling algorithms.

Properties of unique information

We study the measure of unique information $UI(T:X\setminus Y)$ defined by Bertschinger et al. (2014) within the framework of information decompositions. We study uniqueness and support of the solutions to the optimization problem underlying the definition of $UI$. We identify sufficient conditions for non-uniqueness of solutions with full support in terms of conditional independence constraints and in terms of the cardinalities of $T$, $X$ and $Y$. Our results are based on a reformulation of the first order conditions on the objective function as rank constraints on a matrix of conditional probabilities. These results help to speed up the computation of $UI(T:X\setminus Y)$, most notably when $T$ is binary. In the case that all variables are binary, we obtain a complete picture of where the optimizing probability distributions lie.

Conditional probabilities via line arrangements and point configurations

We study the connection between probability distributions satisfying certain conditional independence (CI) constraints, and point and line arrangements in incidence geometry. To a family of CI statements, we associate a polynomial ideal whose algebraic invariants are encoded in a hypergraph. The primary decompositions of these ideals give a characterization of the distributions satisfying the original CI statements. Classically, these ideals are generated by 2-minors of a matrix of variables; however, in the presence of hidden variables, they contain higher degree minors. This leads to the study of the structure of determinantal hypergraph ideals whose decompositions can be understood in terms of point and line configurations in the projective space.

Bayesian hypothesis testing for Gaussian graphical models: Conditional independence and order constraints

Gaussian graphical models (GGM; partial correlation networks) have become increasingly popular in the social and behavioral sciences for studying conditional (in)dependencies between variables. In this work, we introduce exploratory and confirmatory Bayesian tests for partial correlations. For the former, we first extend the customary GGM formulation that focuses on conditional dependence to also consider the null hypothesis of conditional independence for each partial correlation. Here a novel testing strategy is introduced that can provide evidence for a null, negative, or positive effect. We then introduce a test for hypotheses with order constraints on partial correlations. This allows for testing theoretical and clinical expectations in GGMs. The novel matrix-F prior distribution is described that provides increased flexibility in specification compared to the Wishart prior. The methods are applied to PTSD symptoms. In several applications, we demonstrate how the exploratory and confirmatory approaches can work in tandem: hypotheses are formulated from an initial analysis and then tested in an independent dataset. The methodology is implemented in the R package BGGM.

Mutual conditional independence and its applications to model selection in Markov networks

The fundamental concepts underlying Markov networks are the conditional independence and the set of rules called Markov properties that translate conditional independence constraints into graphs. We introduce the concept of mutual conditional independence in an independent set of a Markov network, and we prove its equivalence to the Markov properties under certain regularity conditions. This extends the notion of similarity between separation in graph and conditional independence in probability to similarity between the mutual separation in graph and the mutual conditional independence in probability. Model selection in graphical models remains a challenging task due to the large search space. We show that mutual conditional independence property can be exploited to reduce the search space. We present a new forward model selection algorithm for graphical log-linear models using mutual conditional independence. We illustrate our algorithm with a real data set example. We show that for sparse models the size of the search space can be reduced from mathcal {O} (n^{3}) to mathcal {O}(n^{2}) using our proposed forward selection method rather than the classical forward selection method. We also envision that this property can be leveraged for model selection and inference in different types of graphical models.

Structured Bayesian Networks: From Inference to Learning with Routes

Structured Bayesian networks (SBNs) are a recently proposed class of probabilistic graphical models which integrate background knowledge in two forms: conditional independence constraints and Boolean domain constraints. In this paper, we propose the first exact inference algorithm for SBNs, based on compiling a given SBN to a Probabilistic Sentential Decision Diagram (PSDD). We further identify a tractable subclass of SBNs, which have PSDDs of polynomial size. These SBNs yield a tractable model of route distributions, whose structure can be learned from GPS data, using a simple algorithm that we propose. Empirically, we demonstrate the utility of our inference algorithm, showing that it can be an order-ofmagnitude more efficient than more traditional approaches to exact inference. We demonstrate the utility of our learning algorithm, showing that it can learn more accurate models and classifiers from GPS data.

Minimal Characterization of Shannon-Type Inequalities Under Functional Dependence and Full Conditional Independence Structures

The minimal set of Shannon-type inequalities (or elemental inequalities) plays a central role in efficiently determining whether a given inequality is in fact Shannon-type or not and in computing the linear programming bound for network coding capacity. In many cases, random variables under consideration are subject to additional constraints, such as functional dependence and conditional independence constraints. For example, functional dependence constraints are common in many communication problems due to deterministic encoding and decoding constraints. In other situations, the variables involved may form a Markov chain or in general a Markov random field, leading to conditional independence constraint. Subject to additional constraints, the challenge is how to identify the non-redundant inequalities. While one can always numerically determine the non-redundant inequalities (subject to additional linear equality constraints), it will be instrumental and also important if the non-redundant inequalities can be listed explicitly. In this paper, we show that this is achievable under the functional dependence and full conditional independence constraints.

A Directional-Linear Bayesian Network and Its Application for Clustering and Simulation of Neural Somas

Neural somas perform most of the metabolic activities in the neuron and support the chemical process that generates the basic elements of the synapses, and consequently the brain activity. The morphology of the somas is one of the fundamental features for classifying neurons and their functionality. In this paper, we characterize the morphology of the 39 three-dimensional reconstructed human pyramidal somas in terms of their multiresolutional Reeb graph representation, from which we extract a set of directional and linear variables to perform model-based clustering. To deal with this dataset, we introduce the novel Extended Mardia-Sutton mixture model whose mixture components are distributed according to a newly proposed multivariate probability density function that is able to capture the directional-linear correlations. We exploit the capabilities of Bayesian networks in combination with the Structural Expectation-Maximization algorithm to learn the finite mixture model that clusters the neural somas by their morphology and the conditional independence constraints between variables. We also derive the Kullback-Leibler divergence of the Extended Mardia-Sutton distribution to be used as a measure of similarity between soma clusters. The proposed finite mixture model discovered three subtypes of human pyramidal somas. We performed Weltch t-tests and Watson-Williams tests, as well as rule-based identification of clusters to characterize each group by its most prominent features. Furthermore, the resulting model allows us to simulate the 3D virtual representations of somas from each cluster, which can be a useful tool for neuroscientists to reason and suggest new hypotheses.

Estimating bounds on causal effects in high-dimensional and possibly confounded systems

We present an algorithm for estimating bounds on causal effects from observational data which combines graphical model search with simple linear regression. We assume that the underlying system can be represented by a linear structural equation model with no feedback, and we allow for the possibility of latent confounders. Under assumptions standard in the causal search literature, we use conditional independence constraints to search for an equivalence class of ancestral graphs. Then, for each model in the equivalence class, we perform the appropriate regression (using causal structure information to determine which covariates to adjust for) to estimate a set of possible causal effects. Our approach is based on the IDA procedure of Maathuis et al. [17], which assumes that all relevant variables have been measured (i.e., no latent confounders). We generalize their work by relaxing this assumption, which is often violated in applied contexts. We validate the performance of our algorithm in simulation experiments.

The $K$ -User Vector Gaussian Multiple-Access Channel With General Messages Sets: Capacity, Polymatroidal Structure, and Efficient Computation

The capacity region of the $K$ -user vector Gaussian multiple-access channel with general message sets is established. Furthermore, due to the presence of convexity, both in the discrete and continuous senses, it is shown that this capacity region is efficiently computable. The capacity result is obtained by specializing recent results by the authors for the discrete memoryless multiple-access channel and demonstrating that it suffices to only consider jointly Gaussian input and auxiliary random variables. For this second conclusion, it is shown that jointly Gaussian random variables maximize entropy subject to lattice conditional independence and covariance constraints, a result that is of interest in its own right. Discrete convexity arises since the capacity region is a union of polymatroids . Over each polymatroid, computing the maximal weighted sum rates is simple due to submodularity—a discrete analog of concavity—of the set function associated with the linear inequalities that define the polymatroid. Continuous convexity arises as the set of admissible covariance matrices is convex and the polymatroidal bounds are concave in these covariances.

Algebraic representations of Gaussian Markov combinations

Markov combinations for structural meta-analysis problems provide a way of constructing a statistical model that takes into account two or more marginal distributions by imposing conditional independence constraints between the variables that are not jointly observed. This paper considers Gaussian distributions and discusses how the covariance and concentration matrices of the different combinations can be found via matrix operations. In essence, all these Markov combinations correspond to finding a positive definite completion of the covariance matrix over the set of random variables of interest and respecting the constraints imposed by each Markov combination. The paper further shows the potential of investigating the properties of the combinations via algebraic statistics tools. An illustrative application will motivate the importance of solving problems of this type.

Structural Markov graph laws for Bayesian model uncertainty

This paper considers the problem of defining distributions over graphical structures. We propose an extension of the hyper Markov properties of Dawid and Lauritzen [Ann. Statist. 21 (1993) 1272-1317], which we term structural Markov properties, for both undirected decomposable and directed acyclic graphs, which requires that the structure of distinct components of the graph be conditionally independent given the existence of a separating component. This allows the analysis and comparison of multiple graphical structures, while being able to take advantage of the common conditional independence constraints. Moreover, we show that these properties characterise exponential families, which form conjugate priors under sampling from compatible Markov distributions.

Introduction to Nested Markov Models

Graphical models provide a principled way to take advantage of independence constraints for probabilistic and causal modeling, while giving an intuitive graphical description of “qualitative features” useful for these tasks. A popular graphical model, known as a Bayesian network, represents joint distributions by means of a directed acyclic graph (DAG). DAGs provide a natural representation of conditional independence constraints, and also have a simple causal interpretation. When all variables are observed, the associated statistical models have many attractive properties. However, in many practical data analyses unobserved variables may be present. In general, the set of marginal distributions obtained from a DAG model with hidden variables is a much more complicated statistical model: the likelihood of the marginal is often intractable; the model may contain singularities. There are also an infinite number of such models to consider. It is possible to avoid these difficulties by modeling the observed marginal directly. One strategy is to define a model by means of conditional independence constraints induced on the observed marginal by the hidden variable DAG; we call this the ordinary Markov model. This model will be a supermodel that contains the set of marginal distributions obtained from the original DAG. Richardson and Spirtes (2002) and Evans and Richardson (2013a) gave parametrizations of this model in the Gaussian and discrete case, respectively. However, it has long been known that hidden variable DAG models also imply nonparametric constraints which generalize conditional independences; these are sometimes called “Verma Constraints”. In this paper we describe a natural extension of the ordinary Markov approach, whereby both conditional independences and these generalized constraints are used to define a nested Markov model. The binary nested Markov model may be parametrized via a simple extension of the binary parametrization of the ordinary Markov model of Evans and Richardson (2013a). We also give evidence for a characterization of nested Markov equivalence for models with four observed variables. A consequence of this characterization is that, in some instances, most structural features of hidden variable DAGs can be recovered exactly when a single generalized independence constraint holds under the distribution of the observed variables.

Robustness, canalyzing functions and systems design

We study a notion of knockout robustness of a stochastic map (Markov kernel) that describes a system of several input random variables and one output random variable. Robustness requires that the behaviour of the system does not change if one or several of the input variables are knocked out. Gibbs potentials are used to give a mechanistic description of the behaviour of the system after knockouts. Robustness imposes structural constraints on these potentials. We show that robust systems can be described in terms of suitable interaction families of Gibbs potentials, which allows us to address the problem of systems design. Robustness is also characterized by conditional independence constraints on the joint distribution of input and output. The set of all probability distributions corresponding to robust systems can be decomposed into a finite union of components, and we find parametrizations of the components.

An efficient algorithm for maximum entropy extension of block-circulant covariance matrices

This paper deals with maximum entropy completion of partially specified block-circulant matrices. Since positive definite symmetric circulants happen to be covariance matrices of stationary periodic processes, in particular of stationary reciprocal processes, this problem has applications in signal processing, in particular to image modeling. In fact it is strictly related to maximum likelihood estimation of bilateral AR-type representations of acausal signals subject to certain conditional independence constraints. The maximum entropy completion problem for block-circulant matrices has recently been solved by the authors, although leaving open the problem of an efficient computation of the solution. In this paper, we provide an efficient algorithm for computing its solution which compares very favorably with existing algorithms designed for positive definite matrix extension problems. The proposed algorithm benefits from the analysis of the relationship between our problem and the band-extension problem for block-Toeplitz matrices also developed in this paper.

On conditional independence and log-convexity

Si des contraintes d’independance conditionnelle definissent une famille de distributions positives qui est log-convexe, alors cette famille doit etre un modele de Markov sur un graphe non-dirige. Ceci est demontre pour les distributions sur le produits d’ensembles finis et pour les distributions gaussiennes regulieres. Par consequent, l’assertion connue comme le theoreme de factorisation de Brook, le theoreme de Hammersley–Clifford ou l’equivalence de Gibbs–Markov est obtenue.

Trek separation for Gaussian graphical models

Gaussian graphical models are semi-algebraic subsets of the cone of positive definite covariance matrices. Submatrices with low rank correspond to generalizations of conditional independence constraints on collections of random variables. We give a precise graph-theoretic characterization of when submatrices of the covariance matrix have small rank for a general class of mixed graphs that includes directed acyclic and undirected graphs as special cases. Our new trek separation criterion generalizes the familiar d-separation criterion. Proofs are based on the trek rule, the resulting matrix factorizations and classical theorems of algebraic combinatorics on the expansions of determinants of path polynomials.

Covariance selection for nonchordal graphs via chordal embedding

We describe algorithms for maximum likelihood estimation of Gaussian graphical models with conditional independence constraints. This problem is also known as covariance selection, and it can be expressed as an unconstrained convex optimization problem with a closed-form solution if the underlying graph is chordal. The focus of the paper is on iterative algorithms for covariance selection with nonchordal graphs. We first derive efficient methods for evaluating the gradient and Hessian of the log-likelihood function when the underlying graph is chordal. The algorithms are formulated as simple recursions on a clique tree associated with the graph. We also show that the gradient and Hessian mappings are easily inverted when the underlying graph is chordal. We then exploit these results to obtain efficient implementations of Newton's method and the conjugate gradient method for large nonchordal graphs, by embedding the graph in a chordal graph.

Piecewise linear conditional information inequality

A new information inequality of non-Shannon type is proved for three discrete random variables under conditional independence constraints, using the framework of entropy functions and polymatroids. Tightness of the inequality is described via quasi-groups