Conditional Independence Structure Research Articles

Structured Variational Approximations with Skew Normal Decomposable Graphical Models and Implicit Copulas

Although there is much recent work developing flexible variational methods for Bayesian computation, Gaussian approximations with structured covariance matrices are often preferred computationally in high-dimensional settings. This article considers approximate inference methods for complex latent variable models where the posterior is close to Gaussian, but with some skewness in the posterior marginals. We consider skew decomposable graphical models (SDGMs), which are based on the closed skew normal family of distributions, as variational approximations. These approximations can reflect the true posterior conditional independence structure and capture posterior skewness. To increase flexibility, implicit copula SDGM approximations are also developed, where elementwise transformations of an approximately standardized SDGM random vector are considered. This implicit copula extension is an important contribution of our work, and improves the accuracy of SDGM approximations for only a modest increase in computational cost. Our parameterization of the copula approximation is novel, even in the Gaussian case. Performance of the methods is examined in a number of real examples involving generalized linear mixed models and state space models. Supplemental materials including code and appendix are available online.

Vine copula structure representations using graphs and matrices

A widespread methodology for modeling modern day information, which consists of high-dimensional digital measurements, is to use vine copulas; they can flexibly encode the underlying dependence structure of the data. Here we introduce a new algorithm to encode complete and truncated vines in a matrix, and as such, storing the information content of vines in a virtual environment. The conditional independence structure encoded by a vine can be represented in graph terms. We summarize these representations, and show equivalence between them. We show a new result, namely that when a perfect elimination ordering of a vine structure is given, then it can be uniquely represented with a matrix. Nápoles has shown a way to represent vines in a matrix, and we algorithmify this previous approach, while also showing a new method for constructing such a matrix, through cherry tree sequences, which can also store truncated vines. Moreover, this new algorithm can directly build truncated vines by storing each level separately - without building up the entire vine, which would be necessary in Nápoles' algorithm. We calculate the runtime of these algorithms. Lastly, we prove that these two matrix-building algorithms are equivalent if the same perfect elimination ordering is being used.

Bayesian Learning of Graph Substructures

Graphical models provide a powerful methodology for learning the conditional independence structure in multivariate data. Inference is often focused on estimating individual edges in the latent graph. Nonetheless, there is increasing interest in inferring more complex structures, such as communities, for multiple reasons, including more effective information retrieval and better interpretability. Stochastic blockmodels offer a powerful tool to detect such structure in a network. We thus propose to exploit advances in random graph theory and embed them within the graphical models framework. A consequence of this approach is the propagation of the uncertainty in graph estimation to large-scale structure learning. We consider Bayesian nonparametric stochastic blockmodels as priors on the graph. We extend such models to consider clique-based blocks and to multiple graph settings introducing a novel prior process based on a Dependent Dirichlet process. Moreover, we devise a tailored computation strategy of Bayes factors for block structure based on the Savage-Dickey ratio to test for presence of larger structure in a graph. We demonstrate our approach in simulations as well as on real data applications in finance and transcriptomics.

Selfadhesivity in Gaussian conditional independence structures

Selfadhesivity is a property of entropic polymatroids which guarantees that the polymatroid can be glued to an identical copy of itself along arbitrary restrictions such that the two pieces are independent given the common restriction. We show that positive definite matrices satisfy this condition as well and examine consequences for Gaussian conditional independence structures. New axioms of Gaussian CI are obtained by applying selfadhesivity to the previously known axioms of structural semigraphoids and orientable gaussoids.

Gaussian Variational Approximations for High-dimensional State Space Models

We consider a Gaussian variational approximation of the posterior density in high-dimensional state space models. The number of parameters in the covariance matrix of the variational approximation grows as the square of the number of model parameters, so it is necessary to find simple yet effective parametrisations of the covariance structure when the number of model parameters is large. We approximate the joint posterior density of the state vectors by a dynamic factor model, having Markovian time dependence and a factor covariance structure for the states. This gives a reduced description of the dependence structure for the states, as well as a temporal conditional independence structure similar to that in the true posterior. We illustrate the methodology on two examples. The first is a spatio-temporal model for the spread of the Eurasian collared-dove across North America. Our approach compares favorably to a recently proposed ensemble Kalman filter method for approximate inference in high-dimensional hierarchical spatio-temporal models. Our second example is a Wishart-based multivariate stochastic volatility model for financial returns, which is outside the class of models the ensemble Kalman filter method can handle.

On a wider class of prior distributions for graphical models

AbstractGaussian graphical models are useful tools for conditional independence structure inference of multivariate random variables. Unfortunately, Bayesian inference of latent graph structures is challenging due to exponential growth of $\mathcal{G}_n$ , the set of all graphs in n vertices. One approach that has been proposed to tackle this problem is to limit search to subsets of $\mathcal{G}_n$ . In this paper we study subsets that are vector subspaces with the cycle space $\mathcal{C}_n$ as the main example. We propose a novel prior on $\mathcal{C}_n$ based on linear combinations of cycle basis elements and present its theoretical properties. Using this prior, we implement a Markov chain Monte Carlo algorithm, and show that (i) posterior edge inclusion estimates computed with our technique are comparable to estimates from the standard technique despite searching a smaller graph space, and (ii) the vector space perspective enables straightforward implementation of MCMC algorithms.

Lilac: A Modal Separation Logic for Conditional Probability

We present Lilac, a separation logic for reasoning about probabilistic programs where separating conjunction captures probabilistic independence. Inspired by an analogy with mutable state where sampling corresponds to dynamic allocation, we show how probability spaces over a fixed, ambient sample space appear to be the natural analogue of heap fragments, and present a new combining operation on them such that probability spaces behave like heaps and measurability of random variables behaves like ownership. This combining operation forms the basis for our model of separation, and produces a logic with many pleasant properties. In particular, Lilac has a frame rule identical to the ordinary one, and naturally accommodates advanced features like continuous random variables and reasoning about quantitative properties of programs. Then we propose a new modality based on disintegration theory for reasoning about conditional probability. We show how the resulting modal logic validates examples from prior work, and give a formal verification of an intricate weighted sampling algorithm whose correctness depends crucially on conditional independence structure.

Modeling Massive Highly Multivariate Nonstationary Spatial Data with the Basis Graphical Lasso

We propose a new modeling framework for highly multivariate spatial processes that synthesizes ideas from recent multiscale and spectral approaches with graphical models. The basis graphical lasso writes a univariate Gaussian process as a linear combination of basis functions weighted with entries of a Gaussian graphical vector whose graph is estimated from optimizing an l 1 penalized likelihood. This article extends the setting to a multivariate Gaussian process where the basis functions are weighted with Gaussian graphical vectors. We motivate a model where the basis functions represent different levels of resolution and the graphical vectors for each level are assumed to be independent. Using an orthogonal basis grants linear complexity and memory usage in the number of spatial locations, the number of basis functions, and the number of realizations. An additional fusion penalty encourages a parsimonious conditional independence structure in the multilevel graphical model. We illustrate our method on a large climate ensemble from the National Center for Atmospheric Research’s Community Atmosphere Model that involves 40 spatial processes. Supplementary materials for this article are available online.

Exact and Approximate Heterogeneous Bayesian Decentralized Data Fusion

In Bayesian peer-to-peer decentralized data fusion, the underlying distributions held locally by autonomous agents are frequently assumed to be over the same set of variables (homogeneous). This requires each agent to process and communicate the full global joint distribution, and thus leads to high computation and communication costs irrespective of relevancy to specific local objectives. This work formulates and studies heterogeneous decentralized fusion problems, defined as the set of problems in which either the communicated or the processed distributions describe different, but overlapping, random states of interest that are subsets of a larger full global joint state. We exploit the conditional independence structure of such problems and provide a rigorous derivation of novel exact and approximate conditionally factorized heterogeneous fusion rules. We further develop a new version of the homogeneous Channel Filter algorithm to enable conservative heterogeneous fusion for smoothing and filtering scenarios in dynamic problems. Numerical examples show more than $99.5\%$ potential communication reduction for heterogeneous channel filter fusion, and a multi-target tracking simulation shows that these methods provide consistent estimates while remaining computationally scalable.

CubeSat attitude determination with decomposed Kalman filters

CubeSats are the cost-effective entry to space research and applications. As mission requirements increase to carry out more complex tasks, the constraints on the satellite challenge how attitude estimation and control systems are designed. Limited energy, sensors, and computational capacity require compromises. In this paper, we propose a Kalman filter architecture to reduce the computational cost of attitude estimation, leveraging the conditional independence structure of its physical model. Our method decomposes attitude dynamics and kinematics, leading to a linear attitude quaternion and a nonlinear angular velocity filter. As accommodating all vector measurements would require a nonlinear filter, we propose the virtual sensor paradigm that transforms the nonlinear observation model into a linear one, without relying on approximations. Our numerical experiments showcase superior error dynamics and robustness to epistemic uncertainty compared to a nonlinear quaternionic filter, and we also investigate performance against star tracker measurement frequency and sensitivity to the angle between Sun and Earth magnetic field measurements.

Directed Gaussian graphical models with toric vanishing ideals

Directed Gaussian graphical models are statistical models that use a directed acyclic graph (DAG) to represent the conditional independence structures between a set of jointly normal random variables. The DAG specifies the model through recursive factorization of the parametrization, via restricted conditional distributions. In this paper, we make an attempt to characterize the DAGs whose vanishing ideals are toric ideals. In particular, we give some combinatorial criteria to construct such DAGs from smaller DAGs which have toric vanishing ideals. An associated monomial map called the shortest trek map plays an important role in our description of toric Gaussian DAG models. For DAGs whose vanishing ideal is toric, we prove results about the generating sets of those toric ideals.

Network Structure Learning Under Uncertain Interventions

Gaussian Directed Acyclic Graphs (DAGs) represent a powerful tool for learning the network of dependencies among variables, a task which is of primary interest in many fields and specifically in biology. Different DAGs may encode equivalent conditional independence structures, implying limited ability, with observational data, to identify causal relations. In many contexts however, measurements are collected under heterogeneous settings where variables are subject to exogenous interventions. Interventional data can improve the structure learning process whenever the targets of an intervention are known. However, these are often uncertain or completely unknown, as in the context of drug target discovery. We propose a Bayesian method for learning dependence structures and intervention targets from data subject to interventions on unknown variables of the system. Selected features of our approach include a DAG-Wishart prior on the DAG parameters, and the use of variable selection priors to express uncertainty on the targets. We provide theoretical results on the correct asymptotic identification of intervention targets and derive sufficient conditions for Bayes factor and posterior ratio consistency of the graph structure. Our method is applied in simulations and real-data world settings, to analyze perturbed protein data and assess antiepileptic drug therapies. Details of the MCMC algorithm and proofs of propositions are provided in the supplementary materials, together with more extensive results on simulations and applied studies. Supplementary materials for this article are available online.

Gaussian graphical modeling for spectrometric data analysis

Motivated by the analysis of spectrometric data, a Gaussian graphical model for learning the dependence structure among frequency bands of the infrared absorbance spectrum is introduced. The spectra are modeled as continuous functional data through a B-spline basis expansion and a Gaussian graphical model is assumed as a prior specification for the smoothing coefficients to induce sparsity in their precision matrix. Bayesian inference is carried out to simultaneously smooth the curves and to estimate the conditional independence structure between portions of the functional domain. The proposed model is applied to the analysis of infrared absorbance spectra of strawberry purees.

Reflection of Conditional Independence Structure to Noise Variability for Noise Robust Text Dependent Speaker Verification

In the field of speaker verification (SV), the development of noise-robust systems is a challenge for their deployment in real-world environments. Noise variability compensation is a common strategy for increasing the robustness to noise variations. The performance of noise compensation depends on how well the noise variability, which is inherent in within-class variability, is estimated. However, to date, there is no information about the true noise variability that could reduce the gap between the empirical and true statistics. Most studies assume that true noise covariates are independent. This study aims to demonstrate the assumption that true noise variability has a conditional independence structure rather than an independence structure. This assumption was motivated by our previous findings, which revealed that optimal within-class variability has a conditional independence structure in text-dependent speaker verification (TD-SV) in clean environments. This indicates that the optima of all the variabilities in within-class variability, except noise variability, has a conditional independence structure; however, it is unknown whether this is also true for optimal noise variability. Our assumption was supported by the experimental results obtained under noisy TD-SV trials using systems built with graphical least absolute shrinking and selection operator-based probabilistic linear discriminant analysis, which achieved up to 10% relative equal error rate improvements.

Gaussoids are two-antecedental approximations of Gaussian conditional independence structures

The gaussoid axioms are conditional independence inference rules which characterize regular Gaussian CI structures over a three-element ground set. It is known that no finite set of inference rules completely describes regular Gaussian CI as the ground set grows. In this article we show that the gaussoid axioms logically imply every inference rule of at most two antecedents which is valid for regular Gaussians over any ground set. The proof is accomplished by exhibiting for each inclusion-minimal gaussoid extension of at most two CI statements a regular Gaussian realization. Moreover we prove that all those gaussoids have rational positive-definite realizations inside every ε-ball around the identity matrix. For the proof we introduce the concept of algebraic Gaussians over arbitrary fields and of positive Gaussians over ordered fields and obtain the same two-antecedental completeness of the gaussoid axioms for algebraic and positive Gaussians over all fields of characteristic zero as a byproduct.

Conditional Independence Structures Over Four Discrete Random Variables Revisited: Conditional Ingleton Inequalities

The paper deals with conditional linear information inequalities valid for entropy functions induced by discrete random variables. Specifically, the so-called conditional Ingleton inequalities are in the center of interest: these are valid under conditional independence assumptions on the inducing random variables. We discuss five inequalities of this particular type, four of which has appeared earlier in the literature. Besides the proof of the new fifth inequality, simpler proofs of (some of) former inequalities are presented. These five information inequalities are used to characterize all conditional independence structures induced by four discrete random variables.

A Convex Formulation for Learning from Crowds

Recently crowdsourcing services are often used to collect a large amount of labeled data for machine learning, since they provide us an easy way to get labels at very low cost and in a short period. The use of crowdsourcing has introduced a new challenge in machine learning, that is, coping with the variable quality of crowd-generated data. Although there have been many recent attempts to address the quality problem of multiple workers, only a few of the existing methods consider the problem of learning classifiers directly from such noisy data. All these methods modeled the true labels as latent variables, which resulted in non-convex optimization problems. In this paper, we propose a convex optimization formulation for learning from crowds without estimating the true labels by introducing personal models of the individual crowd workers. We also devise an efficient iterative method for solving the convex optimization problems by exploiting conditional independence structures in multiple classifiers. We evaluate the proposed method against three competing methods on synthetic data sets and a real crowdsourced data set and demonstrate that the proposed method outperforms the other three methods.

Jewel: A Novel Method for Joint Estimation of Gaussian Graphical Models

In this paper, we consider the problem of estimating multiple Gaussian Graphical Models from high-dimensional datasets. We assume that these datasets are sampled from different distributions with the same conditional independence structure, but not the same precision matrix. We propose jewel, a joint data estimation method that uses a node-wise penalized regression approach. In particular, jewel uses a group Lasso penalty to simultaneously guarantee the resulting adjacency matrix’s symmetry and the graphs’ joint learning. We solve the minimization problem using the group descend algorithm and propose two procedures for estimating the regularization parameter. Furthermore, we establish the estimator’s consistency property. Finally, we illustrate our estimator’s performance through simulated and real data examples on gene regulatory networks.

Characterisation of conditional independence structures for polymatroids using vanishing sets

Characterisation of conditional independence structures for polymatroids using vanishing sets

Discovering Causal Models with Optimization: Confounders, Cycles, and Feature Selection

We propose a new method for learning causal structures from observational data, a process known as causal discovery. Our method takes as input observational data over a set of variables and returns a graph in which causal relations are specified by directed edges. We consider a highly general search space that accommodates latent confounders and feedback cycles, which few extant methods do. We formulate the discovery problem as an integer program, and propose a solution technique that leverages the conditional independence structure in the data to identify promising edges for inclusion in the output graph. In the large-sample limit, our method recovers a graph that is equivalent to the true data-generating graph. Computationally, our method is competitive with the state-of-the-art, and can solve in minutes instances that are intractable for alternative causal discovery methods. We demonstrate our approach by showing how it can be used to examine the validity of instrumental variables, which are widely used for causal inference. In particular, we analyze US Census data from the seminal paper on the returns to education by Angrist and Krueger (1991), and find that the causal structures uncovered by our method are consistent with the literature.