Conditional Independence Structure Research Articles

Decomposition and Model Selection for Large Contingency Tables

Large contingency tables summarizing categorical variables arise in many areas. One example is in biology, where large numbers of biomarkers are cross-tabulated according to their discrete expression level. Interactions of the variables are of great interest and are generally studied with log-linear models. The structure of a log-linear model can be visually represented by a graph from which the conditional independence structure can then be easily read off. However, since the number of parameters in a saturated model grows exponentially in the number of variables, this generally comes with a heavy computational burden. Even if we restrict ourselves to models of lower-order interactions or other sparse structures, we are faced with the problem of a large number of cells which play the role of sample size. This is in sharp contrast to high-dimensional regression or classification procedures because, in addition to a high-dimensional parameter, we also have to deal with the analogue of a huge sample size. Furthermore, high-dimensional tables naturally feature a large number of sampling zeros which often leads to the nonexistence of the maximum likelihood estimate. We therefore present a decomposition approach, where we first divide the problem into several lower-dimensional problems and then combine these to form a global solution. Our methodology is computationally feasible for log-linear interaction models with many categorical variables each or some of them having many levels. We demonstrate the proposed method on simulated data and apply it to a bio-medical problem in cancer research.

On Sums of Conditionally Independent Subexponential Random Variables

The asymptotic tail behaviour of sums of independent subexponential random variables is well understood, one of the main characteristics being the principle of the single big jump. We study the case of dependent subexponential random variables, for both deterministic and random sums, using a fresh approach, by considering conditional independence structures on the random variables. We seek sufficient conditions for the results of the theory with independent random variables to still hold. For a subexponential distribution, we introduce the concept of a boundary class of functions, which we hope will be a useful tool in studying many aspects of subexponential random variables. The examples we give demonstrate a variety of effects owing to the dependence, and are also interesting in their own right.

Logical and algorithmic properties of stable conditional independence

The logical and algorithmic properties of stable conditional independence (CI) as an alternative structural representation of conditional independence information are investigated. We utilize recent results concerning a complete axiomatization of stable conditional independence relative to discrete probability measures to derive perfect model properties of stable conditional independence structures. We show that stable CI can be interpreted as a generalization of Markov networks and establish a connection between sets of stable CI statements and propositional formulas in conjunctive normal form. Consequently, we derive that the implication problem for stable CI is coNP-complete. Finally, we show that Boolean satisfiability (SAT) solvers can be employed to efficiently decide the implication problem and to compute concise, non-redundant representations of stable CI, even for instances involving hundreds of random variables.

Posterior Predictive Outlier Detection Using Sample Reweighting

To facilitate checking and improvement of a Bayesian model, we define an outlier as an observation or group of observations that is “surprising” relative to its predictive distribution, under the model, given the remainder of the data. Hence outlyingness can be measured by the posterior predictive case-deleted p-value of any interesting scalar summary of the (possibly multivariate) observation. It is also sometimes useful to condition on a part of the data for the potentially outlying case, such as the pattern of missing data, thus defining a conditional outlier.When parameters of interest have been sampled from their posterior distribution, the case-deleted p-value can be calculated by reweighting the sample to reflect deletion of the target observation and then drawing from the predictive distribution of the observation, facilitating huge computational savings. One efficient extension to the basic reweighting approach exploits the conditional independence structure of most hierarchical models. Another extension uses weighted systematic sampling to accommodate the dependence structure of the MCMC sample.Outlier checks are illustrated in hierarchical models for two datasets, a standard linear hierarchical model for rat growth and a complex ordinal model for survey data with nonignorably missing responses.Sample code and data for the rat growth example are available as an on-line supplement.

The conditional independences between variables derived from two independent identically distributed Markov random fields when pairwise order is ignored

A result for the equivalence of conditional independence graphs of ordered and unordered vector random variables from first-order Markov models is extended to arbitrary forests. The result is relevant to estimating graphical models for linkage disequilibrium between genetic loci. It explains why, in terms of the conditional independence structure, it sometimes does not matter whether you consider haplotypes or genotypes.

A boosting approach to structure learning of graphs with and without prior knowledge

Identifying the network structure through which genes and their products interact can help to elucidate normal cell physiology as well as the genetic architecture of pathological phenotypes. Recently, a number of gene network inference tools have appeared based on Gaussian graphical model representations. Following this, we introduce a novel Boosting approach to learn the structure of a high-dimensional Gaussian graphical model motivated by the applications in genomics. A particular emphasis is paid to the inclusion of partial prior knowledge on the structure of the graph. With the increasing availability of pathway information and large-scale gene expression datasets, we believe that conditioning on prior knowledge will be an important aspect in raising the statistical power of structural learning algorithms to infer true conditional dependencies. Our Boosting approach, termed BoostiGraph, is conceptually and algorithmically simple. It complements recent work on the network inference problem based on Lasso-type approaches. BoostiGraph is computationally cheap and is applicable to very high-dimensional graphs. For example, on graphs of order 5000 nodes, it is able to map out paths for the conditional independence structure in few minutes. Using computer simulations, we investigate the ability of our method with and without prior information to infer Gaussian graphical models from artificial as well as actual microarray datasets. The experimental results demonstrate that, using our method, it is possible to recover the true network topology with relatively high accuracy. This method and all other associated files are freely available from http://www.stats.ox.ac.uk/~anjum/.

Conditional independence structure and its closure: Inferential rules and algorithms

In this paper, we deal with conditional independence models closed with respect to graphoid properties. Such models come from different uncertainty measures, in particular in a probabilistic setting. We study some inferential rules and describe methods and algorithms to compute efficiently the closure of a set of conditional independence statements.

Covariance Estimation in Decomposable Gaussian Graphical Models

Graphical models are a framework for representing and exploiting prior conditional independence structures within distributions using graphs. In the Gaussian case, these models are directly related to the sparsity of the inverse covariance (concentration) matrix and allow for improved covariance estimation with lower computational complexity. We consider concentration estimation with the mean-squared error (MSE) as the objective, in a special type of model known as decomposable. This model includes, for example, the well known banded structure and other cases encountered in practice. Our first contribution is the derivation and analysis of the minimum variance unbiased estimator (MVUE) in decomposable graphical models. We provide a simple closed form solution to the MVUE and compare it with the classical maximum likelihood estimator (MLE) in terms of performance and complexity. Next, we extend the celebrated Stein's unbiased risk estimate (SURE) to graphical models. Using SURE, we prove that the MSE of the MVUE is always smaller or equal to that of the biased MLE, and that the MVUE itself is dominated by other approaches. In addition, we propose the use of SURE as a constructive mechanism for deriving new covariance estimators. Similarly to the classical MLE, all of our proposed estimators have simple closed form solutions but result in a significant reduction in MSE.

Building hyper Dirichlet processes for graphical models

Graphical models are used to describe the conditional independence relations in multivariate data. They have been used for a variety of problems, including log-linear models (Liu and Massam, 2006), network analysis (Holland and Leinhardt, 1981; Strauss and Ikeda, 1990; Wasserman and Pattison, 1996; Pattison and Wasserman, 1999; Robins et al., 1999), graphical Gaussian models (Roverato and Whittaker, 1998; Giudici and Green, 1999; Marrelec and Benali, 2006), and genetics (Dobra et al., 2004). A distribution that satisfies the conditional independence structure of a graph is Markov. A graphical model is a family of distributions that is restricted to be Markov with respect to a certain graph. In a Bayesian problem, one may specify a prior over the graphical model. Such a prior is called a hyper Markov law if the random marginals also satisfy the independence constraints. Previous work in this area includes (Dempster, 1972; Dawid and Lauritzen, 1993; Giudici and Green, 1999; Letac and Massam, 2007). We explore graphical models based on a non-parametric family of distributions, developed from Dirichlet processes.

Convex Rank Tests and Semigraphoids

Convex rank tests are partitions of the symmetric group which have desirable geometric properties. The statistical tests defined by such partitions involve counting all permutations in the equivalence classes. Each class consists of the linear extensions of a partially ordered set specified by data. Our methods refine existing rank tests of nonparametric statistics, such as the sign test and the runs test, and are useful for exploratory analysis of ordinal data. We establish a bijection between convex rank tests and probabilistic conditional independence structures known as semigraphoids. The subclass of submodular rank tests is derived from faces of the cone of submodular functions or from Minkowski summands of the permutohedron. We enumerate all small instances of such rank tests. Of particular interest are graphical tests, which correspond to both graphical models and to graph associahedra.

Robust concentration graph model selection

Concentration graph models are an attractive tool to explore the conditional independence structure in a multivariate normal distribution. In applications, in absence of a priori knowledge, it is possible to select the graph underlying a set of data through an appropriate model selection procedure. The recently proposed procedure, SINful, is appealing but sensitive to outliers, as it utilizes the sample estimator of the covariance matrix. A method to make the SINful procedure robust with respect to the presence of outlying observations, is proposed. This is based on the minimum covariance determinant (MCD) estimator for the variance–covariance matrix. A simulation study shows the advantages of this method.

Effects of dependence in high-dimensional multiple testing problems

BackgroundWe consider effects of dependence among variables of high-dimensional data in multiple hypothesis testing problems, in particular the False Discovery Rate (FDR) control procedures. Recent simulation studies consider only simple correlation structures among variables, which is hardly inspired by real data features. Our aim is to systematically study effects of several network features like sparsity and correlation strength by imposing dependence structures among variables using random correlation matrices.ResultsWe study the robustness against dependence of several FDR procedures that are popular in microarray studies, such as Benjamin-Hochberg FDR, Storey's q-value, SAM and resampling based FDR procedures. False Non-discovery Rates and estimates of the number of null hypotheses are computed from those methods and compared. Our simulation study shows that methods such as SAM and the q-value do not adequately control the FDR to the level claimed under dependence conditions. On the other hand, the adaptive Benjamini-Hochberg procedure seems to be most robust while remaining conservative. Finally, the estimates of the number of true null hypotheses under various dependence conditions are variable.ConclusionWe discuss a new method for efficient guided simulation of dependent data, which satisfy imposed network constraints as conditional independence structures. Our simulation set-up allows for a structural study of the effect of dependencies on multiple testing criterions and is useful for testing a potentially new method on π0 or FDR estimation in a dependency context.

Forecast covariances in the linear multiregression dynamic model

AbstractThe linear multiregression dynamic model (LMDM) is a Bayesian dynamic model which preserves any conditional independence and causal structure across a multivariate time series. The conditional independence structure is used to model the multivariate series by separate (conditional) univariate dynamic linear models, where each series has contemporaneous variables as regressors in its model. Calculating the forecast covariance matrix (which is required for calculating forecast variances in the LMDM) is not always straightforward in its current formulation. In this paper we introduce a simple algebraic form for calculating LMDM forecast covariances. Calculation of the covariance between model regression components can also be useful and we shall present a simple algebraic method for calculating these component covariances. In the LMDM formulation, certain pairs of series are constrained to have zero forecast covariance. We shall also introduce a possible method to relax this restriction. Copyright © 2008 John Wiley & Sons, Ltd.

Cluster allocation design networks

When planning and designing a policy intervention and evaluation, it is important to differentiate between (future) policy interventions we want to evaluate, $F_{T}$, affecting "the world," and experimental allocations, $A_{T}$, affecting "our picture of the world." The policy maker usually has to define a strategy that involves policy assignment and recording mechanisms that will affect the (conditional independence) structure of the data available. Causal inference is sensitive to the specification of these mechanisms. Influence diagrams have been used for causal reasoning within a Bayesian decision-theoretic framework that introduces interventions as decision nodes (Dawid 2002). Design Networks expand this framework by including experimental design decision nodes (Madrigal and Smith 2004). They provide semantics to discuss how a design decision strategy (such as a cluster randomised study) might assist the identification of intervention causal effects. The Design Network framework is extended to Cluster Allocation. It is used to assess identifiability when the experimental unit's level is different from the analysis unit's level, and to discuss the evaluation of cluster- and individual-level future policies. Cases of 'pure' cluster (all individuals in a cluster receiving the same intervention) and 'non-pure' cluster (only a subset receiving the policy) are discussed in terms of causal effects. The representation and analysis of a simplified version of a Mexican social policy programme to alleviate poverty (Progresa) is performed as an illustration of the use of Bayesian hierarchical models to make causal inferences relating to household and community level interventions.

ELICITING A DIRECTED ACYCLIC GRAPH FOR A MULTIVARIATE TIME SERIES OF VEHICLE COUNTS IN A TRAFFIC NETWORK

SummaryThe problem of modelling multivariate time series of vehicle counts in traffic networks is considered. It is proposed to use a model called the linear multiregression dynamic model (LMDM). The LMDM is a multivariate Bayesian dynamic model which uses any conditional independence and causal structure across the time series to break down the complex multivariate model into simpler univariate dynamic linear models.The conditional independence and causal structure in the time series can be represented by a directed acyclic graph (DAG). The DAG not only gives a useful pictorial representation of the multivariate structure, but it is also used to build the LMDM. Therefore, eliciting a DAG which gives a realistic representation of the series is a crucial part of the modelling process.A DAG is elicited for the multivariate time series of hourly vehicle counts at the junction of three major roads in the UK. A flow diagram is introduced to give a pictorial representation of the possible vehicle routes through the network. It is shown how this flow diagram, together with a map of the network, can suggest a DAG for the time series suitable for use with an LMDM.

Conditional independence and chain event graphs

Graphs provide an excellent framework for interrogating symmetric models of measurement random variables and discovering their implied conditional independence structure. However, it is not unusual for a model to be specified from a description of how a process unfolds (i.e. via its event tree), rather than through relationships between a given set of measurements. Here we introduce a new mixed graphical structure called the chain event graph that is a function of this event tree and a set of elicited equivalence relationships. This graph is more expressive and flexible than either the Bayesian network—equivalent in the symmetric case—or the probability decision graph. Various separation theorems are proved for the chain event graph. These enable implied conditional independencies to be read from the graph's topology. We also show how the topology can be exploited to tease out the interesting conditional independence structure of functions of random variables associated with the underlying event tree.

Dynamic matrix-variate graphical models

This paper introduces a novel class of Bayesian models for multivariate time series analysis based on a synthesis of dynamic linear models and graphical models. The synthesis uses sparse graphical modelling ideas to introduce struc- tured, conditional independence relationships in the time-varying, cross-sectional covariance matrices of multiple time series. We dene this new class of models and their theoretical structure involving novel matrix-normal/hyper-inverse Wishart distributions. We then describe the resulting Bayesian methodology and compu- tational strategies for model tting and prediction. This includes novel stochastic evolution theory for time-varying, structured variance matrices, and the full se- quential and conjugate updating, ltering and forecasting analysis. The models are then applied in the context of nancial time series for predictive portfolio analysis. The improvements dened in optimal Bayesian decision analysis in this example context vividly illustrate the practical benets of the parsimony induced via appro- priate graphical model structuring in multivariate dynamic modelling. We discuss theoretical and empirical aspects of the conditional independence structures in such models, issues of model uncertainty and search, and the relevance of this new framework as a key step towards scaling multivariate dynamic Bayesian modelling methodology to time series of increasing dimension and complexity.

Computationally efficient Monte Carlo EM algorithms for generalized linear mixed models

Maximum likelihood estimation in generalized linear mixed models usually involves intractable integrals that may be of high dimension. To reduce the dimensions of the integrals involved in computation, a reduced form of the score equation obtained by exploiting conditional independence and random effects structures can be used. Two Monte Carlo methods, one based on direct Monte Carlo integration and the other based on a stochastic version of the Gauss–Hermite quadrature, are proposed for approximating the integrals involved in the reduced score equation. The proposed Monte Carlo EM algorithms need to simulate random effects only in the first iteration; hence, the computation burden can be reduced and the likelihood that the increasing property of the original EM algorithm can be preserved. A reduced form of the information matrix is proposed for standard error estimation. We illustrate the proposed methods with an application to the salamander data and examine their performance via simulation studies.

Low-Order Conditional Independence Graphs for Inferring Genetic Networks

As a powerful tool for analyzing full conditional (in-)dependencies between random variables, graphical models have become increasingly popular to infer genetic networks based on gene expression data. However, full (unconstrained) conditional relationships between random variables can be only estimated accurately if the number of observations is relatively large in comparison to the number of variables, which is usually not fulfilled for high-throughput genomic data. Recently, simplified graphical modeling approaches have been proposed to determine dependencies between gene expression profiles. For sparse graphical models such as genetic networks, it is assumed that the zero- and first-order conditional independencies still reflect reasonably well the full conditional independence structure between variables. Moreover, low-order conditional independencies have the advantage that they can be accurately estimated even when having only a small number of observations. Therefore, using only zero- and first-order conditional dependencies to infer the complete graphical model can be very useful. Here, we analyze the statistical and probabilistic properties of these low-order conditional independence graphs (called 0-1 graphs). We find that for faithful graphical models, the 0-1 graph contains at least all edges of the full conditional independence graph (concentration graph). For simple structures such as Markov trees, the 0-1 graph even coincides with the concentration graph. Furthermore, we present some asymptotic results and we demonstrate in a simulation study that despite their simplicity, 0-1 graphs are generally good estimators of sparse graphical models. Finally, the biological relevance of some applications is summarized.

Approximation of sums of conditionally independent variables by the translated Poisson distribution

It is shown that the distribution of the sum of a Poisson random variable and an independent approximately normally distributed integer-valued random variable can be well approximated in total variation by a translated Poisson distribution, and further that a mixed translated Poisson distribution is close to a mixed translated Poisson distribution with the same random shift but fixed variance. Using these two results, a general approach is then presented for the approximation of sums of integer-valued random variables, having some conditional independence structure, by a translated Poisson distribution. We illustrate the method by means of two examples. The proofs are mainly based on Stein's method for distributional approximation.