Markov Boundary Research Articles

A Novel Ensemble Method of Divide-and-Conquer Markov Boundary Discovery for Causal Feature Selection

A Novel Ensemble Method of Divide-and-Conquer Markov Boundary Discovery for Causal Feature Selection

Bounding the family-wise error rate in local causal discovery using Rademacher averages

Many algorithms have been proposed to learn local graphical structures around target variables of interest from observational data, focusing on two sets of variables. The first one, called Parent–Children (PC) set, contains all the variables that are direct causes or consequences of the target while the second one, known as Markov boundary (MB), is the minimal set of variables with optimal prediction performances of the target. In this paper we introduce two novel algorithms for the PC and MB discovery tasks with rigorous guarantees on the Family-Wise Error Rate (FWER), that is, the probability of reporting any false positive in output. Our algorithms use Rademacher averages, a key concept from statistical learning theory, to properly account for the multiple-hypothesis testing problem arising in such tasks. Our evaluation on simulated data shows that our algorithms properly control for the FWER, while widely used algorithms do not provide guarantees on false discoveries even when correcting for multiple-hypothesis testing. Our experiments also show that our algorithms identify meaningful relations in real-world data.

Causal Feature Selection With Dual Correction.

Causal feature selection methods aim to identify a Markov boundary (MB) of a class variable, and almost all the existing causal feature selection algorithms use conditional independence (CI) tests to learn the MB. However, in real-world applications, due to data issues (e.g., noisy or small samples), CI tests can be unreliable; thus, causal feature selection algorithms relying on CI tests encounter two types of errors: false positives (i.e., selecting false MB features) and false negatives (i.e., discarding true MB features). Existing algorithms only tackle either false positives or false negatives, and they cannot deal with both types of errors at the same time, leading to unsatisfactory results. To address this issue, we propose a dual-correction-strategy-based MB learning (DCMB) algorithm to correct the two types of errors simultaneously. Specifically, DCMB selectively removes false positives from the MB features currently selected, while selectively retrieving false negatives from the features currently discarded. To automatically determine the optimal number of selected features for the selective removal and retrieval in the dual correction strategy, we design the simulated-annealing-based DCMB (SA-DCMB) algorithm. Using benchmark Bayesian network (BN) datasets, the experimental results demonstrate that DCMB achieves substantial improvements on the MB learning accuracy compared with the existing MB learning methods. Empirical studies in real-world datasets validate the effectiveness of SA-DCMB for classification against state-of-the-art causal and traditional feature selection algorithms.

Local discovery in Bayesian networks by information-connecting

<p>Local discovery plays an important role in Bayesian networks (BNs), mainly addressing PC (parents and children) discovery and MB (Markov boundary) discovery. In this paper, we considered the problem of large local discovery. First, we focused on an assumption about conditional independence (CI) tests: We explained why it was unreasonable to assume all CI tests were reliable in large local discovery, studied how the power and reliability of CI tests changed with the data size and the number of degrees of freedom, and then modified the assumption about CI tests in a more reasonable way. Second, we concentrated on improving local discovery algorithms: We posed the problem of premature termination of the forward search, analyze why it arose frequently in large local discovery when implementing the existing local discovery algorithms, put forward an idea of preventing the premature termination of forward search called information connection (IC), and used IC to build a novel algorithm called ICPC; the theoretical basis of ICPC was detailedly presented. In addition, a more steady incremental algorithm as the subroutine of ICPC was proposed. Third, the way of breaking ties among equal associations was considered and optimized. Finally, we conducted a benchmarking study by means of six synthetic BNs from various domains. The experimental results revealed the applicability and superiority of ICPC in solving the problem of premature termination of the forward search that arose frequently in large local discovery.</p>

Nonlinear learning method for local causal structures

Recent years have witnessed the proliferation of causal learning techniques, aimed at extracting the abundant causal relationships embedded within observational data. In many scenarios, our primary focus lies in predicting a single target variable. In such cases, it becomes both inefficient and unnecessary to learn an entire causal network through advanced global learning methods. To address this challenge, the concept of local causal learning has been introduced to identify the direct causes and effects of a target variable of interest. However, current algorithms exhibit limitations stemming from their reliance on conditional independence tests, which only consider the linear and pairwise relationships but ignore the ubiquitous nonlinear and multivariate causality, making them lose efficacy in practical scenarios. This paper takes significant strides toward facilitating the real-world applications for local causal learning. To identify the nonlinear relations, this paper discovers the Markov boundary (MB) through calculating the minimal conditional covariance operator in reproducing kernel Hilbert space. This approach establishes a theoretical equivalence between the solution and MB. Subsequently, a nonlinear scoring mechanism for structure learning is employed based on the selected subset, yielding the optimal local causal structure. A series of extensive experiments serves to underscore the superiority of the proposed method.

Feature Selection in the Data Stream Based on Incremental Markov Boundary Learning.

Recent years have witnessed the proliferation of techniques for streaming data mining to meet the demands of many real-time systems, where high-dimensional streaming data are generated at high speed, increasing the burden on both hardware and software. Some feature selection algorithms for streaming data are proposed to tackle this issue. However, these algorithms do not consider the distribution shift due to nonstationary scenarios, leading to performance degradation when the underlying distribution changes in the data stream. To solve this problem, this article investigates feature selection in streaming data through incremental Markov boundary (MB) learning and proposes a novel algorithm. Different from existing algorithms focusing on prediction performance on off-line data, the MB is learned by analyzing conditional dependence/independence in data, which uncovers the underlying mechanism and is naturally more robust against the distribution shift. To learn MB in the data stream, the proposal transforms the learned information in previous data blocks to prior knowledge and employs them to assist MB discovery in current data blocks, where the likelihood of distribution shift and reliability of conditional independence test are monitored to avoid the negative impact from invalid prior information. Extensive experiments on synthetic and real-world datasets demonstrate the superiority of the proposed algorithm.

Practical Markov Boundary Learning without Strong Assumptions

Theoretically, the Markov boundary (MB) is the optimal solution for feature selection. However, existing MB learning algorithms often fail to identify some critical features in real-world feature selection tasks, mainly because the strict assumptions of existing algorithms, on either data distribution, variable types, or correctness of criteria, cannot be satisfied in application scenarios. This paper takes further steps toward opening the door to real-world applications for MB. We contribute in particular to a practical MB learning strategy, which can maintain feasibility and effectiveness in real-world data where variables can be numerical or categorical with linear or nonlinear, pairwise or multivariate relationships. Specifically, the equivalence between MB and the minimal conditional covariance operator (CCO) is investigated, which inspires us to design the objective function based on the predictability evaluation of the mapping variables in a reproducing kernel Hilbert space. Based on this, a kernel MB learning algorithm is proposed, where nonlinear multivariate dependence could be considered without extra requirements on data distribution and variable types. Extensive experiments demonstrate the efficacy of these contributions.

Multi-Target Markov Boundary Discovery: Theory, Algorithm, and Application.

Markov boundary (MB) has been widely studied in single-target scenarios. Relatively few works focus on the MB discovery for variable set due to the complex variable relationships, where an MB variable might contain predictive information about several targets. This paper investigates the multi-target MB discovery, aiming to distinguish the common MB variables (shared by multiple targets) and the target-specific MB variables (associated with single targets). Considering the multiplicity of MB, the relation between common MB variables and equivalent information is studied. We find that common MB variables are determined by equivalent information through different mechanisms, which is relevant to the existence of the target correlation. Based on the analysis of these mechanisms, we propose a multi-target MB discovery algorithm to identify these two types of variables, whose variant also achieves superiority and interpretability in feature selection tasks. Extensive experiments demonstrate the efficacy of these contributions.

Fast Relevant Feature Selection Based on Markov Boundary Learning from Data Stream for Classification Tasks

Learning a classification model from a dataset with the presence of irrelevant variables makes such a task more difficult and less accurate. To address this issue, feature selection has been proposed. However, existing feature selection struggles to select accurate and relevant sub-features or subspaces. It has been demonstrated in the literature that learning the Markov boundary (MB) of features helps to extract appropriate subsets of relevant features from the full data space. However, due to the fact that almost all existing methods learn MBs with static data, a state-of-the-art algorithm for learning MBs with streaming data, called SDMB (streaming data-based MB), has recently been proposed. SDMB has successfully improved online feature selection based on learning Markov boundary (MB) from data stream for classification tasks. Nevertheless, we found that SDMB has computational redundancy and there is room for improvement in terms of runtime. Therefore, we propose an improvement to SDMB. The main drawback of SDMB is that it entirely recomputes the conditional independence/dependence tests (i.e., the dependency scores) between a target variable and other variables in each iteration of learning MBs from the full data space. This paper proposes an incremental SDMB, called FastSDMB, to address the above bottleneck by reducing the dependency score computation time of SDMB. FastSDMB avoids the computational redundancy in SDMB by using a novel incremental computing collective dependency scoring process. This incremental process retains and reuses, in current iterations, the calculated conditional independence test score of variables that have not changed from previous iterations. To demonstrate FastSDMB's effectiveness and capability of handling fast data arrival scenarios, comparisons are made first over the state-of-the-art SDMB, and then we apply FastSDMB to various classification models in order to showcase the resulting performance improvement when implemented over a wide variety of synthetic and real-world datasets. The experimental results show that FastSDMB can run up to 4.8 × faster on average than SDMB and it maintains the same accuracy when learning Markov boundary. Thereby proving its efficiency in fast relevant feature selection based on Markov boundary learning from data stream.

Analyzing Markov Boundary Discovery Algorithms in Ideal Conditions Using the d-Separation Criterion

This article proposes the usage of the d-separation criterion in Markov Boundary Discovery algorithms, instead of or alongside the statistical tests of conditional independence these algorithms usually rely on. This is a methodological improvement applicable when designing, studying or improving such algorithms, but it is not applicable for productive use, because computing the d-separation criterion requires complete knowledge of a Bayesian network. Yet Bayesian networks can be made available to the algorithms when studied in controlled conditions. This approach has the effect of removing sources of suboptimal behavior, allowing the algorithms to perform at their theoretical best and providing insights about their properties. The article also discusses an extension of this approach, namely to use d-separation as a complement to the usual statistical tests performed on synthetic datasets in order to ascertain the overall accuracy of the tests chosen by the algorithms, for further insights into their behavior. To exemplify these two approaches, two Markov Boundary Discovery algorithms were used, namely the Incremental Association Markov Blanket algorithm and the Iterative Parent–Child-Based Search of Markov Blanket algorithm. Firstly, these algorithms were configured to use d-separation alone as their conditional independence test, computed on known Bayesian networks. Subsequently, the algorithms were configured to use the statistical G-test complemented by d-separation to evaluate their behavior on synthetic data.

Separation and recovery Markov boundary discovery and its application in EEG-based emotion recognition

In a Bayesian network (BN), the Markov boundary (MB) presents the local causal structure around a target. Due to the interpretability and robustness, it has been widely applied to feature selection and BN structure learning. However, existing MB discovery algorithms might fail to identify some true positives, leading to poor performance in real-world applications. To tackle this issue, we introduce a two-phase-discovery strategy to search more true positives. Based on this strategy, we propose a more accurate and data-efficient algorithm, separation and recovery MB discovery algorithm (SRMB). SRMB first discovers an incomplete parent–child set and spouse set via an MB separation process, and then retrieves the ignored true positives via an MB recovery process, which further exploits a symmetry test to improve accuracy in unfaithful cases. Experiments on standard BN and real-world data sets demonstrate the effectiveness and superiority of SRMB in terms of MB discovery, BN structure learning, and feature selection. To demonstrate the superiority of SRMB in data with distribution shift, we further apply SRMB to EEG-based emotion recognition tasks, where distribution shift exists in multiple unstable sessions. We prove that the most predictive features are from Gamma/Beta frequency bands and are distributed at the lateral temporal area.

Accurate Markov Boundary Discovery for Causal Feature Selection.

Causal feature selection has achieved much attention in recent years, which discovers a Markov boundary (MB) of the class attribute. The MB of the class attribute implies local causal relations between the class attribute and the features, thus leading to more interpretable and robust prediction models than the features selected by the traditional feature selection algorithms. Many causal feature selection methods have been proposed, and almost all of them employ conditional independence (CI) tests to identify MBs. However, many datasets from real-world applications may suffer from incorrect CI tests due to noise or small-sized samples, resulting in lower MB discovery accuracy for these existing algorithms. To tackle this issue, in this article, we first introduce a new concept of PCMasking to explain a type of incorrect CI tests in the MB discovery, then propose a cross-check and complement MB discovery (CCMB) algorithm to repair this type of incorrect CI tests for accurate MB discovery. To improve the efficiency of CCMB, we further design a pipeline machine-based CCMB (PM-CCMB) algorithm. Using benchmark Bayesian network datasets, the experiments demonstrate that both CCMB and PM-CCMB achieve significant improvements on the MB discovery accuracy compared with the existing methods, and PM-CCMB further improves the computational efficiency. The empirical study in the real-world datasets validates the effectiveness of CCMB and PM-CCMB against the state-of-the-art causal and traditional feature selection algorithms.

Markov Boundary Learning With Streaming Data for Supervised Classification

In this paper, we study the problem of Markov boundary (MB) learning with streaming data. A MB is a crucial concept in a Bayesian network (BN) and plays an important role in BN structure learning. In addition, in the supervised learning setting, the MB of a class attribute is optimal to feature selection for classification. Almost all existing MB learning algorithms focus on static data, but few efforts have been proposed to learning MBs with streaming data. In this paper, by linking dynamic AD-trees with streaming data, we proposed a new SDMB (streaming data-based MB) algorithm for learning MBs with streaming data. Specifically, given a target variable, SDMB employs a dynamic AD-tree to summarize the historical data, then the SDMB sequentially learns the MB of the target upon all available data by calculating independence tests using the dynamic AD-tree. In experiments, using the synthetic and real-world data sets, we evaluate the SDMB algorithm and compared it with the state-of-the-art online feature selection algorithms and data stream mining methods, and the experimental results validate the SDMB algorithm.

Alternative models for the modified form of ridge regularized linear model in discovering Markov boundary

It has been proved that the modified form of ridge regularized linear model (MRRLM) can discover a subset of Markov boundary of the target variable under some constrained conditions. However, MRRLM cannot be applied to the data sets with collinear variables due to covariance matrix is employed. To develop a suitable alternative model for MRRLM, we study the relationships of discovery performance of Markov boundary among MRRLM, ridge regression linear model (RRLM), and LASSO combining with permutation test through empirical method. In addition, we also proposed a new NVRRLM to discover Markov boundary of the target variable. The experimental results show that: (1) On the binary data sets, MRRLM has a basically similar performance with LASSO and RRLM; (2) On the continuous data sets, MRRLM has a basically similar discovery performance with LASSO but has higher discovery performance than RRLM; (3) The new NVRRLM can replace MRRLM on the data sets with collinear variables. The above experimental results demonstrate NVRRLM can effectively deal with variable collinearity problems.

Multiscale integration: beyond internalism and externalism

We present a multiscale integrationist interpretation of the boundaries of cognitive systems, using the Markov blanket formalism of the variational free energy principle. This interpretation is intended as a corrective for the philosophical debate over internalist and externalist interpretations of cognitive boundaries; we stake out a compromise position. We first survey key principles of new radical (extended, enactive, embodied) views of cognition. We then describe an internalist interpretation premised on the Markov blanket formalism. Having reviewed these accounts, we develop our positive multiscale account. We argue that the statistical seclusion of internal from external states of the system—entailed by the existence of a Markov boundary—can coexist happily with the multiscale integration of the system through its dynamics. Our approach does not privilege any given boundary (whether it be that of the brain, body, or world), nor does it argue that all boundaries are equally prescient. We argue that the relevant boundaries of cognition depend on the level being characterised and the explanatory interests that guide investigation. We approach the issue of how and where to draw the boundaries of cognitive systems through a multiscale ontology of cognitive systems, which offers a multidisciplinary research heuristic for cognitive science.

Markov Boundary Discovery Based on Variant Ridge Regularized Linear Models

It has been proved that the modified form of ridge regularized linear models (MRRLMs) can get “very close” to identifying a subset of Markov boundary. However, it is assumed that the covariance matrix is non-singular, so MRRLMs cannot be applied to discover the Markov boundary (subset) from data sets when the covariance matrix is singular. The singularity of the covariance matrix means that there are some collinear variables in the data sets, and such data sets exist widely in the real world. In this paper, we present a novel variant of ridge regularized linear models (VRRLMs) to identify a subset of Markov boundary from data sets with collinear and non-collinear variables and, then, reveal the relationship between covariance matrix and collinearity of variables in the theory. In addition, we prove theoretically that the VRRLMs can identify a subset of Markov boundary under some reasonable assumptions and verify the theory on the four discrete data sets. The results show that VRRLMs outperform the MRRLMs in discovering a subset of Markov boundary on the data sets with collinear variables, while both of them have a similar discovery efficiency of the Markov boundary (subset) on the data sets with non-collinear variables.

Efficient grounding of abstract spatial concepts for natural language interaction with robot platforms

Our goal is to develop models that allow a robot to efficiently understand or “ground” natural language instructions in the context of its world representation. Contemporary approaches estimate correspondences between language instructions and possible groundings such as objects, regions, and goals for actions that the robot should execute. However, these approaches typically reason in relatively small domains and do not model abstract spatial concepts such as as “rows,” “columns,” or “groups” of objects and, hence, are unable to interpret an instruction such as “pick up the middle block in the row of five blocks.” In this paper, we introduce two new models for efficient natural language understanding of robot instructions. The first model, which we call the adaptive distributed correspondence graph (ADCG), is a probabilistic model for interpreting abstract concepts that require hierarchical reasoning over constituent concrete entities as well as notions of cardinality and ordinality. grounding variables form a Markov boundary over concrete groundings, effectively de-correlating them from the remaining variables in the graph. This structure reduces the complexity of model training and inference. Inference in the model is posed as an approximate search procedure that orders factor computation such that the estimated probable concrete groundings focus the search for abstract concepts towards likely hypothesis, pruning away improbable portions of the exponentially large space of abstractions. Further, we address the issue of scalability to complex domains and introduce a hierarchical extension to a second model termed the hierarchical adaptive distributed correspondence graph (HADCG). The model utilizes the abstractions in the ADCG but infers a coarse symbolic structure from the utterance and the environment model and then performs fine-grained inference over the reduced graphical model, further improving the efficiency of inference. Empirical evaluation demonstrates accurate grounding of abstract concepts embedded in complex natural language instructions commanding a robotic torso and a mobile robot. Further, the proposed approximate inference method allows significant efficiency gains compared with the baseline, with minimal trade-off in accuracy.

Markov blanket and Markov boundary of multiple variables

Markov blanket (Mb) and Markov boundary (MB) are two key concepts in Bayesian networks (BNs). In this paper, we study the problem of Mb and MB for multiple variables. First, we show that Mb possess...

The alarming problems of confounding equivalence using logistic regression models in the perspective of causal diagrams

BackgroundConfounders can produce spurious associations between exposure and outcome in observational studies. For majority of epidemiologists, adjusting for confounders using logistic regression model is their habitual method, though it has some problems in accuracy and precision. It is, therefore, important to highlight the problems of logistic regression and search the alternative method.MethodsFour causal diagram models were defined to summarize confounding equivalence. Both theoretical proofs and simulation studies were performed to verify whether conditioning on different confounding equivalence sets had the same bias-reducing potential and then to select the optimum adjusting strategy, in which logistic regression model and inverse probability weighting based marginal structural model (IPW-based-MSM) were compared. The “do-calculus” was used to calculate the true causal effect of exposure on outcome, then the bias and standard error were used to evaluate the performances of different strategies.ResultsAdjusting for different sets of confounding equivalence, as judged by identical Markov boundaries, produced different bias-reducing potential in the logistic regression model. For the sets satisfied G-admissibility, adjusting for the set including all the confounders reduced the equivalent bias to the one containing the parent nodes of the outcome, while the bias after adjusting for the parent nodes of exposure was not equivalent to them. In addition, all causal effect estimations through logistic regression were biased, although the estimation after adjusting for the parent nodes of exposure was nearest to the true causal effect. However, conditioning on different confounding equivalence sets had the same bias-reducing potential under IPW-based-MSM. Compared with logistic regression, the IPW-based-MSM could obtain unbiased causal effect estimation when the adjusted confounders satisfied G-admissibility and the optimal strategy was to adjust for the parent nodes of outcome, which obtained the highest precision.ConclusionsAll adjustment strategies through logistic regression were biased for causal effect estimation, while IPW-based-MSM could always obtain unbiased estimation when the adjusted set satisfied G-admissibility. Thus, IPW-based-MSM was recommended to adjust for confounders set.

Enhancing ELM by Markov Boundary based feature selection

ELM, as an efficient classification technology, has been used in many popular application domains. However, ELM has weak generalization performance when the data set is small with respect to its feature space. In this paper, an enhanced ELM algorithm based on representative features is proposed to address the problem. At first, the method automatically generates some discrete intervals for every continuous feature. Then, it removes the irrelevant features by a method considering the feature interaction and reduces the weakly relevant features by a mutual information based method. Further, the reduction of redundancy features is conducted. Instead of constructing a large Bayesian network using all features, we just select the features of high relevance with the object node by an improved Markov Boundary identifying algorithm. Finally, we obtain the enhanced ELM classifier by training ELM using the extracted representative features and a genetic algorithm based weight assignment mechanism. The experiments conducted on real and synthetic small sample data sets demonstrate that the enhanced ELM classifier based on representative features outperforms the other methods used in our comparison study in terms of both efficiency and effectiveness.