Conditional Independence Assumption Research Articles

Promises of parallel outcomes

Summary A key challenge in causal inference from observational studies is the identification and estimation of causal effects in the presence of unmeasured confounding. In this paper, we introduce a novel approach for causal inference that leverages information in multiple outcomes to deal with unmeasured confounding. An important assumption in our approach is conditional independence among multiple outcomes. In contrast to existing proposals in the literature, the roles of multiple outcomes in the conditional independence assumption are symmetric; hence, the name parallel outcomes. We show nonparametric identifiability with at least three parallel outcomes and provide parametric estimation tools under a set of linear structural equation models. Our proposal is evaluated through a set of synthetic and real data analyses.

A copula formulation for multivariate latent Markov models

AbstractWe specify a general formulation for multivariate latent Markov models for panel data, where outcomes are possibly of mixed-type (categorical, discrete, continuous). Conditionally on a time-varying discrete latent variable and covariates, the joint distribution of outcomes simultaneously observed is expressed through a parametric copula. We therefore do not make any conditional independence assumption. The observed likelihood is maximized by means of an expectation–maximization algorithm. In a simulation study, we argue how modeling the residual contemporary dependence might be crucial in order to avoid bias in the parameter estimates. We illustrate through an original application to assessment of poverty through direct and indirect indicators in a cohort of Italian households.

Micro-Macro Mediation Analysis in Social Networks

Mediation analysis is increasingly used in the social sciences. Extension to social network data, however, has proved difficult because statistical network models are formulated at a lower level of analysis (the dyad) than many outcomes of interest. This study introduces a general approach for micro-macro mediation analysis in social networks. The author defines the average mediated micro effect (AMME) as the indirect effect of a network selection process on an individual, group, or organizational outcome through its effect on an intervening network variable. The author shows that the AMME can be nonparametrically identified using a wide range of common statistical network and regression modeling strategies under the assumption of conditional independence among multiple mediators. Nonparametric and parametric algorithms are introduced to generically estimate the AMME in a multitude of research designs. The author illustrates the utility of the method with an applied example using cross-sectional National Longitudinal Study of Adolescent to Adult Health data to examine the friendship selection mechanisms that indirectly shape adolescent school performance through their effect on network structure.

No Eleventh Conditional Ingleton Inequality

A rational probability distribution on four binary random variables X , Y , Z , U is constructed which satisfies the conditional independence relations [ X ⊥ ⊥ Y ] , [ X ⊥ ⊥ Z | U ] , [ Y ⊥ ⊥ U | Z ] and [ Z ⊥ ⊥ U | X Y ] but whose entropy vector violates the Ingleton inequality. This settles a recent question of Studený (IEEE Trans. Inf. Theory vol. 67, no. 11) and shows that there are, up to symmetry, precisely ten inclusion-minimal sets of conditional independence assumptions on four discrete random variables which make the Ingleton inequality hold. The last case in the classification of which of these inequalities are essentially conditional is also settled.

Alleviating conditional independence assumption of naive Bayes

Alleviating conditional independence assumption of naive Bayes

Adjusted Residuals for Evaluating Conditional Independence in IRT Models for Multistage Adaptive Testing.

The key assumption of conditional independence of item responses given latent ability in item response theory (IRT) models is addressed for multistage adaptive testing (MST) designs. Routing decisions in MST designs can cause patterns in the data that are not accounted for by the IRT model. This phenomenon relates to quasi-independence in log-linear models for incomplete contingency tables and impacts certain types of statistical inference based on assumptions on observed and missing data. We demonstrate that generalized residuals for item pair frequencies under IRT models as discussed by Haberman and Sinharay (J Am Stat Assoc 108:1435-1444, 2013. https://doi.org/10.1080/01621459.2013.835660 ) are inappropriate for MST data without adjustments. The adjustments are dependent on the MST design, and can quickly become nontrivial as the complexity of the routing increases. However, the adjusted residuals are found to have satisfactory Type I errors in a simulation and illustrated by an application to real MST data from the Programme for International Student Assessment (PISA). Implications and suggestions for statistical inference with MST designs are discussed.

Spam filter based on naive bayes algorithm

The widespread use of Electronic Mail (E-mail) has led to a significant increase in spam, which has severely impeded the growth and well-being of the Internet. To mitigate this issue, the implementation of email filtering techniques has become necessary, requiring the use of specific technological tools. Presently, the K-Nearest Neighbor (KNN), Support Vector Machine (SVM), and Naive Bayes (NB) algorithms are commonly used in probability statistical classification methods for email filtering. Among these, the NB algorithm is the most classical, with its rich mathematical theory as the basis, high classification efficiency, and straightforward algorithmic approach. However, the algorithm relies on the conditional independence assumption, making the accuracy susceptible to the correlation between attributes. This study focuses on email filtering techniques based on the NB algorithm, conducting experiments to evaluate the classification accuracy and proposing feasible improvements to weaken the independence assumption. The experimental results demonstrated the effectiveness of the employed method.

A Flexible Probabilistic Framework With Concurrent Analysis of Continuous and Categorical Data for Industrial Fault Detection and Diagnosis

Operational data from industrial processes typically consist of continuous and categorical variables. They reveal different aspects of the operational conditions, and both are useful for comprehensive fault detection and diagnosis (FDD). However, due to the multiple production modes, the variables usually do not follow Gaussian or Bernoulli distributions. Furthermore, their correlations can be different across normal and various faulty classes. Thus, the main challenge lies in how to properly fuse the complementary information in the two types of variables and accurately characterize the complicated distribution for each class. This work proposes a flexible probabilistic framework that can concurrently analyze continuous and categorical variables for FDD. Our framework specifies a finite mixture model for each class. Thus, it can handle non-Gaussian and non-Bernoulli variables and capture their correlations under the conditional independence assumption. We then introduce the variational inference for parameter estimation, which makes our framework adaptive to the different distributions of various classes. Furthermore, a unified statistical index is designed for two types of variables, giving our method extra capability to distinguish unknown faults. Finally, the effectiveness of our method is validated on the Tennessee Eastman (TE) process and a practical industrial plant process. The average <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">F1</i> score of our method is improved by 3.4/3.9 percentage points in the single-mode/multimode situation on the TE process compared with traditional mixture discriminant analysis. In the industrial plant process, when unknown faults are added, the average <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">F1</i> score of our method is 91.1% and only 1.2 percentage points lower than that on the test set without unknown faults.

Testing conditional independence in casual inference for time series data

In this paper, we propose a new procedure to test conditional independence assumption in studying casual inference for time series data. The conditional independence assumption is transformed to a nonparametric conditional moment test with the help of auxiliary variables which are allowed to affect policy choice but the dependence can be fully captured by potential outcomes and observable controls. When the policy choice is binary, a nonparametric statistic test is developed further for testing the conditional independence assumption conditional on policy propensity score. Under some regular conditions, we show that the proposed test statistics are asymptotically normal under the null hypotheses for time series data. In addition, the performances of the proposed methods are illustrated through Monte Carlo simulations and a real example considered in Angrist and Kuersteiner (2011).

A Bayesian latent class extension of naive Bayesian classifier and its application to the classification of gastric cancer patients

BackgroundThe Naive Bayes (NB) classifier is a powerful supervised algorithm widely used in Machine Learning (ML). However, its effectiveness relies on a strict assumption of conditional independence, which is often violated in real-world scenarios. To address this limitation, various studies have explored extensions of NB that tackle the issue of non-conditional independence in the data. These approaches can be broadly categorized into two main categories: feature selection and structure expansion.In this particular study, we propose a novel approach to enhancing NB by introducing a latent variable as the parent of the attributes. We define this latent variable using a flexible technique called Bayesian Latent Class Analysis (BLCA). As a result, our final model combines the strengths of NB and BLCA, giving rise to what we refer to as NB-BLCA. By incorporating the latent variable, we aim to capture complex dependencies among the attributes and improve the overall performance of the classifier.MethodsBoth Expectation-Maximization (EM) algorithm and the Gibbs sampling approach were offered for parameter learning. A simulation study was conducted to evaluate the classification of the model in comparison with the ordinary NB model. In addition, real-world data related to 976 Gastric Cancer (GC) and 1189 Non-ulcer dyspepsia (NUD) patients was used to show the model's performance in an actual application. The validity of models was evaluated using the 10-fold cross-validation.ResultsThe presented model was superior to ordinary NB in all the simulation scenarios according to higher classification sensitivity and specificity in test data. The NB-BLCA model using Gibbs sampling accuracy was 87.77 (95% CI: 84.87-90.29). This index was estimated at 77.22 (95% CI: 73.64-80.53) and 74.71 (95% CI: 71.02-78.15) for the NB-BLCA model using the EM algorithm and ordinary NB classifier, respectively.ConclusionsWhen considering the modification of the NB classifier, incorporating a latent component into the model offers numerous advantages, particularly within medical and health-related contexts. By doing so, the researchers can bypass the extensive search algorithm and structure learning required in the local learning and structure extension approach. The inclusion of latent class variables allows for the integration of all attributes during model construction. Consequently, the NB-BLCA model serves as a suitable alternative to conventional NB classifiers when the assumption of independence is violated, especially in domains pertaining to health and medicine.

Evaluating the effectiveness of ground motion intensity measures through the lens of causal inference

AbstractWhen solving the performance‐based earthquake engineering (PBEE) integral, the engineering demand parameters (EDPs) are assumed to be independent of the “upstream” parameters used in ground motion models (e.g., magnitude and source‐to‐site distance) after conditioning on the intensity measure (IM). This paper formulates a methodology for evaluating this conditional independence assumption through the lens of causal inference (CI). From a causal perspective, the effect of an upstream parameter on the EDP of interest is partially mediated by the IM with the remainder having a direct influence on the EDP. An IM is judged to be desirable (from a conditional independence standpoint) if the mediated effect of the upstream parameter is maximized, which implies that the direct effect is minimized. In the language of causal inference (CI), the IM, EDP and upstream parameters are described as the “treatment”, “outcome” and “confounding” variables, respectively. It then follows that the best performing IM is the one that maximizes the effect on the EDP after controlling for the upstream parameters. A semi‐parametric model that employs double machine learning is used to estimate the causal effect. The methodology is demonstrated through a case study application utilizing the responses from five special steel moment frames analyzed using 240 site‐agnostic ground motions. Compared to sufficiency‐based assessments, the casual inference method produces findings that are more explainable using structural dynamics principles and invariant to the number of ground motions used in the evaluation.

A design utility approach for preferentially sampled spatial data

Abstract Spatial preferential sampling occurs when the choice of sampling locations depends stochastically on the process of interest. Ignoring this dependence leads to inaccurate inferences. Our framework models experimenter preferences jointly with the spatial process to adjust for this. We dispense with the unrealistic assumption (required by existing methods) of conditional independence of sampling locations by defining a whole design distribution proportional to a utility function on the space of designs. The proposed model likelihood is generally intractable. We provide fitting techniques based on the noisy Markov chain Monte Carlo and demonstrate their usage on a data set of spatially distributed ammonia concentrations.

A Latent Space Diffusion Item Response Theory Model to Explore Conditional Dependence between Responses and Response Times.

Traditional measurement models assume that all item responses correlate with each other only through their underlying latent variables. This conditional independence assumption has been extended in joint models of responses and response times (RTs), implying that an item has the same item characteristics fors all respondents regardless of levels of latent ability/trait and speed. However, previous studies have shown that this assumption is violated in various types of tests and questionnaires and there are substantial interactions between respondents and items that cannot be captured by person- and item-effect parameters in psychometric models with the conditional independence assumption. To study the existence and potential cognitive sources of conditional dependence and utilize it to extract diagnostic information for respondents and items, we propose a diffusion item response theory model integrated with the latent space of variations in information processing rate of within-individual measurement processes. Respondents and items are mapped onto the latent space, and their distances represent conditional dependence and unexplained interactions. We provide three empirical applications to illustrate (1) how to use an estimated latent space to inform conditional dependence and its relation to person and item measures, (2) how to derive diagnostic feedback personalized for respondents, and (3) how to validate estimated results with an external measure. We also provide a simulation study to support that the proposed approach can accurately recover its parameters and detect conditional dependence underlying data.

Accurate Summary-based Cardinality Estimation Through the Lens of Cardinality Estimation Graphs

We study two classes of summary-based cardinality estimators that use statistics about input relations and small-size joins: (i) optimistic estimators, which were defined in the context of graph database management systems, that make uniformity and conditional independence assumptions; and (ii) the recent pessimistic estimators that use information theoretic linear programs (LPs). We show that optimistic estimators can be modeled as picking bottom-to-top paths in a cardinality estimation graph (CEG), which contains subqueries as nodes and edges whose weights are average degree statistics. We show that existing optimistic estimators have either undefined or fixed choices for picking CEG paths as their estimates and ignore alternative choices. Instead, we outline a space of optimistic estimators to make an estimate on CEGs, which subsumes existing estimators. We show, using an extensive empirical analysis, that effective paths depend on the structure of the queries. We next show that optimistic estimators and seemingly disparate LP-based pessimistic estimators are in fact connected. Specifically, we show that CEGs can also model some recent pessimistic estimators. This connection allows us to provide insights into the pessimistic estimators, such as showing that they have combinatorial solutions.

Validity of Food insecurity experience scale (FIES) for use in rural Bangladesh and prevalence and determinants of household food insecurity: An analysis of data from Bangladesh integrated household survey (BIHS) 2018-2019

“Access” dimension of Food insecurity (FI) is directly measured by the Food Insecurity Experience Scale (FIES). The current study assessed the appropriateness of the FIES for measuring FI in rural Bangladesh, followed by an assessment of FI prevalence and its correlates utilizing Bangladesh Integrated Household Survey (BIHS) data. The internal validity of the FIES and the prevalence of FI were investigated using the Rasch modeling approach. We utilized equating procedure to calibrate the study’s result to the global FIES reference scale and determined FI prevalence rates that were comparable across countries. The external validity of the FIES was evaluated by examining its association with other FI measures using Spearman’s rho correlation analysis. With an overall Rasch reliability of 0.84, the FIES met the Rasch model assumptions of conditional independence and equal discrimination, and as well as the fit statistics standards for all eight items. Infit statistics were within the allowed limit for all FIES items indicating good internal validity. However, we noted a high outfit (>2) for the “unable to eat healthy and nutritious food” item indicating the presence of some unusual response patterns. Our analysis found no significant (>0.4) correlation between FIES items. We also found a significant correlation between FIES and other FI proxies, e.g., the Household hunger scale (HHS), Food consumption score (FCS), and Household dietary diversity score (HDDS). Overall, the prevalence of moderate or severe FI was 18.92% in rural Bangladesh. Geographic areas, access to electricity, household ownership, access to sanitation, livestock ownership, family size, education level, and monthly per capita food expenditure significantly explained the variation in FI. Our analyses suggest that the FIES is internally and externally valid for FI measurement in rural Bangladesh. However, FIES questions may need to be reordered to more accurately evaluate lower levels of FI, and the item “unable to eat healthy and nutritious food” may need cognitive testing.

Analysis of alternative strategies applied to Naïve-Bayes classifier into the recognition of electrofacies: Application in well-log data at Recôncavo Basin, North-East Brazil

This paper is concerned with the applicability of different strategies to improve the definition of prior probabilities and/or likelihoods of naïve Bayes (NB) classifiers. Standard NB method computes likelihoods and priors by means of normal distributions and evaluation of the entire training data-set, respectively. NB is one of the most prolific classification methods in data mining and machine learning. Despite decent efficiency facing good training data, NB classifiers present the intriguing assumption of conditional independence between the attributes. Several algorithms have been proposed to improve the effectiveness of NB classifiers by inserting discriminant approaches into its generative structure. From a reliability perspective, the standard NB approach might not explore the real capabilities of NB classifiers facing the lithologic classification problem. To cover such distrust, a novel approach considering four particular strategies are suggested and compared to the standard NB classification outcomes. At first, a kernel density estimation (KDE) is considered to ameliorate the likelihood models. We also apply the NB classifier in parts by separating the training data-set in individual wells in a committee architecture framework. A tuning strategy is also considered for automatic estimation of prior probabilities in an optimization-scheme. Another novel alternative, named as CRC, adapted to the standard NB classifier consists in defining priors based on depth zones from regional stratigraphic information in which to apply NB classifier. We prepare an extensive statistical investigation, based on precision, recall, classification errors, fscores and confusion matrices to bespeak the most relevant NB strategy for classification of electrofacies. Despite the decent classification outcomes for all above-mentioned strategies, CRC can be considered, by a narrow margin, the most prolific method to be applied as an improvement of the standard NB to classify rock units. Tests are performed on a validation well (i.e., 7-MP-50D-BA) of Massapê Field, in Recôncavo Basin, northeast Brazil to highlight the classification particularities provided by the improved strategy. A significant improvement in the classification of sandstones (i.e., from 68 % to 83 % accuracy) is observed, according to the confusion matrix analysis. Additionally, a minimal decreasing is observed into the classification of shales and slurries (i.e., from 92% to 90% and to 63% to 56%, respectively), which is acceptable according to the fscores and errors. This aspect reinforces the relevance of using the NB classifier jointly with previous geologic information to optimize the lithologic classification.

Complement-Class Harmonized Naïve Bayes Classifier

Naïve Bayes (NB) classification performance degrades if the conditional independence assumption is not satisfied or if the conditional probability estimate is not realistic due to the attributes of correlation and scarce data, respectively. Many works address these two problems, but few works tackle them simultaneously. Existing methods heuristically employ information theory or applied gradient optimization to enhance NB classification performance, however, to the best of our knowledge, the enhanced model generalization capability deteriorated especially on scant data. In this work, we propose a fine-grained boosting of the NB classifier to identify hidden and potential discriminative attribute values that lead the NB model to underfit or overfit on the training data and to enhance their predictive power. We employ the complement harmonic average of the conditional probability terms to measure their distribution divergence and impact on the classification performance for each attribute value. The proposed method is subtle yet significant enough in capturing the attribute values’ inter-correlation (between classes) and intra-correlation (within the class) and elegantly and effectively measuring their impact on the model’s performance. We compare our proposed complement-class harmonized Naïve Bayes classifier (CHNB) with the state-of-the-art Naive Bayes and imbalanced ensemble boosting methods on general and imbalanced machine-learning benchmark datasets, respectively. The empirical results demonstrate that CHNB significantly outperforms the compared methods.

Simulation studies on the estimation of sensitivity and specificity in the absence of a gold standard and with rare positive frequencies

In the absence of a gold standard, sensitivity and specificity could be evaluated using the Bayesian latent class model underlying true positives and true negatives. However, when the frequency of positivity is limited owing to low prevalence, the estimates may be strongly influenced by prior information or small-sample bias. In this study, we evaluated the performance of the Bayesian latent class model with two independent tests under a rare frequency of positivity, varying the strength and way of giving prior information under the conditional independence assumption is satisfied. Throughout the simulation experiments, a small-sample bias led to underestimation and overestimation when the frequency of positivity was low. Furthermore, we observed that placing an informative prior distribution for only one of the two tests resulted in a constant bias of sensitivity or specificity for the other test, regardless of the strength of the prior information.

Null hypothesis test for anomaly detection

We extend the use of Classification Without Labels for anomaly detection with a hypothesis test designed to exclude the background-only hypothesis. By testing for statistical independence of the two discriminating dataset regions, we are able to exclude the background-only hypothesis without relying on fixed anomaly score cuts or extrapolations of background estimates between regions. The method relies on the assumption of conditional independence of anomaly score features and dataset regions, which can be ensured using existing decorrelation techniques. As a benchmark example, we consider the LHC Olympics dataset where we show that mutual information represents a suitable test for statistical independence and our method exhibits excellent and robust performance at different signal fractions even in presence of realistic feature correlations.

Diabetes disease prediction system using HNB classifier based on discretization method.

Diagnosing diabetes early is critical as it helps patients live with the disease in a healthy way - through healthy eating, taking appropriate medical doses, and making patients more vigilant in their movements/activities to avoid wounds that are difficult to heal for diabetic patients. Data mining techniques are typically used to detect diabetes with high confidence to avoid misdiagnoses with other chronic diseases whose symptoms are similar to diabetes. Hidden Naïve Bayes is one of the algorithms for classification, which works under a data-mining model based on the assumption of conditional independence of the traditional Naïve Bayes. The results from this research study, which was conducted on the Pima Indian Diabetes (PID) dataset collection, show that the prediction accuracy of the HNB classifier achieved 82%. As a result, the discretization method increases the performance and accuracy of the HNB classifier.