Conditional Independence Assumption Research Articles

A Flexible Probabilistic Framework With Concurrent Analysis of Continuous and Categorical Data for Industrial Fault Detection and Diagnosis

Operational data from industrial processes typically consist of continuous and categorical variables. They reveal different aspects of the operational conditions, and both are useful for comprehensive fault detection and diagnosis (FDD). However, due to the multiple production modes, the variables usually do not follow Gaussian or Bernoulli distributions. Furthermore, their correlations can be different across normal and various faulty classes. Thus, the main challenge lies in how to properly fuse the complementary information in the two types of variables and accurately characterize the complicated distribution for each class. This work proposes a flexible probabilistic framework that can concurrently analyze continuous and categorical variables for FDD. Our framework specifies a finite mixture model for each class. Thus, it can handle non-Gaussian and non-Bernoulli variables and capture their correlations under the conditional independence assumption. We then introduce the variational inference for parameter estimation, which makes our framework adaptive to the different distributions of various classes. Furthermore, a unified statistical index is designed for two types of variables, giving our method extra capability to distinguish unknown faults. Finally, the effectiveness of our method is validated on the Tennessee Eastman (TE) process and a practical industrial plant process. The average <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">F1</i> score of our method is improved by 3.4/3.9 percentage points in the single-mode/multimode situation on the TE process compared with traditional mixture discriminant analysis. In the industrial plant process, when unknown faults are added, the average <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">F1</i> score of our method is 91.1% and only 1.2 percentage points lower than that on the test set without unknown faults.

Testing conditional independence in casual inference for time series data

In this paper, we propose a new procedure to test conditional independence assumption in studying casual inference for time series data. The conditional independence assumption is transformed to a nonparametric conditional moment test with the help of auxiliary variables which are allowed to affect policy choice but the dependence can be fully captured by potential outcomes and observable controls. When the policy choice is binary, a nonparametric statistic test is developed further for testing the conditional independence assumption conditional on policy propensity score. Under some regular conditions, we show that the proposed test statistics are asymptotically normal under the null hypotheses for time series data. In addition, the performances of the proposed methods are illustrated through Monte Carlo simulations and a real example considered in Angrist and Kuersteiner (2011).

A Bayesian latent class extension of naive Bayesian classifier and its application to the classification of gastric cancer patients

BackgroundThe Naive Bayes (NB) classifier is a powerful supervised algorithm widely used in Machine Learning (ML). However, its effectiveness relies on a strict assumption of conditional independence, which is often violated in real-world scenarios. To address this limitation, various studies have explored extensions of NB that tackle the issue of non-conditional independence in the data. These approaches can be broadly categorized into two main categories: feature selection and structure expansion.In this particular study, we propose a novel approach to enhancing NB by introducing a latent variable as the parent of the attributes. We define this latent variable using a flexible technique called Bayesian Latent Class Analysis (BLCA). As a result, our final model combines the strengths of NB and BLCA, giving rise to what we refer to as NB-BLCA. By incorporating the latent variable, we aim to capture complex dependencies among the attributes and improve the overall performance of the classifier.MethodsBoth Expectation-Maximization (EM) algorithm and the Gibbs sampling approach were offered for parameter learning. A simulation study was conducted to evaluate the classification of the model in comparison with the ordinary NB model. In addition, real-world data related to 976 Gastric Cancer (GC) and 1189 Non-ulcer dyspepsia (NUD) patients was used to show the model's performance in an actual application. The validity of models was evaluated using the 10-fold cross-validation.ResultsThe presented model was superior to ordinary NB in all the simulation scenarios according to higher classification sensitivity and specificity in test data. The NB-BLCA model using Gibbs sampling accuracy was 87.77 (95% CI: 84.87-90.29). This index was estimated at 77.22 (95% CI: 73.64-80.53) and 74.71 (95% CI: 71.02-78.15) for the NB-BLCA model using the EM algorithm and ordinary NB classifier, respectively.ConclusionsWhen considering the modification of the NB classifier, incorporating a latent component into the model offers numerous advantages, particularly within medical and health-related contexts. By doing so, the researchers can bypass the extensive search algorithm and structure learning required in the local learning and structure extension approach. The inclusion of latent class variables allows for the integration of all attributes during model construction. Consequently, the NB-BLCA model serves as a suitable alternative to conventional NB classifiers when the assumption of independence is violated, especially in domains pertaining to health and medicine.

Evaluating the effectiveness of ground motion intensity measures through the lens of causal inference

AbstractWhen solving the performance‐based earthquake engineering (PBEE) integral, the engineering demand parameters (EDPs) are assumed to be independent of the “upstream” parameters used in ground motion models (e.g., magnitude and source‐to‐site distance) after conditioning on the intensity measure (IM). This paper formulates a methodology for evaluating this conditional independence assumption through the lens of causal inference (CI). From a causal perspective, the effect of an upstream parameter on the EDP of interest is partially mediated by the IM with the remainder having a direct influence on the EDP. An IM is judged to be desirable (from a conditional independence standpoint) if the mediated effect of the upstream parameter is maximized, which implies that the direct effect is minimized. In the language of causal inference (CI), the IM, EDP and upstream parameters are described as the “treatment”, “outcome” and “confounding” variables, respectively. It then follows that the best performing IM is the one that maximizes the effect on the EDP after controlling for the upstream parameters. A semi‐parametric model that employs double machine learning is used to estimate the causal effect. The methodology is demonstrated through a case study application utilizing the responses from five special steel moment frames analyzed using 240 site‐agnostic ground motions. Compared to sufficiency‐based assessments, the casual inference method produces findings that are more explainable using structural dynamics principles and invariant to the number of ground motions used in the evaluation.

A design utility approach for preferentially sampled spatial data

Abstract Spatial preferential sampling occurs when the choice of sampling locations depends stochastically on the process of interest. Ignoring this dependence leads to inaccurate inferences. Our framework models experimenter preferences jointly with the spatial process to adjust for this. We dispense with the unrealistic assumption (required by existing methods) of conditional independence of sampling locations by defining a whole design distribution proportional to a utility function on the space of designs. The proposed model likelihood is generally intractable. We provide fitting techniques based on the noisy Markov chain Monte Carlo and demonstrate their usage on a data set of spatially distributed ammonia concentrations.

A Latent Space Diffusion Item Response Theory Model to Explore Conditional Dependence between Responses and Response Times.

Traditional measurement models assume that all item responses correlate with each other only through their underlying latent variables. This conditional independence assumption has been extended in joint models of responses and response times (RTs), implying that an item has the same item characteristics fors all respondents regardless of levels of latent ability/trait and speed. However, previous studies have shown that this assumption is violated in various types of tests and questionnaires and there are substantial interactions between respondents and items that cannot be captured by person- and item-effect parameters in psychometric models with the conditional independence assumption. To study the existence and potential cognitive sources of conditional dependence and utilize it to extract diagnostic information for respondents and items, we propose a diffusion item response theory model integrated with the latent space of variations in information processing rate of within-individual measurement processes. Respondents and items are mapped onto the latent space, and their distances represent conditional dependence and unexplained interactions. We provide three empirical applications to illustrate (1) how to use an estimated latent space to inform conditional dependence and its relation to person and item measures, (2) how to derive diagnostic feedback personalized for respondents, and (3) how to validate estimated results with an external measure. We also provide a simulation study to support that the proposed approach can accurately recover its parameters and detect conditional dependence underlying data.

Accurate Summary-based Cardinality Estimation Through the Lens of Cardinality Estimation Graphs

We study two classes of summary-based cardinality estimators that use statistics about input relations and small-size joins: (i) optimistic estimators, which were defined in the context of graph database management systems, that make uniformity and conditional independence assumptions; and (ii) the recent pessimistic estimators that use information theoretic linear programs (LPs). We show that optimistic estimators can be modeled as picking bottom-to-top paths in a cardinality estimation graph (CEG), which contains subqueries as nodes and edges whose weights are average degree statistics. We show that existing optimistic estimators have either undefined or fixed choices for picking CEG paths as their estimates and ignore alternative choices. Instead, we outline a space of optimistic estimators to make an estimate on CEGs, which subsumes existing estimators. We show, using an extensive empirical analysis, that effective paths depend on the structure of the queries. We next show that optimistic estimators and seemingly disparate LP-based pessimistic estimators are in fact connected. Specifically, we show that CEGs can also model some recent pessimistic estimators. This connection allows us to provide insights into the pessimistic estimators, such as showing that they have combinatorial solutions.

Validity of Food insecurity experience scale (FIES) for use in rural Bangladesh and prevalence and determinants of household food insecurity: An analysis of data from Bangladesh integrated household survey (BIHS) 2018-2019

“Access” dimension of Food insecurity (FI) is directly measured by the Food Insecurity Experience Scale (FIES). The current study assessed the appropriateness of the FIES for measuring FI in rural Bangladesh, followed by an assessment of FI prevalence and its correlates utilizing Bangladesh Integrated Household Survey (BIHS) data. The internal validity of the FIES and the prevalence of FI were investigated using the Rasch modeling approach. We utilized equating procedure to calibrate the study’s result to the global FIES reference scale and determined FI prevalence rates that were comparable across countries. The external validity of the FIES was evaluated by examining its association with other FI measures using Spearman’s rho correlation analysis. With an overall Rasch reliability of 0.84, the FIES met the Rasch model assumptions of conditional independence and equal discrimination, and as well as the fit statistics standards for all eight items. Infit statistics were within the allowed limit for all FIES items indicating good internal validity. However, we noted a high outfit (>2) for the “unable to eat healthy and nutritious food” item indicating the presence of some unusual response patterns. Our analysis found no significant (>0.4) correlation between FIES items. We also found a significant correlation between FIES and other FI proxies, e.g., the Household hunger scale (HHS), Food consumption score (FCS), and Household dietary diversity score (HDDS). Overall, the prevalence of moderate or severe FI was 18.92% in rural Bangladesh. Geographic areas, access to electricity, household ownership, access to sanitation, livestock ownership, family size, education level, and monthly per capita food expenditure significantly explained the variation in FI. Our analyses suggest that the FIES is internally and externally valid for FI measurement in rural Bangladesh. However, FIES questions may need to be reordered to more accurately evaluate lower levels of FI, and the item “unable to eat healthy and nutritious food” may need cognitive testing.

Analysis of alternative strategies applied to Naïve-Bayes classifier into the recognition of electrofacies: Application in well-log data at Recôncavo Basin, North-East Brazil

This paper is concerned with the applicability of different strategies to improve the definition of prior probabilities and/or likelihoods of naïve Bayes (NB) classifiers. Standard NB method computes likelihoods and priors by means of normal distributions and evaluation of the entire training data-set, respectively. NB is one of the most prolific classification methods in data mining and machine learning. Despite decent efficiency facing good training data, NB classifiers present the intriguing assumption of conditional independence between the attributes. Several algorithms have been proposed to improve the effectiveness of NB classifiers by inserting discriminant approaches into its generative structure. From a reliability perspective, the standard NB approach might not explore the real capabilities of NB classifiers facing the lithologic classification problem. To cover such distrust, a novel approach considering four particular strategies are suggested and compared to the standard NB classification outcomes. At first, a kernel density estimation (KDE) is considered to ameliorate the likelihood models. We also apply the NB classifier in parts by separating the training data-set in individual wells in a committee architecture framework. A tuning strategy is also considered for automatic estimation of prior probabilities in an optimization-scheme. Another novel alternative, named as CRC, adapted to the standard NB classifier consists in defining priors based on depth zones from regional stratigraphic information in which to apply NB classifier. We prepare an extensive statistical investigation, based on precision, recall, classification errors, fscores and confusion matrices to bespeak the most relevant NB strategy for classification of electrofacies. Despite the decent classification outcomes for all above-mentioned strategies, CRC can be considered, by a narrow margin, the most prolific method to be applied as an improvement of the standard NB to classify rock units. Tests are performed on a validation well (i.e., 7-MP-50D-BA) of Massapê Field, in Recôncavo Basin, northeast Brazil to highlight the classification particularities provided by the improved strategy. A significant improvement in the classification of sandstones (i.e., from 68 % to 83 % accuracy) is observed, according to the confusion matrix analysis. Additionally, a minimal decreasing is observed into the classification of shales and slurries (i.e., from 92% to 90% and to 63% to 56%, respectively), which is acceptable according to the fscores and errors. This aspect reinforces the relevance of using the NB classifier jointly with previous geologic information to optimize the lithologic classification.

Complement-Class Harmonized Naïve Bayes Classifier

Naïve Bayes (NB) classification performance degrades if the conditional independence assumption is not satisfied or if the conditional probability estimate is not realistic due to the attributes of correlation and scarce data, respectively. Many works address these two problems, but few works tackle them simultaneously. Existing methods heuristically employ information theory or applied gradient optimization to enhance NB classification performance, however, to the best of our knowledge, the enhanced model generalization capability deteriorated especially on scant data. In this work, we propose a fine-grained boosting of the NB classifier to identify hidden and potential discriminative attribute values that lead the NB model to underfit or overfit on the training data and to enhance their predictive power. We employ the complement harmonic average of the conditional probability terms to measure their distribution divergence and impact on the classification performance for each attribute value. The proposed method is subtle yet significant enough in capturing the attribute values’ inter-correlation (between classes) and intra-correlation (within the class) and elegantly and effectively measuring their impact on the model’s performance. We compare our proposed complement-class harmonized Naïve Bayes classifier (CHNB) with the state-of-the-art Naive Bayes and imbalanced ensemble boosting methods on general and imbalanced machine-learning benchmark datasets, respectively. The empirical results demonstrate that CHNB significantly outperforms the compared methods.

Simulation studies on the estimation of sensitivity and specificity in the absence of a gold standard and with rare positive frequencies

In the absence of a gold standard, sensitivity and specificity could be evaluated using the Bayesian latent class model underlying true positives and true negatives. However, when the frequency of positivity is limited owing to low prevalence, the estimates may be strongly influenced by prior information or small-sample bias. In this study, we evaluated the performance of the Bayesian latent class model with two independent tests under a rare frequency of positivity, varying the strength and way of giving prior information under the conditional independence assumption is satisfied. Throughout the simulation experiments, a small-sample bias led to underestimation and overestimation when the frequency of positivity was low. Furthermore, we observed that placing an informative prior distribution for only one of the two tests resulted in a constant bias of sensitivity or specificity for the other test, regardless of the strength of the prior information.

Null hypothesis test for anomaly detection

We extend the use of Classification Without Labels for anomaly detection with a hypothesis test designed to exclude the background-only hypothesis. By testing for statistical independence of the two discriminating dataset regions, we are able to exclude the background-only hypothesis without relying on fixed anomaly score cuts or extrapolations of background estimates between regions. The method relies on the assumption of conditional independence of anomaly score features and dataset regions, which can be ensured using existing decorrelation techniques. As a benchmark example, we consider the LHC Olympics dataset where we show that mutual information represents a suitable test for statistical independence and our method exhibits excellent and robust performance at different signal fractions even in presence of realistic feature correlations.

Diabetes disease prediction system using HNB classifier based on discretization method.

Diagnosing diabetes early is critical as it helps patients live with the disease in a healthy way - through healthy eating, taking appropriate medical doses, and making patients more vigilant in their movements/activities to avoid wounds that are difficult to heal for diabetic patients. Data mining techniques are typically used to detect diabetes with high confidence to avoid misdiagnoses with other chronic diseases whose symptoms are similar to diabetes. Hidden Naïve Bayes is one of the algorithms for classification, which works under a data-mining model based on the assumption of conditional independence of the traditional Naïve Bayes. The results from this research study, which was conducted on the Pima Indian Diabetes (PID) dataset collection, show that the prediction accuracy of the HNB classifier achieved 82%. As a result, the discretization method increases the performance and accuracy of the HNB classifier.

Improving Statistical Matching when Auxiliary Information is Available

Abstract There is growing interest within National Statistical Institutes in combining available datasets containing information on a large variety of social domains. Statistical matching approaches can be used to integrate data sources through a common set of variables where each dataset contains different units that belong to the same target population. However, a common problem is related to the assumption of conditional independence among variables observed in different data sources. In this context, an auxiliary dataset containing all the variables jointly can be used to improve the statistical matching by providing information on the correlation structure of variables observed across different datasets. We propose modifying the prediction models from the auxiliary dataset through a calibration step and show that we can improve the outcome of statistical matching in a variety of settings. We evaluate the proposed approach via simulation and an application based on the European Union Statistics for Income and Living Conditions and Living Costs and Food Survey for the United Kingdom.

A Gaussian copula joint model for longitudinal and time-to-event data with random effects

Longitudinal and survival sub-models are two building blocks for joint modelling of longitudinal and time-to-event data. Extensive research indicates separate analysis of these two processes could result in biased outputs due to their associations. Conditional independence between measurements of biomarkers and event time process given latent classes or random effects is a conventional approach for characterising the association between the two sub-models while taking the heterogeneity among the population into account. However, this assumption is difficult to validate because of the unobservable latent variables. Thus a Gaussian copula joint model with random effects is proposed to accommodate the scenarios where the conditional independence assumption is questionable. The conventional joint model assuming conditional independence is a special case of the proposed model when the association parameters in the Gaussian copula shrink to zero. Simulation studies and real data application are carried out to evaluate the performance of the proposed model with different correlation structures. In addition, personalised dynamic predictions of survival probabilities are obtained based on the proposed model and comparisons are made to the predictions obtained under the conventional joint model.

Probabilistic Overview of Probabilities of Default for Low Default Portfolios by K. Pluto and D. Tasche

This article gives a probabilistic overview of the widely used method of default probability estimation proposed by K. Pluto and D. Tasche. There are listed detailed assumptions and derivation of the inequality where the probability of default is involved under the influence of systematic factor. The author anticipates adding more clarity, especially for early career analysts or scholars, regarding the assumption of borrowers' independence, conditional independence, and interaction between the probability distributions such as binomial, beta, normal, and others. There is also shown the relation between the probability of default and the joint distribution of \sqrt{ϱ}X - \sqrt{1-ϱ}Y, where X, including but not limiting, is the standard normal, Y admits, including but not limiting, the beta-normal distribution and X, Y are independent.

Are Girls' and Boys' Cognitive Test Performance in Adolescence Differently Affected by Deprivation at Earlier Ages?*

AbstractUsing data on the Millennium Children from the Young Lives Survey in Ethiopia, India, Peru, and Vietnam, we find that earlier nutritional growth and household wealth are important predictors of adolescent outcomes in math, reading, and receptive vocabulary for all children. Gender differences in the effect of wealth are significant mostly for non‐poor regions. The cognitive outcomes at age 8 are more strongly associated with growth between ages 1 and 5 for girls than boys. The gender differences reverse after age 8 mostly due to strong associations between growth in preadolescence ages and cognitive outcomes at age 15 for boys. Under the conditional mean independence assumption, the estimators for growth of the children are unbiased and consistent.

Measurement Systems

Economic models often depend on quantities that are unobservable, either for privacy reasons or because they are difficult to measure. Examples of such variables include human capital (or ability), personal income, unobserved heterogeneity (such as consumer “types”), et cetera. This situation has historically been handled either by simply using observable imperfect proxies for each of the unobservables, or by assuming that such unobservables satisfy convenient conditional mean or independence assumptions that enable their elimination from the estimation problem. However, thanks to tremendous increases in both the amount of data available and computing power, it has become possible to take full advantage of recent formal methods to infer the statistical properties of unobservable variables from multiple imperfect measurements of them. The general framework used is the concept of measurement systems in which a vector of observed variables is expressed as a (possibly nonlinear or nonparametric) function of a vector of all unobserved variables (including unobserved error terms or “disturbances” that may have nonadditively separable affects). The framework emphasizes important connections with related fields, such as nonlinear panel data, limited dependent variables, game theoretic models, dynamic models, and set identification. This review reports the progress made toward the central question of whether there exist plausible assumptions under which one can identify the joint distribution of the unobservables from the knowledge of the joint distribution of the observables. It also overviews empirical efforts aimed at exploiting such identification results to deliver novel findings that formally account for the unavoidable presence of unobservables. (JEL C30, C55, C57, D12, E21, E23, J24)

Rethinking the Statistical Analysis of Neuromechanical Data.

Researchers in neuromechanics should upgrade their statistical toolbox. We propose linear mixed-effects models in place of commonly used statistical tests to better capture subject-specific baselines and treatment-associated effects that naturally occur in neuromechanics. Researchers can use this approach to handle sporadic missing data, avoid the assumption of conditional independence in observations, and successfully model complex experimental protocols.

Score Test for Assessing the Conditional Dependence in Latent Class Models and its Application to Record Linkage

AbstractThe Fellegi–Sunter model has been widely used in probabilistic record linkage despite its often invalid conditional independence assumption. Prior research has demonstrated that conditional dependence latent class models yield improved match performance when using the correct conditional dependence structure. With a misspecified conditional dependence structure, these models can yield worse performance. It is, therefore, critically important to correctly identify the conditional dependence structure. Existing methods for identifying the conditional dependence structure include the correlation residual plot, the log-odds ratio check, and the bivariate residual, all of which have been shown to perform inadequately. Bootstrap bivariate residual approach and score test have also been proposed and found to have better performance, with the score test having greater power and lower computational burden. In this paper, we extend the score-test-based approach to account for different conditional dependence structures. Through a simulation study, we develop practical recommendations on the utilisation of the score test and assess the match performance with conditional dependence identified by the proposed method. Performance of the proposed method is further evaluated using a real-world record linkage example. Findings show that the proposed method leads to improved matching accuracy relative to the Fellegi–Sunter model.