Attribute Independence Assumption Research Articles

Adaptive Two-Index Fusion Attribute-Weighted Naive Bayes

Naive Bayes (NB) is one of the essential algorithms in data mining. However, it is rarely used in reality because of the attribute independence assumption. Researchers have proposed many improved NB methods to alleviate this assumption. Among these methods, due to its high efficiency and easy implementation, the filter-attribute-weighted NB methods have received great attentions. However, there still exist several challenges, such as the poor representation ability for a single index and the fusion problem of two indexes. To overcome the above challenges, we propose a general framework of an adaptive two-index fusion attribute-weighted NB (ATFNB). Two types of data description category are used to represent the correlation between classes and attributes, the intercorrelation between attributes and attributes, respectively. ATFNB can select any one index from each category. Then, we introduce a regulatory factor β to fuse two indexes, which can adaptively adjust the optimal ratio of any two indexes on various datasets. Furthermore, a range query method is proposed to infer the optimal interval of regulatory factor β. Finally, the weight of each attribute is calculated using the optimal value β and is integrated into an NB classifier to improve the accuracy. The experimental results on 50 benchmark datasets and a Flavia dataset show that ATFNB outperforms the basic NB and state-of-the-art filter-weighted NB models. In addition, the ATFNB framework can improve the existing two-index NB model by introducing the adaptive regulatory factor β. Auxiliary experimental results demonstrate the improved model significantly increases the accuracy compared to the original model without the adaptive regulatory factor β.

Identification of informational and probabilistic independence by adaptive thresholding

The independence assumptions help Bayesian network classifier (BNC), e.g., Naive Bayes (NB), reduce structure complexity and perform surprisingly well in many real-world applications. Semi-naive Bayesian techniques seek to improve the classification performance by relaxing the attribute independence assumption. However, the study of dependence rather than independence has received more attention during the past decade and the validity of independence assumptions needs to be further explored. In this paper, a novel learning technique, called Adaptive Independence Thresholding (AIT), is proposed to automatically identify the informational independence and probabilistic independence. AIT can respectively tune the network topologies of BNC learned from training data and testing instance under the framework of target learning. Zero-one loss, bias, variance and conditional log likelihood are introduced to compare the classification performance in the experimental study. The extensive experimental results on a collection of 36 benchmark datasets from the UCI machine learning repository show that AIT is more effective than other learning techniques (such as structure extension, attribute weighting) and helps make the final BNCs achieve remarkable classification improvements.

Attribute Weighted Na飗e Bayes Classifier

The naïve Bayes classifier is one of the commonly used data mining methods for classification. Despite its simplicity, naïve Bayes is effective and computationally efficient. Although the strong attribute independence assumption in the naïve Bayes classifier makes it a tractable method for learning, this assumption may not hold in real-world applications. Many enhancements to the basic algorithm have been proposed in order to alleviate the violation of attribute independence assumption. While these methods improve the classification performance, they do not necessarily retain the mathematical structure of the naïve Bayes model and some at the expense of computational time. One approach to reduce the naïveté of the classifier is to incorporate attribute weights in the conditional probability. In this paper, we proposed a method to incorporate attribute weights to naïve Bayes. To evaluate the performance of our method, we used the public benchmark datasets. We compared our method with the standard naïve Bayes and baseline attribute weighting methods. Experimental results show that our method to incorporate attribute weights improves the classification performance compared to both standard naïve Bayes and baseline attribute weighting methods in terms of classification accuracy and F1, especially when the independence assumption is strongly violated, which was validated using the Chi-square test of independence.

Alleviating the independence assumptions of averaged one-dependence estimators by model weighting

Of numerous proposals to refine naive Bayes by weakening its attribute independence assumption, averaged one-dependence estimators (AODE) has been shown to be able to achieve significantly higher classification accuracy at a moderate cost in classification efficiency. However, all one-dependence estimators (ODEs) in AODE have the same weights and are treated equally. To address this issue, model weighting, which assigns discriminate weights to ODEs and then linearly combine their probability estimates, has been proved to be an efficient and effective approach. Most information-theoretic weighting metrics, including mutual information, Kullback-Leibler measure and the information gain, place more emphasis on the correlation between root attribute (value) and class variable. We argue that the topology of each ODE can be divided into a set of local directed acyclic graphs (DAGs) based on the independence assumption, and multivariate mutual information is introduced to measure the extent to which the DAGs fit data. Based on this premise, in this study we propose a novel weighted AODE algorithm, called AWODE, that adaptively selects weights to alleviate the independence assumption and make the learned probability distribution fit the instance. The proposed approach is validated on 40 benchmark datasets from UCI machine learning repository. The experimental results reveal that, AWODE achieves bias-variance trade-off and is a competitive alternative to single-model Bayesian learners (such as TAN and KDB) and other weighted AODEs (such as WAODE).

Fine-grained attribute weighted inverted specific-class distance measure for nominal attributes

The inverted specific-class distance measure (ISCDM) ranks first in the list of distance metrics that deal solely with nominal attributes, especially when values are missing and noise exists in non-class attributes. However, the attribute independence assumption still inevitably exists, which is almost untenable in many real-world applications with sophisticated attribute dependencies. Many improved versions based on different attribute weighting schemes have been proposed to relax this unrealistic assumption and circumvent its damage to measuring performance. However, existing attribute weighting schemes are limited to assigning a weight corresponding to each attribute; they ignore the more fine-grained dependence relationships between attributes and classes. Thus, in this study, we derive a novel fine-grained attribute weighting scheme, which first calculates the initial fine-grained attribute weights according to different attribute values and class labels and then uses random walk with restart to optimize them. We titled our improved measure fine-grained attribute-weighted ISCDM (FAWISCDM). Extensive experimental results on 66 datasets from a machine learning repository, collected by the University of California at Irvine illustrate that the FAWISCDM is notably superior to the original ISCDM and some other state-of-the-art competing methods, in terms of the negative conditional log likelihood and root relative squared error.

Gain ratio weighted inverted specific-class distance measure for nominal attributes

Enhancing distance measures is key to improving the performances of many machine learning algorithms, such as instance-based learning algorithms. Although the inverted specific-class distance measure (ISCDM) is among the top performing distance measures addressing nominal attributes with the presence of missing values and non-class attribute noise in the training set, this still requires the attribute independence assumption. It is obvious that the attribute independence assumption required by the ISCDM is rarely true in reality, which harms its performance in applications with complex attribute dependencies. Thus, in this study we propose an improved ISCDM by utilizing attribute weighting to circumvent the attribute independence assumption. In our improved ISCDM, we simply define the weight of each attribute as its gain ratio. Thus, we denote our improved ISCDM as the gain ratio weighted ISCDM (GRWISCDM for short). We tested the GRWISCDM experimentally on 29 University of California at Irvine datasets, and found that it significantly outperforms the original ISCDM and some other state-of-the-art competitors in terms of the negative conditional log likelihood and root relative squared error.

Self-Adaptive Attribute Value Weighting for Averaged One-Dependence Estimators

Of numerous proposals for weakening the attribute independence assumption of Naive Bayes, averaged one-dependence estimators (AODE) learns by extrapolation from marginal to full-multivariate probability distributions, and has demonstrated reasonable improvement in terms of classification performance. However, all the one-dependence estimators in AODE are assigned with the same weight, and their probability estimates are combined linearly. This work presents an efficient and effective attribute value weighting approach that assigns discriminative weights to different super-parent one-dependence estimators for different instances by identifying the differences among these one-dependence estimators in terms of log likelihood. The proposed approach is validated on widely used benchmark datasets from UCI machine learning repository. Experimental results show that the proposed approach achieves bias-variance trade-off and is a competitive alternative to state-of-the-art Bayesian and non-Bayesian learners (e.g., tree augmented Naive Bayes and logistic regression).

Averaged one-dependence inverted specific-class distance measure for nominal attributes

ABSTRACTScores of machine learning algorithms depend on a good distance measure to achieve high performance. The inverted specific-class distance measure, simply ISCDM, is proposed to find reasonable distance measure between each pair of instances with nominal attributes only. ISCDM does not depend on the attribute value of the training instance, which makes it less sensitive to missing values in the training set and more robust to non-class attribute noise. However, in ISCDM, all attributes are assumed to be fully independent. It is obvious that the attribute independence assumption in ISCDM is rarely true in reality, which would harm its performance in the applications with complex attribute dependencies. In this paper, we single out an improved inverted specific-class distance measure by relaxing its unrealistic attribute independence assumption. We call it averaged one-dependence inverted specific-class distance measure, simply AODISCDM. We experimentally tested it on 29 classification problems from the University of California at Irvine (UCI) repository and found that it significantly outperforms ISCDM in terms of the negative conditional log likelihood (-CLL) and the root relative squared error (RRSE). Besides, the proposed AODISCDM maintains the computational simplicity (no search involved) and robustness that characterise ISCDM.

Structure Extension of Tree-Augmented Naive Bayes.

Due to the simplicity and competitive classification performance of the naive Bayes (NB), researchers have proposed many approaches to improve NB by weakening its attribute independence assumption. Through the theoretical analysis of Kullback–Leibler divergence, the difference between NB and its variations lies in different orders of conditional mutual information represented by these augmenting edges in the tree-shaped network structure. In this paper, we propose to relax the independence assumption by further generalizing tree-augmented naive Bayes (TAN) from 1-dependence Bayesian network classifiers (BNC) to arbitrary k-dependence. Sub-models of TAN that are built to respectively represent specific conditional dependence relationships may “best match” the conditional probability distribution over the training data. Extensive experimental results reveal that the proposed algorithm achieves bias-variance trade-off and substantially better generalization performance than state-of-the-art classifiers such as logistic regression.

A systematic literature review on attribute independent assumption of Naive Bayes: research trend, datasets, methods and frameworks

Recent studies of attribute independent assumptions on Naïve Bayes (NB) typically generate data sets, methods and frameworks that enable researchers to focus on development activities in terms of finding solutions to attribute independent issues, thereby enhancing the quality of NB classification and better utilizing resources. Many data sets deal with the NB attribute independence issues and different frameworks, so the overall picture of the independent assumption of the current NB attribute is not yet complete. This literature review aims to identify and analyze the research trends, data sets, methods and frameworks used in the attribute independence assumption research on NB for data classification between 2010 and 2018. The results of this research identified three frameworks that are highly cited and therefore influential in the software defect prediction field. They are Langley and Sage Framework, Friedman et al. Framework, and Wu et al. Framework

Two improved attribute weighting schemes for value difference metric

Due to its simplicity, efficiency and efficacy, value difference metric (VDM) has continued to perform well against more sophisticated newcomers and thus has remained of great interest to the distance metric learning community. Of numerous approaches to improving VDM by weakening its attribute independence assumption, attribute weighting has received less attention (only two attribute weighting schemes) but demonstrated remarkable class probability estimation performance. Among two existing attribute weighting schemes, one is non-symmetric and the other is symmetric. In this paper, we propose two simple improvements for setting attribute weights for use with VDM. One is the non-symmetric Kullback–Leibler divergence weighted value difference metric (KLD-VDM) and the other is the symmetric gain ratio weighted value difference metric (GR-VDM). We performed extensive evaluations on a large number of datasets and found that KLD-VDM and GR-VDM significantly outperform two existing attribute weighting schemes in terms of the negative conditional log likelihood and root relative squared error, yet at the same time maintain the computational simplicity and robustness that characterize VDM.

Exploiting efficient and effective lazy Semi-Bayesian strategies for text classification

Abstract Automatic Document Classification (ADC) has become the basis of many important applications, e.g., authorship identification, opinion mining, spam filtering, content organizers, etc. Due to their simplicity, efficiency, absence of parameters, and effectiveness in several scenarios, Naive Bayes (NB) approaches are widely used as a classification paradigm. Due to some characteristics of real document collections, e.g., class imbalance and feature sparseness, NB solutions do not present competitive effectiveness in some ADC tasks when compared to other supervised learning strategies, e.g., SVMs. In this article, we investigate whether a proper combination of some alternative NB learning models with different feature weighting techniques is able to improve the NB effectiveness in ADC tasks and verify that comparable or even superior results when compared to the state-of-the-art in ADC can be achieved. Moreover, we also present an investigation on the relaxation of the NB attribute independence assumption (aka, Semi-Naive approaches) in large text collections, something missing in the literature. Given the high computational costs of these investigations, we take advantage of current many core GPU and multi-GPU architectures to perform such investigation, presenting a massively parallelized version of the NB approach. Finally, supported by the parallel implementations, we propose four novel Lazy Semi-NB approaches to overcome potential overfitting problems. In our experiments, the new lazy solutions are not only more efficient and effective than existing Semi-NB approaches, but also surpass, in terms of effectiveness, all other alternatives in the majority of the cases.

Alleviating naive Bayes attribute independence assumption by attribute weighting

Despite the simplicity of the Naive Bayes classifier, it has continued to perform well against more sophisticated newcomers and has remained, therefore, of great interest to the machine learning community. Of numerous approaches to refining the naive Bayes classifier, attribute weighting has received less attention than it warrants. Most approaches, perhaps influenced by attribute weighting in other machine learning algorithms, use weighting to place more emphasis on highly predictive attributes than those that are less predictive. In this paper, we argue that for naive Bayes attribute weighting should instead be used to alleviate the conditional independence assumption. Based on this premise, we propose a weighted naive Bayes algorithm, called WANBIA, that selects weights to minimize either the negative conditional log likelihood or the mean squared error objective functions. We perform extensive evaluations and find that WANBIA is a competitive alternative to state of the art classifiers like Random Forest, Logistic Regression and A1DE.

Attribute Value Weighted Average of One-Dependence Estimators

Of numerous proposals to improve the accuracy of naive Bayes by weakening its attribute independence assumption, semi-naive Bayesian classifiers which utilize one-dependence estimators (ODEs) have been shown to be able to approximate the ground-truth attribute dependencies; meanwhile, the probability estimation in ODEs is effective, thus leading to excellent performance. In previous studies, ODEs were exploited directly in a simple way. For example, averaged one-dependence estimators (AODE) weaken the attribute independence assumption by directly averaging all of a constrained class of classifiers. However, all one-dependence estimators in AODE have the same weights and are treated equally. In this study, we propose a new paradigm based on a simple, efficient, and effective attribute value weighting approach, called attribute value weighted average of one-dependence estimators (AVWAODE). AVWAODE assigns discriminative weights to different ODEs by computing the correlation between the different root attribute value and the class. Our approach uses two different attribute value weighting measures: the Kullback–Leibler (KL) measure and the information gain (IG) measure, and thus two different versions are created, which are simply denoted by AVWAODE-KL and AVWAODE-IG, respectively. We experimentally tested them using a collection of 36 University of California at Irvine (UCI) datasets and found that they both achieved better performance than some other state-of-the-art Bayesian classifiers used for comparison.

Toward value difference metric with attribute weighting

In distance metric learning, recent work has shown that value difference metric (VDM) with a strong attribute independence assumption outperforms other existing distance metrics. However, an open q...

Structure extended multinomial naive Bayes

Multinomial naive Bayes (MNB) assumes that all attributes (i.e., features) are independent of each other given the context of the class, and it ignores all dependencies among attributes. However, in many real-world applications, the attribute independence assumption required by MNB is often violated and thus harms its performance. To weaken this assumption, one of the most direct ways is to extend its structure to represent explicitly attribute dependencies by adding arcs between attributes. On the other hand, although a Bayesian network can represent arbitrary attribute dependencies, learning an optimal Bayesian network from high-dimensional text data is almost impossible. The main reason is that learning the optimal structure of a Bayesian network from high-dimensional text data is extremely time and space consuming. Thus, it would be desirable if a multinomial Bayesian network model can avoid structure learning and be able to represent attribute dependencies to some extent. In this paper, we propose a novel model called structure extended multinomial naive Bayes (SEMNB). SEMNB alleviates the attribute independence assumption by averaging all of the weighted one-dependence multinomial estimators. To learn SEMNB, we propose a simple but effective learning algorithm without structure searching. The experimental results on a large suite of benchmark text datasets show that SEMNB significantly outperforms MNB and is even markedly better than other three state-of-the-art improved algorithms including TDM, DWMNB, and Rw, cMNB.

SODE: Self-Adaptive One-Dependence Estimators for classification

SuperParent-One-Dependence Estimators (SPODEs) represent a family of semi-naive Bayesian classifiers which relax the attribute independence assumption of Naive Bayes (NB) to allow each attribute to depend on a common single attribute (superparent). SPODEs can effectively handle data with attribute dependency but still inherent NB׳s key advantages such as computational efficiency and robustness for high dimensional data. In reality, determining an optimal superparent for SPODEs is difficult. One common approach is to use weighted combinations of multiple SPODEs, each having a different superparent with a properly assigned weight value (i.e., a weight value is assigned to each attribute). In this paper, we propose a self-adaptive SPODEs, namely SODE, which uses immunity theory in artificial immune systems to automatically and self-adaptively select the weight for each single SPODE. SODE does not need to know the importance of individual SPODE nor the relevance among SPODEs, and can flexibly and efficiently search optimal weight values for each SPODE during the learning process. Extensive experiments and comparisons on 56 benchmark data sets, and validations on image and text classification, demonstrate that SODE outperforms state-of-the-art weighted SPODE algorithms and is suitable for a wide range of learning tasks. Results also confirm that SODE provides an appropriate balance between runtime efficiency and accuracy.

Not always simple classification: Learning SuperParent for class probability estimation

Of numerous proposals to improve naive Bayes (NB) by weakening its attribute independence assumption, SuperParent (SP) has demonstrated remarkable classification performance. In many real-world applications, however, accurate class probability estimation of instances is more desirable than simple classification. For example, we often need to recommend commodities to customers with the higher likelihood (class probability) of purchase. Conditional log likelihood (CLL) is currently a well-accepted measure for the quality of class probability estimation. Inspired by this, in this paper, we firstly investigate the class probability estimation performance of SP in terms of CLL and find that its class probability estimation performance almost ties the original distribution-based tree augmented naive Bayes (TAN). In order to scale up its class probability estimation performance, we then propose an improved CLL-based SuperParent algorithm (CLL-SP). In CLL-SP, a CLL-based approach, instead of a classification-based approach, is used to find the augmenting arcs. The experimental results on a large suite of benchmark datasets show that our CLL-based approach (CLL-SP) significantly outperforms the classification-based approach (SP) and the original distribution-based approach (TAN) in terms of CLL, yet at the same time maintains the high classification accuracy that characterizes the classification-based approach (SP).

Adapting naive Bayes tree for text classification

Naive Bayes (NB) is one of the top 10 algorithms thanks to its simplicity, efficiency, and interpretability. To weaken its attribute independence assumption, naive Bayes tree (NBTree) has been prop...

Selective Value Difference Metric

Value Difference Metric (VDM) is one of the widely used distance functions to define the distance between a pair of instances with nominal attributes only. Many approaches have been proposed to improve the performance of VDM. In this paper, we focus on the attribute selection approach and propose another improved Value Difference Metric. We call it Selective Value Difference Metric (SVDM). In order to learn SVDM, we investigate the attribute independence assumption held by VDM and then single out two effective attribute selection methods for SVDM. The experimental results on 36 UCI benchmark datasets validate the effectiveness of the proposed SVDM.