Bayesian Network Classifiers Research Articles

Compiling Bayesian Network Classifiers into Decision Graphs

We propose an algorithm for compiling Bayesian network classifiers into decision graphs that mimic the input and output behavior of the classifiers. In particular, we compile Bayesian network classifiers into ordered decision graphs, which are tractable and can be exponentially smaller in size than decision trees. This tractability facilitates reasoning about the behavior of Bayesian network classifiers, including the explanation of decisions they make. Our compilation algorithm comes with guarantees on the time of compilation and the size of compiled decision graphs. We apply our compilation algorithm to classifiers from the literature and discuss some case studies in which we show how to automatically explain their decisions and verify properties of their behavior.

DNA methylation-based classification and identification of renal cell carcinoma prognosis-subgroups

BackgroundRenal cell carcinoma (RCC) is the most common kidney cancer and includes several molecular and histological subtypes with different clinical characteristics. The combination of DNA methylation and gene expression data can improve the classification of tumor heterogeneity, by incorporating differences at the epigenetic level and clinical features.MethodsIn this study, we identified the prognostic methylation and constructed specific prognosis-subgroups based on the DNA methylation spectrum of RCC from the TCGA database.ResultsSignificant differences in DNA methylation profiles among the seven subgroups were revealed by consistent clustering using 3389 CpGs that indicated that were significant differences in prognosis. The specific DNA methylation patterns reflected differentially in the clinical index, including TNM classification, pathological grade, clinical stage, and age. In addition, 437 CpGs corresponding to 477 genes of 151 samples were identified as specific hyper/hypomethylation sites for each specific subgroup. A total of 277 and 212 genes corresponding to DNA methylation at promoter sites were enriched in transcription factor of GKLF and RREB-1, respectively. Finally, Bayesian network classifier with specific methylation sites was constructed and was used to verify the test set of prognoses into DNA methylation subgroups, which was found to be consistent with the classification results of the train set. DNA methylation-based classification can be used to identify the distinct subtypes of renal cell carcinoma.ConclusionsThis study shows that DNA methylation-based classification is highly relevant for future diagnosis and treatment of renal cell carcinoma as it identifies the prognostic value of each epigenetic subtype.

Structure Learning of Bayesian Network Based on Adaptive Thresholding.

Direct dependencies and conditional dependencies in restricted Bayesian network classifiers (BNCs) are two basic kinds of dependencies. Traditional approaches, such as filter and wrapper, have proved to be beneficial to identify non-significant dependencies one by one, whereas the high computational overheads make them inefficient especially for those BNCs with high structural complexity. Study of the distributions of information-theoretic measures provides a feasible approach to identifying non-significant dependencies in batch that may help increase the structure reliability and avoid overfitting. In this paper, we investigate two extensions to the k-dependence Bayesian classifier, MI-based feature selection, and CMI-based dependence selection. These two techniques apply a novel adaptive thresholding method to filter out redundancy and can work jointly. Experimental results on 30 datasets from the UCI machine learning repository demonstrate that adaptive thresholds can help distinguish between dependencies and independencies and the proposed algorithm achieves competitive classification performance compared to several state-of-the-art BNCs in terms of 0–1 loss, root mean squared error, bias, and variance.

Young driver fatal motorcycle accident analysis by jointly maximizing accuracy and information

While young drivers (YDs) constitute ∼10% of the driver population, their fatality rate in motorcycle accidents is up to three times higher. Thus, we are interested in predicting fatal motorcycle accidents (FMAs), and in identifying their key factors and possible causes. Accurate prediction of YD FMAs from data by risk minimization using the 0/1 loss function (i.e., the ordinary classification accuracy) cannot be guaranteed because these accidents are only ∼1% of all YD motorcycle accidents, and classifiers tend to focus on the majority class of minor accidents at the expense of the minority class of fatal ones. Also, classifiers are usually uninformative (providing no information about the distribution of misclassifications), insensitive to error severity (making no distinction between misclassification of fatal accidents as severe or minor), and limited in identifying key factors. We propose to use an information measure (IM) that jointly maximizes accuracy and information and is sensitive to the error distribution and severity. Using a database of ∼3600 motorcycle accidents, a Bayesian network classifier optimized by IM predicted FMAs better than classifiers maximizing accuracy or other predictive or information measures, and identified fatal accident key factors and causal relations.

A deep neural network based classifier for brain tumor diagnosis

Classification process plays a key role in diagnosing brain tumors. Earlier research works are intended for identifying brain tumors using different classification techniques. However, the False Alarm Rates (FARs) of existing classification techniques are high. To improve the early-stage brain tumor diagnosis via classification the Weighted Correlation Feature Selection Based Iterative Bayesian Multivariate Deep Neural Learning (WCFS-IBMDNL) technique is proposed in this work. The WCFS-IBMDNL algorithm considers medical dataset for classifying the brain tumor diagnosis at an early stage. At first, the WCFS-IBMDNL technique performs Weighted Correlation-Based Feature Selection (WC-FS) by selecting subsets of medical features that are relevant for classification of brain tumors. After completing the feature selection process, the WCFS-IBMDNL technique uses Iterative Bayesian Multivariate Deep Neural Network (IBMDNN) classifier for reducing the misclassification error rate of brain tumor identification. The WCFS-IBMDNL technique was evaluated in JAVA language using Disease Diagnosis Rate (DDR), Disease Diagnosis Time (DDT), and FAR parameter through the epileptic seizure recognition dataset.

Discriminatory Target Learning: Mining Significant Dependence Relationships from Labeled and Unlabeled Data.

Machine learning techniques have shown superior predictive power, among which Bayesian network classifiers (BNCs) have remained of great interest due to its capacity to demonstrate complex dependence relationships. Most traditional BNCs tend to build only one model to fit training instances by analyzing independence between attributes using conditional mutual information. However, for different class labels, the conditional dependence relationships may be different rather than invariant when attributes take different values, which may result in classification bias. To address this issue, we propose a novel framework, called discriminatory target learning, which can be regarded as a tradeoff between probabilistic model learned from unlabeled instance at the uncertain end and that learned from labeled training data at the certain end. The final model can discriminately represent the dependence relationships hidden in unlabeled instance with respect to different possible class labels. Taking k-dependence Bayesian classifier as an example, experimental comparison on 42 publicly available datasets indicated that the final model achieved competitive classification performance compared to state-of-the-art learners such as Random forest and averaged one-dependence estimators.

Data-Driven Design Space Exploration and Exploitation for Design for Additive Manufacturing

Recently, design for additive manufacturing has been proposed to maximize product performance through the rational and integrated design of the product, its materials, and their manufacturing processes. Searching design solutions in such a multidimensional design space is a challenging task. Notably, no existing design support method is both rapid and tailored to the design process. In this study, we propose a holistic approach that applies data-driven methods in design search and optimization at successive stages of a design process. More specifically, a two-step surrogate model-based design method is proposed for the embodiment and detailed design stages. The Bayesian network classifier is used as the reasoning framework to explore the design space in the embodiment design stage, while the Gaussian process regression model is used as the evaluation function for an optimization method to exploit the design space in detailed design. These models are constructed based on one dataset that is created by the Latin hypercube sampling method and then refined by the Markov Chain Monte Carlo sampling method. This cost-effective data-driven approach is demonstrated in the design of a customized ankle brace that has a tunable mechanical performance by using a highly stretchable design concept with tailored stiffnesses.

Neo-adjuvant chemoradiotherapy response prediction using MRI based ensemble learning method in rectal cancer patients

ObjectivesThe aim of this study was to investigate and validate the performance of individual and ensemble machine learning models (EMLMs) based on magnetic resonance imaging (MRI) to predict neo-adjuvant chemoradiation therapy (nCRT) response in rectal cancer patients. We also aimed to study the effect of Laplacian of Gaussian (LOG) filter on EMLMs predictive performance. Methods98 rectal cancer patients were divided into a training (n = 53) and a validation set (n = 45). All patients underwent MRI a week before nCRT. Several features from intensity, shape and texture feature sets were extracted from MR images. SVM, Bayesian network, neural network and KNN classifiers were used individually and together for response prediction. Predictive performance was evaluated using the area under the receiver operator characteristic (ROC) curve (AUC). ResultsPatients' nCRT responses included 17 patients with Grade 0, 28 with Grade 1, 34 with Grade 2, and 19 with Grade 3 according to AJCC/CAP pathologic grading. In without preprocessing MR Image the best result was for Bayesian network classifier with AUC and accuracy of 75.2% and 80.9% respectively, which was confirmed in the validation set with an AUC and accuracy of 74% and 79% respectively. In EMLMs the best result was for 4 (SVM.NN.BN.KNN) classifier EMLM with AUC and accuracy of 97.8% and 92.8% in testing and 95% and 90% in validation set respectively. ConclusionsIn conclusion, we observed that machine learning methods can used to predict nCRT response in patients with rectal cancer. Preprocessing LOG filters and EL models can improve the prediction process.

Discriminative Structure Learning of Bayesian Network Classifiers from Training Dataset and Testing Instance.

Over recent decades, the rapid growth in data makes ever more urgent the quest for highly scalable Bayesian networks that have better classification performance and expressivity (that is, capacity to respectively describe dependence relationships between attributes in different situations). To reduce the search space of possible attribute orders, k-dependence Bayesian classifier (KDB) simply applies mutual information to sort attributes. This sorting strategy is very efficient but it neglects the conditional dependencies between attributes and is sub-optimal. In this paper, we propose a novel sorting strategy and extend KDB from a single restricted network to unrestricted ensemble networks, i.e., unrestricted Bayesian classifier (UKDB), in terms of Markov blanket analysis and target learning. Target learning is a framework that takes each unlabeled testing instance as a target and builds a specific Bayesian model Bayesian network classifiers (BNC) to complement BNC learned from training data . UKDB respectively introduced UKDB and UKDB to flexibly describe the change in dependence relationships for different testing instances and the robust dependence relationships implicated in training data. They both use UKDB as the base classifier by applying the same learning strategy while modeling different parts of the data space, thus they are complementary in nature. The extensive experimental results on the Wisconsin breast cancer database for case study and other 10 datasets by involving classifiers with different structure complexities, such as Naive Bayes (0-dependence), Tree augmented Naive Bayes (1-dependence) and KDB (arbitrary k-dependence), prove the effectiveness and robustness of the proposed approach.

An application of Bayesian Network approach for selecting energy efficient HVAC systems

The conventional approach of selecting HVAC systems is based on a designer's knowledge and experience, and this may lead to flawed decisions. The ever-growing accumulation of building performance data makes the machine learning algorithm based HVAC system selection possible. This study presents an innovative approach wherein the Bayesian Network technique is applied to select the most energy efficient primary HVAC systems. The database upon which the approach is developed is the 2012 Commercial Building Energy Consumption Survey (CBECS). The first step of this research involves clustering a group of similar buildings for the target building. Euclidean distance is adopted to calculate the similarity of a building with the target building. In the meantime, a survey is carried out to investigate the major factors that designers considered when selecting the primary HVAC system. The survey results show that climate zone, design cooling/heating load, and principal activity type are three main factors considered by designers and only 13% designers value the building energy consumption. In this study, all factors that could potentially influence the HVAC system's energy consumption are used to construct the directed acyclic graph of the proposed Bayesian Network Classifier. This classifier is trained with high energy efficient buildings data after the filtering out of irrational outliers. Then, the trained classifier is applied to select the primary cooling systems for three case buildings. The results indicate that the selected systems coincide with common HVAC design logic. The proposed method provides designers with an innovative approach to select energy efficient HVAC systems by using the data of hundreds of high energy efficient buildings. This study demonstrates the feasibility and capability of data-driven building design.

Evaluating complex relationships between ecological indicators and environmental factors in the Baltic Sea: A machine learning approach

The state of marine ecosystems is increasingly evaluated using indicators. The indicator assessment results need to be understood in the context of the whole ecosystem in order to understand the key factors determining the status of these environmental components. Data available from the system’s different components are, however, often heterogeneous: they may represent different spatial and temporal scales, and different parameters can be measured with different accuracy. This makes it difficult to evaluate the relationship between these variables and status of the environment using indicators. We studied whether probabilistic, machine learning-based classifiers could provide for assessing the relationships between multiple environmental factors and ecological indicators. This paper demonstrates the use of Bayesian network classifiers (Tree-augmented Naive Bayes classifier, TAN as the specific case example), used together with structural learning from data and Entropy Minimization Discretization (IEMD) algorithm to study environment-indicator relationships within coastal fish communities in the Baltic Sea. By using two Baltic-wide indicators of coastal fish community status and a heterogeneous set of potentially influential natural and anthropogenic variables, we explore and discuss the potential of the approach. Given pre-defined cutting points for the indicators, such as the classification thresholds of the indicator, the method enables identifying relevant variables and estimating their relative importance. This information could be used in environmental management to demonstrate at which threshold value the state of an indicator is likely to respond to a pressure or a combination of pressures. In contrast to many other multivariate statistical methodologies, the presented approach can handle missing data as well as data of varying types, from fully quantitative to presence-absence, in the same analysis.

Optimizing the Topology of Bayesian Network Classifiers by Applying Conditional Entropy to Mine Causal Relationships Between Attributes

Due to the excellent classification performance and expressivity, the study of Bayesian network classifiers (BNCs) has attracted great attention ever since the success of Naive Bayes (NB). Information theory has established mathematical basis for the rapid development of BNC. In this paper we propose the definition of entropy function H B (D), which corresponds to the optimal number of bits encoded in the network structure B and can roughly measure the amount of information implicated in training data D. Each factor in H B (D) explicitly represents statements about causal relationships. An efficient heuristic search strategy is introduced to minimize H B (D) and explore the optimal topology of BNC. Our extensive experimental evaluation on 40 datasets reveals that this out-of-core algorithm achieves competitive classification performance compared to state-of-the-art learners such as tree augmented Naive Bayes, k-dependence Bayesian classifier, support vector machine, logistic regression and neural network.

Integration of Data Technology for Analyzing University Dropout

Dropout, defined as the abandonment of a career before obtaining the corresponding degree, considering a significant time period to rule out the possibility of return. Higher education students´ dropout generates several issues that affect students and universities. The results obtained from the data provided by the Engineering departments of the University of Mumbai, in India, determine that the variables that best explain a student’s dropout are the socioeconomic factors and the income score provided by the University Admission Test (UAT). According to the decision tree technique, it is concluded that the retention is 78.3%. The quality of the classifiers allows to ensure that their predictions are correct, with statistical levels of ROC curve are 76%, 75%, and 83% successful for Bayesian network classifiers, decision tree, and neural network respectively.

Bnclassify: Learning Bayesian Network Classifiers

The bnclassify package provides state-of-the art algorithms for learning Bayesian network classifiers from data. For structure learning it provides variants of the greedy hill-climbing search, a well-known adaptation of the Chow-Liu algorithm and averaged one-dependence estimators. It provides Bayesian and maximum likelihood parameter estimation, as well as three naive-Bayes-specific methods based on discriminative score optimization and Bayesian model averaging. The implementation is efficient enough to allow for time-consuming discriminative scores on medium-sized data sets. bnclassify provides utilities for model evaluation, such as cross-validated accuracy and penalized log-likelihood scores, and analysis of the underlying networks, including network plotting via the Rgraphviz package. It is extensively tested, with over 200 automated tests that give a code coverage of 94%. Here we present the main functionalities, illustrate them with a number of data sets, and comment on related software.

Model Matching: A Novel Framework to use Clustering Strategy to Solve the Classification Problem

It is a common practice to handle labeled data with classifiers and unlabeled ones with clusterings. The traditional Bayesian network classifiers (BNC$^{\mathcal {T}}\text{s}$ ) learned from labeled training set $\mathcal {T}$ directly map the unlabeled test instance into the network structure to calculate the conditional probability for the classification, which neglects the information hidden in the unlabeled data and will result in classification bias. To address this issue, we propose a novel learning framework, called model matching, that uses the “clustering” strategy to solve the classification problem. The labeled data is divided into several clusters according to the different class label to learn a set of BNC$^{\mathcal {T}}\text{s}$ and a corresponding set of BNC$^{p}\text{s}$ is built for each unlabeled test instance. To make a classification, the cross entropy method is applied to compare the structural similarity between BNC$^{\mathcal {T}}$ and BNC p . The extensive experimental results on 46 datasets from the University of California at Irvine (UCI) machine learning repository demonstrate that for BNCs model matching helps improve the generalization performance and outperforms the several state-of-the-art classifiers like tree-augmented naive Bayes and Random forest.

Dynamic Sensor-Frontend Tuning for Resource Efficient Embedded Classification

Emerging portable applications, including human activity recognition and robot navigation, require always-on sensing technologies to continuously monitor the environment. Continuous sensing, however, strongly dominates the hardware devices’ power consumption and consequently hampers the systems’ always-on functionality. In this paper, we propose a circuit-aware machine learning scheme that exploits the devices’ ability to dynamically tune the quality of its sensors to trade-off system level accuracy versus total system-level power consumption. To this end, we: 1) analytically derive the power-quality trade-off space in sensory front-ends; 2) use these equations to make the probabilistic relations between sensory features and their degraded versions explicit in a Bayesian network classifier; and 3) propose a methodology building upon this model to control the required sensory information quality at run-time. We show how this enables to tune the circuit’s power consumption versus inference accuracy trade-off space with fine granularity, and achieve significant power savings at almost no accuracy loss. In addition, our methodology is able to cope with sensor failure and with a dynamically changing environment by means of an efficient on-line tuning strategy. This dynamic power-vs-quality scalability is empirically shown on various machine learning benchmarking data sets.

Using Kinect to classify Parkinson\u2019s disease stages related to severity of gait impairment

BackgroundParkinson’s Disease (PD) is a chronic neurodegenerative disease associated with motor problems such as gait impairment. Different systems based on 3D cameras, accelerometers or gyroscopes have been used in related works in order to study gait disturbances in PD. Kinect Ⓡ has also been used to build these kinds of systems, but contradictory results have been reported: some works conclude that Kinect does not provide an accurate method of measuring gait kinematics variables, but others, on the contrary, report good accuracy results.MethodsIn this work, we have built a Kinect-based system that can distinguish between different PD stages, and have performed a clinical study with 30 patients suffering from PD belonging to three groups: early PD patients without axial impairment, more evolved PD patients with higher gait impairment but without Freezing of Gait (FoG), and patients with advanced PD and FoG. Those patients were recorded by two Kinect devices when they were walking in a hospital corridor. The datasets obtained from the Kinect were preprocessed, 115 features identified, some methods were applied to select the relevant features (correlation based feature selection, information gain, and consistency subset evaluation), and different classification methods (decision trees, Bayesian networks, neural networks and K-nearest neighbours classifiers) were evaluated with the goal of finding the most accurate method for PD stage classification.ResultsThe classifier that provided the best results is a particular case of a Bayesian Network classifier (similar to a Naïve Bayesian classifier) built from a set of 7 relevant features selected by the correlation-based on feature selection method. The accuracy obtained for that classifier using 10-fold cross validation is 93.40%. The relevant features are related to left shin angles, left humerus angles, frontal and lateral bents, left forearm angles and the number of steps during spin.ConclusionsIn this paper, it is shown that using Kinect is adequate to build a inexpensive and comfortable system that classifies PD into three different stages related to FoG. Compared to the results of previous works, the obtained accuracy (93.40%) can be considered high. The relevant features for the classifier are: a) movement and position of the left arm, b) trunk position for slightly displaced walking sequences, and c) left shin angle, for straight walking sequences. However, we have obtained a better accuracy (96.23%) for a classifier that only uses features extracted from slightly displaced walking steps and spin walking steps. Finally, the obtained set of relevant features may lead to new rehabilitation therapies for PD patients with gait problems.

Efficient Heuristics for Structure Learning of k-Dependence Bayesian Classifier.

The rapid growth in data makes the quest for highly scalable learners a popular one. To achieve the trade-off between structure complexity and classification accuracy, the k-dependence Bayesian classifier (KDB) allows to represent different number of interdependencies for different data sizes. In this paper, we proposed two methods to improve the classification performance of KDB. Firstly, we use the minimal-redundancy-maximal-relevance analysis, which sorts the predictive features to identify redundant ones. Then, we propose an improved discriminative model selection to select an optimal sub-model by removing redundant features and arcs in the Bayesian network. Experimental results on 40 UCI datasets demonstrate that these two techniques are complementary and the proposed algorithm achieves competitive classification performance, and less classification time than other state-of-the-art Bayesian network classifiers like tree-augmented naive Bayes and averaged one-dependence estimators.

A meta‐heuristic Bayesian network classification for intrusion detection

SummarySoftware‐defined networking (SDN) is an innovative network paradigm much in demand today in academics and industry. In this network, the SDN controller must be able to observe and examine traffic flow through the network systems. However, intrusion‐based data packets affect the whole system is a major drawback. To overcome this issue, we propose a Novel Agent Program (NAP) framework for preventing switches from the external compromised attacks. A Meta‐Heuristic Bayesian Network Classification (MHBNC) algorithm for intrusion detection is proposed in this paper. The proposed algorithm follows certain procedures for preprocessing, feature selection, feature optimization, and classification. Normal and anomaly‐based data packets are classified successfully with its improved detection capabilities based on the optimization technique. The simulation results of the proposed ID_MBC (intrusion detection based on meta‐heuristic Bayesian classifier) technique is compared with existing techniques such as the association rule, PSO+GA, and the GA+RVM. The proposed MHBNC classifier performs better than existing methods.

Design Exploration of Reliably Manufacturable Materials and Structures With Applications to Negative Stiffness Metamaterials and Microstereolithography1

One of the challenges in designing metamaterials for additive manufacturing (AM) is accounting for the differences between as-designed and as-built geometries and material properties. From a designer's perspective, these differences can lead to degradation of part and metamaterial performance, which is especially difficult to accommodate in small-lot or one-of-a-kind production. In this context, each part is unique, and therefore, extensive iteration is costly. Designers need a means of exploring the design space while simultaneously considering the reliability of additively manufacturing particular candidate designs. In this work, a design exploration approach, based on Bayesian network classifiers (BNC), is extended to incorporate manufacturing variation into the design exploration process and identify designs that reliably meet performance requirements when this variation is taken into account. The example application is the design of negative stiffness (NS) metamaterials, in which small volume fractions of NS inclusions are embedded within a host material. The resulting metamaterial or composite exhibits macroscopic mechanical stiffness and loss properties that exceed those of the base matrix material. The inclusions are fabricated with microstereolithography with features on the scale of tens of microns, but variability is observed in material properties and dimensions from specimen to specimen. This variability is measured and modeled via design, fabrication, and characterization of metrology parts. The quantified manufacturing variability is incorporated into the BNC approach as a manufacturability classifier to identify candidate designs that achieve performance targets reliably, even when manufacturing variability is taken into account.