Classifier Chains Research Articles

Tree-based dynamic classifier chains

Classifier chains are an effective technique for modeling label dependencies in multi-label classification. However, the method requires a fixed, static order of the labels. While in theory, any order is sufficient, in practice, this order has a substantial impact on the quality of the final prediction. Dynamic classifier chains denote the idea that for each instance to classify, the order in which the labels are predicted is dynamically chosen. The complexity of a naïve implementation of such an approach is prohibitive, because it would require to train a sequence of classifiers for every possible permutation of the labels. To tackle this problem efficiently, we propose a new approach based on random decision trees which can dynamically select the label ordering for each prediction. We show empirically that a dynamic selection of the next label improves over the use of a static ordering under an otherwise unchanged random decision tree model. In addition, we also demonstrate an alternative approach based on extreme gradient boosted trees, which allows for a more target-oriented training of dynamic classifier chains. Our results show that this variant outperforms random decision trees and other tree-based multi-label classification methods. More importantly, the dynamic selection strategy allows to considerably speed up training and prediction.

Multilabel Classification Methods for Human Activity Recognition: A Comparison of Algorithms.

As the world’s population is aging, and since access to ambient sensors has become easier over the past years, activity recognition in smart home installations has gained increased scientific interest. The majority of published papers in the literature focus on single-resident activity recognition. While this is an important area, especially when focusing on elderly people living alone, multi-resident activity recognition has potentially more applications in smart homes. Activity recognition for multiple residents acting concurrently can be treated as a multilabel classification problem (MLC). In this study, an experimental comparison between different MLC algorithms is attempted. Three different techniques were implemented: RAkEL, classifier chains, and binary relevance. These methods are evaluated using the ARAS and CASAS public datasets. Results obtained from experiments have shown that using MLC can recognize activities performed by multiple people with high accuracy. While RAkEL had the best performance, the rest of the methods had on-par results.

A new label ordering method in Classifier Chains based on imprecise probabilities

In Multi-Label Classification (MLC), Classifier Chains (CC) are considered simple and effective methods to exploit correlations between labels. A CC considers a binary classifier per label, in which the previous labels, according to an established order, are used as additional features. The label order strongly influences the performance of the CC, and there is no way to determine the optimal order so far. In this work, a new label ordering method based on label correlations is proposed. It uses a non-parametric model based on imprecise probabilities to estimate the correlations between pairs of labels. Then, it employs a greedy procedure that, to insert the labels in the chain, considers the correlations among the candidate labels and the ones already inserted, as well as the correlations between the candidate labels and the ones non-inserted yet. We argue that our proposal presents some advantages over the label ordering methods in CC developed so far based on label correlations. It is also shown that our proposal achieves better experimental results than the label ordering methods proposed so far that use label correlations in CC.

Ensemble of classifier chains and decision templates for multi-label classification

Ensemble of classifier chains and decision templates for multi-label classification

Linear Ordering Problem based Classifier Chain using Genetic Algorithm for multi-label classification

One of the most challenging tasks in multi-label classification is to identify label interdependence. Classifier Chain is the most prevalent method that utilizes label interdependence for improving classification accuracy as it requires only the number of classifiers equal to the number of labels. It uses a random sequence of labels. However, the order of labels in these sequences affects the classification performance. Nevertheless, despite many proposals in the literature, deciding the order in which these classifiers provide optimum results is a challenge to date. This paper proposes two methods for the ordering problem of the Classifier chain. The first proposed method termed as Linear Ordering Problem based Classifier Chain (LOP-CC) finds the chain order by considering it as a Linear Ordering Problem (LOP). The LOP utilizes a matrix and finds the optimal permutation of rows and corresponding columns that maximizes the sum of all the elements in the upper triangular matrix. This paper utilizes pairwise conditional entropy for creating the matrix to be used with the LOP and solves it using the Genetic Algorithm. It also proposes an extension to LOP-CC method termed as Linear Ordering Problem based partial Classifier Chain (LOP-pCC). It uses the same order of labels as LOP-CC. However, as opposed to LOP-CC, it utilizes partial sequences in the classifier chain rather than a full sequence. Experimentation performed on ten benchmark datasets consisting of a varying number of labels using different performance metrics demonstrates the proposed methods’ effectiveness compared to the other state-of-the-art classifier chain models.

Ml-Kfhe: Multi-Label Ensemble Classification Algorithm Exploiting Sensor Fusion Properties of the Kalman Filter

Despite the success of ensemble classification methods in multi-class classification problems, ensemble methods based on approaches other than bagging have not been widely explored for multi-label classification problems. The Kalman Filter-based Heuristic Ensemble (KFHE) is a recent ensemble method that exploits the sensor fusion properties of the Kalman filter to combine several classifier models, and that has been shown to be very effective. This work proposes a multi-label version of KFHE, ML-KFHE, demonstrating the effectiveness of the KFHE method on multi-label datasets. Two variants are introduced based on the underlying component classifier algorithm, ML-KFHE-HOMER, and ML-KFHE-CC which uses HOMER and Classifier Chain (CC) as the underlying multi-label algorithms respectively. ML-KFHE-HOMER and ML-KFHE-CC sequentially trains multiple HOMER and CC multi-label classifiers and aggregates their outputs using the sensor fusion properties of the Kalman filter. Experiments and detailed analysis performed on thirteen multi-label datasets and eight other algorithms, including state-of-the-art ensemble methods, show that for both versions, the ML-KFHE framework improves the ensembling process significantly with respect to bagging based combinations of HOMER and CC, thus demonstrating the effectiveness of ML-KFHE. Also, the ML-KFHE-HOMER variant was found to perform consistently and significantly better than existing multi-label methods including existing approaches based on ensembles.

A Bayesian network correlation-based classifier chain algorithm for multilabel learning

A Bayesian network correlation-based classifier chain algorithm for multilabel learning

Association Rules-Based Classifier Chains Method

The order for label learning is very important to the classifier chains method, and improper order can limit learning performance and make the model very random. Therefore, this paper proposes a classifier chains method based on the association rules (ARECC in short). ARECC first designs strong association rules based label dependence measurement strategy by combining the idea of frequent patterns; then based on label dependence relationship, a directed acyclic graph is constructed to topologically sort all vertices in the graph; next, the linear topological sequence obtained is used as the learning order of labels to train each label’s classifier; finally, ARECC uses association rules to modify and update the probability of the prediction for each label. By mining the label dependencies, ARECC writes the correlation information between labels in the topological sequence, which improves the utilization of the correlation information. Experimental results of a variety of public multi-label datasets show that ARECC can effectively improve classification performance.

Bagging method-based classifier chains for multi-label classification

Bagging method-based classifier chains for multi-label classification

Bagging method-based classifier chains for multi-label classification

Bagging method-based classifier chains for multi-label classification

Multi-label classification for multi-drug resistance prediction of Escherichia coli

Antimicrobial resistance (AMR) is a global health and development threat. In particular, multi-drug resistance (MDR) is increasingly common in pathogenic bacteria. It has become a serious problem to public health, as MDR can lead to the failure of treatment of patients. MDR is typically the result of mutations and the accumulation of multiple resistance genes within a single cell. Machine learning methods have a wide range of applications for AMR prediction. However, these approaches typically focus on single drug resistance prediction and do not incorporate information on accumulating antimicrobial resistance traits over time. Thus, identifying multi-drug resistance simultaneously and rapidly remains an open challenge. In our study, we could demonstrate that multi-label classification (MLC) methods can be used to model multi-drug resistance in pathogens. Importantly, we found the ensemble of classifier chains (ECC) model achieves accurate MDR prediction and outperforms other MLC methods. Thus, our study extends the available tools for MDR prediction and paves the way for improving diagnostics of infections in patients. Furthermore, the MLC methods we introduced here would contribute to reducing the threat of antimicrobial resistance and related deaths in the future by improving the speed and accuracy of the identification of pathogens and resistance.

Protein Subcellular Localization Based on Evolutionary Information and Segmented Distribution

The prediction of protein subcellular localization not only is important for the study of protein structure and function but also can facilitate the design and development of new drugs. In recent years, feature extraction methods based on protein evolution information have attracted much attention and made good progress. Based on the protein position-specific score matrix (PSSM) obtained by PSI-BLAST, PSSM-GSD method is proposed according to the data distribution characteristics. In order to reflect the protein sequence information as much as possible, AAO method, PSSM-AAO method, and PSSM-GSD method are fused together. Then, conditional entropy-based classifier chain algorithm and support vector machine are used to locate multilabel proteins. Finally, we test Gpos-mPLoc and Gneg-mPLoc datasets, considering the severe imbalance of data, and select SMOTE algorithm to expand a few sample; the experiment shows that the AAO + PSSM ∗ method in the paper achieved 83.1% and 86.8% overall accuracy, respectively. After experimental comparison of different methods, AAO + PSSM ∗ has good performance and can effectively predict protein subcellular location.

An experimental framework for evaluating loss minimization in multi‐label classification via stochastic process

AbstractOne major challenge multi‐label classification faces, are the conditions for evaluating multi‐label algorithms. Simplistic experimental setups based on artificial data may not capture crucial situations for analyzing these algorithms. This article introduces an experimental framework for evaluating multi‐label algorithms by artificially generating the probabilistic label distributions. The proposed framework has the benefits of considering a wide variety of labels distributions, and enables users to simulate probability label distributions with a better control of the label dependence and the difficulty of the problem. An experimental study was conducted using the framework where new findings with respect to five methods, binary relevance, classifier chain, dependent binary relevance, calibrated label ranking by pairwise comparison and probabilistic classifier chain, were revealed. This framework will facilitate conducting new experimental studies for analysing the effects of changing label dependence and the difficulty of the problem on the performance of new multi‐label algorithms.

Multilabel convolution neural network for facial expression recognition and ordinal intensity estimation.

Facial Expression Recognition (FER) has gained considerable attention in affective computing due to its vast area of applications. Diverse approaches and methods have been considered for a robust FER in the field, but only a few works considered the intensity of emotion embedded in the expression. Even the available studies on expression intensity estimation successfully assigned a nominal/regression value or classified emotion in a range of intervals. Most of the available works on facial expression intensity estimation successfully present only the emotion intensity estimation. At the same time, others proposed methods that predict emotion and its intensity in different channels. These multiclass approaches and extensions do not conform to man heuristic manner of recognising emotion and its intensity estimation. This work presents a Multilabel Convolution Neural Network (ML-CNN)-based model, which could simultaneously recognise emotion and provide ordinal metrics as the intensity estimation of the emotion. The proposed ML-CNN is enhanced with the aggregation of Binary Cross-Entropy (BCE) loss and Island Loss (IL) functions to minimise intraclass and interclass variations. Also, ML-CNN model is pre-trained with Visual Geometric Group (VGG-16) to control overfitting. In the experiments conducted on Binghampton University 3D Facial Expression (BU-3DFE) and Cohn Kanade extension (CK+) datasets, we evaluate ML-CNN’s performance based on accuracy and loss. We also carried out a comparative study of our model with some popularly used multilabel algorithms using standard multilabel metrics. ML-CNN model simultaneously predicts emotion and intensity estimation using ordinal metrics. The model also shows appreciable and superior performance over four standard multilabel algorithms: Chain Classifier (CC), distinct Random K label set (RAKEL), Multilabel K Nearest Neighbour (MLKNN) and Multilabel ARAM (MLARAM).

Radiomics-based prediction of multiple gene alteration incorporating mutual genetic information in glioblastoma and grade 4 astrocytoma, IDH-mutant

PurposeIn glioma, molecular alterations are closely associated with disease prognosis. This study aimed to develop a radiomics-based multiple gene prediction model incorporating mutual information of each genetic alteration in glioblastoma and grade 4 astrocytoma, IDH-mutant.MethodsFrom December 2014 through January 2020, we enrolled 418 patients with pathologically confirmed glioblastoma (based on the 2016 WHO classification). All selected patients had preoperative MRI and isocitrate dehydrogenase (IDH) mutation, O-6-methylguanine-DNA methyltransferase (MGMT) promoter methylation, epidermal growth factor receptor amplification, and alpha-thalassemia/mental retardation syndrome X-linked (ATRX) loss status. Patients were randomly split into training and test sets (7:3 ratio). Enhancing tumor and peritumoral T2-hyperintensity were auto-segmented, and 660 radiomics features were extracted. We built binary relevance (BR) and ensemble classifier chain (ECC) models for multi-label classification and compared their performance. In the classifier chain, we calculated the mean absolute Shapley value of input features.ResultsThe micro-averaged area under the curves (AUCs) for the test set were 0.804 and 0.842 in BR and ECC models, respectively. IDH mutation status was predicted with the highest AUCs of 0.964 (BR) and 0.967 (ECC). The ECC model showed higher AUCs than the BR model for ATRX (0.822 vs. 0.775) and MGMT promoter methylation (0.761 vs. 0.653) predictions. The mean absolute Shapley values suggested that predicted outcomes from the prior classifiers were important for better subsequent predictions along the classifier chains.ConclusionWe built a radiomics-based multiple gene prediction chained model that incorporates mutual information of each genetic alteration in glioblastoma and grade 4 astrocytoma, IDH-mutant and performs better than a simple bundle of binary classifiers using prior classifiers’ prediction probability.

Multi-label classification approach for quranic verses labeling

Machine learning involves the task of training systems to be able to make decisions without being explicitly programmed. Important among machine learning tasks is classification involving the process of training machines to make predictions from predefined labels. Classification is broadly categorized into three distinct groups: single-label (SL), multi-class, and multi-label (ML) classification. This research work presents an application of a multi-label classification (MLC) technique in automating Quranic verses labeling. MLC has been gaining attention in recent years. This is due to the increasing amount of works based on real-world classification problems of multi-label data. In traditional classification problems, patterns are associated with a single-label from a set of disjoint labels. However, in MLC, an instance of data is associated with a set of labels. In this paper, three standard <em>MLC</em> methods: <span>binary relevance (BR), classifier chain (CC), and label powerset (LP) algorithms are implemented with four baseline classifiers: support vector machine (SVM), naïve Bayes (NB), k-nearest neighbors (k-NN), and J48. The research methodology adopts the multi-label problem transformation (PT) approach. The results are validated using six conventional performance metrics. These include: hamming loss, accuracy, one error, micro-F1, macro-F1, and avg. precision. From the results, the classifiers effectively achieved above 70% accuracy mark. Overall, SVM achieved the best results with CC and LP algorithms.</span>

A fair classifier chain for multi‐label bank marketing strategy classification

AbstractRecently, the usage of machine learning algorithms is subject to discussion from a legal and ethical point of view. Unwanted discrimination regarding gender or race of a prediction model can lead to legal consequences. Therefore, during predictive model learning, one needs to be aware of possible bias and adjust the model to be fair. However, in bank marketing applications, one customer can receive multiple offers instead of just one. Because of their correlation between, a multi‐label classification approach is the most suitable one. This paper proposes a fair classifier chain machine learning model for multi‐label classification. Our algorithm solves the multi‐label classification problem in an efficient manner, and it is suitable for real‐life application employment. The proposed approach allows for controlling fairness constraints during the process of machine learning. It is based on the logistic regression model, thus enabling high efficiency and understandability. We apply our model to a real‐life model from bank marketing campaign response prediction. The obtained results are promising. More specifically, our model achieves high fairness measures having an increase from 7% to 17%. However, fairness has a price of a decrease in predictive performance, up to 9% of AUC. To the best of our knowledge, this is the first algorithm that introduce fairness constraints in multi‐label classification problems.

Social Media Monitoring of the COVID-19 Pandemic and Influenza Epidemic With Adaptation for Informal Language in Arabic Twitter Data: Qualitative Study.

BackgroundTwitter is a real-time messaging platform widely used by people and organizations to share information on many topics. Systematic monitoring of social media posts (infodemiology or infoveillance) could be useful to detect misinformation outbreaks as well as to reduce reporting lag time and to provide an independent complementary source of data compared with traditional surveillance approaches. However, such an analysis is currently not possible in the Arabic-speaking world owing to a lack of basic building blocks for research and dialectal variation.ObjectiveWe collected around 4000 Arabic tweets related to COVID-19 and influenza. We cleaned and labeled the tweets relative to the Arabic Infectious Diseases Ontology, which includes nonstandard terminology, as well as 11 core concepts and 21 relations. The aim of this study was to analyze Arabic tweets to estimate their usefulness for health surveillance, understand the impact of the informal terms in the analysis, show the effect of deep learning methods in the classification process, and identify the locations where the infection is spreading.MethodsWe applied the following multilabel classification techniques: binary relevance, classifier chains, label power set, adapted algorithm (multilabel adapted k-nearest neighbors [MLKNN]), support vector machine with naive Bayes features (NBSVM), bidirectional encoder representations from transformers (BERT), and AraBERT (transformer-based model for Arabic language understanding) to identify tweets appearing to be from infected individuals. We also used named entity recognition to predict the place names mentioned in the tweets.ResultsWe achieved an F1 score of up to 88% in the influenza case study and 94% in the COVID-19 one. Adapting for nonstandard terminology and informal language helped to improve accuracy by as much as 15%, with an average improvement of 8%. Deep learning methods achieved an F1 score of up to 94% during the classifying process. Our geolocation detection algorithm had an average accuracy of 54% for predicting the location of users according to tweet content.ConclusionsThis study identified two Arabic social media data sets for monitoring tweets related to influenza and COVID-19. It demonstrated the importance of including informal terms, which are regularly used by social media users, in the analysis. It also proved that BERT achieves good results when used with new terms in COVID-19 tweets. Finally, the tweet content may contain useful information to determine the location of disease spread.

Harnessing Multi-label Classification Approaches for Economic Phenomena Categorization

One fashion to report a country’s economic state is by compiling economic phenomena from several sources. The collected data may be explored based on their sentiments and economic categories. This research attempted to perform and analyze multiple approaches to multi-label text classification in addition to providing sentiment analysis on the economic phenomena. The sentiment and single-label category classification was performed utilizing the logistic regression model. Meanwhile, the multi-label category classification was fulfilled using a combination of logistic regression, support vector machines, k-nearest neighbor, naïve Bayes, and decision trees as base classifiers, with binary relevance, classifier chain, and label power set as the implementation approaches. The results showed that logistic regression works well in sentiment and single-label classification, with a classification accuracy of 80.08% and 92.71%, respectively. However, it was also discovered that it works poorly as a base classifier in multi-label classification, indicated by the classification accuracy dropping to 13.35%, 15.40%, and 30.65% for binary relevance, classifier chain, and label power set, respectively. Alternatively, naïve Bayes works best as a base classifier in the label power set approach for multi-label classification, with a classification accuracy of 63.22%, followed by decision trees and support vector machines.

Incorporating clinical knowledge with constrained classifier chain into a multimodal deep network for melanoma detection

In recent years, vast developments in Computer-Aided Diagnosis (CAD) for skin diseases have generated much interest from clinicians and other eventual end-users of this technology. Introducing clinical domain knowledge to these machine learning strategies can help dispel the black box nature of these tools, strengthening clinician trust. Clinical domain knowledge also provides new information channels which can improve CAD diagnostic performance. In this paper, we propose a novel framework for malignant melanoma (MM) detection by fusing clinical images and dermoscopic images. The proposed method combines a multi-labeled deep feature extractor and clinically constrained classifier chain (CC). This allows the 7-point checklist, a clinician diagnostic algorithm, to be included in the decision level while maintaining the clinical importance of the major and minor criteria in the checklist. Our proposed framework achieved an average accuracy of 81.3% for detecting all criteria and melanoma when testing on a publicly available 7-point checklist dataset. This is the highest reported results, outperforming state-of-the-art methods in the literature by 6.4% or more. Analyses also show that the proposed system surpasses the single modality system of using either clinical images or dermoscopic images alone and the systems without adopting the approach of multi-label and clinically constrained classifier chain. Our carefully designed system demonstrates a substantial improvement over melanoma detection. By keeping the familiar major and minor criteria of the 7-point checklist and their corresponding weights, the proposed system may be more accepted by physicians as a human-interpretable CAD tool for automated melanoma detection.