Classifier Chains Research Articles

Probabilistic regressor chains with Monte Carlo methods

A large number and diversity of techniques have been offered in the literature in recent years for solving multi-label classification tasks, including classifier chains where predictions are cascaded to other models as additional features. Chaining methods have often providing state of the art results, and the idea of extending it to multi-output regression has already been trialed. However, these ‘regressor chains’ have seen limited applicability, on account of yielding relatively little predictive performance compared to individual regression models, and also limited interpretability. In this work we identify and discuss the main limitations of regressor chains, including an analysis of different base models, loss functions, explainability, and other desiderata of real-world applications. We develop and examine techniques to overcome these limitations. In particular we present Monte Carlo schemes in the framework of probabilistic chains. We show they can be effective, flexible and useful in different areas. Overall, we also place regressor chains in context among general multi-output learning with continuous outputs, and in doing this shed additional light on the applicability of chaining to machine learning tasks.

Comparison of base classifiers for multi-label learning

Multi-label learning methods can be categorised into algorithm adaptation, problem transformation and ensemble methods. Some of these methods depend on a base classifier and the relationship is not well understood. In this paper the sensitivity of five problem transformation and two ensemble methods to four types of classifiers is studied. Their performance across 11 benchmark datasets is measured using 16 evaluation metrics. The best classifier is shown to depend on the method: Support Vector Machines (SVM) for binary relevance, classifier chains, calibrated label ranking, quick weighted multi-label learning and RAndom k-labELsets; k-Nearest Neighbours (k-NN) and Naïve Bayes (NB) for Hierarchy Of Multilabel classifiERs; and Decision Trees (DT) for ensemble of classifier chains. The statistical performance of a classifier is also found to be generally consistent across the metrics for any given method. Overall, DT and SVM have the best performance–computational time trade-off followed by k-NN and NB.

Improving the $$\epsilon $$-approximate algorithm for Probabilistic Classifier Chains

Probabilistic Classifier Chains are a multi-label classification method which has gained the attention of researchers in recent years. This is because of their ability to optimally estimate the entire joint conditional probability of a label combination through the product rule of probability. Their main drawback is that they require performing an exhaustive search in order to obtain Bayes optimal predictions. This means computing this probability for all possible label combinations before taking a label combination with the highest value of probability. This is the reason why several works have been published in recent years that avoid exploring all combinations, while maintaining optimality. Approaches such as greedy search, beam search and Monte Carlo reduce the computational cost, but at the cost of not ensuring Bayes optimal predictions (although, in general, they provide close to optimal solutions). Methods based on a heuristic search provide optimal predictions, but the computational time has not been as good as expected. In this respect, the $$\epsilon $$-approximate algorithm has been found to be the best inference approach among those that provide Bayes optimal predictions, not only for its optimality, but also for its computational time. However, this paper both theoretically and experimentally shows that it sometimes performs some backtracking during the search for optimal predictions which may prolong the prediction time. The aim of this paper is thus to improve this algorithm by achieving a more direct search. Specifically, it enhances the criterion under which the next node to be expanded is chosen by adding heuristic information, although it is only applicable for linear-based models. The experiments carried out confirm that the improved $$\epsilon $$-approximate algorithm explores fewer nodes and reduces the computational time of the original version.

AIEpred: An Ensemble Predictive Model of Classifier Chain to Identify Anti-Inflammatory Peptides

Anti-inflammatory peptides (AIEs) have recently emerged as promising therapeutic agent for treatment of various inflammatory diseases, such as rheumatoid arthritis and Alzheimer's disease. Therefore, detecting the correlation between amino acid sequence and its anti-inflammatory property is of great importance for the discovery of new AIEs. To address this issue, we propose a novel prediction tool for accurate identification of peptides as anti-inflammatory epitopes or non anti-inflammatory epitopes. Most of all, we encode the original peptide sequence for better mining and exploring the information and patterns, based on the three feature representations as amino acid contact, position specific scoring matrix, physicochemical property. At the same time, we exploit several feature extraction models and utilize one feature selection model, in order to construct many base classifiers from various feature representations. More specifically, we develop an effective classification model, with which we can extract and learn a set of informative features from the ensemble classifier chain model with different group of base classifiers. Furthermore, in order to test the predictive power of our model, we conduct the comparative experiments on the leave-one-out cross-validation and the independent test. It shows that our novel predictor performs great accurate for identification of AIEs as well as existing outstanding prediction tools. Source codes are available at https://github.com/guofei-tju/Ensemble-classifier-chain-model.

Aspect-Based Sentiment Analysis and Emotion Detection for Code-Mixed Review

Review can affect customer decision making because by reading it, people manage to know whether the review is positive, or negative. However, positive, negative, and neutral, without considering the emotion will be not enough because emotion can strengthen the sentiment result. This study explains about the comparison of machine learning and deep learning in sentiment as well as emotion classification with multi-label classification. In machine learning comparison, the problem transformation that we used are Binary Relevance (BR), Classifier Chain (CC), and Label Powerset (LP), with Decision Tree (DT), Random Forest (RF), Support Vector Machine (SVM), and Extra Tree Classifier (ET) as algorithms of machine learning. The features we compared are n-gram language model (unigram, bigram, unigram-bigram). For deep learning, algorithms that we applied are Gated Recurrent Unit (GRU) and Bidirectional Long Short-Term Memory (BiLSTM), using self-developed word embedding. The comparison results show RF dominates with 88.4% and 89.54% F1 scores with CC method for food aspect, and LP for price, respectively. For service and ambience aspects, ET leads with 92.65% and 87.1% with LP and CC methods, respectively. On the other hand, in deep learning comparison, GRU and BiLSTM obtained similar F1- score for food aspect, 88.16%. On price aspect, GRU leads with 83.01%. However, for service and ambience, BiLSTM achieved higher F1-score, 89.03% and 84.78%.

Label Specific Features-Based Classifier Chains for Multi-Label Classification

Multi-label classification tackles the problems in which each instance is associated with multiple labels. Due to the interdependence among labels, exploiting label correlations is the main means to enhance the performances of classifiers and a variety of corresponding multi-label algorithms have been proposed. Among those algorithms Classifier Chains (CC) is one of the most effective methods. It induces binary classifiers for each label, and these classifiers are linked in a chain. In the chain, the labels predicted by previous classifiers are used as additional features for the current classifier. The original CC has two shortcomings which potentially decrease classification performances: random label ordering, noise in original and additional features. To deal with these problems, we propose a novel and effective algorithm named LSF-CC, i.e. Label Specific Features based Classifier Chain for multi-label classification. At first, a feature estimating technique is employed to produce a list of most relevant features and labels for each label. According to these lists, we define a chain to guarantee that the most frequent labels that appear in these lists are top-ranked. Then, label specific features can be selected from the original feature space and label space. Based on these label specific features, corresponding binary classifiers are learned for each label. Experiments on several multi-label data sets from various domains have shown that the proposed method outperforms well-established approaches.

User Clustering for MIMO NOMA via Classifier Chains and Gradient-Boosting Decision Trees

In this article, we propose a data-driven approach to group users in a Non-Orthogonal Multiple Access (NOMA) MIMO setting. Specifically, we formulate user clustering as a multi-label classification problem and solve it by coupling a Classifier Chain (CC) with a Gradient Boosting Decision Tree (GBDT), namely, the LightGBM algorithm. The performance of the proposed CC-LightGBM scheme is assessed via numerical simulations. For benchmarking, we consider two classical adaptation learning schemes: Multi-Label k-Nearest Neighbours (ML-KNN) and Multi-Label Twin Support Vector Machines (ML-TSVM); as well as other naive approaches. Besides, we also compare the computational complexity of the proposed scheme with those of the aforementioned benchmarks.

A Machine Learning-Based Recommender System for Improving Students Learning Experiences

Outcome-based education (OBE) is a well-proven teaching strategy based upon a predefined set of expected outcomes. The components of OBE are Program Educational Objectives (PEOs), Program Outcomes (POs), and Course Outcomes (COs). These latter are assessed at the end of each course and several recommended actions can be proposed by faculty members' to enhance the quality of courses and therefore the overall educational program. Considering a large number of courses and the faculty members' devotion, bad actions could be recommended and therefore undesirable and inappropriate decisions may occur. In this paper, a recommender system, using different machine learning algorithms, is proposed for predicting suitable actions based on course specifications, academic records, and course learning outcomes' assessments. We formulated the problem as a multi-label multi-class binary classification problem and the dataset was translated into different problem transformation and adaptive methods such as one-vs.-all, binary relevance, label powerset, classifier chain, and ML-KNN adaptive classifier. As a case study, the proposed recommender system is applied to the college of Computer and Information Sciences, Jouf University, Kingdom of Saudi Arabia (KSA) for helping academic staff improving the quality of teaching strategies. The obtained results showed that the proposed recommender system presents more recommended actions for improving students' learning experiences.

Logistics engineering optimization based on machine learning and artificial intelligence technology

In the era of the Internet of Things, smart logistics has become an important means to improve people’s life rhythm and quality of life. At present, some problems in logistics engineering have caused logistics efficiency to fail to meet people’s expected goals. Based on this, this paper proposes a logistics engineering optimization system based on machine learning and artificial intelligence technology. Moreover, based on the classifier chain and the combined classifier chain, this paper proposes an improved multi-label chain learning method for high-dimensional data. In addition, this study combines the actual needs of logistics transportation and the constraints of the logistics transportation process to use multi-objective optimization to optimize logistics engineering and output the optimal solution through an artificial intelligence model. In order to verify the effectiveness of the model, the performance of the method proposed in this paper is verified by designing a control experiment. The research results show that the logistics engineering optimization based on machine learning and artificial intelligence technology proposed in this paper has a certain practical effect.

Toward systems-centered analysis of patient safety events: Improving root cause analysis by optimized incident classification and information presentation

BackgroundSystems-centered root cause analysis (RCA) of patient safety events presents unique advantages as it aims to disclose vulnerabilities of healthcare systems. However, the increasing number of collected events poses the problems of low efficiency and information overload for traditional RCA. ObjectivesThis study aims to improve systems-centered RCA by developing optimized information extraction and presentation. MethodsWe experimented supervised machine-learning methods to extract safety-related information from 3333 de-identified patient safety event reports from two independent sources. Based on the extracted information, we further evaluated how optimized information presentation could help facilitate the disclosure of system vulnerabilities in traditional RCA. ResultsMultilabel text classification is effective in identifying safety-related information from the narrative description of patient safety events. The Pruned Sets in conjunction with Naïve Bayes are the outperformed algorithm in one dataset, with an overall F score of 60.0 % and the highest F score of 96.0 % for identifying “Adverse Drug Reaction”. The Classifier Chains in conjunction with Naïve Bayes are the outperformed algorithm in another dataset, with an overall F score of 43.2 % and the highest F score of 64.0 % for identifying “Medication”. During the RCA, human experts applied the optimized presentation of information which showed advantages of identifying system vulnerabilities. ConclusionOur study demonstrated the feasibility of using multilabel text classification for identifying safety-related information from the narrative description of patient safety events. The extracted information when grouped by safety-related information can better aid human experts to conduct systems-centered RCA and disclose system vulnerabilities.

Dealing with class imbalance in classifier chains via random undersampling

Class imbalance is an intrinsic characteristic of multi-label data. Most of the labels in multi-label data sets are associated with a small number of training examples, much smaller compared to the size of the data set. Class imbalance poses a key challenge that plagues most multi-label learning methods. Ensemble of Classifier Chains (ECC), one of the most prominent multi-label learning methods, is no exception to this rule, as each of the binary models it builds is trained from all positive and negative examples of a label. To make ECC resilient to class imbalance, we first couple it with random undersampling. We then present two extensions of this basic approach, where we build a varying number of binary models per label and construct chains of different sizes, in order to improve the exploitation of majority examples with approximately the same computational budget. Experimental results on 16 multi-label datasets demonstrate the effectiveness of the proposed approaches in a variety of evaluation metrics.

Automatic Classification of Open-Ended Questions: Check-All-That-Apply Questions

Text data from open-ended questions in surveys are challenging to analyze and are often ignored. Open-ended questions are important though because they do not constrain respondents’ answers. Where open-ended questions are necessary, often human coders manually code answers. When data sets are large, it is impractical or too costly to manually code all answer texts. Instead, text answers can be converted into numerical variables, and a statistical/machine learning algorithm can be trained on a subset of manually coded data. This statistical model is then used to predict the codes of the remainder. We consider open-ended questions where the answers are coded into multiple labels (all-that-apply questions). For example, in the open-ended question in our Happy example respondents are explicitly told they may list multiple things that make them happy. Algorithms for multilabel data take into account the correlation among the answer codes and may therefore give better prediction results. For example, when giving examples of civil disobedience, respondents talking about “minor nonviolent offenses” were also likely to talk about “crimes.” We compare the performance of two different multilabel algorithms (random k-labelsets [RAKEL], classifier chains [CC]) to the default method of binary relevance (BR) which applies single-label algorithms to each code separately. Performance is evaluated on data from three open-ended questions (Happy, Civil Disobedience, and Immigrant). We found weak bivariate label correlations in the Happy data (90th percentile: 7.6%), and stronger bivariate label correlations in the Civil Disobedience (90th percentile: 17.2%) and Immigrant (90th percentile: 19.2%) data. For the data with stronger correlations, we found both multilabel methods performed substantially better than BR using 0/1 loss (“at least one label is incorrect”) and had little effect when using Hamming loss (average error). For data with weak label correlations, we found no difference in performance between multilabel methods and BR. We conclude that automatic classification of open-ended questions that allow multiple answers may benefit from using multilabel algorithms for 0/1 loss. The degree of correlations among the labels may be a useful prognostic tool.

Automated Billing Code Retrieval from MRI Scanner Log Data

Although the level of digitalization and automation steadily increases in radiology, billing coding for magnetic resonance imaging (MRI) exams in the radiology department is still based on manual input from the technologist. After the exam completion, the technologist enters the corresponding exam codes that are associated with billing codes in the radiology information system. Moreover, additional billing codes are added or removed, depending on the performed procedure. This workflow is time-consuming and we showed that billing codes reported by the technologists contain errors. The coding workflow can benefit from an automated system, and thus a prediction model for automated assignment of billing codes for MRI exams based on MRI log data is developed in this work. To the best of our knowledge, it is the first attempt to focus on the prediction of billing codes from modality log data. MRI log data provide a variety of information, including the set of executed MR sequences, MR scanner table movements, and given a contrast medium. MR sequence names are standardized using a heuristic approach and incorporated into the features for the prediction. The prediction model is trained on 9754 MRI exams and tested on 1 month of log data (423 MRI exams) from two MRI scanners of the radiology site for the Swiss medical tariffication system Tarmed. The developed model, an ensemble of classifier chains with multilayer perceptron as a base classifier, predicts medical billing codes for MRI exams with a micro-averaged F1-score of 97.8% (recall 98.1%, precision 97.5%). Manual coding reaches a micro-averaged F1-score of 98.1% (recall 97.4%, precision 98.8%). Thus, the performance of automated coding is close to human performance. Integrated into the clinical environment, this work has the potential to free the technologist from a non-value adding an administrative task, therefore enhance the MRI workflow, and prevent coding errors.

Chained ensemble classifier for image annotation

Image annotation is the task of assigning keywords or identifiers to images, holistically or in specific regions. These keywords serve as descriptors of high-level semantics to facilitate retrieval and organization of visual information. It plays an important role in content-based image understanding, as well as in areas such as object recognition in robotics, content-based image searching and knowledge extraction. Automatic image annotation is usually approached by means of supervised classification, where a set of previously annotated images is required to train a learning algorithm that later predicts the labels for new images. This paper proposes a novel ensemble classifier for the supervised image annotation task inspired in chain classifiers. In the proposed approach a chain of individual classifiers is build, where each classifier is trained by using a different modality. In addition, the input space of models in the chain is augmented with the output of the preceding model in the sequence. Each model in the chain deals with the same classification problem, making the proposed method an ensemble model build from multimodal data. To the best of our knowledge, chain classifiers have not been used in this particular setting. Experimental results in a challenging image collection show that the proposed method is able to obtain an f −value superior to 0.5, outperforming related work.

Identification of unhealthy Panax notoginseng from different geographical origins by means of multi-label classification

Root-knot nematode is a common plant-parasitic pest with a highly destructive that infects more than 2000 plant species. Panax notoginseng (P. notoginseng) is one of the most susceptible traditional medicine. More importantly, it is difficult to distinguish the powders of P. notoginseng infected with root-knot nematode from those of healthy P. notoginseng due to the color and shape are same after being ground into powder. In this paper, Attenuated Total Reflection-Fourier Transform Infrared (ATR-FTIR) was used to identify P. notoginseng samples. Multiplicative scatter correction (MSC) was applied to preprocess the spectral data. Competitive adaptive reweighted sampling (CARS) and successive projection algorithm (SPA) were employed to select feature variables. Density-based spatial clustering of application with noise (DBSCAN) was adopted to discover groups within the data. Also, we found that the geographical origin is a pivotal factor to consider when identifying unhealthy P. notoginseng. Therefore, we introduced a novel multi-label classification (MLC) method to identify healthy and unhealthy P. notoginseng powders from three different geographical origins. In addition, binary relevance method (BR), classifier chain (CC), ensembles of classifier chains (ECC), and multilayer perceptron classifier (MLPC) were applied to create classification models, ECC exhibits superior performance in particular.

Evaluating multi-label classifiers and recommender systems in the financial service sector

The objective of this paper is to evaluate multi-label classification techniques and recommender systems for cross-sell purposes in the financial services sector. We carried out three analyses using data obtained from an international financial services provider. First, we tested four multi-label classification techniques, of which the two problem transformation methods were combined with several base classifiers. Second, we benchmarked the performance of five state-of-the-art recommender approaches. Third, we compared the best performing multi-label classification and recommender approaches with each other. The results identify user-based collaborative filtering as the top performing recommender system, with a cross-validated F1 measure of 42.20% and G-mean of 42.64%. Classifier chains binary relevance with adaboost and binary relevance with random forest are the top performing multi-label classification algorithms for respectively F1 measure and G-mean, yielding a cross-validated F1 measure of 53.33% and G-mean of 54.37%. The statistical comparison between the best performing approaches confirms the superiority of multi-label classification techniques. Our study provides important recommendations for financial services providers, who are interested in the most effective methods to determine cross-sell opportunities. In previous studies, multi-label classification techniques and recommender systems were always investigated independently of each other. To the best of our knowledge, our study is therefore the first to compare both techniques in the financial services sector.

Efficient label ordering for improving multi-label classifier chain accuracy

Classifier chain (CC) algorithms have been introduced for multi label classification predictions in recent years. The accuracy of these algorithms is considered better than the other state-of-the art algorithms in this domain. In addition to accuracy, an effort is made to improve the complexity of the algorithms in order to predict an optimal order in which the binary classifiers are executed. Existing label ordering algorithms are executed twice, once for the generation of label ordering and another time for improving classifier chain accuracy with predicted order. In this paper, we discuss the current chain classifier algorithms and their comparison in terms of both accuracy and execution time. Moreover, we have introduced a new Label Ordering for Classifier Chain (LOCC), which exploits the semantic relationships among the labels of a dataset. The predicted label’s order is computed without the execution of the classification algorithm. The semantic relations among the labels are analysed and an order is generated, which is fed to a classifier chain algorithm. The proposed algorithm is better in terms of accuracy and computational time than the available classifier chain algorithms.

Conditional entropy based classifier chains for multi-label classification

In many real-world problems, data samples are simultaneously associated with multiple labels, instead of a single label. Multi-label classification deals with such problems, and has extensive applications in many fields. Among the many methods proposed for multi-label classification tasks, classifier chains (CC) is an appealing one. In the classifier chains method, the label order has a strong effect on the classification performance. However, it is difficult to determine a proper order. In this paper, we propose ordering methods based on the conditional entropy of labels. We generate a single order instead of multiple orders. Unlike existing ordering methods, there is no need to train more classifiers than CC. Experimental results on nine benchmark datasets evaluated by eight measures show that the proposed methods achieve good performance.

Ensemble of classifier chains and Credal C4.5 for solving multi-label classification

In this work, we have considered the ensemble of classifier chains (ECC) algorithm in order to solve the multi-label classification (MLC) task. It starts from binary relevance algorithm (BR), a simple and direct approach to MLC that has been shown to provide good results in practice. Nevertheless, unlike BR, ECC aims to exploit the correlations between labels. ECC uses an algorithm of traditional supervised classification in order to approach the binary problems. Within this field, Credal C4.5 (CC4.5) is a new version of the well-known C4.5 algorithm that uses imprecise probabilities in order to estimate the probability distribution of the class variable. This new version of C4.5 algorithm has been shown to provide better performance when noisy datasets are classified. In MLC, the intrinsic noise might be higher than in traditional supervised classification. The reason is very simple: in MLC, there are multiple labels, whereas in traditional classification there is just a class variable. Thus, there is more probability of error for an instance. For the previous reasons, the performance of ECC with CC4.5 as base classifier is studied in this work. We have carried out an extensive experimental analysis with several multi-label datasets, different noise levels and a large number of evaluation metrics for MLC. This experimental study has shown that, generally, ECC has better performance with CC4.5 as base classifier than using C4.5. The higher is the label noise level introduced in the data, the more significative is this improvement. Therefore, it is probably suitable to use imprecise probabilities in Decision Trees within MLC.

Multi-label Arabic text categorization: A benchmark and baseline comparison of multi-label learning algorithms

Multi-label text categorization refers to the problem of assigning each document to a subset of categories by means of multi-label learning algorithms. Unlike English and most other languages, the unavailability of Arabic benchmark datasets prevents evaluating multi-label learning algorithms for Arabic text categorization. As a result, only a few recent studies have dealt with multi-label Arabic text categorization on non-benchmark and inaccessible datasets. Therefore, this work aims to promote multi-label Arabic text categorization through (a) introducing “RTAnews”, a new benchmark dataset of multi-label Arabic news articles for text categorization and other supervised learning tasks. The benchmark is publicly available in several formats compatible with the existing multi-label learning tools, such as MEKA and Mulan. (b) Conducting an extensive comparison of most of the well-known multi-label learning algorithms for Arabic text categorization in order to have baseline results and show the effectiveness of these algorithms for Arabic text categorization on RTAnews. The evaluation involves four multi-label transformation-based algorithms: Binary Relevance, Classifier Chains, Calibrated Ranking by Pairwise Comparison and Label Powerset, with three base learners (Support Vector Machine, k-Nearest-Neighbors and Random Forest); and four adaptation-based algorithms (Multi-label kNN, Instance-Based Learning by Logistic Regression Multi-label, Binary Relevance kNN and RFBoost). The reported baseline results show that both RFBoost and Label Powerset with Support Vector Machine as base learner outperformed other compared algorithms. Results also demonstrated that adaptation-based algorithms are faster than transformation-based algorithms.