Multi-label Classification Algorithms Research Articles

Linear Ordering Problem based Classifier Chain using Genetic Algorithm for multi-label classification

One of the most challenging tasks in multi-label classification is to identify label interdependence. Classifier Chain is the most prevalent method that utilizes label interdependence for improving classification accuracy as it requires only the number of classifiers equal to the number of labels. It uses a random sequence of labels. However, the order of labels in these sequences affects the classification performance. Nevertheless, despite many proposals in the literature, deciding the order in which these classifiers provide optimum results is a challenge to date. This paper proposes two methods for the ordering problem of the Classifier chain. The first proposed method termed as Linear Ordering Problem based Classifier Chain (LOP-CC) finds the chain order by considering it as a Linear Ordering Problem (LOP). The LOP utilizes a matrix and finds the optimal permutation of rows and corresponding columns that maximizes the sum of all the elements in the upper triangular matrix. This paper utilizes pairwise conditional entropy for creating the matrix to be used with the LOP and solves it using the Genetic Algorithm. It also proposes an extension to LOP-CC method termed as Linear Ordering Problem based partial Classifier Chain (LOP-pCC). It uses the same order of labels as LOP-CC. However, as opposed to LOP-CC, it utilizes partial sequences in the classifier chain rather than a full sequence. Experimentation performed on ten benchmark datasets consisting of a varying number of labels using different performance metrics demonstrates the proposed methods’ effectiveness compared to the other state-of-the-art classifier chain models.

LP-MLTSVM: Laplacian Multi-Label Twin Support Vector Machine for Semi-Supervised Classification

In the machine learning jargon, multi-label classification refers to a task where multiple mutually non-exclusive class labels are assigned to a single instance. Generally, the lack of sufficient labeled training data demanded by a classification task is met by an approach known as semi-supervised learning. This type of learning extracts the decision rules of classification by utilizing both labeled and unlabeled data. Regarding multi-label data, however, current semi-supervised learning methods are unable to classify them accurately. Therefore, with the goal of generalizing the state-of-the-art semi-supervised approaches to multi-label data, this paper proposes a novel two-stage method for multi-label semi-supervised classification. The first stage determines the label(s) of the unlabeled training data by means of a smooth graph constructed using the manifold regularization. In the second stage, thanks to the capability of the twin support vector machine to relax the requirement that hyperplanes should be parallel in classical SVM, we employ it to establish a multi-label classifier called LP-MLTSVM. In the experiments, this classifier is applied on benchmark datasets. The simulation results substantiate that compared to the existing multi-label classification algorithms, LP-MLTSVM shows superior performance in terms of the Hamming loss, average precision, coverage, ranking loss, and one-error metrics.

Application of Uncertainty Thought Environment in Judicial Adjudication Based on Cognitive Psychology.

The uncertainty of judicial decision-making has a deep and extensive theoretical foundation. Theoretical analysis starts with a reflection on legal rationalism that challenges the legal certainty before delving deeply into the case's facts and the entire legal system. In light of this, this paper explores a novel approach to enhance the reasoning mechanism of trial documents from the viewpoint of modern cognitive psychology, concentrating on the parties' and the public's cognitive processes to justice. It is suggested to use an inert hierarchical multilabel classification algorithm. In order to predict the category of invisible examples, the extended multilabel training set is first searched for adjacent samples of invisible examples, and the classification weight and confidence of each category are then determined in accordance with these adjacent samples. The group of invisible examples is then anticipated. Experimental comparison demonstrates that this algorithm outperforms other prediction techniques; the macro accuracy, macro recall, and macro F1 of this method are, respectively, 0.896, 0.871, and 0.814. It has some advantages in many multilabel evaluation indexes when compared to other multilabel algorithms.

Reducing the Label Space a Predefined Ratio for a More Efficient Multilabel Classification

The multi-label classification task has been widely used to solve problems where each of the instances may be related not only to one class but to many of them simultaneously. Many of these problems usually comprise a high number of labels in the output space, so learning a predictive model from such datasets may turn into a challenging task since the computational complexity of most algorithms depends on the number of labels. In this paper, we propose a methodology to reduce the label space a predefined ratio of labels, aiming to improve the runtime of the multi-label classification algorithms. Obviously, such reduction should be done without producing a significant drop in their final predictive performance. The experimental analysis carried out over 25 well-known multi-label datasets, demonstrates a drastic reduction in the runtime. Besides, it is statistically proven that reducing 20% the number of labels does not lead to a decrease in the predictive performance of the multi-label algorithms using four well-known evaluation measures. Even more, in many cases, although reductions of up to 50% of the output space are made, the predictive performance of the algorithms is not significantly different from using the whole set of labels.

Disturbance Evaluation in Power System Based on Machine Learning

The operation complexity of the distribution system increases as a large number of distributed generators (DG) and electric vehicles were introduced, resulting in higher demands for fast online reactive power optimization. In a power system, the characteristic selection criteria for power quality disturbance classification are not universal. The classification effect and efficiency needs to be improved, as does the generalization potential. In order to categorize the quality in the power signal disturbance, this paper proposes a multi-layer severe learning computer auto-encoder to optimize the input weights and extract the characteristics of electric power quality disturbances. Then, a multi-label classification algorithm based on rating is proposed to understand the relationship between the labels and identify the various power quality disturbances. The two algorithms are combined to construct a multi-label classification model based on a multi-level extreme learning machine, and the optimal network structure of the multi-level extreme learning machine as well as the optimal multi-label classification threshold are developed. The proposed method can be used to classify the single and compound power quality disturbances with improved classification effect, reliability, robustness, and anti-noise performance, according to the experimental results. The hamming loss obtained by the proposed algorithm is about 0.17 whereas ML-RBF, SVM and ML-KNN schemes have 0.28, 0.23 and 0.22 respectively at a noise intensity of 20 dB. The average precision obtained by the proposed algorithm 0.85 whereas the ML-RBF, SVM and ML-KNN schemes indicates 0.7, 0.77 and 0.78 respectively.

On the Effects of Different Types of Label Noise in Multi-Label Remote Sensing Image Classification

The development of accurate methods for multi-label classification (MLC) of remote sensing (RS) images is one of the most important research topics in RS. To address MLC problems, the use of deep neural networks that require a high number of reliable training images annotated by multiple land-cover class labels (multi-labels) has been found popular in RS. However, collecting such annotations is time-consuming and costly. A common procedure to obtain annotations at zero labeling cost is to rely on thematic products or crowdsourced labels. As a drawback, these procedures come with the risk of label noise that can distort the learning process of the MLC algorithms. In the literature, most label noise robust methods are designed for single-label classification (SLC) problems in computer vision (CV), where each image is annotated by a single label. Unlike SLC, label noise in MLC can be associated with: 1) subtractive label-noise (a land cover class label is not assigned to an image while that class is present in the image); 2) additive label-noise (a land cover class label is assigned to an image although that class is not present in the given image); and 3) mixed label-noise (a combination of both). In this paper, we investigate three different noise robust CV SLC methods and adapt them to be robust for multi-label noise scenarios in RS. During experiments, we study the effects of different types of multi-label noise and evaluate the adapted methods rigorously. To this end, we also introduce a synthetic multi-label noise injection strategy that is more adequate to simulate operational scenarios compared to the uniform label noise injection strategy, in which the labels of absent and present classes are flipped at uniform probability. Further, we study the relevance of different evaluation metrics in MLC problems under noisy multi-labels.

A machine learning framework for risk prediction of multi-label cardiovascular events based on focused carotid plaque B-Mode ultrasound: A Canadian study

MotivationMachine learning (ML) algorithms can provide better cardiovascular event (CVE) prediction. However, ML algorithms are mostly explored for predicting a single CVE at a time. The objective of this study is to design and develop an ML-based system to predict multi-label CVEs, such as (i) coronary artery disease, (ii) acute coronary syndrome, and (iii) a composite CVE—a class of AtheroEdge 3.0 (ML) system. MethodsFocused carotid B-mode ultrasound and coronary angiography are performed on a group of 459 participants consisting of three cardiovascular labels. Initially, 23 risk predictors comprising (i) patients’ demographics, (ii) clinical blood-biomarkers, and (iii) carotid ultrasound image-based phenotypes are collected. Six types of classification techniques comprising (a) four problem transformation methods (PTM) and (b) two algorithm adaptation methods (AAM) are used for multi-label CVE prediction. The performance of the proposed system is evaluated for accuracy, sensitivity, specificity, F1-score, and area-under-the-curve (AUC) using 10-fold cross-validation. The proposed system is also verified using another database of 522 participants. ResultsFor the primary database, PTM demonstrated a better multi-label CVE prediction than AAM (mean accuracy: 80.89% vs. 62.83%, mean AUC: 0.89 vs. 0.63), validating our hypothesis. The PTM-based binary relevance (BR) technique provided optimal performance in multi-label CVE prediction. The overall multi-label classification accuracy, sensitivity, specificity, F1-score, and AUC using BR are 81.2 ± 3.01%, 76.5 ± 8.8%, 83.8 ± 3.8%, 75.37 ± 5.8%, and 0.89 ± 0.02 (p < 0.0001), respectively. When used on the second Canadian database with seven cardiovascular events (acute coronary syndrome, myocardial infarction, angina, stroke, transient ischemic attack, heart failure, and death), the proposed system showed an accuracy of 96.36 ± 0.87% (AUC: 0.61 ± 0.06, p < 0.0001). ConclusionML-based multi-label classification algorithms, such as binary relevance, yielded the best predictions for three cardiovascular endpoints.

Research on Multi-label Text Classification Method Based on tALBERT-CNN

Single-label classification technology has difficulty meeting the needs of text classification, and multi-label text classification has become an important research issue in natural language processing (NLP). Extracting semantic features from different levels and granularities of text is a basic and key task in multi-label text classification research. A topic model is an effective method for the automatic organization and induction of text information. It can reveal the latent semantics of documents and analyze the topics contained in massive information. Therefore, this paper proposes a multi-label text classification method based on tALBERT-CNN: an LDA topic model and ALBERT model are used to obtain the topic vector and semantic context vector of each word (document), a certain fusion mechanism is adopted to obtain in-depth topic and semantic representations of the document, and the multi-label features of the text are extracted through the TextCNN model to train a multi-label classifier. The experimental results obtained on standard datasets show that the proposed method can extract multi-label features from documents, and its performance is better than that of the existing state-of-the-art multi-label text classification algorithms.

Non-intrusive energy estimation using random forest based multi-label classification and integer linear programming

Home energy management system is proposed to reduce the influences caused by the high ratio penetration of renewable energy generation, through managing and dispatching the residential power and energy consumption in the demand side. Being aware of how the electric energy is consumed is a key step of this system. Non-intrusive Load Monitoring is regarded as the most potential method to address this problem, which aims to separate individual appliances in households by decomposing the total power consumption. In recent years, NILM is framed as a multi-label classification problem and many researches has been investigated in this field. In this paper, a non-intrusive method which can identify appliances power usage information from the total power consumption is proposed and thoroughly investigated. Firstly, the random k-labelset multi-label classification algorithm is enhanced by introducing random forest algorithm as base classifier. Then, grid search method and cross validation method are integrated to determine the optimal paraments set. This algorithm is used to achieve the appliances identification. Finally, based on the identification result, the integer linear programming is employed for power estimation of each appliance, especially multi-state appliances. Experimental results on low voltage networks simulator demonstrate that the proposed method has a high identification accuracy compared with the traditional random k-labelset multi-label classification methods with other base classifiers, and it is capable of identifying the power usages of different appliances accurately. The desirable performance of power estimation has broadened the applications of machine learning based non-intrusive energy monitoring.

An efficient Pareto-based feature selection algorithm for multi-label classification

Multi-label learning algorithms have significant challenges due to high-dimensional feature space and noises in multi-label datasets. Feature selection methods are effective techniques to deal with these problems. ParetoCluster is an effective multi-label feature selection algorithm based on Pareto dominance and cluster analysis concepts which considers each label an objective function. This algorithm loses its effectiveness to differentiate features when dealing with high labeled datasets and makes most features incomparable (e.g., when most features fall into the first layer). Thus, a cluster analysis criterion in ParetoCluster will play a decisive role in determining the most relevant features. Bearing this in mind, in this paper, we have modeled the multi-label feature selection problem into a bi-objective optimization problem regarding the relevancy and redundancy degree of the features. We then handle it using Pareto dominance in this bi-objective domain. To illustrate the optimality and efficiency of the proposed method, we have compared our approach against some similar techniques.

Multi-label classification for simultaneous fault diagnosis of marine machinery: A comparative study

Fault diagnosis of marine machinery is of utmost importance in modern ships. The widely used machine learning techniques have made it possible to realize intelligent diagnosis by using large amounts of sensory data. However, the detection of simultaneous faults is still a challenge in the absence of simultaneous fault data. Multi-label classification has recently gained popularity in simultaneous fault diagnosis with promising results. The contribution of this work is to carry out a comparative study of several state-of-the-art multi-label classification algorithms for simultaneous fault diagnosis of marine machinery based on single fault data. The proposed method is experimentally validated with a dataset generated from a real data validated simulator of a Frigate. The experimental results show the effectiveness of the proposed method, which can provide decision support for the application of multi-label classification in the simultaneous fault diagnosis of similar marine systems.

Multi-label classification with local pairwise and high-order label correlations using graph partitioning

In multi-label learning problems, the class labels are correlated and the label correlations can be leveraged to improve the predictive performance of a classifier. Methods that consider high-order correlations in the label space, usually do not utilize pairwise correlations. In most of these methods, label correlations are considered as prior knowledge, which can be misleading in problems with noisy or missing labels. In such cases, learning the label correlation as part of the model training task is more effective. In this paper, a rule-based evolutionary multi-label classification method is proposed that incorporates the local label correlations through the high-order label subsets and pairwise dependencies. Graph structures are employed to model the label dependencies and the estimated label similarities are used to obtain more accurate label sets for the classification rules. To refine the high-order label relations, a novel hierarchical density-based clustering method is proposed to obtain a k-way partitioning for the label graphs based on their pairwise correlations. The effectiveness of the proposed method is experimented on multiple benchmark datasets from different domains and compared with several well-known multi-label classification algorithms. The proposed method has shown the highest average rank along multiple metrics and the results are consistently better than or similar to the compared methods with statistical significance.

Auto-adaptive Grammar-Guided Genetic Programming algorithm to build Ensembles of Multi-Label Classifiers

Multi-label classification has been used to solve a wide range of problems where each example in the dataset may be related either to one class (as in traditional classification problems) or to several class labels at the same time. Many ensemble-based approaches have been proposed in the literature, aiming to improve the performance of traditional multi-label classification algorithms. However, most of them do not consider the data characteristics to build the ensemble, and those that consider them need to tune many parameters to maximize their performance.In this paper, we propose an Auto-adaptive algorithm based on Grammar-Guided Genetic Programming to generate Ensembles of Multi-Label Classifiers based on projections of k labels (AG3P-kEMLC). It creates a tree-shaped ensemble, where each leaf is a multi-label classifier focused on a subset of k labels. Unlike other methods in the literature, our proposal can deal with different values of k in the same ensemble, instead of fixing one specific value. It also includes an auto-adaptive process to reduce the number of hyper-parameters to tune, prevent overfitting and reduce the runtime required to execute it. Three versions of the algorithm are proposed. The first, fixed, uses the same value of k for all multi-label classifiers in the ensemble. The remaining two deal with different k values in the ensemble: uniform gives the same probability to choose each available value of k, and gaussian favors the selection of smaller values of k.The experimental study carried out considering twenty reference datasets and five evaluation metrics, compared with eleven ensemble methods demonstrates that our proposal performs significantly better than the state-of-the-art methods.

Spatial Distribution Assessment of Terrorist Attack Types Based on I-MLKNN Model

Terrorist attacks are harmful to lives and property and seriously affect the stability of the international community and economic development. Exploring the regularity of terrorist attacks and building a model for assessing the risk of terrorist attacks (a kind of public safety risk, and it means the possibility of a terrorist attack) are of great significance to the security and stability of the international community and to global anti-terrorism. We propose a fusion of Inverse Distance Weighting (IDW) and a Multi-label k-Nearest Neighbor (I-MLKNN)-based assessment model for terrorist attacks, which is in a grid-scale and considers 17 factors of socio-economic and natural environments, and applied the I-MLKNN assessment model to assess the risk of terrorist attacks in Southeast Asia. The results show the I-MLKNN multi-label classification algorithm is proven to be an ideal tool for the assessment of the spatial distribution of terrorist attacks, and it can assess the risk of different types of terrorist attacks, thus revealing the law of distribution of different types of terrorist attacks. The terrorist attack risk assessment results indicate that Armed Attacks, Bombing/Explosions and Facility/Infrastructure Attacks in Southeast Asia are high-risk terrorist attack events, and the southernmost part of Thailand and the Philippines are high-risk terrorist attack areas for terrorism. We do not only provide a reference for incorporating spatial features in multi-label classification algorithms, but also provide a theoretical basis for decision-makers involved in terrorist attacks, which is meaningful to the implementation of the international counter-terrorism strategy.

A federated approach for detecting the chest diseases using DenseNet for multi-label classification

Multi-label disease classification algorithms help to predict various chronic diseases at an early stage. Diverse deep neural networks are applied for multi-label classification problems to foresee multiple mutually non-exclusive classes or diseases. We propose a federated approach for detecting the chest diseases using DenseNets for better accuracy in prediction of various diseases. Images of chest X-ray from the Kaggle repository is used as the dataset in the proposed model. This new model is tested with both sample and full dataset of chest X-ray, and it outperforms existing models in terms of various evaluation metrics. We adopted transfer learning approach along with the pre-trained network from scratch to improve performance. For this, we have integrated DenseNet121 to our framework. DenseNets have a few focal points as they help to overcome vanishing gradient issues, boost up the feature propagation and reuse and also to reduce the number of parameters. Furthermore, gradCAMS are used as visualization methods to visualize the affected parts on chest X-ray. Henceforth, the proposed architecture will help the prediction of various diseases from a single chest X-ray and furthermore direct the doctors and specialists for taking timely decisions.

Multi-label text classification via joint learning from label embedding and label correlation

For the multi-label text classification problems with many classes, many existing multi-label classification algorithms become infeasible or suffer an unaffordable cost. Some researches hence perform the Label Space Dimension Reduction(LSDR) to solve this problem, but a number of methods ignore the sequence information of texts and the label correlation in the original label space, and treat the label as a meaningless multi-hot vector. In this paper, we put forward a multi-label text classification algorithm LELC(joint learning from Label Embedding and Label Correlation) based on the multi-layer attention and label correlation to solve the issue of multi-label text classification with a large number of class labels. Specifically, we firstly extract features through Bidirectional Gated Recurrent Unit Network(Bi-GRU), multi-layer attention and linear layers. Bi-GRU will capture the content information and sequence information of the text at the same time, and the attention mechanism can help us select the valid features related to labels. Then, we use matrix factorization to perform LSDR, and consider label correlation of the original label space in this process, which allows us to implicitly encode the latent space and simplify the model learning. Finally, Deep Canonical Correlation Analysis(CCA) technology is exploited to couple features and the latent space in an end-to-end pattern, so that these two can influence each other to learn the mapping of feature space to latent space. Experiments on 11 real-world datasets show the comparability between our proposed model and the state-of-the-art methods.

Social Media Content Analysis and Classification Using Data Mining and ML

Students' natural conversations on social media such as Twitter and Whatsapp are useful to understand their learning experiences feelings. Collecting and analyzing data from such media can be a difficult task. However, the large scale of data is required for automatic data analysis techniques to classify Twitter data. The proposed new system is a combination of qualitative analysis and large-scale data mining and ML techniques. This system focuses on engineering students' Twitter posts, which are collected from engineering colleges, to understand issues and problems in their learning. The authors first conduct a qualitative analysis using ML studio on tweets collected from engineering colleges using term #DStudentsproblems, engineeringProblem, Aluminisuggestions, and ladyEngineer. Collected tweets are related to engineering students' college lives. In the proposed system, a multi-label classification algorithm to classify tweets reflecting students' problems such as soft skill issues, heavy study load, lack of social engagement, and sleep problems is used.

Multi-label classification of fundus images based on graph convolutional network

BackgroundDiabetic Retinopathy (DR) is the most common and serious microvascular complication in the diabetic population. Using computer-aided diagnosis from the fundus images has become a method of detecting retinal diseases, but the detection of multiple lesions is still a difficult point in current research.MethodsThis study proposed a multi-label classification method based on the graph convolutional network (GCN), so as to detect 8 types of fundus lesions in color fundus images. We collected 7459 fundus images (1887 left eyes, 1966 right eyes) from 2282 patients (1283 women, 999 men), and labeled 8 types of lesions, laser scars, drusen, cup disc ratio (C/D>0.6), hemorrhages, retinal arteriosclerosis, microaneurysms, hard exudates and soft exudates. We constructed a specialized corpus of the related fundus lesions. A multi-label classification algorithm for fundus images was proposed based on the corpus, and the collected data were trained.ResultsThe average overall F1 Score (OF1) and the average per-class F1 Score (CF1) of the model were 0.808 and 0.792 respectively. The area under the ROC curve (AUC) of our proposed model reached 0.986, 0.954, 0.946, 0.957, 0.952, 0.889, 0.937 and 0.926 for detecting laser scars, drusen, cup disc ratio, hemorrhages, retinal arteriosclerosis, microaneurysms, hard exudates and soft exudates, respectively.ConclusionsOur results demonstrated that our proposed model can detect a variety of lesions in the color images of the fundus, which lays a foundation for assisting doctors in diagnosis and makes it possible to carry out rapid and efficient large-scale screening of fundus lesions.

A novel binary many-objective feature selection algorithm for multi-label data classification

For Multi-label classification, redundant and irrelevant features degrade the performance of classification. To select the best features based on several conflicting objectives, feature selection can be modeled as a large-scale optimization problem. However, most existing multi-objective feature selection methods select the features based on minimizing two well-known objectives, the number of features and classification error, additional objectives can be considered to improve the classification performance. In this study, for the first time, a many-objective optimization method is proposed to select the efficient features for multi-label classification based on not only two mentioned objectives, but also maximizing the correlation between features and labels and minimizing the computational complexity of features. Maximizing the correlation could lead to increasing the accuracy of classification. On the other hand, selecting less complex features decreases the computational complexity of feature extraction phase. The most important aim of this paper is to tackle the multi-label feature selection based on the number of features, classification error, correlation between features and labels, and computational complexity of features, simultaneously. The conducted many-objective feature selection problem is solved using a proposed binary version of NSGA-III algorithm. The binary operator improves the exploration power of optimizer to search the large-scale space. In order to evaluate the proposed algorithm (called binary NSGA-III), a benchmarking experiments is conducted on eight multi-label datasets in terms of several multi-objective assessment metrics, including Hypervolume indicator, Pure Diversity, and Set-coverage. Experimental results show significant improvements for proposed method in comparison with other algorithms.

Sentiment analysis of GO-JEK services quality using Multi-Label Classification

One of the most popular online transportation providers in Indonesia is GO-JEK. At the first establishment, GO-JEK is only an online taxi motorbike service (in Bahasa Indonesia: ojek) that transforms from a conventional taxi motorbike. After several years GO-JEK began to develop more services, like GO-FOOD, GO-SEND, GO-CAR, GO-MART, GO-RIDE, GO-PAY, GO-TIX, GO-BOX, and GO-MED. As the GO-JEK services develop into more categories, it is more challenging to automatically analyze the sentiment polarity for each category of services category. An ordinary classification algorithm or single-label classification is concerned with learning from a set of examples associated with a single label classification. However, in this case, we want to classify GO-JEK services based on two-class targets, which are GO-JEK service categories and polarity sentiment classification. This research methodology contains Dataset Preparation, Feature Selection, Basic text mining process, train, and split dataset, and Classification. We implemented two classification methods: Multi-Label Classification and a simultaneous classification using the Random Forest Algorithm as a comparison. Based on this dataset, the most mentioned GO-JEK service is GO-FOOD followed by GO-SEND, GO-RIDE, GO-CAR, and GO-MART. Based on the service category, GO-FOOD gets the most positive reviews following by GO-RIDE and GOJEK 90. Some service categories like GO-SEND and GO-MART get more negative reviews than positive reviews. The accuracy of the Multi-label classification method raised 76%. Simultaneous classification accuracy using the Random Forest algorithm produce for service category classification yields 97% and only 78% for sentiment polarity classification. We can see that both algorithm, multi-label classification, and Random Forest algorithm, yields almost the same classification accuracy for polarity sentiment classification. We can conclude that the imperfect accuracy in polarity sentiment classification is related to the difficulty of identifying the most suitable polarity for each tweet. Many sarcasm tweets and slang words that cannot be identified.