Multi-label Evaluation Research Articles

CAFA-evaluator: a Python tool for benchmarking ontological classification methods.

https://pypi.org/project/cafaeval.

A syndrome differentiation model of TCM based on multi-label deep forest using biomedical text mining.

Syndrome differentiation and treatment is the basic principle of traditional Chinese medicine (TCM) to recognize and treat diseases. Accurate syndrome differentiation can provide a reliable basis for treatment, therefore, establishing a scientific intelligent syndrome differentiation method is of great significance to the modernization of TCM. With the development of biomdical text mining technology, TCM has entered the era of intelligence that based on data, and model training increasingly relies on the large-scale labeled data. However, it is difficult to form a large standard data set in the field of TCM due to the low degree of standardization of TCM data collection and the privacy protection of patients' medical records. To solve the above problem, a multi-label deep forest model based on an improved multi-label ReliefF feature selection algorithm, ML-PRDF, is proposed to enhance the representativeness of features within the model, express the original information with fewer features, and achieve optimal classification accuracy, while alleviating the problem of high data processing cost of deep forest models and achieving effective TCM discriminative analysis under small samples. The results show that the proposed model finally outperforms other multi-label classification models in terms of multi-label evaluation criteria, and has higher accuracy in the TCM syndrome differentiation problem compared with the traditional multi-label deep forest, and the comparative study shows that the use of PCC-MLRF algorithm for feature selection can better select representative features.

Accuracy of the Sentence-BERT Semantic Search System for a Japanese Database of Closed Medical Malpractice Claims

In this study, we developed a similar text retrieval system using Sentence-BERT (SBERT) for our database of closed medical malpractice claims and investigated its retrieval accuracy. We assigned each case in the database a short Japanese summary of the accident as well as two labels: the category was classified as a hospital department mainly, and the process indicated a failed medical procedure. We evaluated the accuracies of a similar text retrieval system with the two labels using three different multilabel evaluation metrics. For the encoders of SBERT, we employed two pretrained BERT models, UTH-BERT and NICT-BERT, that were trained on huge Japanese corpora, and we performed iterative optimization to train the SBERTs. The accuracies of the similar text retrieval systems using the trained SBERTs were more than 15 points higher than those of the Okapi BM25 system and the pretrained SBERT system.

CNN-BiLSTM-Attention: A multi-label neural classifier for short texts with a small set of labels

We propose a CNN-BiLSTM-Attention classifier to classify online short messages in Chinese posted by users on government web portals, so that a message can be directed to one or more government offices. Our model leverages every bit of information to carry out multi-label classification, to make use of different hierarchical text features and the labels information. In particular, our designed method extracts label meaning, the CNN layer extracts local semantic features of the texts, the BiLSTM layer fuses the contextual features of the texts and the local semantic features, and the attention layer selects the most relevant features for each label. We evaluate our model on two public large corpuses, and our high-quality handcraft e-government multi-label dataset, which is constructed by the text annotation tool doccano and consists of 29920 data points. Experimental results show that our proposed method is effective under common multi-label evaluation metrics, achieving micro-f1 of 77.22%, 84.42%, 87.52%, and marco-f1 of 77.68%, 73.37%, 83.57% on these three datasets respectively, confirming that our classifier is robust. We conduct ablation study to evaluate our label embedding method and attention mechanism. Moreover, case study on our handcraft e-government multi-label dataset verifies that our model integrates all types of semantic information of short messages based on different labels to achieve text classification.

Deep learning for religious and continent-based toxic content detection and classification

With time, numerous online communication platforms have emerged that allow people to express themselves, increasing the dissemination of toxic languages, such as racism, sexual harassment, and other negative behaviors that are not accepted in polite society. As a result, toxic language identification in online communication has emerged as a critical application of natural language processing. Numerous academic and industrial researchers have recently researched toxic language identification using machine learning algorithms. However, Nontoxic comments, including particular identification descriptors, such as Muslim, Jewish, White, and Black, were assigned unrealistically high toxicity ratings in several machine learning models. This research analyzes and compares modern deep learning algorithms for multilabel toxic comments classification. We explore two scenarios: the first is a multilabel classification of Religious toxic comments, and the second is a multilabel classification of race or toxic ethnicity comments with various word embeddings (GloVe, Word2vec, and FastText) without word embeddings using an ordinary embedding layer. Experiments show that the CNN model produced the best results for classifying multilabel toxic comments in both scenarios. We compared the outcomes of these modern deep learning model performances in terms of multilabel evaluation metrics.

Application of Uncertainty Thought Environment in Judicial Adjudication Based on Cognitive Psychology.

The uncertainty of judicial decision-making has a deep and extensive theoretical foundation. Theoretical analysis starts with a reflection on legal rationalism that challenges the legal certainty before delving deeply into the case's facts and the entire legal system. In light of this, this paper explores a novel approach to enhance the reasoning mechanism of trial documents from the viewpoint of modern cognitive psychology, concentrating on the parties' and the public's cognitive processes to justice. It is suggested to use an inert hierarchical multilabel classification algorithm. In order to predict the category of invisible examples, the extended multilabel training set is first searched for adjacent samples of invisible examples, and the classification weight and confidence of each category are then determined in accordance with these adjacent samples. The group of invisible examples is then anticipated. Experimental comparison demonstrates that this algorithm outperforms other prediction techniques; the macro accuracy, macro recall, and macro F1 of this method are, respectively, 0.896, 0.871, and 0.814. It has some advantages in many multilabel evaluation indexes when compared to other multilabel algorithms.

Randomized neural networks for multilabel classification

Multilabel classification is a supervised learning problem in which input instances belong to multiple output labels. In this paper, we propose noniterative randomization-based neural networks for multilabel classification. These multilabel neural networks are named as Multilabel Random Vector Functional Link Network (ML-RVFL), Multilabel Kernelized Random Vector Functional Link Network (ML-KRVFL), Multilabel Broad Learning System (ML-BLS), and Multilabel Fuzzy Broad Learning System (ML-FBLS). The output weights of these neural networks are computed using pseudoinverse. At the output layer, multilabel classification is performed by using an adaptive threshold function. The computation of output weights using pseudoinverse retains the faster computation power of these algorithms compared to iterative learning algorithms. The adaptive threshold function used in the proposed approach can consider the correlation among the output labels and the whole dataset for threshold computation. Five multilabel evaluation metrics evaluate the proposed multilabel neural networks on 12 benchmark datasets of various domains such as text, image, and genomics. The ML-KRVFL provides the overall best Friedman rankings on five evaluation metrics followed by ML-RVFL, ML-FBLS, and ML-BLS, respectively. Based on the experimentation results, the proposed ML-KRVFL, ML-RVFL, ML-FBLS, and ML-BLS perform better than other relevant multilabel approaches in the mentioned order.The proposed approaches are faster than other state-of-the-art iterative approaches and noniterative approaches in terms of running time.

Unsupervised law article mining based on deep pre-trained language representation models with application to the Italian civil code

Modeling law search and retrieval as prediction problems has recently emerged as a predominant approach in law intelligence. Focusing on the law article retrieval task, we present a deep learning framework named LamBERTa, which is designed for civil-law codes, and specifically trained on the Italian civil code. To our knowledge, this is the first study proposing an advanced approach to law article prediction for the Italian legal system based on a BERT (Bidirectional Encoder Representations from Transformers) learning framework, which has recently attracted increased attention among deep learning approaches, showing outstanding effectiveness in several natural language processing and learning tasks. We define LamBERTa models by fine-tuning an Italian pre-trained BERT on the Italian civil code or its portions, for law article retrieval as a classification task. One key aspect of our LamBERTa framework is that we conceived it to address an extreme classification scenario, which is characterized by a high number of classes, the few-shot learning problem, and the lack of test query benchmarks for Italian legal prediction tasks. To solve such issues, we define different methods for the unsupervised labeling of the law articles, which can in principle be applied to any law article code system. We provide insights into the explainability and interpretability of our LamBERTa models, and we present an extensive experimental analysis over query sets of different type, for single-label as well as multi-label evaluation tasks. Empirical evidence has shown the effectiveness of LamBERTa, and also its superiority against widely used deep-learning text classifiers and a few-shot learner conceived for an attribute-aware prediction task.

Ant-TD: Ant colony optimization plus temporal difference reinforcement learning for multi-label feature selection

In recent years, multi-label learning becomes a trending topic in machine learning and data mining. This type of learning deals with data that each instance is associated with more than one label. Feature selection is a pre-processing method, which can significantly improve the performance of the multi-label classification. In this paper, we propose a new multi-label feature selection method based on Ant Colony Optimization (ACO). The proposed method makes a significant difference among all ACO-based feature selection methods so that instead of using a static heuristic function, it uses a heuristic learning approach. Because heuristic functions influence the decision-making process of the ACO during the search process, using a heuristic learning approach can help the algorithm to search better in the search space. Here we propose a heuristic learning approach for the ACO heuristic function, which learns from experiences directly by using the Temporal Difference (TD) reinforcement learning algorithm. For heuristic learning, we need to model the ACO problem into a reinforcement learning problem. Thus, we model the feature selection search space into a Markovian Decision Process (MDP) where features represent the states (S), and selecting the unvisited features by each ant represents a set of actions (A). Reward signals (R) are composed of two criteria when ants take action. The ACO state transition rule, which is a combination of both probabilistic and greedy rules, forms the transition function (T) in MDP. In addition to the pheromone that is updated by the “global updating rule” in the ACO, the state-value function (V) is directly updated by the temporal difference formulation to form a learned heuristic function. We conduct various experiments on nine benchmark data and compare the classification performance over three multi-label evaluation measures against nine state-of-the-art multi-label feature selection methods. The results show that the proposed method outperforms competing methods significantly.

Classifier Chains: A Review and Perspectives

The family of methods collectively known as classifier chains has become a popular approach to multi-label learning problems. This approach involves chaining together off-the-shelf binary classifiers in a directed structure, such that individual label predictions become features for other classifiers. Such methods have proved flexible and effective and have obtained state-of-the-art empirical performance across many datasets and multi-label evaluation metrics. This performance led to further studies of the underlying mechanism and efficacy, and investigation into how it could be improved. In the recent decade, numerous studies have explored the theoretical underpinnings of classifier chains, and many improvements have been made to the training and inference procedures, such that this method remains among the best options for multi-label learning. Given this past and ongoing interest, which covers a broad range of applications and research themes, the goal of this work is to provide a review of classifier chains, a survey of the techniques and extensions provided in the literature, as well as perspectives for this approach in the domain of multi-label classification in the future. We conclude positively, with a number of recommendations for researchers and practitioners, as well as outlining key issues for future research.

A ranking-based feature selection for multi-label classification with fuzzy relative discernibility

Feature selection is a crucial pre-processing step for learning tasks to mitigate the “curse of dimensionality”, which is caused by irrelevant and redundant features in high-dimensional feature space. The fuzzy rough set is an effective tool in feature selection, and the fuzzy discernibility matrix, one of the generalized model of the fuzzy rough set, has attracted significant attention recently. However, with the complicated discrimination relation of multiple labels, the traditional fuzzy discernibility matrix does not fit well for multi-label data. To address this problem, in this paper, the fuzzy label discernibility relation and the fuzzy relative discernibility relation are defined firstly, which are derived from fuzzy discernibility relation. Subsequently, we present the feature discernibility significance which measures the discernibility ability of the conditional feature and selects the most relevant features using the value of discernibility significance. On this basis, we propose a ranking-based feature selection algorithm for multi-label classification with fuzzy relative discernibility. Finally, in terms of six widely-accepted multi-label evaluation metrics, a series of experiments is conducted based on the multi-label classifier with ten different multi-label datasets from Mulan library and MLL resource, to compare the performance of the proposed algorithm with four state-of-the-art multi-label feature selection algorithms. The experimental results demonstrate the superiority and effectiveness of the proposed algorithm in multi-label classification.

A multi-objective algorithm for multi-label filter feature selection problem

Feature selection is an important data preprocessing method before classification. Multi-objective optimization algorithms have been proved an effective way to solve feature selection problems. However, there are few studies on multi-objective optimization feature selection methods for multi-label data. In this paper, a multi-objective multi-label filter feature selection algorithm based on two particle swarms (MOMFS) is proposed. We use mutual information to measure the relevance between features and label sets, and the redundancy between features, which are taken as two objectives. In order to avoid Particle Swarm Optimization (PSO) from falling into the local optimum and obtaining a false Pareto front, we employ two swarms to optimize the two objectives separately and propose an improved hybrid topology based on particle’s fitness value. Furthermore, an archive maintenance strategy is introduced to maintain the distribution of archive. In order to study the effectiveness of the proposed algorithm, we select five multi-label evaluation criteria and perform experiments on seven multi-label data sets. MOMFS is compared with classic single-objective multi-label feature selection algorithms, multi-objective filter and wrapper feature selection algorithms. The experimental results show that MOMFS can effectively reduce the multi-label data dimension and perform better than other approaches on five evaluation criteria.

MLACO: A multi-label feature selection algorithm based on ant colony optimization

Nowadays, with emerge the multi-label datasets, the multi-label learning processes attracted interest and increasingly applied to different fields. In such learning processes, unlike single-label learning, instances have more than one class label simultaneously. Also, multi-label learning suffers from the curse of dimensionality, and thus, the feature selection becomes a difficult task. In this paper, we propose a novel multi-label relevance–redundancy feature selection method based on Ant colony optimization (ACO) for the first time, called MLACO. By introducing two unsupervised and supervised heuristic functions, MLACO tries to search in the features space to find the most promising features with the lowest redundancy (unsupervised) and highest relevancy with class labels (supervised) through several iterations. For speeding up the convergence of the algorithm, the normalized cosine similarity between features and class labels have been used as the initial pheromone of each ant. The proposed method does not take into account any learning algorithm, and it can be classified as a filter-based method. We compare the performance of the MLACO against five well-known and state-of-the-art feature selection methods using ML-KNN classifier. The experimental results on several frequently used datasets show the superiority of the MLACO in different multi-label evaluation measures criteria and runtime.

Multi-label optimal margin distribution machine

Multi-label support vector machine (Rank-SVM) is a classic and effective algorithm for multi-label classification. The pivotal idea is to maximize the minimum margin of label pairs, which is extended from SVM. However, recent studies disclosed that maximizing the minimum margin does not necessarily lead to better generalization performance, and instead, it is more crucial to optimize the margin distribution. Inspired by this idea, in this paper, we first introduce margin distribution to multi-label learning and propose multi-label Optimal margin Distribution Machine (mlODM), which optimizes the margin mean and variance of all label pairs efficiently. Extensive experiments in multiple multi-label evaluation metrics illustrate that mlODM outperforms SVM-style multi-label methods. Moreover, empirical study presents the best margin distribution and verifies the fast convergence of our method.

Correlation analysis of performance measures for multi-label classification

In many important application domains, such as text categorization, scene classification, biomolecular analysis and medical diagnosis, examples are naturally associated with more than one class label, giving rise to multi-label classification problems. This fact has led, in recent years, to a substantial amount of research in multi-label classification. In order to evaluate and compare multi-label classifiers, researchers have adapted evaluation measures from the single-label paradigm, like Precision and Recall; and also have developed many different measures specifically for the multi-label paradigm, like Hamming Loss and Subset Accuracy. However, these evaluation measures have been used arbitrarily in multi-label classification experiments, without an objective analysis of correlation or bias. This can lead to misleading conclusions, as the experimental results may appear to favor a specific behavior depending on the subset of measures chosen. Also, as different papers in the area currently employ distinct subsets of measures, it is difficult to compare results across papers. In this work, we provide a thorough analysis of multi-label evaluation measures, and we give concrete suggestions for researchers to make an informed decision when choosing evaluation measures for multi-label classification.

Learning safe multi-label prediction for weakly labeled data

In this paper we study multi-label learning with weakly labeled data, i.e., labels of training examples are incomplete, which commonly occurs in real applications, e.g., image classification, document categorization. This setting includes, e.g., (i) semi-supervised multi-label learning where completely labeled examples are partially known; (ii) weak label learning where relevant labels of examples are partially known; (iii) extended weak label learning where relevant and irrelevant labels of examples are partially known. Previous studies often expect that the learning method with the use of weakly labeled data will improve the performance, as more data are employed. This, however, is not always the cases in reality, i.e., weakly labeled data may sometimes degenerate the learning performance. It is desirable to learn safe multi-label prediction that will not hurt performance when weakly labeled data is involved in the learning procedure. In this work we optimize multi-label evaluation metrics ( $$\hbox {F}_1$$ score and Top-k precision) given that the ground-truth label assignment is realized by a convex combination of base multi-label learners. To cope with the infinite number of possible ground-truth label assignments, cutting-plane strategy is adopted to iteratively generate the most helpful label assignments. The whole optimization is cast as a series of simple linear programs in an efficient manner. Extensive experiments on three weakly labeled learning tasks, namely, (i) semi-supervised multi-label learning; (ii) weak label learning and (iii) extended weak label learning, clearly show that our proposal improves the safeness of using weakly labeled data compared with many state-of-the-art methods.

An Improved ML-kNN Algorithm by Fusing Nearest Neighbor Classification

Based on the ML-kNN multi-label classification algorithm of probability and statistics, the k neighbors of unclassified sample are implicitly deemed that they have the same effect on classification result while ignoring the influence of the distances between k neighbors and unclassified sample. This paper proposes an improved ML-kNN algorithm by fusing nearest neighbor classification: IML-kNN. On the basis of the traditional ML-kNN algorithm, the algorithm considers the influence of the nearest neighbor and k neighbors of unclassified sample. Numerical simulation results show that the IML-kNN algorithm can have a good classification effect on the multi-label evaluation metrics.

An ensemble-based approach for multi-view multi-label classification

Multi-label classification with multiple data views is a recent research field not much explored. This more flexible learning approach allows each pattern to be represented by several sets of attributes and each pattern can have simultaneously associated several labels. In this work, an ensemble-based approach, which enables the fusion of views at decision level by majority voting, is proposed. The study carried out on four data sets considering 27 multi-label evaluation metrics shows that our proposal overcomes and improves the results obtained by the individual views as well as the execution time and the performance of the classic approach which concatenates all the views in a single set of features.

Multi-label class assignment in land-use modelling

During the last two decades, a variety of models have been applied to understand and predict changes in land use. These models assign a single-attribute label to each spatial unit at any particular time of the simulation. This is not realistic because mixed use of land is quite common. A more detailed classification allowing the modelling of mixed land use would be desirable for better understanding and interpreting the evolution of the use of land. A possible solution is the multi-label (ML) concept where each spatial unit can belong to multiple classes simultaneously. For example, a cluster of summer houses at a lake in a forested area should be classified as water, forest and residential (built-up). The ML concept was introduced recently, and it belongs to the machine learning field. In this article, the ML concept is introduced and applied in land-use modelling. As a novelty, we present a land-use change model that allows ML class assignment using the k nearest neighbour (kNN) method that derives a functional relationship between land use and a set of explanatory variables. A case study with a rich data-set from Luxembourg using biophysical data from aerial photography is described. The model achieves promising results based on the well-known ML evaluation criteria. The application described in this article highlights the value of the multi-label k nearest neighbour method (MLkNN) for land-use modelling.

Multi-label classification of chronically ill patients with bag of words and supervised dimensionality reduction algorithms

ObjectiveThis research is motivated by the issue of classifying illnesses of chronically ill patients for decision support in clinical settings. Our main objective is to propose multi-label classification of multivariate time series contained in medical records of chronically ill patients, by means of quantization methods, such as bag of words (BoW), and multi-label classification algorithms. Our second objective is to compare supervised dimensionality reduction techniques to state-of-the-art multi-label classification algorithms. The hypothesis is that kernel methods and locality preserving projections make such algorithms good candidates to study multi-label medical time series. MethodsWe combine BoW and supervised dimensionality reduction algorithms to perform multi-label classification on health records of chronically ill patients. The considered algorithms are compared with state-of-the-art multi-label classifiers in two real world datasets. Portavita dataset contains 525 diabetes type 2 (DT2) patients, with co-morbidities of DT2 such as hypertension, dyslipidemia, and microvascular or macrovascular issues. MIMIC II dataset contains 2635 patients affected by thyroid disease, diabetes mellitus, lipoid metabolism disease, fluid electrolyte disease, hypertensive disease, thrombosis, hypotension, chronic obstructive pulmonary disease (COPD), liver disease and kidney disease. The algorithms are evaluated using multi-label evaluation metrics such as hamming loss, one error, coverage, ranking loss, and average precision. ResultsNon-linear dimensionality reduction approaches behave well on medical time series quantized using the BoW algorithm, with results comparable to state-of-the-art multi-label classification algorithms. Chaining the projected features has a positive impact on the performance of the algorithm with respect to pure binary relevance approaches. ConclusionsThe evaluation highlights the feasibility of representing medical health records using the BoW for multi-label classification tasks. The study also highlights that dimensionality reduction algorithms based on kernel methods, locality preserving projections or both are good candidates to deal with multi-label classification tasks in medical time series with many missing values and high label density.