Error Correcting Output Codes Algorithms Research Articles

An adaptive error-correcting output codes algorithm based on gene expression programming and similarity measurement matrix

The multi-class classification task is one of the most common tasks in machine learning. As a typical solution based on a partitioning strategy, Error-Correcting Output Codes (ECOC) can transform a multi-class classification problem into multiple binary classification problems. The key of ECOC is to construct an effective codematrix to represent a set of class decomposition schemes, which transforms a multiclass problem into a group of binary class problems. Consequently, the design of a fast and effective ECOC codematrix generation method is of great research significance and value for solving multi-class classification problems. In ECOC algorithms, the design of codematrix is treated as a combination problem between different code columns, in which the evolutionary algorithm shows a great advantage. Based on this consideration, the Gene Expression Programming (GEP) is applied to search for the codematrix with high performance because its expressive tree structure makes it well represent codematrcies for subsequent optimization operations. This paper proposes an adaptive ECOC algorithm based on Gene Expression Programming (GEP) and similarity measurement matrix, named GEP-ECOC. In our GEP, each individual represents a set of columns to form a random ECOC codematrix, which is optimized in the evolutionary process. Meanwhile, the crossover and mutation operations are modified to include a legality checking process to ensure that the generated codematrix satisfies the ECOC constraints. The GEP-based ECOC codematrix generation algorithm can quickly produce a codematrix with better performance, which ensures the efficiency of the algorithm to a certain extent. In addition, an adaptive algorithm based on a similarity measurement matrix is proposed to add new columns to the current codematrix, aiming to better handle hard classes. Our algorithm is compared with other algorithms on various data sets, and the experimental results confirm that our GEP-ECOC can balance the efficiency and performance of the algorithm and achieve higher performance.

A self-adaptive soft-recoding strategy for performance improvement of error-correcting output codes

The technique of error-correcting output codes (ECOC) has been proven to be of high discriminative ability in many classification applications. However, most algorithms on the ECOC were designed based on the binary or ternary codes (referred to as the hard codes), which might fail to precisely correct errors in dealing with tough tasks. In this study, a Soft-Recoding strategy based on a self-adaptive algorithm is proposed, which replaces the traditional hard codes with the real-value elements to better fit the output distribution of the base learners. This is achieved by minimizing the ratio of two distances: the distance of the output vector to the ground-truth class, and the average distance of the output vector to the remaining classes. Extensive experiments using five different hard ECOC algorithms and the corresponding softened versions on twenty datasets with diversified numbers of features and classes confirm the effectiveness of our Soft-Recoding strategy in promoting the performance of the original ECOC algorithms. Our source code and additional results are available at: github.com/MLDMXM2017/SA-soft-recoding.

A novel soft-coded error-correcting output codes algorithm

Error-Correcting Output Codes (ECOC) algorithms enable multiclass classification by reassigning multiple classes to the positive/negative group with the class reassignment schemes being recorded as binary/ternary hard-coded (HC) codematrices. Different classes tend to get diverse subordination degrees to the positive/negative group, providing clues to correct potential errors. However, the HC codematrices are unable to provide the information in the subordination degrees. In this paper, a Soft-Coded ECOC (SC-ECOC) scheme, namely, the Sequential Forward Floating Selection algorithm, is proposed by filling codematrices with real values instead of hard codes to improve classification performance. This algorithm divides multiple classes into two groups by maximizing the ratio of inter-group distance to intra-group distance. Then a new measure coverage is designed to evaluate the subordination degrees of different classes to both groups, which are set as the elements to form a codematrix. Furthermore, a self-adaptive strategy adjusts the value of each element to fit learners better. Experiments are carried out to verify the performance of our algorithm on various data sets, and results confirm that our algorithm can achieve more balanced results compared with the traditional HC ECOC algorithms. Besides, the values of soft codes correlate with the difficulty level of various classes to improve the multiclass classification ability.

Atrial Fibrillation Discrimination for Real-Time ECG Monitoring Based On QT Interval Variation

Background/Objectives: An occasional Atrial Fibrillation (AF) event in heart rhythm should be monitored regularly, in continuous intervals. Timely detection of these anomalies in heart rhythm is required to save patients from sudden cardiac arrest. Method: A long-duration ECG categorization algorithm named AFECOC is proposed. For this one-minute-long 71 signals are attained from the Physionet’s “MIT-BIH arrhythmia (MA)” and “AF” database. Two-stage filtering of noisy signals is employed before signal analysis. Four algorithms i.e. Error-Correcting Output Code (ECOC), Naïve Bayes, Decision Tree, and K-Nearest Neighbor(K-NN) are applied to reduce the feature set and then signals are classified with ECOC classifier. Findings: It was found that the ECOC algorithm gives the highest accuracy of 81.95% on the complete feature set. To exclude the irrelevant features, the highest performing algorithm ECOC was used that extracts the combination of the feature sets that get most affected during AF. The combination of ’heart-beat’ and ’mean QT-interval’ are found to be the most relevant features affected during AF events. The accuracy of these two features was evaluated with four classifiers namely ECOC, Naïve Bayes, Decision tree-based and K mean classifier and the accuracy obtained was 89.6%, 76.19%, 76.19%, and 61% respectively. It concludes that the proposed methodology achieved the highest accuracy of 89.6% with the ECOC classifier. Finally, all the AF rhythms have been checked using annotated labels for spontaneous change in QT-interval to verify the designed methodologies. Novelty: Instead of missing P-waves and RR-interval variation, recognition of mean QT interval variation-based AF event detection algorithm gives better accuracy for longer signals. Hence, it can be implemented in Real- Time continuous monitoring. Keywords: Atrial Fibrillation (AF); Classification; Error Correcting Output Code (ECOC); Feature extraction; QTinterval

Classification of Tree Species in Different Seasons and Regions Based on Leaf Hyperspectral Images

This paper aims to establish a tree species identification model suitable for different seasons and regions based on leaf hyperspectral images, and to mine a more effective hyperspectral identification algorithm. Firstly, the reflectance spectra of leaves in different seasons and regions were analyzed. Then, to solve the problem that 0-element in sparse random (SR) coding matrices affects the classification performance of error-correcting output codes (ECOC), two versions of supervision-mechanism-based ECOC algorithms, namely SM-ECOC-V1 and SM-ECOC-V2, were proposed in this paper. In addition, the performance of the proposed algorithms was compared with that of six traditional algorithms based on all bands and feature bands. The experiment results show that seasonal and regional changes have an effect on the reflectance spectra of leaves, especially in the near-infrared region of 760–1000 nm. When the spectral information of different seasons and different regions is added into the identification model, tree species can be effectively classified. SM-ECOC-V2 achieves the best classification performance based on both all bands and feature bands. Furthermore, both SM-ECOC-V1 and SM-ECOC-V2 outperform the ECOC method under SR coding strategy, indicating the proposed methods can effectively avoid the influence of 0-element in SR coding matrix on classification performance.

The design of dynamic ensemble selection strategy for the error-correcting output codes family

Error-Correcting Output Codes (ECOC) is widely deployed to tackle the multiclass classification problem by reducing the original multi-class problem to several binary sub-problems. This study attempts to design a dynamic ensemble selection strategy to promote the performance of ECOC algorithms. Concretely, each column in a coding matrix is matched with a set of feature subsets generated by various feature selection methods. In the decoding process, a novel criterion based on the data complexity theory is proposed to pick up an optimal feature subset from the candidate subsets, so as to better distinguish unknown samples. As this strategy can be embedded in all types of ECOC algorithms, seven classical ECOC algorithms are deployed to verify the effectiveness of our strategy. Experiments are carried out on a set of UCI data sets, and the results confirm that despite different working principle, the proposed strategy can further improve the performance of various ECOC algorithms in most cases. Our python source code is available at: https://github.com/MLDMXM2017/ECOC_DES .

A novel multi-objective genetic algorithm based error correcting output codes

Up to now, different genetic algorithm (GA) based error correcting output codes (ECOC) algorithms have been proposed by setting accuracy as the optimization objective. However, it was demonstrated that diversity among learners is of great significance to a robust ensemble. In this paper, we propose a multi-objective GA with setting accuracy and diversity as two objectives. To further promote diversity in an ensemble, a new individual structure is designed to accommodate heterogeneous dichotomizers. Three multi-objective ranking strategies are deployed to balance two objectives respectively. A novel genetic operator is designed to produce ECOC-compatible offspring in the evolutionary process, and a local improvement algorithm is designed to promote individuals’ fitness values. To verify the performance of our GA, a single objective ranking strategy and the design of homogeneous learner based GA are also adopted. Ten widely used ECOC algorithms and three famous ensemble algorithms are deployed for performance comparisons based on a set of the UCI data and microarray data sets. Results show that compared with other algorithms, our GA obtains higher performance in most cases due to the trade-off between performance and diversity. Besides, the accommodation of heterogeneous dichotomizers in an ensemble provides higher generalization ability compared with homogeneous ensembles.

A new re-encoding ECOC using reject option

When training base classifier by ternary Error Correcting Output Codes (ECOC), it is well know that some classes are ignored. On this account, a non-competent classifier emerges when it classify an instance whose real label does not belong to the meta-subclasses. Meanwhile, the classic ECOC dichotomizers can only produce binary outputs and have no capability of rejection for classification. To overcome the non-competence problem and better model the multi-class problem for reducing the classification cost, we embed reject option to ECOC and present a new variant of ECOC algorithm called as Reject-Option-based Re-encoding ECOC (ROECOC). The cost-sensitive classification model and cost-loss function based on Receiver Operating Characteristic (ROC) curve are built respectively. The optimal reject threshold values are obtained by combing the condition to be met for minimizing the loss function and the ROC convex hull. In so doing, reject option (t1, t2) provides a three-symbol output to make dichotomizers more competent and ROECOC more universal and practical for cost-sensitive classification issue. Experimental results on two kinds of datasets show that our scheme with low-degree freedom of initialized ECOC can effectively enhance accuracy and reduce cost.

Imprecise Deep Forest for Partial Label Learning

In partial label (PL) learning, each instance corresponds to a set of candidate labels, among which only one is valid. The objective of PL learning is to obtain a multi-class classifier from the training instances. Because the true label of a PL training instance is hidden in the candidate label set and inaccessible to the learning algorithm, the training process of the classifier is significantly challenging. This study proposes a novel deep learning method for PL learning based on the improved error-correcting output codes (ECOC) algorithm and deep forest (DF) framework. For the ECOC algorithm, we extract the prior knowledge of the candidate label sets from the PL training set to optimize the generation of its coding matrix, where different binary training sets can be derived from the PL training set based on the dichotomy corresponding to each column code. For the DF framework, this improved ECOC algorithm is embedded as a unit in its cascade structure; moreover, an imprecise evaluation method is designed to determine the growth of the cascade of the DF. The effectiveness of the proposed method is verified by conducting several experiments on artificial and real-world PL datasets.

A Hybrid Reducing Error Correcting Output Code for Lithology Identification

Lithology information is critical to the adjustment of drilling control strategies, and can be identified by training a classification model from the well logging data. However, achieving accurate lithology identification is rather difficult owing to complex characteristics, such as data imbalance, data-overlapping, and multi-classification. In this brief, a hybrid lithology identification method is developed based on the Reducing Error Correcting Output Code algorithm with the Kernel Fisher Discriminant Analysis (RECOC-KFDA). The effectiveness of the proposed method is demonstrated based on case studies with the UCI machine learning database and the real logging data. The results show that the proposed method has superior performances compared to conventional methods.

A novel Error-Correcting Output Codes algorithm based on genetic programming

Error-Correcting Output Codes (ECOC) is widely used in the field of multiclass classification. As an optimal codematrix is key to the performance of an ECOC algorithm, this paper proposes a genetic programming (GP) based ECOC algorithm (GP-ECOC). In the design of individual of our GP, each terminal node represents a class, and nonterminal nodes combine the classes in their child nodes. In this way, an individual is a class combination tree, and represents an ECOC codematrix. A legality checking process is embedded in our algorithm to check each codematrix, so as to ensure each codematrix satisfying ECOC constraints. Those violating the constraints will be corrected by a proposed Guided Mutation operator. Before fitness evaluation, a local optimization algorithm is proposed to append new columns for tough classes, so as to improve the generalization ability of each individual and accelerate the evolutionary speed. In this way, our GP can evolve optimal codematrices through the evolutionary process. Experiments show that compared with other ensemble algorithms, our algorithm can achieve stable and high performances with relatively small ensemble scales on various UCI data sets. To the best of our knowledge, it is the first time that GP has been applied to implement the ECOC encoding algorithm. Our Python code is available at https://github.com/samuellees/gpecoc.

A novel ECOC algorithm for multiclass microarray data classification based on data complexity analysis

Nowadays, a lot of new classification and clustering techniques have been proposed for microarray data analysis. However, the multiclass microarray data classification is still regarded as a tough task because of the small sample size problem and the class imbalance problem. In this paper, we propose a novel error correcting output code (ECOC) algorithm for the classification of multiclass microarray data based on the data complexity (DC) theory. In this algorithm, an ECOC coding matrix is generated based on a hierarchical partition of the class space with the aim of Minimizing Data Complexity (named as ECOC-MDC). As the partition process can be mapped as a binary tree, a compact ensemble with high discrimination power is produced. The performance of ECOC-MDC is compared with some state-of-art ECOC algorithms on six multiclass microarray data sets, and it is found that the proposed algorithm can obtain better results in most cases. The correlation between DC measures and the dichotomizers’ performances is checked, and the observations confirm that high complexity in data usually leads to high error rates of the connected dichotomizers. But the error correcting mechanism in the ECOC framework can effectively improve our algorithm's generalization ability. In short, ECOC-MDC can produce a compact ensemble system with high error correction capability through the application of diverse DC measures. Our Matlab code is available at: github.com/MLDMXM2017/ECOC-MDC.

Research on Intrusion Detection Algorithm Based on Multi-Class SVM in Wireless Sensor Networks

A multi-class method is proposed based on Error Correcting Output Codes algorithm in order to get better performance of attack recognition in Wireless Sensor Networks. Aiming to enhance the accuracy of attack detection, the multi-class method is constructed with Hadamard matrix and two-class Support Vector Machines. In order to minimize the complexity of the algorithm, sparse coding method is applied in this paper. The comprehensive experimental results show that this modified multi-class method has better attack detection rate compared with other three coding algorithms, and its time efficiency is higher than Hadamard coding algorithm.