Error-Correcting Output Codes Method Research Articles

Multiclass classification by Min–Max ECOC with Hamming distance optimization

Two questions often arise in the field of the ensemble in multiclass classification problems, (i) how to combine base classifiers and (ii) how to design possible binary classifiers. Error-correcting output codes (ECOC) methods answer these questions, but they focused on only the general goodness of the classifier. The main purpose of our research was to strengthen the bottleneck of the ensemble method, i.e., to minimize the largest values of two types of error ratios in the deep neural network-based classifier. The research was theoretical and experimental, the proposed Min–Max ECOC method suggests a theoretically proven optimal solution, which was verified by experiments on image datasets. The optimal solution was based on the maximization of the lowest value in the Hamming matrix coming from the ECOC matrix. The largest ECOC matrix, the so-called full matrix is always a Min–Max ECOC matrix, but smaller matrices generally do not reach the optimal Hamming distance value, and a recursive construction algorithm was proposed to get closer to it. It is not easy to calculate optimal values for large ECOC matrices, but an interval with upper and lower limits was constructed by two theorems, and they were proved. Convolutional Neural Networks with Min–Max ECOC matrix were tested on four real datasets and compared with OVA (one versus all) and variants of ECOC methods in terms of known and two new indicators. The experimental results show that the suggested method surpasses the others, thus our method is promising in the ensemble learning literature.

Classification of Tree Species in Different Seasons and Regions Based on Leaf Hyperspectral Images

This paper aims to establish a tree species identification model suitable for different seasons and regions based on leaf hyperspectral images, and to mine a more effective hyperspectral identification algorithm. Firstly, the reflectance spectra of leaves in different seasons and regions were analyzed. Then, to solve the problem that 0-element in sparse random (SR) coding matrices affects the classification performance of error-correcting output codes (ECOC), two versions of supervision-mechanism-based ECOC algorithms, namely SM-ECOC-V1 and SM-ECOC-V2, were proposed in this paper. In addition, the performance of the proposed algorithms was compared with that of six traditional algorithms based on all bands and feature bands. The experiment results show that seasonal and regional changes have an effect on the reflectance spectra of leaves, especially in the near-infrared region of 760–1000 nm. When the spectral information of different seasons and different regions is added into the identification model, tree species can be effectively classified. SM-ECOC-V2 achieves the best classification performance based on both all bands and feature bands. Furthermore, both SM-ECOC-V1 and SM-ECOC-V2 outperform the ECOC method under SR coding strategy, indicating the proposed methods can effectively avoid the influence of 0-element in SR coding matrix on classification performance.

Method of Code Description of Classes for Solving Multi-Class Problem

The method of code description of classes, which is a development of the ECOC (error-correcting output codes) method, is grounded theoretically. The main difference is as follows: a multiset of code descriptions of its training objects is used instead of one object code. It is shown that under certain conditions the two methods are equivalent. Ways for improving the recognition quality if code descriptions are used are shown.

An empirical comparison on state-of-the-art multi-class imbalance learning algorithms and a new diversified ensemble learning scheme

Class-imbalance learning is one of the most challenging problems in machine learning. As a new and important direction in this field, multi-class imbalanced data classification has attracted a great many research focus in recent years. In this paper, we first make a very comprehensive review on state-of-the-art classification algorithms for multi-class imbalanced data. Moreover, we propose a new multi-class imbalance classification algorithm, which is hereafter referred to as the Diversified Error Correcting Output Codes (DECOC) method. The main idea of DECOC is to combine the improved ECOC (Error Correcting Output Codes) method for tackling class imbalance, and the diversified ensemble learning framework, which finds the best classification algorithm (out of many heterogeneous classification algorithms) for each individual sub-dataset resampled from the original data. We conduct experiments on 19 public datasets to empirically compare the performance of DECOC with 17 state-of-the-art multi-class imbalance learning algorithms, using 4 different accuracy measures: overall accuracy, Geometric mean, F-measure, and Area Under Curve. Experimental results demonstrate that DECOC achieves significantly better accuracy performance than the other 17 algorithms on these accuracy metrics. To advance research in this field, we make all the source codes of DECOC and the above-mentioned 17 state-of-the-art algorithms for imbalanced data classification be available at GitHub: https://github.com/chongshengzhang/Multi_Imbalance.

Improving multiclass classification using neighborhood search in error correcting output codes

Error Correcting Output Code (ECOC) is an effective approach for multiclass classification problems. This method decomposes a multiclass problem to many binary sub-problems and makes a dichotomizer for each sub-problem. It then tries to classify samples by combining outputs of all dichotomizers. One of the main points in ECOC method is to construct an ensemble of independent binary classifiers. Many studies have been conducted to design an optimal ECOC matrix. However, most of these methods aim to construct an ECOC code Matrix without considering the relations between data to design an ensemble of binary classifiers. In this study, a new method is presented based on ECOC which improves the performance of sparse ECOC by considering the neighborhood of samples. The proposed method is evaluated using 16 UCI datasets. The results indicate that our method not only significantly improves the classification accuracy compared to other commonly used ECOC based methods, but it also can result in a lower number of classifiers in comparison with random dense ECOC with the same accuracy.

Error-Correcting Output Codes in Classification of Human Induced Pluripotent Stem Cell Colony Images.

The purpose of this paper is to examine how well the human induced pluripotent stem cell (hiPSC) colony images can be classified using error-correcting output codes (ECOC). Our image dataset includes hiPSC colony images from three classes (bad, semigood, and good) which makes our classification task a multiclass problem. ECOC is a general framework to model multiclass classification problems. We focus on four different coding designs of ECOC and apply to each one of them k-Nearest Neighbor (k-NN) searching, naïve Bayes, classification tree, and discriminant analysis variants classifiers. We use Scaled Invariant Feature Transformation (SIFT) based features in classification. The best accuracy (62.4%) is obtained with ternary complete ECOC coding design and k-NN classifier (standardized Euclidean distance measure and inverse weighting). The best result is comparable with our earlier research. The quality identification of hiPSC colony images is an essential problem to be solved before hiPSCs can be used in practice in large-scale. ECOC methods examined are promising techniques for solving this challenging problem.

A Comparison of Error Correcting Output Coding Methods for Multiclass Classification by Using Support Vector Machine: The Prediction of Self-Monitoring of Blood Sugar

SVMs were initially developed to perform binary classification. However, in many real-world problems, particularly pattern recocnation studies, aimed to determine the distinctive features of large number of class or group. For this reason, a number of methods to generate multiclass SVMs from binary SVMs have been proposed by researchers and this is still a continuing research topic. In this study we aimed to compare classification accuracy and computational cost of four multiclass approaches using a original and simulated data sets. Error Corrected Output Coding (ECOC) based multiclass approaches that is used in this study creates many binary classifiers and combines their results to determine the class label of a test pixel. As a result of the comparisons for all conditions examined in this study, it’s found that the classification accuracy and computational cost of One vs. One multiclass approach is better than the other multiclass approaches. In classification or pattern recognization problems, some of supervised machine learning methods or algorithms can be easily extended to multiclass problems. However, some other powerful and popular classifiers, such as AdaBoost and Support Vector Machines, do not extend to multiclass easily. In those situations, the usual way to proceed is to reduce the complexity of the multiclass problem into multiple simpler binary classification problems.

Multi-label classification using error correcting output codes

A framework for multi-label classification extended by Error Correcting Output Codes (ECOCs) is introduced and empirically examined in the article. The solution assumes the base multi-label classifiers to be a noisy channel and applies ECOCs in order to recover the classification errors made by individual classifiers. The framework was examined through exhaustive studies over combinations of three distinct classification algorithms and four ECOC methods employed in the multi-label classification problem. The experimental results revealed that (i) the Bode-Chaudhuri-Hocquenghem (BCH) code matched with any multi-label classifier results in better classification quality; (ii) the accuracy of the binary relevance classification method strongly depends on the coding scheme; (iii) the label power-set and the RAkEL classifier consume the same time for computation irrespective of the coding utilized; (iv) in general, they are not suitable for ECOCs because they are not capable to benefit from ECOC correcting abilities; (v) the all-pairs code combined with binary relevance is not suitable for datasets with larger label sets.

Error-correcting output codes based ensemble feature extraction

This paper proposes a novel feature extraction method based on ensemble learning. Using the error-correcting output codes (ECOC) to design binary classifiers (dichotomizers) for separating subsets of classes, the outputs of the dichotomizers are linear or nonlinear features that provide powerful separability in a new space. In this space, the vector quantization based meta classifier can be viewed as an ECOC decoder, where each learned prototype of a class can be seen as a codeword of the class in the new representation space. We conducted extensive experiments on 16 multi-class data sets from the UCI machine learning repository. The results demonstrate the superiority of the proposed method over both existing ECOC approaches and classic feature extraction approaches. In particular, the decoding strategy using a meta classifier is shown to be more computationally efficient than the linear loss-weighted decoding in state-of-the-art ECOC methods.

A subspace approach to error correcting output codes

Among the proposed methods to deal with multi-class classification problems, the error-correcting output codes (ECOCs) represents a powerful framework. A key factor in designing any ECOC matrix is the independency of the binary classifiers, without which the ECOC method would be ineffective. This paper proposes an efficient new approach to the classical ECOC design in order to improve independency among classifiers. The main idea of the proposed method is based on using different feature subsets for each binary classifier, named subspace ECOC. In addition to creating more independent classifiers in the proposed technique, ECOC matrices with longer codes can be built. The numerical experiments in this study compare the classification accuracy of subspace ECOC, classical ECOC, one-versus-one, and one-versus-all methods over a set of UCI machine learning repository datasets and two image vision applications. The results show that the proposed technique increases the classification accuracy in comparison with the state of the art coding methods.

Pedestrian Detection in Video Images via Error Correcting Output Code Classification of Manifold Subclasses

Pedestrian detection in images and video frames is challenged by the view and posture problem. In this paper, we propose a new pedestrian detection approach by error correcting output code (ECOC) classification of manifold subclasses. The motivation is that pedestrians across views and postures form a manifold and that the ECOC method constructs a nonlinear classification boundary that can discriminate the manifold from negative samples. The pedestrian manifold is first constructed with a local linear embedding algorithm and then divided into subclasses with a -means clustering algorithm. The neighboring relationships of these subclasses are used to make the encoding rule for ECOCs, which we use to train multiple base classifiers with histogram of oriented gradient features and linear support vector machines. In the detection procedure, image windows are tested with all base classifiers, and their output codes are fed into an ECOC decoding procedure to decide whether it is a pedestrian or not. Experiments on three data sets show that the results of our approach improve the state of the art.

Biomarker selection and sample prediction for multi-category disease on MALDI-TOF data

Diseases normally progress through several stages. Therefore, biomarkers corresponding to each stage may exist. To deal with such a multi-category problem, including sample stage prediction and biomarker selection, we propose methods for classification and feature selection. The proposed classification method is based on two schemes: error-correcting output coding (ECOC) and pairwise coupling (PWC). The final decision for a test sample prediction is an integration of these two schemes. The biomarker pattern for distinguishing each disease category from another one is achieved by the development of an extended Markov blanket (EMB) feature selection method. In this study, a liver cancer matrix-assisted laser desorption/ionization time-of-flight (MALDI-TOF) mass spectrometry (MS) dataset was used, which comprises hepatocellular carcinoma (HCC), cirrhosis, and healthy spectra. Peak patterns were discovered for distinguishing pairwise categories among the three classes. Importance and reliability of individual peaks were presented by the measurements of certain weight values and frequencies. The classification capability of the proposed approach was compared with classical ECOC, random forest, Naive Bayes, and J48 methods. Supplementary materials are available at http://visionlab.uta.edu/biomarker/bioinfo.htm.

Data-driven decomposition for multi-class classification

This paper presents a new study on a method of designing a multi-class classifier: Data-driven Error Correcting Output Coding (DECOC). DECOC is based on the principle of Error Correcting Output Coding (ECOC), which uses a code matrix to decompose a multi-class problem into multiple binary problems. ECOC for multi-class classification hinges on the design of the code matrix. We propose to explore the distribution of data classes and optimize both the composition and the number of base learners to design an effective and compact code matrix. Two real world applications are studied: (1) the holistic recognition (i.e., recognition without segmentation) of touching handwritten numeral pairs and (2) the classification of cancer tissue types based on microarray gene expression data. The results show that the proposed DECOC is able to deliver competitive accuracy compared with other ECOC methods, using parsimonious base learners than the pairwise coupling (one-vs-one) decomposition scheme. With a rejection scheme defined by a simple robustness measure, high reliabilities of around 98% are achieved in both applications.

Design of Output Codes for Fast Covering Learning using Basic Decomposition Techniques

We propose the design of output codes for solving the classification problem in Fast Covering Learning Algorithm (FCLA). For a complex multi-class problem normally the classifiers are constructed by combining the outputs of several binary ones. In this paper, we use the basic methods of decomposition; one per class (OPC) and Error Correcting Output Code (ECOC) with FCLA, binary to binary mapping algorithm as a base binary learner. The methods have been tested on Fisher’s well-known Iris data set and experimental results show that the classification ability is improved by using ECOC method.

Effectiveness of error correcting output coding methods in ensemble and monolithic learning machines

Error Correcting Output Coding (ECOC) methods for multiclass classification present several open problems ranging from the trade-off between their error recovering capabilities and the learnability of the induced dichotomies to the selection of proper base learners and to the design of well-separated codes for a given multiclass problem. We experimentally analyse some of the main factors affecting the effectiveness of ECOC methods. We show that the architecture of ECOC learning machines influences the accuracy of the ECOC classifier, highlighting that ensembles of parallel and independent dichotomic Multi-Layer Perceptrons are well-suited to implement ECOC methods. We quantitatively evaluate the dependence among codeword bit errors using mutual information based measures, experimentally showing that a low dependence enhances the generalisation capabilities of ECOC. Moreover we show that the proper selection of the base learner and the decoding function of the reconstruction stage significantly affects the performance of the ECOC ensemble. The analysis of the relationships between the error recovering power, the accuracy of the base learners, and the dependence among codeword bits show that all these factors concur to the effectiveness of ECOC methods in a not straightforward way, very likely dependent on the distribution and complexity of the data.

Stochastic organization of output codes in multiclass learning problems.

The best-known decomposition schemes of multiclass learning problems are one per class coding (OPC) and error-correcting output coding (ECOC). Both methods perform a prior decomposition, that is, before training of the classifier takes place. The impact of output codes on the inferred decision rules can be experienced only after learning. Therefore, we present a novel algorithm for the code design of multiclass learning problems. This algorithm applies a maximum-likelihood objective function in conjunction with the expectation-maximization (EM) algorithm. Minimizing the augmented objective function yields the optimal decomposition of the multiclass learning problem in two-class problems. Experimental results show the potential gain of the optimized output codes over OPC or ECOC methods.