Error-Correcting Output Codes Framework Research Articles

A STUDY PAPER ON ERROR CORRECTING OUTPUT CODE BUILD ON MULTICLASS CLASSIFICATION

Error-correcting output codes is a technique for using binary classification models on multi-class classification prediction tasks. Error-Correcting Output Codes (ECOC) represents a successful framework to deal with these kinds of problems. Recent works in the ECOC framework appear notable performance improvements. The ECOC framework is a high-level tool to deal with multi-class categorization problems. As the error correcting output codes have error correcting ability and improve the generalization ability to base classification. This library contains both modern coding (one-versus-one, one-versus- all, dense-random, sparserandom, DECOC, forest- ECOC, and ECOC-ONE) and decoding designs (hamming, Euclidean, inverse hamming, laplacian, β-density, density, attenuated, lossbased, probabilistic kernel-based, and loss weighted) with the framework defined by the authors, as well as the option to include your own coding, decoding, and base classifier.

Categorized Image Classification uing CNN Features with ECOC Framework

Image Classification technique is used to classify images into categories. In this study, an application is presented to examine category based image classification by combining Support Vector Machine with error correcting output codes (ECOC) framework. The ResNet50 used as Network architecture, our image dataset include caltech101 images from 9 categories (classes) which builds our classification task a multiclass problem. ECOC is a commonly used framework to model multiclass classification problem. We present one-verses-all coding design of ECOC and apply to SVM classifier. A pre-trained CNN (convolution neural network) is used for extracting image feature and as a classifier Multiclass Support Vector Machine is used. The extracted features are then passed for classification via ECOC approach. The final classification result predicts the class labels. The application is implemented in Matlab using pre-trained CNN. The prediction accuracy of each category is evaluated and presented. The experimental result shows an accuracy of 97.6%. Further experiments are carried out on different dataset which showed that best accuracy is achieved using CNN with ECOC for multiclass problem.

A novel ECOC algorithm for multiclass microarray data classification based on data complexity analysis

Nowadays, a lot of new classification and clustering techniques have been proposed for microarray data analysis. However, the multiclass microarray data classification is still regarded as a tough task because of the small sample size problem and the class imbalance problem. In this paper, we propose a novel error correcting output code (ECOC) algorithm for the classification of multiclass microarray data based on the data complexity (DC) theory. In this algorithm, an ECOC coding matrix is generated based on a hierarchical partition of the class space with the aim of Minimizing Data Complexity (named as ECOC-MDC). As the partition process can be mapped as a binary tree, a compact ensemble with high discrimination power is produced. The performance of ECOC-MDC is compared with some state-of-art ECOC algorithms on six multiclass microarray data sets, and it is found that the proposed algorithm can obtain better results in most cases. The correlation between DC measures and the dichotomizers’ performances is checked, and the observations confirm that high complexity in data usually leads to high error rates of the connected dichotomizers. But the error correcting mechanism in the ECOC framework can effectively improve our algorithm's generalization ability. In short, ECOC-MDC can produce a compact ensemble system with high error correction capability through the application of diverse DC measures. Our Matlab code is available at: github.com/MLDMXM2017/ECOC-MDC.

A discriminant binarization transform using genetic algorithm and error-correcting output code for face template protection

In a biometric cryptosystem, biometric templates are encrypted before being stored in the database. This approach often requires a binarization phase to transform the original real-valued templates into their binary versions. Most of the existing binarization schemes do not take into consideration the need to discriminate binary templates; which in turn degrade the system recognition accuracy. This paper proposes a new binarization transform for face template protection using Error-Correcting Output Code (ECOC) and genetic algorithm (GA) which offsets this drawback. The proposed binarization transform novelty lies in the fact that the specific crossover, mutation and extension operators are defined by considering the theoretical properties of the ECOC framework in order to find an optimized coding matrix. The specification of these operations reduce the search space and speed up the convergence rate of GA. In addition, the proposed method is immune from being trapped in local minima. The quality of our method is evaluated using three well-known face databases: CMU PIE, Extended Yale B and FEI. The results demonstrate that the proposed binarization transform can result in a higher recognition accuracy without sacrificing the security of the protocol compared to alternative methods, such as binary discriminant analysis (BDA), discriminability-preserving (DP), chaos biohashing, and Discriminant-Genetic.

Learning ECOC Code Matrix for Multiclass Classification with Application to Glaucoma Diagnosis.

Classification of different mechanisms of angle closure glaucoma (ACG) is important for medical diagnosis. Error-correcting output code (ECOC) is an effective approach for multiclass classification. In this study, we propose a new ensemble learning method based on ECOC with application to classification of four ACG mechanisms. The dichotomizers in ECOC are first optimized individually to increase their accuracy and diversity (or interdependence) which is beneficial to the ECOC framework. Specifically, the best feature set is determined for each possible dichotomizer and a wrapper approach is applied to evaluate the classification accuracy of each dichotomizer on the training dataset using cross-validation. The separability of the ECOC codes is maximized by selecting a set of competitive dichotomizers according to a new criterion, in which a regularization term is introduced in consideration of the binary classification performance of each selected dichotomizer. The proposed method is experimentally applied for classifying four ACG mechanisms. The eye images of 152 glaucoma patients are collected by using anterior segment optical coherence tomography (AS-OCT) and then segmented, from which 84 features are extracted. The weighted average classification accuracy of the proposed method is 87.65 % based on the results of leave-one-out cross-validation (LOOCV), which is much better than that of the other existing ECOC methods. The proposed method achieves accurate classification of four ACG mechanisms which is promising to be applied in diagnosis of glaucoma.

HuPBA8k+: Dataset and ECOC-Graph-Cut based segmentation of human limbs

Human multi-limb segmentation in RGB images has attracted a lot of interest in the research community because of the huge amount of possible applications in fields like Human–Computer Interaction, Surveillance, eHealth, or Gaming. Nevertheless, human multi-limb segmentation is a very hard task because of the changes in appearance produced by different points of view, clothing, lighting conditions, occlusions, and number of articulations of the human body. Furthermore, this huge pose variability makes the availability of large annotated datasets difficult. In this paper, we introduce the HuPBA8k+ dataset. The dataset contains more than 8000 labeled frames at pixel precision, including more than 120000 manually labeled samples of 14 different limbs. For completeness, the dataset is also labeled at frame-level with action annotations drawn from an 11 action dictionary which includes both single person actions and person–person interactive actions. Furthermore, we also propose a two-stage approach for the segmentation of human limbs. In the first stage, human limbs are trained using cascades of classifiers to be split in a tree-structure way, which is included in an Error-Correcting Output Codes (ECOC) framework to define a body-like probability map. This map is used to obtain a binary mask of the subject by means of GMM color modelling and Graph-Cuts theory. In the second stage, we embed a similar tree-structure in an ECOC framework to build a more accurate set of limb-like probability maps within the segmented user mask that are fed to a multi-label Graph-Cut procedure to obtain final multi-limb segmentation. The methodology is tested on the novel HuPBA8k+ dataset, showing performance improvements in comparison to state-of-the-art approaches. In addition, a baseline of standard action recognition methods for the 11 actions categories of the novel dataset is also provided.

On the design of an ECOC-Compliant Genetic Algorithm

Genetic Algorithms (GA) have been previously applied to Error-Correcting Output Codes (ECOC) in state-of-the-art works in order to find a suitable coding matrix. Nevertheless, none of the presented techniques directly take into account the properties of the ECOC matrix. As a result the considered search space is unnecessarily large. In this paper, a novel Genetic strategy to optimize the ECOC coding step is presented. This novel strategy redefines the usual crossover and mutation operators in order to take into account the theoretical properties of the ECOC framework. Thus, it reduces the search space and lets the algorithm to converge faster. In addition, a novel operator that is able to enlarge the code in a smart way is introduced. The novel methodology is tested on several UCI datasets and four challenging computer vision problems. Furthermore, the analysis of the results done in terms of performance, code length and number of Support Vectors shows that the optimization process is able to find very efficient codes, in terms of the trade-off between classification performance and the number of classifiers. Finally, classification performance per dichotomizer results shows that the novel proposal is able to obtain similar or even better results while defining a more compact number of dichotomies and SVs compared to state-of-the-art approaches.

Generalized multi-scale stacked sequential learning for multi-class classification

In many classification problems, neighbor data labels have inherent sequential relationships. Sequential learning algorithms take benefit of these relationships in order to improve generalization. In this paper, we revise the multi-scale sequential learning approach (MSSL) for applying it in the multi-class case (MMSSL). We introduce the error-correcting output codesframework in the MSSL classifiers and propose a formulation for calculating confidence maps from the margins of the base classifiers. In addition, we propose a MMSSL compression approach which reduces the number of features in the extended data set without a loss in performance. The proposed methods are tested on several databases, showing significant performance improvement compared to classical approaches.

A genetic-based subspace analysis method for improving Error-Correcting Output Coding

Two key factors affecting the performance of Error Correcting Output Codes (ECOC) in multiclass classification problems are the independence of binary classifiers and the problem-dependent coding design. In this paper, we propose an evolutionary algorithm-based approach to the design of an application-dependent codematrix in the ECOC framework. The central idea of this work is to design a three-dimensional codematrix, where the third dimension is the feature space of the problem domain. In order to do that, we consider the feature space in the design process of the codematrix with the aim of improving the independence and accuracy of binary classifiers. The proposed method takes advantage of some basic concepts of ensemble classification, such as diversity of classifiers, and also benefits from the evolutionary approach for optimizing the three-dimensional codematrix, taking into account the problem domain. We provide a set of experimental results using a set of benchmark datasets from the UCI Machine Learning Repository, as well as two real multiclass Computer Vision problems. Both sets of experiments are conducted using two different base learners: Neural Networks and Decision Trees. The results show that the proposed method increases the classification accuracy in comparison with the state-of-the-art ECOC coding techniques.

Automatic Detection of Dominance and Expected Interest

Social Signal Processing is an emergent area of research that focuses on the analysis of social constructs. Dominance and interest are two of these social constructs. Dominance refers to the level of influence a person has in a conversation. Interest, when referred in terms of group interactions, can be defined as the degree of engagement that the members of a group collectively display during their interaction. In this paper, we argue that only using behavioral motion information, we are able to predict the interest of observers when looking at face-to-face interactions as well as the dominant people. First, we propose a simple set of movement-based features from body, face, and mouth activity in order to define a higher set of interaction indicators. The considered indicators are manually annotated by observers. Based on the opinions obtained, we define an automatic binary dominance detection problem and amulticlass interest quantification problem. Error-Correcting Output Codes framework is used to learn to rank the perceived observer's interest in face-to-face interactions meanwhile Adaboost is used to solve the dominant detection problem. The automatic system shows good correlation between the automatic categorization results and the manual ranking made by the observers in both dominance and interest detection problems.

Error-Correcting Ouput Codes Library

In this paper, we present an open source Error-Correcting Output Codes (ECOC) library. The ECOC framework is a powerful tool to deal with multi-class categorization problems. This library contains ...

On the Decoding Process in Ternary Error-Correcting Output Codes

A common way to model multiclass classification problems is to design a set of binary classifiers and to combine them. Error-Correcting Output Codes (ECOC) represent a successful framework to deal with these type of problems. Recent works in the ECOC framework showed significant performance improvements by means of new problem-dependent designs based on the ternary ECOC framework. The ternary framework contains a larger set of binary problems because of the use of a "do not care" symbol that allows us to ignore some classes by a given classifier. However, there are no proper studies that analyze the effect of the new symbol at the decoding step. In this paper, we present a taxonomy that embeds all binary and ternary ECOC decoding strategies into four groups. We show that the zero symbol introduces two kinds of biases that require redefinition of the decoding design. A new type of decoding measure is proposed, and two novel decoding strategies are defined. We evaluate the state-of-the-art coding and decoding strategies over a set of UCI Machine Learning Repository data sets and into a real traffic sign categorization problem. The experimental results show that, following the new decoding strategies, the performance of the ECOC design is significantly improved.

Re-coding ECOCs without re-training

A standard way to deal with multi-class categorization problems is by the combination of binary classifiers in a pairwise voting procedure. Recently, this classical approach has been formalized in the Error-Correcting Output Codes (ECOC) framework. In the ECOC framework, the one-versus-one coding demonstrates to achieve higher performance than the rest of coding designs. The binary problems that we train in the one-versus-one strategy are significantly smaller than in the rest of designs, and usually easier to be learnt, taking into account the smaller overlapping between classes. However, a high percentage of the positions coded by zero of the coding matrix, which implies a high sparseness degree, does not codify meta-class membership information. In this paper, we show that using the training data we can redefine without re-training, in a problem-dependent way, the one-versus-one coding matrix so that the new coded information helps the system to increase its generalization capability. Moreover, the new re-coding strategy is generalized to be applied over any binary code. The results over several UCI Machine Learning repository data sets and two real multi-class problems show that performance improvements can be obtained re-coding the classical one-versus-one and Sparse random designs compared to different state-of-the-art ECOC configurations.

Blurred Shape Model for binary and grey-level symbol recognition

Many symbol recognition problems require the use of robust descriptors in order to obtain rich information of the data. However, the research of a good descriptor is still an open issue due to the high variability of symbols appearance. Rotation, partial occlusions, elastic deformations, intra-class and inter-class variations, or high variability among symbols due to different writing styles, are just a few problems. In this paper, we introduce a symbol shape description to deal with the changes in appearance that these types of symbols suffer. The shape of the symbol is aligned based on principal components to make the recognition invariant to rotation and reflection. Then, we present the Blurred Shape Model descriptor (BSM), where new features encode the probability of appearance of each pixel that outlines the symbols shape. Moreover, we include the new descriptor in a system to deal with multi-class symbol categorization problems. Adaboost is used to train the binary classifiers, learning the BSM features that better split symbol classes. Then, the binary problems are embedded in an Error-Correcting Output Codes framework (ECOC) to deal with the multi-class case. The methodology is evaluated on different synthetic and real data sets. State-of-the-art descriptors and classifiers are compared, showing the robustness and better performance of the present scheme to classify symbols with high variability of appearance.

Separability of ternary codes for sparse designs of error-correcting output codes

Error-correcting output codes (ECOC) represent a successful framework to deal with multi-class categorization problems based on combining binary classifiers. With the extension of the binary ECOC to the ternary ECOC framework, ECOC designs have been proposed in order to better adapt to distributions of the data. In order to decode ternary matrices, recent works redefined many decoding strategies that were formulated to deal with just two symbols. However, the coding step also is affected, and therefore, it requires to be reconsidered. In this paper, we present a new formulation of the ternary ECOC distance and the error-correcting capabilities in the ternary ECOC framework. Based on the new measure, we stress on how to design coding matrices preventing codification ambiguity and propose a new sparse random coding matrix with ternary distance maximization. The results on a wide set of UCI Machine Learning Repository data sets and in a real speed traffic sign categorization problem show that when the coding design satisfies the new ternary measures, significant performance improvement is obtained independently of the decoding strategy applied.

An improved error-correcting output coding framework with kernel-based decoding

This paper presents a novel framework of error-correcting output coding (ECOC) addressing the problem of multi-class classification. By weighting the output space of each base classifier which is trained independently, the distance function of decoding is adapted so that the samples are more discriminative. A criterion generated over the Extended Pair Samples (EPS) is proposed to train the weights of output space. We first conduct empirical studies on UCI datasets to verify the presented framework with four frequently used coding matrixes and then apply it in RoboCup domain to enhance the performance of agent control. Color-based image segmentation is also tested in this paper. Experimental results show that our supervised learned decoding scheme improves the accuracy of classification significantly and betters the ball control of agents in a soccer game after learning from experience. Some properties still hold in the new framework: any classifier, as well as distance function, is still applicable. Besides, it is allowed in the new framework to implement nonlinear decoding processes using kernel tricks.

Intravascular Ultrasound Tissue Characterization with Sub-class Error-Correcting Output Codes

Intravascular ultrasound (IVUS) represents a powerful imaging technique to explore coronary vessels and to study their morphology and histologic properties. In this paper, we characterize different tissues based on radial frequency, texture-based, and combined features. To deal with the classification of multiple tissues, we require the use of robust multi-class learning techniques. In this sense, error-correcting output codes (ECOC) show to robustly combine binary classifiers to solve multi-class problems. In this context, we propose a strategy to model multi-class classification tasks using sub-classes information in the ECOC framework. The new strategy splits the classes into different sub-sets according to the applied base classifier. Complex IVUS data sets containing overlapping data are learnt by splitting the original set of classes into sub-classes, and embedding the binary problems in a problem-dependent ECOC design. The method automatically characterizes different tissues, showing performance improvements over the state-of-the-art ECOC techniques for different base classifiers. Furthermore, the combination of RF and texture-based features also shows improvements over the state-of-the-art approaches.

Subclass Problem-Dependent Design for Error-Correcting Output Codes

A common way to model multi-class classification problems is by means of Error-Correcting Output Codes (ECOC). Given a multi-class problem, the ECOC technique designs a code word for each class, where each position of the code identifies the membership of the class for a given binary problem. A classification decision is obtained by assigning the label of the class with the closest code. One of the main requirements of the ECOC design is that the base classifier is capable of splitting each sub-group of classes from each binary problem. However, we can not guarantee that a linear classifier model convex regions. Furthermore, non-linear classifiers also fail to manage some type of surfaces. In this paper, we present a novel strategy to model multi-class classification problems using sub-class information in the ECOC framework. Complex problems are solved by splitting the original set of classes into sub-classes, and embedding the binary problems in a problem-dependent ECOC design. Experimental results show that the proposed splitting procedure yields a better performance when the class overlap or the distribution of the training objects conceil the decision boundaries for the base classifier. The results are even more significant when one has a sufficiently large training size.

Combining multiple decisions: applications to bioinformatics

Multi-class classification is one of the fundamental tasks in bioinformatics and typically arises in cancer diagnosis studies by gene expression profiling. This article reviews two recent approaches to multi-class classification by combining multiple binary classifiers, which are formulated based on a unified framework of error-correcting output coding (ECOC). The first approach is to construct a multi-class classifier in which each binary classifier to be aggregated has a weight value to be optimally tuned based on the observed data. In the second approach, misclassification of each binary classifier is formulated as a bit inversion error with a probabilistic model by making an analogy to the context of information transmission theory. Experimental studies using various real-world datasets including cancer classification problems reveal that both of the new methods are superior or comparable to other multi-class classification methods.

Coding and decoding strategies for multi-class learning problems

It is known that the error correcting output code (ECOC) technique, when applied to multi-class learning problems, can improve generalisation performance. One reason for the improvement is its ability to decompose the original problem into complementary two-class problems. Binary classifiers trained on the sub-problems are diverse and can benefit from combining using a simple distance-based strategy. However there is some discussion about why ECOC performs as well as it does, particularly with respect to the significance of the coding/decoding strategy. In this paper we consider the binary (0,1) code matrix conditions necessary for reduction of error in the ECOC framework, and demonstrate the desirability of equidistant codes. It is shown that equidistant codes can be generated by using properties related to the number of 1’s in each row and between any pair of rows. Experimental results on synthetic data and a few popular benchmark problems show how performance deteriorates as code length is reduced for six decoding strategies.