Real-world Classification Problems Research Articles

A spectral-spatial kernel-based method for hyperspectral imagery classification

Spectral-based classification methods have gained increasing attention in hyperspectral imagery classification. Nevertheless, the spectral cannot fully represent the inherent spatial distribution of the imagery. In this paper, a spectral-spatial kernel-based method for hyperspectral imagery classification is proposed. Firstly, the spatial feature was extracted by using area median filtering (AMF). Secondly, the result of the AMF was used to construct spatial feature patch according to different window sizes. Finally, using the kernel technique, the spectral feature and the spatial feature were jointly used for the classification through a support vector machine (SVM) formulation. Therefore, for hyperspectral imagery classification, the proposed method was called spectral-spatial kernel-based support vector machine (SSF-SVM). To evaluate the proposed method, experiments are performed on three hyperspectral images. The experimental results show that an improvement is possible with the proposed technique in most of the real world classification problems.

Heterogeneous face recognition based on modality‐independent Kernel Fisher discriminant analysis joint sparse auto‐encoder

A novel method called modality-independent Kernel discriminant analysis joint sparse auto-encoder, for solving heterogeneous face recognition problem is proposed. A projection matrix to map multimodal data into a common feature space for representing cross-modal image data is first learnt. Then extend the model via sparse auto-encoder in an unsupervised manner with the combination of a regularisation term and a Kullback–Leiber divergence term. Different from classical approaches, this model does not require the data correspondences when collecting external cross-modal data. Thus, it is practical for real-world cross-modal classification problem. Experiments conducted on two heterogeneous face datasets demonstrate the effectiveness of the proposed approach.

Sparse Computation for Large-Scale Data Mining

Leading machine learning techniques rely on inputs in the form of pairwise similarities between objects in the data set. The number of pairwise similarities grows quadratically in the size of the data set which poses a challenge in terms of scalability. One way to achieve practical efficiency for similarity-based techniques is to sparsify the similarity matrix. However, existing sparsification approaches consider the complete similarity matrix and remove some of the non-zero entries. This requires quadratic time and storage and is thus intractable for large-scale data sets. We introduce here a method called sparse computation that generates a sparse similarity matrix which contains only relevant similarities without computing first all pairwise similarities. The relevant similarities are identified by projecting the data onto a low-dimensional space in which groups of objects that share the same grid neighborhood are deemed of potential high similarity whereas pairs of objects that do not share a neighborhood are considered to be dissimilar and thus their similarities are not computed. The projection is performed efficiently even for massively large data sets. We apply sparse computation for the $K$ -nearest neighbors algorithm (KNN), for graph-based machine learning techniques of supervised normalized cut and K-supervised normalized cut (SNC and KSNC) and for support vector machines with radial basis function kernels (SVM), on real-world classification problems. Our empirical results show that the approach achieves a significant reduction in the density of the similarity matrix, resulting in a substantial reduction in tuning and testing times, while having a minimal effect (and often none) on accuracy. The low-dimensional projection is of further use in massively large data sets where the grid structure allows to easily identify groups of “almost identical” objects. Such groups of objects are then replaced by representatives, thus reducing the size of the matrix. This approach is effective, as illustrated here for data sets comprising up to 8.5 million objects.

A novel and powerful hybrid classifier method: Development and testing of heuristic k-nn algorithm with fuzzy distance metric

Weight-tuning methods and distance metrics have a significant impact on the k-nearest neighbor-based classification. A major challenge is the issue of how to explore the optimal weight values of the features and how to measure distances between the neighbors affecting the classification accuracy of the k-nn. In this paper, a powerful similarity measurement method, which is called the fuzzy distance metric, is explained and extended to measure the distances between the test and training observations. Depending on the fuzzy metric, similarity arrays can be produced more efficiently than the classic and other weighted distance measurements. Finally, the weighting methods are combined with the fuzzy metric-based similarity measurement and the k-nearest neighbor algorithm to increase the classification accuracy of the proposed algorithm. The effectiveness of the proposed approaches is proven by comparing their performances with the performances of the classic and the population-based heuristic methods on the well-known, real-world classification problems obtained from the UCI machine-learning benchmark repository. The experimental results show that the proposed hybrid algorithms significantly explore more optimal weight vectors significantly and provide more accurate classification results than the powerful and well-known instance-based intuitive and heuristic classification algorithms and classic approaches over real datasets.

On improving the classification accuracy of extension theory

Extension theory (ET) has been criticized mainly because of the following: (1) requirement of specifying the classical domain by user-experience; (2) lack of providing useful summaries of asymmetric data; and (3) failure to implement data classification when attributes are categorical. This paper proposes two well-motivated modifications over the ET framework, called mode-based extension theory (mbET) and frequency-based extension theory (FET), which alleviate the three difficulties of traditional extension theory. Experiment results on 13 widely used UCI data sets show that mbET and FET can achieve superior classification accuracy compared to ET. The FET achieved the most outstanding improvement over ET in classification accuracy, which is statistically significant. According to the experiment results, FET was also found to be superior or comparable to other state-of-the-art classifiers. The experimental results also encourage the application of FET to more real-world classification problems.

Financial credit analysis via a clustering weightless neural classifier

Credit analysis is a real-world classification problem where it is quite common to find datasets with a large amount of noisy data. State-of-the-art classifiers that employ error minimisation techniques, on the other hand, require a long time to converge, in order to achieve robustness. This paper explores ClusWiSARD, a clustering customisation of the WiSARD weightless neural network model, applied to two different credit analysis real-world problems. Experimental evidence shows that ClusWiSARD is very competitive with Support Vector Machine (SVM) w.r.t. accuracy, with the advantage of being capable of online learning. ClusWiSARD outperforms SVM in training time, by two orders of magnitude, and is slightly faster in test time.

Classification with asymmetric label noise: Consistency and maximal denoising

In many real-world classification problems, the labels of training examples are randomly corrupted. Most previous theoretical work on classification with label noise assumes that the two classes are separable, that the label noise is independent of the true class label, or that the noise proportions for each class are known. In this work, we give conditions that are necessary and sufficient for the true class-conditional distributions to be identifiable. These conditions are weaker than those analyzed previously, and allow for the classes to be nonseparable and the noise levels to be asymmetric and unknown. The conditions essentially state that a majority of the observed labels are correct and that the true class-conditional distributions are “mutually irreducible,” a concept we introduce that limits the similarity of the two distributions. For any label noise problem, there is a unique pair of true class-conditional distributions satisfying the proposed conditions, and we argue that this pair corresponds in a certain sense to maximal denoising of the observed distributions. Our results are facilitated by a connection to “mixture proportion estimation,” which is the problem of estimating the maximal proportion of one distribution that is present in another. We establish a novel rate of convergence result for mixture proportion estimation, and apply this to obtain consistency of a discrimination rule based on surrogate loss minimization. Experimental results on benchmark data and a nuclear particle classification problem demonstrate the efficacy of our approach.

A Comparative Study of Locality Preserving Projection and Principle Component Analysis on Classification Performance Using Logistic Regression

There are a variety of classification techniques such as neural network, decision tree, support vector machine and logistic regression. The problem of dimensionality is pertinent to many learning algorithms, and it denotes the drastic raise of computational complexity, however, we need to use dimensionality reduction methods. These methods include principal component analysis (PCA) and locality preserving projection (LPP). In many real-world classification problems, the local structure is more important than the global structure and dimensionality reduction techniques ignore the local structure and preserve the global structure. The objectives is to compare PCA and LPP in terms of accuracy, to develop appropriate representations of complex data by reducing the dimensions of the data and to explain the importance of using LPP with logistic regression. The results of this paper find that the proposed LPP approach provides a better representation and high accuracy than the PCA approach.

Model-based Clustering of High-Dimensional Data in Astrophysics

The nature of data in Astrophysics has changed, as in other scientific fields, in the past decades due to the increase of the measurement capabilities. As a consequence, data are nowadays frequently of high dimensionality and available in mass or stream. Model-based techniques for clustering are popular tools which are renowned for their probabilistic foundations and their flexibility. However, classical model-based techniques show a disappointing behavior in high-dimensional spaces which is mainly due to their dramatical over-parametrization. The recent developments in model-based classification overcome these drawbacks and allow to efficiently classify high-dimensional data, even in the small n / large p situation. This work presents a comprehensive review of these recent approaches, including regularization-based techniques, parsimonious modeling, subspace classification methods and classification methods based on variable selection. The use of these model-based methods is also illustrated on real-world classification problems in Astrophysics using R packages.

Are random forests truly the best classifiers

The JMLR study Do we need hundreds of classifiers to solve real world classification problems? benchmarks 179 classifiers in 17 families on 121 data sets from UCI repository and claims that the random forest is clearly best family of classifier. In this response, we show that study's results are biased by lack of a held-out test set and exclusion of trials with errors. Further, study's own statistical tests indicate that random forests do not have significantly higher percent accuracy than support vector machines and neural networks, calling into question conclusion that random forests are best classifiers.

Comparative study of balancing methods: case of imbalanced medical data

Imbalanced learning problems contain unequal distribution of data samples among different classes, where most of the samples belong to some classes and the rest to the other classes. Learning from the imbalanced data is of utmost important to the research community as it is present in many vital real-world classification problems, such as medical diagnosis. There have been many works dealing with classification of imbalanced data sets. In medical data classification, we often face the imbalanced number of data samples where at least one of the classes constitutes only a very small minority of the data. In this paper, we proposed a learning method based on a cost-sensitive extension of Least Mean Square (LMS) algorithm that penalises errors of different samples with different weights and some rules of thumb to determine those weights. After the balancing phase, we apply different classifiers (Support Vector Machine [SVM], k-Nearest Neighbour [k-NN] and Multilayer Perceptron [MLP]) for the new balanced data set. We have also compared the results obtained by the LMS algorithm with the results obtained by the sampling techniques (under-sampling, oversampling and Synthetic Minority Oversampling Technique (SMOTE)).

Multiobjective learning algorithm based on membrane systems for optimizing the parameters of extreme learning machine

For adaptively learning the parameters of extreme learning machine (ELM), a novel learning algorithm is proposed on the basis of a multiobjective membrane algorithm. More specifically, first, a multiobjective mathematical model is established to learn the parameters of ELM, which is constructed by three objective functions. These objective functions include the root mean squared error, L_1 norm of output weights and the number of hidden nodes. Second, a series of the trade-off solutions with respect to the above-mentioned objective functions are found by the multiobjective membrane algorithm. Finally, a trade-off solution with the best generalization performance of ELM, which is chosen from the Pareto front obtained by the multiobjective algorithm, will become the final parameters for initializing the ELM network. The simulation experiments are run on the approximation of ‘SinC’ function, real-world regression problems and real-world classification problems. Experimental results indicate that the proposed framework is able to achieve good generalization performance in the most cases with many compact networks.

Enhancing Multiclass Classification in FARC-HD Fuzzy Classifier: On the Synergy Between $n$-Dimensional Overlap Functions and Decomposition Strategies

There are many real-world classification problems involving multiple classes, e.g., in bioinformatics, computer vision, or medicine. These problems are generally more difficult than their binary counterparts. In this scenario, decomposition strategies usually improve the performance of classifiers. Hence, in this paper, we aim to improve the behavior of fuzzy association rule-based classification model for high-dimensional problems (FARC-HD) fuzzy classifier in multiclass classification problems using decomposition strategies, and more specifically One-versus-One (OVO) and One-versus-All (OVA) strategies. However, when these strategies are applied on FARC-HD, a problem emerges due to the low-confidence values provided by the fuzzy reasoning method. This undesirable condition comes from the application of the product t -norm when computing the matching and association degrees, obtaining low values, which are also dependent on the number of antecedents of the fuzzy rules. As a result, robust aggregation strategies in OVO, such as the weighted voting obtain poor results with this fuzzy classifier. In order to solve these problems, we propose to adapt the inference system of FARC-HD replacing the product t -norm with overlap functions . To do so, we define n-dimensional overlap functions . The usage of these new functions allows one to obtain more adequate outputs from the base classifiers for the subsequent aggregation in OVO and OVA schemes. Furthermore, we propose a new aggregation strategy for OVO to deal with the problem of the weighted voting derived from the inappropriate confidences provided by FARC-HD for this aggregation method. The quality of our new approach is analyzed using 20 datasets and the conclusions are supported by a proper statistical analysis. In order to check the usefulness of our proposal, we carry out a comparison against some of the state-of-the-art fuzzy classifiers. Experimental results show the competitiveness of our method.

MI-ELM: Highly efficient multi-instance learning based on hierarchical extreme learning machine

Multi-instance learning (MIL) is one of promising paradigms in the supervised learning aiming to handle real world classification problems where a classification target contains several featured sections, e.g., an image typically contains several salient regions. In this paper, we propose a highly efficient learning method for MI classification based on hierarchical extreme learning machine (ELM), called MI-ELM. Specifically, a double-hidden layers feedforward network (DLFN) is designed to serve as the MI classifier. Then, the MI classification is formulated as an optimization problem. Moreover, the output weights of DLFN can be analytically determined by solving the aforementioned optimization problem. The merits of MI-ELM are as follows: (i) MI-ELM extends the single-layer ELM to be a hierarchical one that well fits for training DLFNs in MI classification. (ii) The input and hidden-layer parameters of DLFNs are assigned randomly rather than tuned iteratively, and the output weights of DLFNs can be determined analytically in one step. Therefore, MI-ELM significantly enhances the efficiency of the DLFN without notable loss of the classification accuracy. Experimental results over several real-world data sets demonstrate that the proposed MI-ELM method significantly outperforms existing kernel methods for MI classification in terms of the classification accuracy and the classification time.

Building feature space of extreme learning machine with sparse denoising stacked-autoencoder

The random-hidden-node extreme learning machine (ELM) is a much more generalized cluster of single-hidden-layer feed-forward neural networks (SLFNs) which has three parts: random projection, non-linear transformation, and ridge regression (RR) model. Networks with deep architectures have demonstrated state-of-the-art performance in a variety of settings, especially with computer vision tasks. Deep learning algorithms such as stacked autoencoder (SAE) and deep belief network (DBN) are built on learning several levels of representation of the input. Beyond simply learning features by stacking autoencoders (AE), there is a need for increasing its robustness to noise and reinforcing the sparsity of weights to make it easier to discover interesting and prominent features. The sparse AE and denoising AE was hence developed for this purpose. This paper proposes an approach: SSDAE-RR (stacked sparse denoising autoencoder – ridge regression) that effectively integrates the advantages in SAE, sparse AE, denoising AE, and the RR implementation in ELM algorithm. We conducted experimental study on real-world classification (binary and multiclass) and regression problems with different scales among several relevant approaches: SSDAE-RR, ELM, DBN, neural network (NN), and SAE. The performance analysis shows that the SSDAE-RR tends to achieve a better generalization ability on relatively large datasets (large sample size and high dimension) that were not pre-processed for feature abstraction. For 16 out of 18 tested datasets, the performance of SSDAE-RR is more stable than other tested approaches. We also note that the sparsity regularization and denoising mechanism seem to be mandatory for constructing interpretable feature representations. The fact that a SSDAE-RR approach often has a comparable training time to ELM makes it useful in some real applications.

Example-dependent cost-sensitive decision trees

Several real-world classification problems are example-dependent cost-sensitive in nature, where the costs due to misclassification vary between examples. However, standard classification methods do not take these costs into account, and assume a constant cost of misclassification errors. State-of-the-art example-dependent cost-sensitive techniques only introduce the cost to the algorithm, either before or after training, therefore, leaving opportunities to investigate the potential impact of algorithms that take into account the real financial example-dependent costs during an algorithm training. In this paper, we propose an example-dependent cost-sensitive decision tree algorithm, by incorporating the different example-dependent costs into a new cost-based impurity measure and a new cost-based pruning criteria. Then, using three different databases, from three real-world applications: credit card fraud detection, credit scoring and direct marketing, we evaluate the proposed method. The results show that the proposed algorithm is the best performing method for all databases. Furthermore, when compared against a standard decision tree, our method builds significantly smaller trees in only a fifth of the time, while having a superior performance measured by cost savings, leading to a method that not only has more business-oriented results, but also a method that creates simpler models that are easier to analyze.

Learning Subspace-Based RBFNN Using Coevolutionary Algorithm for Complex Classification Tasks.

Many real-world classification problems are characterized by samples of a complex distribution in the input space. The classification accuracy is determined by intrinsic properties of all samples in subspaces of features. This paper proposes a novel algorithm for the construction of radial basis function neural network (RBFNN) classifier based on subspace learning. In this paper, feature subspaces are obtained for every hidden node of the RBFNN during the learning process. The connection weights between the input layer and the hidden layer are adjusted to produce various subspaces with dominative features for different hidden nodes. The network structure and dominative features are encoded in two subpopulations that are cooperatively coevolved using the coevolutionary algorithm to achieve a better global optimality for the estimated RBFNN. Experimental results illustrate that the proposed algorithm is able to obtain RBFNN models with both better classification accuracy and simpler network structure when compared with other learning algorithms. Thus, the proposed model provides a more flexible and efficient approach to complex classification tasks by employing the local characteristics of samples in subspaces.

On the evolution of ellipsoidal recognition regions in Artificial Immune Systems

Using different shapes of recognition regions in Artificial Immune Systems (AIS) are not a new issue. Especially, ellipsoidal shapes seem to be more intriguing as they have also been used very effectively in other shape space-based classification methods. Some studies have done in AIS through generating ellipsoidal detectors but they are restricted in their detector generating scheme – Genetic Algorithms (GA). In this study, an AIS was developed with ellipsoidal recognition regions by inspiring from the clonal selection principle and an effective search procedure for ellipsoidal regions was applied. Performance evaluation tests were conducted as well as application results on some real-world classification problems taken from UCI machine learning repository were obtained. Comparison with GA was also done in some of these problems. Very effective and comparatively good classification ratios were recorded.

Semisupervised spherical separation

We embed the concept of spherical separation of two disjoint finite sets of points into the semisupervised framework. This approach improves efficiency in the solution of real-world classification problems in which the number of unlabeled points is very large and labeling data is in general expensive. We develop a model characterized by an error function which is nonconvex and nondifferentiable, that we minimize by means of a bundle method. Numerical results on some small/large datasets drawn from literature are reported.

Immune-inspired extreme learning machine: Performance evaluation and conceptual analyses

Recently, it was proposed a novel hybrid approach to train MLPs which combines the advantages of a powerful artificial immune system, called GAIS, with the advantages of Extreme Learning Machine ELM. In that proposal, the GAIS algorithm is responsible for finding a proper set of input weights whereas the output weights are determined by the Moore-Penrose generalized inverse. The methodology was evaluated only in classification problems and its performance compares favorably with that presented by state-of-the-art-algorithms. Motivated by this scenario, this paper better formalizes the proposal and performs a deeper investigation of its usefulness for synthesizing MLP and RBF neural networks on several real-world classification and regression problems. The computational experiments have shown that the proposed methodology outperforms other approaches in both quantitative and qualitative aspects.