High-dimensional Classification Research Articles

Feature selection methods and their combinations in high-dimensional classification of speaker likability, intelligibility and personality traits

This study focuses on feature selection in paralinguistic analysis and presents recently developed supervised and unsupervised methods for feature subset selection and feature ranking. Using the standard k-nearest-neighbors (kNN) rule as the classification algorithm, the feature selection methods are evaluated individually and in different combinations in seven paralinguistic speaker trait classification tasks. In each analyzed data set, the overall number of features highly exceeds the number of data points available for training and evaluation, making a well-generalizing feature selection process extremely difficult. The performance of feature sets on the feature selection data is observed to be a poor indicator of their performance on unseen data. The studied feature selection methods clearly outperform a standard greedy hill-climbing selection algorithm by being more robust against overfitting. When the selection methods are suitably combined with each other, the performance in the classification task can be further improved. In general, it is shown that the use of automatic feature selection in paralinguistic analysis can be used to reduce the overall number of features to a fraction of the original feature set size while still achieving a comparable or even better performance than baseline support vector machine or random forest classifiers using the full feature set. The most typically selected features for recognition of speaker likability, intelligibility and five personality traits are also reported.

Incorporating feature ranking and evolutionary methods for the classification of high-dimensional DNA microarray gene expression data

DNA microarray gene expression classification poses a challenging task to the machine learning domain. Typically, the dimensionality of gene expression data sets could go from several thousands to over 10,000 genes. A potential solution to this issue is using feature selection to reduce the dimensionality. The aim of this paper is to investigate how we can use feature quality information to improve the precision of microarray gene expression classification tasks. We propose two evolutionary machine learning models based on the eXtended Classifier System (XCS) and a typical feature selection methodology. The first one, which we call FS-XCS, uses feature selection for feature reduction purposes. The second model is GRD-XCS, which uses feature ranking to bias the rule discovery process of XCS. The results indicate that the use of feature selection/ranking methods is essential for tackling highdimensional classification tasks, such as microarray gene expression classification. However, the results also suggest that using feature ranking to bias the rule discovery process performs significantly better than using the feature reduction method. In other words, using feature quality information to develop a smarter learning procedure is more efficient than reducing the feature set. Our findings have shown that extracting feature quality information can assist the learning process and improve classification accuracy. On the other hand, relying exclusively on the feature quality information might potentially decrease the classification performance (e.g., using feature reduction). Therefore, we recommend a hybrid approach that uses feature quality information to direct the learning process by highlighting the more informative features, but at the same time not restricting the learning process to explore other features.

Assessing similarity of feature selection techniques in high-dimensional domains

Recent research efforts attempt to combine multiple feature selection techniques instead of using a single one. However, this combination is often made on an “ad hoc” basis, depending on the specific problem at hand, without considering the degree of diversity/similarity of the involved methods. Moreover, though it is recognized that different techniques may return quite dissimilar outputs, especially in high dimensional/small sample size domains, few direct comparisons exist that quantify these differences and their implications on classification performance. This paper aims to provide a contribution in this direction by proposing a general methodology for assessing the similarity between the outputs of different feature selection methods in high dimensional classification problems. Using as benchmark the genomics domain, an empirical study has been conducted to compare some of the most popular feature selection methods, and useful insight has been obtained about their pattern of agreement.

Identification Scheme of Surface Electromyography of Upper Limb Movement

In this study four pairs of Ag/AgCl surface electrodes respectively that pasted on the muscle belly of the biceps, triceps in the upper arm, palmaris longus and the brachioradialis muscle in the forearm are used to collect the electromyography (EMG) signals of six patterns of upper limb movements. The feature vectors of the EMG are extracted by wavelet decomposition method, and then these features are classified by the method of the support vector machine. Experimental results show that the wavelet decomposition is an effective feature extraction method. For small samples, nonlinear and high dimensional classification problems, the support vector machine method has better classification effect and generalization performance than the traditional BP neural network. To speed up the computation of the SVM and adapt the increase of the samples, the incremental SVM is also introduced. By comparing with the traditional SVM method, the advantage of the incremental SVM is demonstrated.

A multiclassifier approach for topology-based WiFi indoor localization

People localization is required for many novel applications like for instance proactive care for the elders or people suffering degenerative dementia such as Alzheimer's disease. This paper introduces a new system for people localization in indoor environments. It is based on a topology-based WiFi signal strength fingerprint approach. Accordingly, it is a robust, cheap, ubiquitous and non-intrusive system which does require neither the installation of extra hardware nor prior knowledge about the structure of the environment under consideration. The well-known curse of dimensionality critically emerges when dealing with complex environments. The localization task turns into a high dimensional classification task. Therefore, the core of the proposed framework is a fuzzy rule-based multiclassification system, using standard methodologies for the component classifier generation such as bagging and random subspace, along with fuzzy logic to deal with the huge uncertainty that is characteristic of WiFi signals. Achieved results in two real environments are encouraging, since they clearly overcome those ones provided by the well-known nearest neighbor fingerprint matching algorithm, which is usually considered as a baseline for WiFi localization.

Asymptotic convergence of dimension reduction based boosting in classification

In high dimensional classification problem, two stage method, reducing the dimension of predictor first and then applying the classification method, is a natural solution and has been widely used in many fields. The consistency of the two stage method is an important issue, since errors induced by dimension reduction method inevitably have impacts on the following classification method. As an effective method for classification problem, boosting has been widely used in practice. In this paper, we study the consistency of two stage method–dimension reduction based boosting algorithm (briefly DRB) for classification problem. Theoretical results show that Lipschitz condition on the base learner is required to guarantee the consistency of DRB. This theoretical findings provide useful guideline for application.

SeqFold: Genome-scale reconstruction of RNA secondary structure integrating high-throughput sequencing data

We present an integrative approach, SeqFold, that combines high-throughput RNA structure profiling data with computational prediction for genome-scale reconstruction of RNA secondary structures. SeqFold transforms experimental RNA structure information into a structure preference profile (SPP) and uses it to select stable RNA structure candidates representing the structure ensemble. Under a high-dimensional classification framework, SeqFold efficiently matches a given SPP to the most likely cluster of structures sampled from the Boltzmann-weighted ensemble. SeqFold is able to incorporate diverse types of RNA structure profiling data, including parallel analysis of RNA structure (PARS), selective 2′-hydroxyl acylation analyzed by primer extension sequencing (SHAPE-Seq), fragmentation sequencing (FragSeq) data generated by deep sequencing, and conventional SHAPE data. Using the known structures of a wide range of mRNAs and noncoding RNAs as benchmarks, we demonstrate that SeqFold outperforms or matches existing approaches in accuracy and is more robust to noise in experimental data. Application of SeqFold to reconstruct the secondary structures of the yeast transcriptome reveals the diverse impact of RNA secondary structure on gene regulation, including translation efficiency, transcription initiation, and protein-RNA interactions. SeqFold can be easily adapted to incorporate any new types of high-throughput RNA structure profiling data and is widely applicable to analyze RNA structures in any transcriptome.

HANDLING HIGHLY-DIMENSIONAL CLASSIFICATION TASKS WITH HIERARCHICAL GENETIC FUZZY RULE-BASED CLASSIFIERS

Many modern classification tasks are defined in highly-dimensional feature spaces. The derivation of high-performing genetic fuzzy rule-based classification systems (GFRBCSs) in such scenarios is a non-trivial task. This paper presents a framework for increasing the performance of GFRBCSs by creating a hierarchical fuzzy rule-based classifier. The proposed system is constructed through repeated invocations to a base GFRBCS procedure, considering at each step an input space fuzzy partition of a certain granularity. The best performing rules are inserted in the hierarchical rule base and the process is repeated again, considering a thicker granularity. The employed boosting scheme guides the algorithm in creating new rules to treat uncovered or misclassified patterns, thus monotonically increasing the performance of the classifier. Extensive experimental analysis in a number of real-world high-dimensional classification tasks proves the effectiveness of the proposed approach in increasing the performance of the base classifier, maintaining its interpretability to a considerable degree.

A Model Selection Criterion for Discriminant Analysis of Several Groups When the Dimension is Larger than the Total Sample Size

This article deals with a criterion for selection of variables for the multiple group discriminant analysis in high-dimensional data. The variable selection models considered for discriminant analysis in Fujikoshi (1985, 2002) are the ones based on additional information due to Rao (1948, 1970). Our criterion is based on Akaike information criterion (AIC) for this model. The AIC has been successfully used in the literature in model selection when the dimension p is smaller than the sample size N. However, the case when p > N has not been considered in the literature, because MLE can not be estimated corresponding to singularity of the within-group covariance matrix. A popular method used to address the singularity problem in high-dimensional classification is the regularized method, which replaces the within-group sample covariance matrix with a ridge-type covariance estimate to stabilize the estimate. In this article, we propose AIC-type criterion by replacing MLE of the within-group covariance matrix with ridge-type estimator. This idea follows Srivastava and Kubokawa (2008). Simulations revealed that our proposed criterion performs well.

Efficient Weighted Voting Random Subspace Support Vector Machine for High-Dimensional Classification

Efficient Weighted Voting Random Subspace Support Vector Machine for High-Dimensional Classification

A Road to Classification in High Dimensional Space: The Regularized Optimal Affine Discriminant

SummaryFor high dimensional classification, it is well known that naively performing the Fisher discriminant rule leads to poor results due to diverging spectra and accumulation of noise. Therefore, researchers proposed independence rules to circumvent the diverging spectra, and sparse independence rules to mitigate the issue of accumulation of noise. However, in biological applications, often a group of correlated genes are responsible for clinical outcomes, and the use of the covariance information can significantly reduce misclassification rates. In theory the extent of such error rate reductions is unveiled by comparing the misclassification rates of the Fisher discriminant rule and the independence rule. To materialize the gain on the basis of finite samples, a regularized optimal affine discriminant (ROAD) is proposed. The ROAD selects an increasing number of features as the regularization relaxes. Further benefits can be achieved when a screening method is employed to narrow the feature pool before applying the ROAD method. An efficient constrained co-ordinate descent algorithm is also developed to solve the associated optimization problems. Sampling properties of oracle type are established. Simulation studies and real data analysis support our theoretical results and demonstrate the advantages of the new classification procedure under a variety of correlation structures. A delicate result on continuous piecewise linear solution paths for the ROAD optimization problem at the population level justifies the linear interpolation of the constrained co-ordinate descent algorithm.

An enhanced XCS rule discovery module using feature ranking

XCS is a genetics-based machine learning model that combines reinforcement learning with evolutionary algorithms to evolve a population of classifiers in the form of condition-action rules. Like many other machine learning algorithms, XCS is less effective on high-dimensional data sets. In this paper, we describe a new guided rule discovery mechanisms for XCS, inspired by feature selection techniques commonly used in machine learning. In our approach, feature quality information is used to bias the evolutionary operators. A comprehensive set of experiments is used to investigate how the number of features used to bias the evolutionary operators, population size, and feature ranking technique, affect model performance. Numerical simulations have shown that our guided rule discovery mechanism improves the performance of XCS in terms of accuracy, execution time and more generally in terms of classifier diversity in the population, especially for high-dimensional classification problems. We present a detailed discussion of the effects of model parameters and recommend settings for large scale problems.

Bias-Corrected Hierarchical Bayesian Classification With a Selected Subset of High-Dimensional Features

Class prediction based on high-dimensional features has received a great deal of attention in many areas of application. For example, biologists are interested in using microarray gene expression profiles for diagnosis or prognosis of a certain disease (e.g., cancer). For computational and other reasons, it is necessary to select a subset of features before fitting a statistical model, by evaluating how strongly the features are related to the response. However, such a feature selection procedure will result in overconfident predictive probabilities for future cases, because the signal-to-noise ratio in the retained features is exacerbated by the feature selection. In this article we develop a hierarchical Bayesian classification method that can correct for this feature selection bias. Our method, which we term bias-corrected Bayesian classification with selected features (BCBCSF), uses the partial information from the feature selection procedure, in addition to the retained features, to form a correct (unbiased) posterior distribution of certain hyperparameters in the hierarchical Bayesian model that control the signal-to-noise ratio of the dataset. We take a Markov chain Monte Carlo (MCMC) approach to inferring the model parameters. We then use MCMC samples to make predictions for future cases. Because of the simplicity of the models, the inferred parameters from MCMC are easy to interpret, and the computation is very fast. Simulation studies and tests with two real microarray datasets related to complex human diseases show that our BCBCSF method provides better predictions than two widely used high-dimensional classification methods, prediction analysis for microarrays and diagonal linear discriminant analysis. The R package BCBCSF for the method described here is available from http://math.usask.ca/longhai/software/BCBCSF and CRAN.

Covariance structure approximation via gLasso in high-dimensional supervised classification

Recent work has shown that the Lasso-based regularization is very useful for estimating the high-dimensional inverse covariance matrix. A particularly useful scheme is based on penalizing the ℓ1 norm of the off-diagonal elements to encourage sparsity. We embed this type of regularization into high-dimensional classification. A two-stage estimation procedure is proposed which first recovers structural zeros of the inverse covariance matrix and then enforces block sparsity by moving non-zeros closer to the main diagonal. We show that the block-diagonal approximation of the inverse covariance matrix leads to an additive classifier, and demonstrate that accounting for the structure can yield better performance accuracy. Effect of the block size on classification is explored, and a class of asymptotically equivalent structure approximations in a high-dimensional setting is specified. We suggest a variable selection at the block level and investigate properties of this procedure in growing dimension asymptotics. We present a consistency result on the feature selection procedure, establish asymptotic lower an upper bounds for the fraction of separative blocks and specify constraints under which the reliable classification with block-wise feature selection can be performed. The relevance and benefits of the proposed approach are illustrated on both simulated and real data.

Recent Advances in Discriminant Analysis for High-dimensional Data Classification

There are serious challenges posed by high-dimensional data sets. With the arrival of new technologies, high-throughput modeling is becoming a norm in many disciplines such as statistical genetics, epidemiology, astronomy, high energy physics, and ecology. Highdimensional data have emerged from various sources such as digital images, documents, next-gen sequencing, mass spectrometry, metabolomics, microarray, proteomics, online videos and web pages. One area with a growing need for new statistical methods and theory for high-dimensional data is the classification of subgroups. For example, cancer classification has primarily been based on histopathological appearance of tumor. However, patients with similar tumor appearance can have different prognosis and response to treatment. The traditional way to classify cancer by pathological review may cause biased results and misclassify the tumor subtypes for patients. The availability of microarray data allows simultaneous measures of thousands of genes. These high-dimensional data have become a standard tool for biomedical studies and are now commonly collected from patients in clinical trials. The identification of informative genes may result in potential molecular markers for tumor class prediction. Correct classifications can help practitioners identify the right treatment for patients. Due to the cost and/or experimental difficulties in obtaining sufficient biological materials, it is common to see studies with sample size much smaller than the number of dimensions. These problems are referred to as “large p small n” issues, where p is the number of dimensions (or say genes) and n is the sample size. High-dimensional data pose challenges to traditional statistical methods. For instance, owing to small n, there are increased uncertainties in the standard estimations of parameters such as means and variances. As a consequence, statistical analyses based on such parameters estimation are usually unreliable. To have improved parameters estimation, researchers have come up with innovative ways to deal with this.

Recursive Learning of Genetic Algorithm Featuring Incremental Attribute Learning for Higher Dimensional Classification Problems

Rule-based classifiers trained by Genetic Algorithms (GAs) have been one of the most prevailing solutions for pattern classification problem. This paper introduces an algorithm named Recursive Learning of Genetic Algorithms featuring Incremental Attribute Learning (RLGA-IAL) developed from the Recursive Learning of Genetic Algorithms with Task Decomposition and Varied Rule Set (RLGA). Instead of training all the attributes in a batch, RLGA-IAL integrates the attributes sequentially. By projecting a large multidimensional search space to single-dimensional space spaces with integration, it reduces the difficulty in deriving the classification rules.Though a series of experiments, RLGA-IAL shows a successful and promising performance in classification problems with the dimension of the datasets ranging from 5 to 60.

Impact of class-imbalance on multi-class high-dimensional class prediction

The goal of multi-class supervised classification is to develop a rule that accurately predicts the class membership of new samples when the number of classes is larger than two. In this paper we consider high-dimensional class-imbalanced data: the number of variables greatly exceeds the number of samples and the number of samples in each class is not equal. We focus on Friedman's one-versus-one approach for three-class problems and show how its class probabilities depend on the class probabilities from the binary classification sub-problems. We further explore its performance using diagonal linear discriminant analysis (DLDA) as a base classifier and compare its performance with multi-class DLDA, using simulated and real data. Our results show that the class-imbalance has a significant effect on the classification results: the classification is biased towards the majority class as in the two-class problems and the problem is magnified when the number of variables is large. The amount of the bias depends also, jointly, on the magnitude of the differences between the classes and on the sample size: the bias diminishes when the difference between the classes is larger or the sample size is increased. Also variable selection plays an important role in the class-imbalance problem and the most effective strategy depends on the type of differences that exist between classes. DLDA seems to be among the least sensible classifiers to class-imbalance and its use is recommended also for multi-class problems. Whenever possible the experiments should be planned using balanced data in order to avoid the class-imbalance problem.

Multiple-input multiple-output causal strategies for gene selection

BackgroundTraditional strategies for selecting variables in high dimensional classification problems aim to find sets of maximally relevant variables able to explain the target variations. If these techniques may be effective in generalization accuracy they often do not reveal direct causes. The latter is essentially related to the fact that high correlation (or relevance) does not imply causation. In this study, we show how to efficiently incorporate causal information into gene selection by moving from a single-input single-output to a multiple-input multiple-output setting.ResultsWe show in synthetic case study that a better prioritization of causal variables can be obtained by considering a relevance score which incorporates a causal term. In addition we show, in a meta-analysis study of six publicly available breast cancer microarray datasets, that the improvement occurs also in terms of accuracy. The biological interpretation of the results confirms the potential of a causal approach to gene selection.ConclusionsIntegrating causal information into gene selection algorithms is effective both in terms of prediction accuracy and biological interpretation.

HGA: Hybrid genetic algorithm in fuzzy rule-based classification systems for high-dimensional problems

The aim of this work is to propose a hybrid heuristic approach (called hGA) based on genetic algorithm (GA) and integer-programming formulation (IPF) to solve high dimensional classification problems in linguistic fuzzy rule-based classification systems. In this algorithm, each chromosome represents a rule for specified class, GA is used for producing several rules for each class, and finally IPF is used for selection of rules from a pool of rules, which are obtained by GA. The proposed algorithm is experimentally evaluated by the use of non-parametric statistical tests on seventeen classification benchmark data sets. Results of the comparative study show that hGA is able to discover accurate and concise classification rules.

Direct Parallel Perceptrons (DPPs): Fast Analytical Calculation of the Parallel Perceptrons Weights With Margin Control for Classification Tasks

Parallel perceptrons (PPs) are very simple and efficient committee machines (a single layer of perceptrons with threshold activation functions and binary outputs, and a majority voting decision scheme), which nevertheless behave as universal approximators. The parallel delta (P-Delta) rule is an effective training algorithm, which, following the ideas of statistical learning theory used by the support vector machine (SVM), raises its generalization ability by maximizing the difference between the perceptron activations for the training patterns and the activation threshold (which corresponds to the separating hyperplane). In this paper, we propose an analytical closed-form expression to calculate the PPs' weights for classification tasks. Our method, called Direct Parallel Perceptrons (DPPs), directly calculates (without iterations) the weights using the training patterns and their desired outputs, without any search or numeric function optimization. The calculated weights globally minimize an error function which simultaneously takes into account the training error and the classification margin. Given its analytical and noniterative nature, DPPs are computationally much more efficient than other related approaches (P-Delta and SVM), and its computational complexity is linear in the input dimensionality. Therefore, DPPs are very appealing, in terms of time complexity and memory consumption, and are very easy to use for high-dimensional classification tasks. On real benchmark datasets with two and multiple classes, DPPs are competitive with SVM and other approaches but they also allow online learning and, as opposed to most of them, have no tunable parameters.