Feature Selection For High-dimensional Data Research Articles

Diagonal Discriminant Analysis With Feature Selection for High-Dimensional Data

We introduce a new method of performing high-dimensional discriminant analysis (DA), which we call multiDA. Starting from multiclass diagonal DA classifiers which avoid the problem of high-dimensional covariance estimation we construct a hybrid model that seamlessly integrates feature selection components. Our feature selection component naturally simplifies to weights which are simple functions of likelihood ratio test statistics allowing natural comparisons with traditional hypothesis testing methods. We provide heuristic arguments suggesting desirable asymptotic properties of our algorithm with regard to feature selection. We compare our method with several other approaches, showing marked improvements in regard to prediction accuracy, interpretability of chosen features, and fast run time. We demonstrate such strengths of our model by showing strong classification performance on publicly available high-dimensional datasets, as well as through multiple simulation studies. We make an R package available implementing our approach. Supplementary materials for this article are available online.

Ensemble feature selection for high-dimensional data: a stability analysis across multiple domains

Selecting a subset of relevant features is crucial to the analysis of high-dimensional datasets coming from a number of application domains, such as biomedical data, document and image analysis. Since no single selection algorithm seems to be capable of ensuring optimal results in terms of both predictive performance and stability (i.e. robustness to changes in the input data), researchers have increasingly explored the effectiveness of “ensemble” approaches involving the combination of different selectors. While interesting proposals have been reported in the literature, most of them have been so far evaluated in a limited number of settings (e.g. with data from a single domain and in conjunction with specific selection approaches), leaving unanswered important questions about the large-scale applicability and utility of ensemble feature selection. To give a contribution to the field, this work presents an empirical study which encompasses different kinds of selection algorithms (filters and embedded methods, univariate and multivariate techniques) and different application domains. Specifically, we consider 18 classification tasks with heterogeneous characteristics (in terms of number of classes and instances-to-features ratio) and experimentally evaluate, for feature subsets of different cardinalities, the extent to which an ensemble approach turns out to be more robust than a single selector, thus providing useful insight for both researchers and practitioners.

Hybrid fast unsupervised feature selection for high-dimensional data

The emergence of ``curse of dimensionality” issue as a result of high reduces datasets deteriorates the capability of learning algorithms, and also requires high memory and computational costs. Selection of features by discarding redundant and irrelevant features functions as a crucial machine learning technique aimed at reducing the dimensionality of these datasets, which improves the performance of the learning algorithm. Feature selection has been extensively applied in many application areas relevant to expert and intelligent systems, such as data mining and machine learning. Although many algorithms have been developed so far, they are still unsatisfying confronting high-dimensional data. This paper presented a new hybrid filter-based feature selection algorithm based on acombination of clustering and the modified Binary Ant System (BAS), called FSCBAS, to overcome the search space and high-dimensional data processing challenges efficiently. This model provided both global and local search capabilities between and within clusters. In the proposed method, inspired by genetic algorithm and simulated annealing, a damped mutation strategy was introduced that avoided falling into local optima, and a new redundancy reduction policy adopted to estimate the correlation between the selected features further improved the algorithm. The proposed method can be applied in many expert system applications such as microarray data processing, text classification and image processing in high-dimensional data to handle the high dimensionality of the feature space and improve classification performance simultaneously. The performance of the proposed algorithm was compared to that of state-of-the-art feature selection algorithms using different classifiers on real-world datasets. The experimental results confirmed that the proposed method reduced computational complexity significantly, and achieved better performance than the other feature selection methods.

A Multicriteria Approach to Find Predictive and Sparse Models with Stable Feature Selection for High-Dimensional Data.

Finding a good predictive model for a high-dimensional data set can be challenging. For genetic data, it is not only important to find a model with high predictive accuracy, but it is also important that this model uses only few features and that the selection of these features is stable. This is because, in bioinformatics, the models are used not only for prediction but also for drawing biological conclusions which makes the interpretability and reliability of the model crucial. We suggest using three target criteria when fitting a predictive model to a high-dimensional data set: the classification accuracy, the stability of the feature selection, and the number of chosen features. As it is unclear which measure is best for evaluating the stability, we first compare a variety of stability measures. We conclude that the Pearson correlation has the best theoretical and empirical properties. Also, we find that for the stability assessment behaviour it is most important that a measure contains a correction for chance or large numbers of chosen features. Then, we analyse Pareto fronts and conclude that it is possible to find models with a stable selection of few features without losing much predictive accuracy.

Data field for mining big data

Big data is a highlighted challenge for many fields with the rapid expansion of large-volume, complex, and fast-growing sources of data. Mining from big data is required for exploring the essence of data and providing meaningful information. To this end, we have previously introduced the theory of physical field to explore relations between objects in data space and proposed a framework of data field to discover the underlying distribution of big data. This paper concerns an overview of big data mining by the use of data field. It mainly discusses the theory of data field and different aspects of applications including feature selection for high-dimensional data, clustering, and the recognition of facial expression in human–computer interaction. In these applications, data field is employed to capture the intrinsic distribution of data objects for selecting meaningful features, fast clustering, and describing variation of facial expression. It is expected that our contributions would help overcome the problems in accordance with big data.

Feature selection for high-dimensional data

Feature selection for high-dimensional data

GISP Algorithm for identifying strong expression and structural pattern of gene in mouse brain data in the field of neuroinformatics

GISP Algorithm for identifying strong expression and structural pattern of gene in mouse brain data in the field of neuroinformatics

Scalable Feature Selection in High-Dimensional Data Based on GRASP

Feature selection in high-dimensional data is one of the active areas of research in pattern recognition. Most of the algorithms in this area try to select a subset of features in a way to maximize the accuracy of classification regardless of the number of selected features that affect classification time. In this article, a new method for feature selection algorithm in high-dimensional data is proposed that can control the trade-off between accuracy and classification time. This method is based on a greedy metaheuristic algorithm called greedy randomized adaptive search procedure (GRASP). It uses an extended version of a simulated annealing (SA) algorithm for local search. In this version of SA, new parameters are embedded that allow the algorithm to control the trade-off between accuracy and classification time. Experimental results show supremacy of the proposed method over previous versions of GRASP for feature selection. Also, they show how the trade-off between accuracy and classification time is controllable by the parameters introduced in the proposed method.

The EBIC and a sequential procedure for feature selection in interactive linear models with high-dimensional data

High-dimensional data arises in many important scientific fields. The analysis of high-dimensional data poses great challenges to statisticians. In high-dimensional data, the relationship among the variables is complex. It involves main effects as well as interaction effects of the covariates. The effect of some covariates is only realized through their interaction with the others. This makes the consideration of interactive models imperative in the analysis of high-dimensional data. Because of the existence of high spurious correlation among the covariates in high-dimensional data, conventional tools for dealing with interactive models become inappropriate. In this paper, we develop specific tools for feature selection in high-dimensional data with interactive models, including a version of the extended BIC (EBIC) for interactive models and a sequential feature selection procedure. Main-effect and interaction features are treated differently in the EBIC for interactive models and the sequential procedure due to their different natures. The selection consistency of the EBIC for interactive models and the sequential procedure is established. Simulation studies are carried out to vindicate the asymptotic property in finite samples as well as to compare with non-sequential procedures. The approach developed in this paper is also applied to a real data set.

Sequential random k-nearest neighbor feature selection for high-dimensional data

Feature selection based on an ensemble classifier has been recognized as a crucial technique for modeling high-dimensional data. Feature selection based on the random forests model, which is constructed by aggregating multiple decision tree classifiers, has been widely used. However, a lack of stability and balance in decision trees decreases the robustness of random forests. This limitation motivated us to propose a feature selection method based on newly designed nearest-neighbor ensemble classifiers. The proposed method finds significant features by using an iterative procedure. We performed experiments with 20 datasets of microarray gene expressions to examine the property of the proposed method and compared it with random forests. The results demonstrated the effectiveness and robustness of the proposed method, especially when the number of features exceeds the number of observations.

Feature Selection for High-Dimensional Data Based on Ridge Regression and SVM and Its Application in Peptide QSAR Modeling

Absolute weight values estimated from test data by ridge regression (RR) can reflect the significance of corresponding features.Based on RR and support vector machine (SVM),a new feature selection algorithm for high-dimensional data is proposed.Examples from bitter tasting thresholds (BTT) and cytotoxic T lymphocyte (CTL) epitopes are presented.All 531 physicochemical property parameters were employed to express each residue of one peptide,thus 1062 and 4779 descriptors were obtained for BTT and CTL,respectively.Each sample was divided into training and test sets,and weight estimates of all training set descriptors were generated by RR.According to the descending order of the weights,corresponding features were gradually selected until the mean square error (MSE) of leave-one-out cross validation (LOOCV) increased significantly.Based on smaller training datasets obtained from the previous step,the reserved features were available from multiple elimination rounds.7 and 18 descriptors were selected by the new method for BTT and CTL,respectively.A quantitative structure-activity relationship (QSAR) model based on support vector regression (SVR) was established on extracted data with the reserved descriptors,and was then used for test data prediction.The fitting,LOOCV,and external prediction accuracies were significantly improved with respect to reported literature values.Because of the calculation speed,clear physicochemical meaning,and ease of interpretation,the new method is widely applicable to regression forecasting of high-dimensional data such as QSAR modeling of peptide or proteins.

Neighborhood Component Feature Selection for High-Dimensional Data

Feature selection is of considerable importance in data mining and machine learning, especially for high dimensional data. In this paper, we propose a novel nearest neighbor-based feature weighting algorithm, which learns a feature weighting vector by maximizing the expected leave-one-out classification accuracy with a regularization term. The algorithm makes no parametric assumptions about the distribution of the data and scales naturally to multiclass problems. Experiments conducted on artificial and real data sets demonstrate that the proposed algorithm is largely insensitive to the increase in the number of irrelevant features and performs better than the state-of-the-art methods in most cases.

Bayesian feature selection for classification with possibly large number of classes

In what follows, we introduce two Bayesian models for feature selection in high-dimensional data, specifically designed for the purpose of classification. We use two approaches to the problem: one which discards the components which have “almost constant” values (Model 1) and another which retains the components for which variations in-between the groups are larger than those within the groups (Model 2). We assume that p ⪢ n , i.e. the number of components p is much larger than the number of samples n, and that only few of those p components are useful for subsequent classification. We show that particular cases of the above two models recover familiar variance or ANOVA-based component selection. When one has only two classes and features are a priori independent, Model 2 reduces to the Feature Annealed Independence Rule (FAIR) introduced by Fan and Fan (2008) and can be viewed as a natural generalization of FAIR to the case of L > 2 classes. The performance of the methodology is studies via simulations and using a biological dataset of animal communication signals comprising 43 groups of electric signals recorded from tropical South American electric knife fishes.

Feature selection for high-dimensional data

This paper focuses on feature selection for problems dealing with high-dimensional data. We discuss the benefits of adopting a regularized approach with L1 or L1–L2 penalties in two different applications—microarray data analysis in computational biology and object detection in computer vision. We describe general algorithmic aspects as well as architecture issues specific to the two domains. The very promising results obtained show how the proposed approach can be useful in quite different fields of application.

A fast separability-based feature-selection method for high-dimensional remotely sensed image classification

Because of the difficulty of obtaining an analytic expression for Bayes error, a wide variety of separability measures has been proposed for feature selection. In this paper, we show that there is a general framework based on the criterion of mutual information (MI) that can provide a realistic solution to the problem of feature selection for high-dimensional data. We give a theoretical argument showing that the MI of multi-dimensional data can be broken down into several one-dimensional components, which makes numerical evaluation much easier and more accurate. It also reveals that selection based on the simple criterion of only retaining features with high associated MI values may be problematic when the features are highly correlated. Although there is a direct way of selecting features by jointly maximising MI, this suffers from combinatorial explosion. Hence, we propose a fast feature-selection scheme based on a ‘greedy’ optimisation strategy. To confirm the effectiveness of this scheme, simulations are carried out on 16 land-cover classes using the 92AV3C data set collected from the 220-dimensional AVIRIS hyperspectral sensor. We replicate our earlier positive results (which used an essentially heuristic method for MI-based band-selection) but with much reduced computational cost and a much sounder theoretical basis.

Feature selection for high-dimensional data in astronomy

With an exponentially increasing amount of astronomical data, the complexity and dimension of astronomical data are likewise growing rapidly. Extracting information from such data becomes a critical and challenging problem. For example, some algorithms can only be employed in the low-dimensional spaces, so feature selection and feature extraction become important topics. Here we describe the difference between feature selection and feature extraction methods, and introduce the taxonomy of feature selection methods as well as the characteristics of each method. We present a case study comparing the performance and computational cost of different feature selection methods. For the filter method, ReliefF and fisher filter are adopted; for the wrapper method, improved CHAID, linear discriminant analysis (LDA), Naive Bayes (NB) and C4.5 are taken as learners. Applied on the sample, the result indicates that from the viewpoints of computational cost the filter method is superior to the wrapper method. Moreover, different learning algorithms combined with appropriate feature selection methods may arrive at better performance.