Huge Number Of Features Research Articles

Large-scale structural learning and predicting via hashing approximation

By combining the structural information with nonparallel support vector machine, structural nonparallel support vector machine (SNPSVM) can fully exploit prior knowledge to directly improve the algorithm’s generalization capacity. However, the scalability issue how to train SNPSVM efficiently on data with huge dimensions has not been studied. In this paper, we integrate linear SNPSVM with b-bit minwise hashing scheme to speedup the training phase for large-scale and high-dimensional statistical learning, and then we address the problem of speeding-up its prediction phase via locality-sensitive hashing. For one-against-one multi-class classification problems, a two-stage strategy is put forward: a series of hash-based classifiers are built in order to approximate the exact results and filter the hypothesis space in the first stage and then the classification can be refined by solving a multi-class SNPSVM on the remaining classes in the second stage. The proposed method can deal with large-scale classification problems with a huge number of features. Experimental results on two large-scale datasets (i.e., news20 and webspam) demonstrate the efficiency of structural learning via b-bit minwise hashing. Experimental results on the ImageNet-BOF dataset, and several large-scale UCI datasets show that the proposed hash-based prediction can be more than two orders of magnitude faster than the exact classifier with minor losses in quality.

Feature clustering based support vector machine recursive feature elimination for gene selection

In a DNA microarray dataset, gene expression data often has a huge number of features(which are referred to as genes) versus a small size of samples. With the development of DNA microarray technology, the number of dimensions increases even faster than before, which could lead to the problem of the curse of dimensionality. To get good classification performance, it is necessary to preprocess the gene expression data. Support vector machine recursive feature elimination (SVM-RFE) is a classical method for gene selection. However, SVM-RFE suffers from high computational complexity. To remedy it, this paper enhances SVM-RFE for gene selection by incorporating feature clustering, called feature clustering SVM-RFE (FCSVM-RFE). The proposed method first performs gene selection roughly and then ranks the selected genes. First, a clustering algorithm is used to cluster genes into gene groups, in each which genes have similar expression profile. Then, a representative gene is found to represent a gene group. By doing so, we can obtain a representative gene set. Then, SVM-RFE is applied to rank these representative genes. FCSVM-RFE can reduce the computational complexity and the redundancy among genes. Experiments on seven public gene expression datasets show that FCSVM-RFE can achieve a better classification performance and lower computational complexity when compared with the state-the-art-of methods, such as SVM-RFE.

Occam's razor in dimension reduction: Using reduced row Echelon form for finding linear independent features in high dimensional microarray datasets

Microarray high dimensional datasets suffer from small sample size and extreme large number of features. Therefore, feature selection plays crucial roles on the performance of the trained models on those datasets. A typical feature selection method consists of two main parts, problem criterion and a search strategy. The common datasets don’t have huge number of features with respect to their number of samples; hence, a search strategy in their feature selection methods were able to seek the search space. In contrast, microarray high dimensional datasets have huge number of features; therefore, their search space is very large and searching that space is a prohibitive action. In this paper, we take into account the philosophy of Occam's razor in feature subset selection in order to release high dimensional datasets from computational search methods. The proposed method uses two stages for feature selection. In the first stage features are rearranged by their importance in the dataset and in the second stage, the fundamental concept of reduced row Echelon form is applied on dataset in order to find linear independent features. For determining the effectiveness of the proposed method some experiments are carried out on nine binary microarray high dimensional datasets. The obtained results are compared with eleven state-of-the-art feature selection algorithms including Correlation based Feature Selection (CFS), Fast Correlation Based Filter (FCBF), Interact (INT) and Maximum Relevancy Minimum Redundancy (MRMR). The average outcomes of the results are analyzed by a statistical non-parametric test and it reveals that the proposed method has a meaningful superiority to the others in terms of accuracy, sensitivity, specificity, G-mean, number of selected features and computational complexity.

A lexical-semantics-based method for multi label text categorisation using word net

Text categorisation is an upcoming area in the field of text mining. The text documents possess huge number of features due to their unstructured nature. In this paper, an algorithm for multi label categorisation of text documents based on the concepts of lexical and semantics using word net (MC-LSW) is proposed. The proposed algorithm is based on the concepts of lexical (tokens) and semantics of a language. It aims at minimising the number of tokens used for categorising text documents. MC-LSW uses word net to extract the semantic information of tokens. The proposed algorithm is implemented and tested on five datasets of text domain and is compared with the existing multi label categorisation algorithms. The proposed algorithm (MC-LSW) shows more efficient and promising results in terms of space and time complexity than the existing methods. Accuracy and precision measures have been improved by the proposed algorithm as well as hamming loss has been reduced.

Effective Feature Selection for Classification of Promoter Sequences.

Exploring novel computational methods in making sense of biological data has not only been a necessity, but also productive. A part of this trend is the search for more efficient in silico methods/tools for analysis of promoters, which are parts of DNA sequences that are involved in regulation of expression of genes into other functional molecules. Promoter regions vary greatly in their function based on the sequence of nucleotides and the arrangement of protein-binding short-regions called motifs. In fact, the regulatory nature of the promoters seems to be largely driven by the selective presence and/or the arrangement of these motifs. Here, we explore computational classification of promoter sequences based on the pattern of motif distributions, as such classification can pave a new way of functional analysis of promoters and to discover the functionally crucial motifs. We make use of Position Specific Motif Matrix (PSMM) features for exploring the possibility of accurately classifying promoter sequences using some of the popular classification techniques. The classification results on the complete feature set are low, perhaps due to the huge number of features. We propose two ways of reducing features. Our test results show improvement in the classification output after the reduction of features. The results also show that decision trees outperform SVM (Support Vector Machine), KNN (K Nearest Neighbor) and ensemble classifier LibD3C, particularly with reduced features. The proposed feature selection methods outperform some of the popular feature transformation methods such as PCA and SVD. Also, the methods proposed are as accurate as MRMR (feature selection method) but much faster than MRMR. Such methods could be useful to categorize new promoters and explore regulatory mechanisms of gene expressions in complex eukaryotic species.

Neighborhood rough set reduction with fish swarm algorithm

Feature reduction refers to the problem of deleting those input features that are less predictive of a given outcome; a problem encountered in many areas such as pattern recognition, machine learning and data mining. In particular, it has been successfully applied in tasks that involve datasets containing huge numbers of features. Rough set theory has been used as such a data set preprocessor with much success, but current methods are inadequate at solving the problem of numerical feature reduction. As the classical rough set model can just be used to evaluate categorical features, we introduce a neighborhood rough set model to deal with numerical datasets by defining a neighborhood relation. However, this method is still not enough to find the optimal subsets regularly. In this paper, we propose a new feature reduction mechanism based on fish swarm algorithm (FSA) in an attempt to polish up this. The method is then applied to the problem of finding optimal feature subsets in the neighborhood rough set reduction process. We define three foraging behaviors of fish to find the optimal subsets and a fitness function to evaluate the best solutions. We construct the neighborhood feature reduction algorithm based on FSA and design some experiments comparing with a heuristic neighborhood feature reduction method. Experimental results show that the FSA-based neighborhood reduction method is suitable to deal with numerical data and more possibility to find an optimal reduct.

Improving feature selection performance using pairwise pre-evaluation.

BackgroundBiological data such as microarrays contain a huge number of features. Thus, it is necessary to select a small number of novel features to characterize the entire dataset. All combinations of the features subset must be evaluated to produce an ideal feature subset, but this is impossible using currently available computing power. Feature selection or feature subset selection provides a sub-optimal solution within a reasonable amount of time.ResultsIn this study, we propose an improved feature selection method that uses information based on all the pairwise evaluations for a given dataset. We modify the original feature selection algorithms to use pre-evaluation information. The pre-evaluation captures the quality and interactions between two features. The feature subset should be improved by using the top ranking pairs for two features in the selection process.ConclusionsExperimental results demonstrated that the proposed method improved the quality of the feature subset produced by modified feature selection algorithms. The proposed method can be applied to microarray and other high-dimensional data.

Relevant based structure learning for feature selection

Feature selection is an important task in many problems occurring in pattern recognition, bioinformatics, machine learning and data mining applications. The feature selection approach enables us to reduce the computation burden and the falling accuracy effect of dealing with huge number of features in typical learning problems. There is a variety of techniques for feature selection in supervised learning problems based on different selection metrics. In this paper, we propose a novel unified framework for feature selection built on the graphical models and information theoretic tools. The proposed approach exploits the structure learning among features to select more relevant and less redundant features to the predictive modeling problem according to a primary novel likelihood based criterion. In line with the selection of the optimal subset of features through the proposed method, it provides us the Bayesian network classifier without the additional cost of model training on the selected subset of features. The optimal properties of our method are established through empirical studies and computational complexity analysis. Furthermore the proposed approach is evaluated on a bunch of benchmark datasets based on the well-known classification algorithms. Extensive experiments confirm the significant improvement of the proposed approach compared to the earlier works.

Speech Emotion Recognition based on SVM as Both Feature Selector and Classifier

The aim of this paper is to utilize Support Vector Machine (SVM) as feature selection and classification techniques for audio signals to identify human emotional states. One of the major bottlenecks of common speech emotion recognition techniques is to use a huge number of features per utterance which could significantly slow down the learning process, and it might cause the problem known as ―the curse of dimensionality‖. Consequently, to ease this challenge this paper aims to achieve high accuracy system with a minimum set of features. The proposed model uses two methods, namely ―SVM features selection‖ and the common ―Correlation-based Feature Subset Selection (CFS)‖ for the feature dimensions reduction part. In addition, two different classifiers, one Support Vector Machine and the other Neural Network are separately adopted to identify the six emotional states of anger, disgust, fear, happiness, sadness and neutral. The method has been verified using Persian (Persian ESD) and German (EMO-DB) emotional speech databases, which yield high recognition rates in both databases. The results show that SVM feature selection method provides better emotional speech-recognition performance compared to CFS and baseline feature set. Moreover, the new system is able to achieve a recognition rate of (99.44%) on the Persian ESD and (87.21%) on Berlin Emotion Database for speaker-dependent classification. Besides, promising result (76.12%) is obtained for speaker-independent classification case; which is among the best-known accuracies reported on the mentioned database relative to its little number of features.

Multiclass Classification Problem of Large-Scale Biomedical Meta-Data

One of the important data mining method in biomedical research is classification task. Recent advances in biomedicine provide opportunities for molecular biology, such as measurement of activity of thousands of molecular tissue biomarkers. For example we can use data of gene expression measured by DNA microarrays or RNA-Seq technique, DNA methylation levels measured by DNA methylation microarrays or protein and phosphoprotein levels measured by reverse phase protein array. A big problem in applying large-scale genomic and proteomic data for classification problem is the dimension of these data. In this work, we propose novel multiclass feature selection and classification system for merged data from different molecular biomedical techniques. However, when we merge these data the biggest problem is the huge number of features with a limited number of samples. For that reason the feature selection step is crucial in high dimension data classification problem. Our results have shown that integrated analysis with proper feature selection and classification techniques used for large-scale meta-data can improve the classification accuracy and feature selection stability index. We have proofed, that for merged data we observe significantly higher classification accuracy for the same number of selected features as for single technique dataset.

Automated Imaging System for Pigmented Skin Lesion Diagnosis

Through the study of pigmented skin lesions risk factors, the appearance of malignant melanoma turns the anomalous occurrence of these lesions to annoying sign. The difficulty of differentiation between malignant melanoma and melanocytic naive is the error-bone problem that usually faces the physicians in diagnosis. To think through the hard mission of pigmented skin lesions diagnosis different clinical diagnosis algorithms were proposed such as pattern analysis, ABCD rule of dermoscopy, Menzies method, and 7-points checklist. Computerized monitoring of these algorithms improves the diagnosis of melanoma compared to simple naked-eye of physician during examination. Toward the serious step of melanoma early detection, aiming to reduce melanoma mortality rate, several computerized studies and procedures were proposed. Through this research different approaches with a huge number of features were discussed to point out the best approach or methodology could be followed to accurately diagnose the pigmented skin lesion. This paper proposes automated system for diagnosis of melanoma to provide quantitative and objective evaluation of skin lesion as opposed to visual assessment, which is subjective in nature. Two different data sets were utilized to reduce the effect of qualitative interpretation problem upon accurate diagnosis. Set of clinical images that are acquired from a standard camera while the other set is acquired from a special dermoscopic camera and so named dermoscopic images. System contribution appears in new, complete and different approaches presented for the aim of pigmented skin lesion diagnosis. These approaches result from using large conclusive set of features fed to different classifiers. The three main types of different features extracted from the region of interest are geometric, chromatic, and texture features. Three statistical methods were proposed to select the most significant features that will cause a valuable effect in diagnosis; Fisher score method, t-test, and F-test. The selected features of high-ranking score based on the statistical methods are used for the diagnosis of the two lesion groups using Artificial Neural Network (ANN), K-Nearest Neighbor (KNN) and Support Vector Machine (SVM) as three different classifiers proposed. The overall System performance was then measured in regards to Specificity, Sensitivity and Accuracy. According to the different approaches that will be mentioned later the best result was showen by the ANN designed with the feature selected according to fisher score method enables a diagnostic accuracy of 96.25% and 97% for dermoscopic and clinical images respectively.

A discriminative random sampling strategy with individual-author feature selection for writeprint recognition of Chinese texts

The auto authorship recognition has become a novel technique to investigate cybercrimes. But the challenge of the research is that a huge number of features exist in the moderate-sized corpus, which causes the curse of over-training. Besides, it is hard to distinguish between potential authors only by a single feature set. In this paper, we proposed a random sampling style ensemble method with individual-author feature selection to exploit the high-dimensional feature space. The proposed method randomly picks writing-style features on each individual-author feature set (IAFS) partitioned from the whole feature set. The IAFSs are heuristically selected with training set of each author. Then, multiple base classifiers (BCs) are formed on the sampled feature sets. Finally, all BCs are fused to get a final decision. Experimental results on the real-life Chinese forum data verify the robustness of the proposed method compared with conventional ensemble methods. We also analyze the diversity of algorithm to reveal that the ensemble strategy is more effective and can construct more diverse BCs than random subspace methods.

Membership-margin based feature selection for mixed type and high-dimensional data: Theory and applications

The present paper describes a new feature weighting method based on a membership margin. Distinctive properties of the proposed method include its capability to process problems characterized by mixed-type data (quantitative, qualitative and interval) as well as a huge number of features. The key idea is to map simultaneously all the features of different types into a common space; the membership space. Once all features are represented in a homogeneous space, a feature weighting task can be performed in unified way. This weighting approach is integrated here within a fuzzy classifier through a fuzzy rule weighted concept in order to improve its performance. Each antecedent fuzzy set in the fuzzy if–then rule is weighted to characterize the importance of each proposition and therefore its corresponding feature. Weight estimation process is based on membership margin maximization to estimate a fuzzy weight of each feature in the membership space. Experiments on low and high dimensional real-world datasets demonstrate that the proposed approach can improve significantly the performance of the fuzzy rule-based as well as other state of the art classifiers and can even outperform classical feature weighting approaches. In particular, we show that this approach can yield meaningful results on two real-world applications for cancer prognosis and industrial process diagnosis.

Finding rough and fuzzy-rough set reducts with SAT

Feature selection refers to the problem of selecting those input features that are most predictive of a given outcome; a problem encountered in many areas such as machine learning, pattern recognition and signal processing. In particular, solution to this has found successful application in tasks that involve datasets containing huge numbers of features (in the order of tens of thousands), which would otherwise be impossible to process further. Recent examples include text processing and web content classification. Rough set theory has been used as such a dataset pre-processor with much success, but current methods are inadequate at finding globally minimal reductions, the smallest sets of features possible. This paper proposes a technique that considers this problem from a propositional satisfiability perspective. In this framework, globally minimal subsets can be located and verified.

Grassmann multimodal implicit feature selection

In pattern recognition field, objects are usually represented by multiple features (multimodal features). For example, to characterize a natural scene image, it is essential to extract a set of visual features representing its color, texture, and shape information. However, integrating multimodal features for recognition is challenging because: (1) each feature has its specific statistical property and physical interpretation, (2) huge number of features may result in the curse of dimensionality (When data dimension is high, the distances between pairwise objects in the feature space become increasingly similar due to the central limit theory. This phenomenon influences negatively to the recognition performance), and (3) some features may be unavailable. To solve these problems, a new multimodal feature selection algorithm, termed Grassmann manifold feature selection (GMFS), is proposed. In particular, by defining a clustering criterion, the multimodal features are transformed into a matrix, and further treated as a point on the Grassmann manifold in Hamm and Lee (Grassmann discriminant analysis: a unifying view on subspace-based learning. In: Proceedings of the 25th international conference on machine learning (ICML), pp. 376---383, Helsinki, Finland [2008]). To deal with the unavailable features, L2-Hausdorff distance, a metric between different-sized matrices, is computed and the kernel is obtained accordingly. Based on the kernel, we propose supervised/unsupervised feature selection algorithms to achieve a physically meaningful embedding of the multimodal features. Experimental results on eight data sets validate the effectiveness the proposed approach.

Unsupervised Clustering of Multivariate Time Series Microarray Experiments based on Incremental Non-Gaussian Analysis

Multiple expression levels of genes obtained using time series microarray experiments have been exploited effectively to enhance understanding of a wide range of biological phenomena. However, the unique nature of microarray data is usually in the form of large matrices of expression genes with high dimensions. Among the huge number of genes presented in microarrays, only a small number of genes are expected to be effective for performing a certain task. Hence, discounting the majority of unaffected genes is the crucial goal of gene selection to improve accuracy for disease diagnosis. In this paper, a non-Gaussian weight matrix obtained from an incremental model is proposed to extract useful features of multivariate time series microarrays. The proposed method can automatically identify a small number of significant features via discovering hidden variables from a huge number of features. An unsupervised hierarchical clustering representative is then taken to evaluate the effectiveness of the proposed methodology. The proposed method achieves promising results based on predictive accuracy of clustering compared to existing methods of analysis. Furthermore, the proposed method offers a robust approach with low memory and computation costs.

A Novel Random Subspace Method for Online Writeprint Identification

With the widespread application of computer network technology, diverse anonymous cyber crimes begin to appear in the online community. The anonymous nature of online-information distribution makes writeprint identification a critical forensic problem. But the difficulty of the task is the huge number of features in even a moderate-sized available text corpus, which causes the problem of over-training. In this paper, we proposed a novel random subspace method by constructing a set of stable classifiers to take advantage of nearly all the discriminative information in the high dimensional feature space. In the construction of base classifiers, an optimized synergetic neural network is employed to provide probabilistic information for each class. Performance results on the subset of Reuters Corpus Volume 1 (RCV1) show that the proposed random subspace method achieves the better identification performance than a single classifier and conventional random subspace methods.

Rough set and scatter search metaheuristic based feature selection for credit scoring

As the credit industry has been growing rapidly, credit scoring models have been widely used by the financial industry during this time to improve cash flow and credit collections. However, a large amount of redundant information and features are involved in the credit dataset, which leads to lower accuracy and higher complexity of the credit scoring model. So, effective feature selection methods are necessary for credit dataset with huge number of features. In this paper, a novel approach, called RSFS, to feature selection based on rough set and scatter search is proposed. In RSFS, conditional entropy is regarded as the heuristic to search the optimal solutions. Two credit datasets in UCI database are selected to demonstrate the competitive performance of RSFS consisted in three credit models including neural network model, J48 decision tree and Logistic regression. The experimental result shows that RSFS has a superior performance in saving the computational costs and improving classification accuracy compared with the base classification methods.

Improved Gini-Index Algorithm to Correct Feature-Selection Bias in Text Classification

This paper presents an improved Gini-Index algorithm to correct feature-selection bias in text classification. Gini-Index has been used as a split measure for choosing the most appropriate splitting attribute in decision tree. Recently, an improved Gini-Index algorithm for feature selection, designed for text categorization and based on Gini-Index theory, was introduced, and it has proved to be better than the other methods. However, we found that the Gini-Index still shows a feature selection bias in text classification, specifically for unbalanced datasets having a huge number of features. The feature selection bias of the Gini-Index in feature selection is shown in three ways: 1) the Gini values of low-frequency features are low (on purity measure) overall, irrespective of the distribution of features among classes, 2) for high-frequency features, the Gini values are always relatively high and 3) for specific features belonging to large classes, the Gini values are relatively lower than those belonging to small classes. Therefore, to correct that bias and improve feature selection in text classification using Gini-Index, we propose an improved Gini-Index (I-GI) algorithm with three reformulated Gini-Index expressions. In the present study, we used global dimensionality reduction (DR) and local DR to measure the goodness of features in feature selections. In experimental results for the I-GI algorithm, we obtained unbiased feature values and eliminated many irrelevant general features while retaining many specific features. Furthermore, we could improve the overall classification performances when we used the local DR method. The total averages of the classification performance were increased by 19.4%, 15.9%, 3.3%, 2.8% and 2.9% (kNN) in Micro-F1, 14%, 9.8%, 9.2%, 3.5% and 4.3% (SVM) in Micro-F1, 20%, 16.9%, 2.8%, 3.6% and 3.1% (kNN) in Macro-F1, 16.3%, 14%, 7.1%, 4.4%, 6.3% (SVM) in Macro-F1, compared with tf*idf, χ2, Information Gain, Odds Ratio and the existing Gini-Index methods according to each classifier.

Rough set and Tabu search based feature selection for credit scoring

Abstract As the credit industry has been growing rapidly, huge number of consumers’ credit data are collected by the credit department of the bank and credit scoring has become a very important issue. Usually, a large amount of redundant information and features are involved in the credit dataset, which leads to lower accuracy and higher complexity of the credit scoring model. So, effective feature selection methods are necessary for credit dataset with huge number of features. In this paper, a novel approach, called FSRT, to feature selection based on rough set and tabu search is proposed. In FSRT, conditional entropy is regarded as the heuristic to search the optimal solutions. The proposed method is introduced to credit scoring and Japan credit dataset in UCI database is selected to demonstrate the competitive performance of the proposed method. Moreover, FSRT shows a superior performance in saving the computational costs and improving classification accuracy.