Feature Subset Selection Problem Research Articles

Classification of Medical Datasets Using SVMs with Hybrid Evolutionary Algorithms Based on Endocrine-Based Particle Swarm Optimization and Artificial Bee Colony Algorithms.

The classification and analysis of data is an important issue in today's research. Selecting a suitable set of features makes it possible to classify an enormous quantity of data quickly and efficiently. Feature selection is generally viewed as a problem of feature subset selection, such as combination optimization problems. Evolutionary algorithms using random search methods have proven highly effective in obtaining solutions to problems of optimization in a diversity of applications. In this study, we developed a hybrid evolutionary algorithm based on endocrine-based particle swarm optimization (EPSO) and artificial bee colony (ABC) algorithms in conjunction with a support vector machine (SVM) for the selection of optimal feature subsets for the classification of datasets. The results of experiments using specific UCI medical datasets demonstrate that the accuracy of the proposed hybrid evolutionary algorithm is superior to that of basic PSO, EPSO and ABC algorithms, with regard to classification accuracy using subsets with a reduced number of features.

A Semidefinite Programming Based Search Strategy for Feature Selection with Mutual Information Measure.

Feature subset selection, as a special case of the general subset selection problem, has been the topic of a considerable number of studies due to the growing importance of data-mining applications. In the feature subset selection problem there are two main issues that need to be addressed: (i) Finding an appropriate measure function than can be fairly fast and robustly computed for high-dimensional data. (ii) A search strategy to optimize the measure over the subset space in a reasonable amount of time. In this article mutual information between features and class labels is considered to be the measure function. Two series expansions for mutual information are proposed, and it is shown that most heuristic criteria suggested in the literature are truncated approximations of these expansions. It is well-known that searching the whole subset space is an NP-hard problem. Here, instead of the conventional sequential search algorithms, we suggest a parallel search strategy based on semidefinite programming (SDP) that can search through the subset space in polynomial time. By exploiting the similarities between the proposed algorithm and an instance of the maximum-cut problem in graph theory, the approximation ratio of this algorithm is derived and is compared with the approximation ratio of the backward elimination method. The experiments show that it can be misleading to judge the quality of a measure solely based on the classification accuracy, without taking the effect of the non-optimum search strategy into account.

Multi-Objective Feature Subset Selection using Non-dominated Sorting Genetic Algorithm

This paper presents an evolutionary algorithm based technique to solve multi-objective feature subset selection problem. The data used for classification contains large number of features called attributes. Some of these attributes are not relevant and needs to be eliminated. In classification procedure, each feature has an effect on the accuracy, cost and learning time of the classifier. So, there is a strong requirement to select a subset of the features before building the classifier. This proposed technique treats feature subset selection as multi-objective optimization problem. This research uses one of the latest multi-objective genetic algorithms (NSGA - II). The fitness value of a particular feature subset is measured by using ID3. The testing accuracy acquired is then assigned to the fitness value. This technique is tested on several datasets taken from the UCI machine repository. The experiments demonstrate the feasibility of using NSGA-II for feature subset selection.

Knowledge Discovery in Medical Data by using Rough Set Rule Induction Algorithms

In the performance of data mining and knowledge discovery activities, rough set theory has been regarded as a powerful, feasible and effective methodology. There is a need for analysis of medical data that deals with incomplete and inconsistent information with the tremendous manipulation at different levels. In this context, rough set rule induction algorithms are capable of generating decision rules which can potentially provide new medical insight and profound medical knowledge. By taking into consideration all the above aspects, the present investigation is carried out. The results clearly show that rough set approach is certainly a useful tool for medical applications. Relationships and patterns within this data could provide new medical knowledge. The genetic algorithms offer an attractive approach for solving the feature subset selection problem. The process of finding useful patterns or meaning in raw data has been called knowledge discovery in databases. The algorithms used for the present study are: Exhaustive search, Covering, LEM2 and Genetic algorithms. Rules are generated and improved in the case of the above mentioned four algorithms. Further, the generated rules are improved by applying the shortening ratio as 0.8. Some of the important results of the present investigation include maximum coverage of 100% is observed in the case of exhaustive and genetic algorithms.

Knowledge Discovery in Medical Data by using Rough Set Rule Induction Algorithms

In the performance of data mining and knowledge discovery activities, rough set theory has been regarded as a powerful, feasible and effective methodology. There is a need for analysis of medical data that deals with incomplete and inconsistent information with the tremendous manipulation at different levels. In this context, rough set rule induction algorithms are capable of generating decision rules which can potentially provide new medical insight and profound medical knowledge. By taking into consideration all the above aspects, the present investigation is carried out. The results clearly show that rough set approach is certainly a useful tool for medical applications. Relationships and patterns within this data could provide new medical knowledge. The genetic algorithms offer an attractive approach for solving the feature subset selection problem. The process of finding useful patterns or meaning in raw data has been called knowledge discovery in databases. The algorithms used for the present study are: Exhaustive search, Covering, LEM2 and Genetic algorithms. Rules are generated and improved in the case of the above mentioned four algorithms. Further, the generated rules are improved by applying the shortening ratio as 0.8. Some of the important results of the present investigation include maximum coverage of 100% is observed in the case of exhaustive and genetic algorithms. Keywords: Decomposition Tree, Rough Set, Rules, Rule Induction Algorithms

A method for determining the number of features in the kernel space required for preserving classifiability

Feature generation techniques that sort the generated features in terms of their importance, such as principal component analysis (PCA), reduce the problem of feature subset selection to only determining the number of features to be retained. For databases with linearly inseparable classes, kernel PCA can be used as the feature generation method instead of the linear PCA. However, determining the number of features in the kernel space that needs to be retained for preserving the classifiability is a difficult problem, since the data vectors are not available in an explicit form in that space. In this paper, we propose a criterion and an algorithm for determining the number of required features in kernel PCA using only the elements of the kernel matrix. In order to show the effectiveness of the proposed criterion, the new algorithm is applied to the USPS handwritten digit, Yale Face and Caltech 101 databases. The proposed algorithm is also investigated for its robustness to noise that corrupts the data samples.

Speeding up incremental wrapper feature subset selection with Naive Bayes classifier

This paper deals with the problem of wrapper feature subset selection (FSS) in classification-oriented datasets with a (very) large number of attributes. In high-dimensional datasets with thousands of variables, wrapper FSS becomes a laborious computational process because of the amount of CPU time it requires. In this paper we study how under certain circumstances the wrapper FSS process can be speeded up by embedding the classifier into the wrapper algorithm, instead of dealing with it as a black-box. Our proposal is based on the combination of the NB classifier (which is known to be largely beneficial for FSS) with incremental wrapper FSS algorithms. The merit of this approach is analyzed both theoretically and experimentally, and the results show an impressive speed-up for the embedded FSS process.

Fast feature selection aimed at high-dimensional data via hybrid-sequential-ranked searches

We address the feature subset selection problem for classification tasks. We examine the performance of two hybrid strategies that directly search on a ranked list of features and compare them with two widely used algorithms, the fast correlation based filter (FCBF) and sequential forward selection (SFS). The proposed hybrid approaches provide the possibility of efficiently applying any subset evaluator, with a wrapper model included, to large and high-dimensional domains. The experiments performed show that our two strategies are competitive and can select a small subset of features without degrading the classification error or the advantages of the strategies under study.

Analysis of Markov Boundary Induction in Bayesian Networks: A New View From Matroid Theory

Learning Markov boundaries from data without having to learn a Bayesian network first can be viewed as a feature subset selection problem and has received much attention due to its significance in ...

IMPROVING INCREMENTAL WRAPPER-BASED SUBSET SELECTION VIA REPLACEMENT AND EARLY STOPPING

This paper deals with the problem of feature subset selection in classification-oriented datasets with a (very) large number of attributes. In such datasets complex classical wrapper approaches become intractable due to the high number of wrapper evaluations to be carried out. One way to alleviate this problem is to use the so-called filter-wrapper approach or Incremental Wrapper-based Subset Selection (IWSS), which consists of the construction of a ranking among the predictive attributes by using a filter measure, and then a wrapper approach is used by following the rank. In this way the number of wrapper evaluations is linear on the number of predictive attributes. In this paper we present two contributions to the IWSS approach. The first one is related with obtaining more compact subsets, and enables not only the addition of new attributes but also their interchange with some of those already included in the selected subset. Our second contribution, termed early stopping, sets an adaptive threshold on the number of attributes in the ranking to be considered. The advantages of these new approaches are analyzed both theoretically and experimentally. The results over a set of 12 high-dimensional datasets corroborate the success of our proposals.

Cost Effective Approach on Feature Selection Using Genetic Algorithms and Fuzzy logic for Diabetes Diagnosis

A way to enhance the performance of a model that combines genetic algorithms and fuzzy logic for feature selection and classification is proposed. Early diagnosis of any disease with less cost is preferable. Diabetes is one such disease. Diabetes has become the fourth leading cause of death in developed countries and there is substantial evidence that it is reaching epidemic proportions in many developing and newly industrialized nations. In medical diagnosis, patterns consist of observable symptoms along with the results of diagnostic tests. These tests have various associated costs and risks. In the automated design of pattern classification, the proposed system solves the feature subset selection problem. It is a task of identifying and selecting a useful subset of pattern-representing features from a larger set of features. Using fuzzy rule-based classification system, the proposed system proves to improve the classification accuracy.

Analysis of Markov Boundary Induction in Bayesian Networks: A New View From Matroid Theory

Learning Markov boundaries from data without having to learn a Bayesian network first can be viewed as a feature subset selection problem and has received much attention due to its significance in the wide applications of AI techniques. Popular constraint based methods suffer from high computational complexity and are usually unstable in spaces of high dimensionality. We propose a new perspective from matroid theory towards the discovery of Markov boundaries of random variable in the domain, and develop a learning algorithm which guarantees to recover the true Markov boundaries by a greedy learning algorithm. Then we use the precision matrix of the original distribution as a measure of independence to make our algorithm feasible in large scale problems, which is essentially an approximation of the probabilistic relations with Gaussians and can find possible variables in Markov boundaries with low computational complexity. Experimental results on standard Bayesian networks show that our analysis and approximation can efficiently and accurately identify Markov boundaries in complex networks from data.

A discrete particle swarm optimization method for feature selection in binary classification problems

This paper investigates the feature subset selection problem for the binary classification problem using logistic regression model. We developed a modified discrete particle swarm optimization (PSO) algorithm for the feature subset selection problem. This approach embodies an adaptive feature selection procedure which dynamically accounts for the relevance and dependence of the features included the feature subset. We compare the proposed methodology with the tabu search and scatter search algorithms using publicly available datasets. The results show that the proposed discrete PSO algorithm is competitive in terms of both classification accuracy and computational performance.

Feature Subset Selection Problem using Wrapper Approach in Supervised Learning

Feature subset selection is of immense importance in the field of data mining. The increased dimensionality of data makes testing and training of general classification method difficult. Mining on the reduced set of attributes reduces computation time and also helps to make the patterns easier to understand. In this paper a wrapper approach for feature selection is proposed. As a part of feature selection step we used wrapper approach with Genetic algorithm as random search technique for subset generation ,wrapped with different classifiers/ induction algorithm namely decision tree C4.5, NaiveBayes, Bayes networks and Radial basis function as subset evaluating mechanism on four standard datasets namely Pima Indians Diabetes Dataset, Breast Cancer, Heart Stat log and Wisconsin Breast Cancer. Further the relevant attributes identified by proposed wrapper are validated using classifiers. Experimental results illustrate, employing feature subset selection using proposed wrapper approach has enhanced classification accuracy.

Feature subset selection from positive and unlabelled examples

The feature subset selection problem has a growing importance in many machine learning applications where the amount of variables is very high. There is a great number of algorithms that can approach this problem in supervised databases but, when examples from one or more classes are not available, supervised feature subset selection algorithms cannot be directly applied. One of these algorithms is the correlation based filter selection (CFS). In this work we propose an adaptation of this algorithm that can be applied when only positive and unlabelled examples are available. As far as we know, this is the first time the feature subset selection problem is studied in the positive unlabelled learning context. We have tested this adaptation on synthetic datasets obtained by sampling Bayesian network models where we know which variables are (in)dependent of the class. We have also tested our adaptations on real-life databases where the absence of negative examples has been simulated. The results show that, having enough positive examples, it is possible to obtain good solutions to the feature subset selection problem when only positive and unlabelled instances are available.

Pap smear diagnosis using a hybrid intelligent scheme focusing on genetic algorithm based feature selection and nearest neighbor classification

The term pap-smear refers to samples of human cells stained by the so-called Papanicolaou method. The purpose of the Papanicolaou method is to diagnose pre-cancerous cell changes before they progress to invasive carcinoma. In this paper a metaheuristic algorithm is proposed in order to classify the cells. Two databases are used, constructed in different times by expert MDs, consisting of 917 and 500 images of pap smear cells, respectively. Each cell is described by 20 numerical features, and the cells fall into 7 classes but a minimal requirement is to separate normal from abnormal cells, which is a 2 class problem. For finding the best possible performing feature subset selection problem, an effective genetic algorithm scheme is proposed. This algorithmic scheme is combined with a number of nearest neighbor based classifiers. Results show that classification accuracy generally outperforms other previously applied intelligent approaches.

Canonical subsets of image features

Many object recognition and localization techniques utilize multiple levels of local representations. These local feature representations are common, and one way to improve the efficiency of algorithms that use them is to reduce the size of the local representations. There has been previous work on selecting subsets of image features, but the focus here is on a systematic study of the feature selection problem. We have developed a combinatorial characterization of the feature subset selection problem that leads to a general optimization framework. This framework optimizes multiple objectives and allows the encoding of global constraints. The features selected by this algorithm are able to achieve improved performance on the problem of object localization. We present a dataset of synthetic images, along with ground-truth information, which allows us to precisely measure and compare the performance of feature subset algorithms. Our experiments show that subsets of image features produced by our method, stable bounded canonical sets (SBCS), outperform subsets produced by K-Means clustering, GA, and threshold-based methods for the task of object localization under occlusion.

Neighborhood rough set based heterogeneous feature subset selection

Feature subset selection is viewed as an important preprocessing step for pattern recognition, machine learning and data mining. Most of researches are focused on dealing with homogeneous feature selection, namely, numerical or categorical features. In this paper, we introduce a neighborhood rough set model to deal with the problem of heterogeneous feature subset selection. As the classical rough set model can just be used to evaluate categorical features, we generalize this model with neighborhood relations and introduce a neighborhood rough set model. The proposed model will degrade to the classical one if we specify the size of neighborhood zero. The neighborhood model is used to reduce numerical and categorical features by assigning different thresholds for different kinds of attributes. In this model the sizes of the neighborhood lower and upper approximations of decisions reflect the discriminating capability of feature subsets. The size of lower approximation is computed as the dependency between decision and condition attributes. We use the neighborhood dependency to evaluate the significance of a subset of heterogeneous features and construct forward feature subset selection algorithms. The proposed algorithms are compared with some classical techniques. Experimental results show that the neighborhood model based method is more flexible to deal with heterogeneous data.

NATURE INSPIRED INTELLIGENCE IN MEDICINE: ANT COLONY OPTIMIZATION FOR PAP-SMEAR DIAGNOSIS

During the last years nature inspired intelligent techniques have become attractive for analyzing large data sets and solving complex optimization problems. In this paper, one of the most interesting of them, the Ant Colony Optimization (ACO), is used for the construction of a hybrid algorithmic scheme which effectively handles the Pap Smear Cell classification problem. This algorithmic approach is properly combined with a number of nearest neighbor based approaches for performing the requested classification task, through the solution of the so-called optimal feature subset selection problem. The proposed complete algorithmic scheme is tested in two sets of data. The first one consists of 917 images of pap smear cells and the second set consists of 500 images, classified carefully by expert cyto-technicians and doctors. Each cell is described by 20 numerical features, and the cells fall into seven (7) classes, four (4) representing normal cells and three (3) abnormal cases. Nevertheless, from the medical diagnosis viewpoint, a minimum requirement corresponds to the general two-class problem of correct separation between normal from abnormal cells.

APPLICATION OF ANT COLONY OPTIMIZATION TO CREDIT RISK ASSESSMENT

This paper presents a novel approach to solve feature subset selection problems using an Ant Colony Optimization (ACO) algorithm. ACO is one of the important naturally inspired intelligent techniques. It is based on the foraging behavior of real ants in nature. The proposed ACO is combined with a number of nearest neighbor classifiers. The resulting ACO algorithm is applied to classify credit risk using data belonging to 1,411 firms obtained from a leading Greek commercial bank. The objective is to classify subject firms into several groups representing different levels of credit risk. The results of the proposed algorithm are compared with those of others including SVM, CART, and with two other metaheuristic algorithms using tabu search and genetic algorithms, both of which use nearest neighbor classifiers in the classification phase. The results suggest that the proposed method is more accurate than others that have been tested in classifying credit risk.