Margin Based Feature Selection Research Articles

Robust weighted fuzzy margin-based feature selection with three-way decision

Feature selection has shown noticeable benefits to the tasks of machine learning and data mining, and an extensive variety of feature selection methods has been proposed to remove redundant and irrelevant features. However, most of the existing methods aim to find a feature subset to perfectly fit data with the minimum empirical risk, thus causing the problems of overfitting and noise sensitivity. In this study, a robust weighted fuzzy margin-based feature selection is proposed for uncertain data with noise. Concretely, a robust weighted fuzzy margin based on fuzzy rough sets is first introduced to evaluate the significance of different features. Then, a gradient ascent algorithm based on the noise filtering strategy and three-way decision is developed to optimize the sample and feature weights to further enlarge the fuzzy margin. Finally, an adaptive feature selection algorithm based on the robust weighted fuzzy margin is presented to generate an optimal feature subset with a large margin. Extensive experiments on the UCI benchmark datasets show that the proposed method could obtain high-quality feature subsets and outperform other representative methods under different noise rates.

Margin-Based Feature Selection in Incomplete Data

This study considers the problem of feature selection in incomplete data. The intuitive approach is to first impute the missing values, and then apply a standard feature selection method to select relevant features. In this study, we show how to perform feature selection directly, without imputing missing values. We define the objective function of the uncertainty margin-based feature selection method to maximize each instance’s uncertainty margin in its own relevant subspace. In optimization, we take into account the uncertainty of each instance due to the missing values. The experimental results on synthetic and 6 benchmark data sets with few missing values (less than 25%) provide evidence that our method can select the same accurate features as the alternative methods which apply an imputation method first. However, when there is a large fraction of missing values (more than 25%) in data, our feature selection method outperforms the alternatives, which impute missing values first.

IMMIGRATE: A Margin-Based Feature Selection Method with Interaction Terms.

Traditional hypothesis-margin researches focus on obtaining large margins and feature selection. In this work, we show that the robustness of margins is also critical and can be measured using entropy. In addition, our approach provides clear mathematical formulations and explanations to uncover feature interactions, which is often lack in large hypothesis-margin based approaches. We design an algorithm, termed IMMIGRATE (Iterative max-min entropy margin-maximization with interaction terms), for training the weights associated with the interaction terms. IMMIGRATE simultaneously utilizes both local and global information and can be used as a base learner in Boosting. We evaluate IMMIGRATE in a wide range of tasks, in which it demonstrates exceptional robustness and achieves the state-of-the-art results with high interpretability.

ENHANCED SVM CLASSIFIER FOR BREAST CANCER DIAGNOSIS

Breast cancer is the leading disease to cause death especially in women. In this paper, a frame work based algorithm for the classification of cancerous/non-cancerous data is developed using application of supervised machine learning. In feature selection, we derive basis set in the kernel space and then we extend the margin based feature selection algorithm. We are trying to explore several feature selection, extraction techniques and combine the optimal feature subsets with various learning classification methods such as KNN, PNN and Support Vector Machine (SVM) classifiers. The best classification performance for breast cancer diagnosis is attained equal to 99.17% between radius and compact features using SVM classifier. And also derive the features of a breast image in the WBCD dataset.

Unsupervised margin-based feature selection using linear transformations with neighbor preservation

We present a new margin-based feature selection method for unsupervised learning and use the selected salient features to discover interesting clusters using clustering algorithms. A basic characteristic of clustering motivates our method: each data instance usually belongs to the same cluster as its nearest neighbors and to different clusters than its farthest neighbors. Intuitively, the nearest and farthest neighbors would be primarily driven by certain features, which are defined as the salient features. Our method uses a maximum margin criterion to derive these salient features from the separability of the neighbors in the original representation space and of those in another space. Our method thus differs from existing wrapper-based methods in unsupervised learning because other methods usually need to partition data into clusters, which are then used to select features in the feature selection process. The experimental results indicate that our method outperforms several benchmark filter- and wrapper-based methods in tests on benchmark image datasets in terms of selecting features for discovering interesting clusters.

English

Feature Selection is the preprocessing process of identifying the subset of data from large dimension data. To identifying the required data, using some Feature Selection algorithms. Like ReliefF, Parzen-ReliefF algorithms, it attempts to directly maximize the classification accuracy and naturally reflects the Bayes error in the objective. Proposed algorithmic framework selects a subset of features by minimizing the Bayes error rate estimated by a nonparametric estimator. A set of existing algorithms as well as new ones can be derived naturally from this framework. As an example, we show that the Relief algorithm greedily attempts to minimize the Bayes error estimated by the k-Nearest-Neighbor (kNN) method. This new interpretation insightfully reveals the secret behind the family of margin-based feature selection algorithms and also offers a principled way to establish new alternatives for performance improvement. In particular, by exploiting the proposed framework, we establish the Parzen-Relief (PRelief) algorithm based on Parzen window estimator. The RELIEF algorithm is a popular approach for feature weight estimation. Many extensions of the RELIEF algorithm are developed. Because of the randomicity and the uncertainty of the instances used for calculating the feature weight vector in the RELEIF algorithm, the results will fluctuate with the instances, which lead to poor evaluation accuracy. To solve this problem, a feature selection algorithm parzen+reliefF based algorithm is proposed. It takes both the mean and the variance of the discrimination among instances and weights into account as the criterion of feature weight estimation, which makes the result more stable and accurate. And the main idea is how to estimate the performance of the both algorithms, for this we are using two algorithms for calculating the quality of the generated out puts. They are Leader and sub-leader algorithm and Davies– Bouldin index (DBI) algorithm. Both are clustering algorithms. Which are used for knowing the cluster quality and cluster similarity.

CLASSIFYING TEMPORAL MICROARRAY DATA BY SELECTING INFORMATIVE GENES

In order to more accurately predict an individual's health status, in clinical applications it is often important to perform analysis of high-dimensional gene expression data that varies with time. A major challenge in predicting from such temporal microarray data is that the number of biomarkers used as features is typically much larger than the number of labeled subjects. One way to address this challenge is to perform feature selection as a preprocessing step and then apply a classification method on selected features. However, traditional feature selection methods cannot handle multivariate temporal data without applying techniques that flatten temporal data into a single matrix in advance. In this study, a feature selection filter that can directly select informative features from temporal gene expression data is proposed. In our approach, we measure the distance between multivariate temporal data from two subjects. Based on this distance, we define the objective function of temporal margin based feature selection to maximize each subject's temporal margin in its own relevant subspace. The experimental results on synthetic and two real flu data sets provide evidence that our method outperforms the alternatives, which flatten the temporal data in advance.

Comparative analysis on margin based feature selection algorithms

Feature evaluation and selection is an important preprocessing step in classification and regression learning. As large quantity of irrelevant information is gathered, selecting the most informative features may help users to understand the task, and enhance the performance of the models. Margin has been widely accepted and used in evaluating feature quality these years. A collection of feature selection algorithms were developed using margin based loss functions and various search strategies. However, there is no comparative research conducted to study the effectiveness of these algorithms. In this work, we compare 14 margin based feature selections from the viewpoints of reduction capability, classification performance of reduced data and robustness, where four margin based loss functions and three search strategies are considered. Moreover, we also compare these techniques with two well-known margin based feature selection algorithms ReliefF and Simba. The derived conclusions give some guidelines for selecting features in practical applications.

Discriminative Feature Selection by Nonparametric Bayes Error Minimization

Feature selection is fundamental to knowledge discovery from massive amount of high-dimensional data. In an effort to establish theoretical justification for feature selection algorithms, this paper presents a theoretically optimal criterion, namely, the discriminative optimal criterion (DoC) for feature selection. Compared with the existing representative optimal criterion (RoC, [CHECK END OF SENTENCE]) which retains maximum information for modeling the relationship between input and output variables, DoC is pragmatically advantageous because it attempts to directly maximize the classification accuracy and naturally reflects the Bayes error in the objective. To make DoC computationally tractable for practical tasks, we propose an algorithmic framework, which selects a subset of features by minimizing the Bayes error rate estimated by a nonparametric estimator. A set of existing algorithms as well as new ones can be derived naturally from this framework. As an example, we show that the Relief algorithm [CHECK END OF SENTENCE] greedily attempts to minimize the Bayes error estimated by the k-Nearest-Neighbor (kNN) method. This new interpretation insightfully reveals the secret behind the family of margin-based feature selection algorithms [CHECK END OF SENTENCE], [CHECK END OF SENTENCE] and also offers a principled way to establish new alternatives for performance improvement. In particular, by exploiting the proposed framework, we establish the Parzen-Relief (P-Relief) algorithm based on Parzen window estimator, and the MAP-Relief (M-Relief) which integrates label distribution into the max-margin objective to effectively handle imbalanced and multiclass data. Experiments on various benchmark data sets demonstrate the effectiveness of the proposed algorithms.

Large-margin feature selection for monotonic classification

Monotonic classification plays an important role in the field of decision analysis, where decision values are ordered and the samples with better feature values should not be classified into a worse class. The monotonic classification tasks seem conceptually simple, but difficult to utilize and explain the order structure in practice. In this work, we discuss the issue of feature selection under the monotonicity constraint based on the principle of large margin. By introducing the monotonicity constraint into existing margin based feature selection algorithms, we design two new evaluation algorithms for monotonic classification. The proposed algorithms are tested with some artificial and real data sets, and the experimental results show its effectiveness.

Applying effective feature selection techniques with hierarchical mixtures of experts for spam classification

E-mail abuse has been steadily increasing during the last decade. E-mail users find themselves targeted by massive quantities of unsolicited bulk e-mail, which often contains offensive language or has fraudulent intentions. Internet Service Providers (ISPs) on the other hand, have to face a considerable system overloading as the incoming mail consumes network and storage resources. Among the plethora of solutions, the most prominent in terms of cost efficiency and complexity are the text filtering approaches. Most of the approaches model the problem using linear statistical models. Despite their popularity - due both to their simplicity and relative ease of interpretation - the non-linearity assumption of data samples is inappropriate in practice. This is mainly due to the inability of other approaches to capture the apparent non-linear relationships, which characterize these samples. In this paper, we propose a margin-based feature selection approach integrated with a Hierarchical Mixtures of Experts (HME) system, which attempts to overcome limitations common to other machine-learning based approaches. By reducing the data dimensionality using effective algorithms for feature selection we evaluated our system with publicly available corpora of e-mails, characterized by very high similarity between legitimate and bulk e-mail (and thus low discriminative potential). We experimented with two different architectures, a hierarchical HME and a perceptron HME. As a result, we confirm the domination of our Spam Filtering (SF) - HME method against other machine learning approaches, which present lesser degree of recall, as well as against traditional rule-based approaches, which lack considerably in the achieved degrees of precision.

Margin-based feature selection for hyperspectral data

A margin-based feature selection approach is explored for hyperspectral data. This approach is based on measuring the confidence of a classifier when making predictions on a test data. Greedy feature flip and iterative search algorithms, which attempts to maximise the margin-based evaluation functions, were used in the present study. Evaluation functions use linear, zero–one and sigmoid utility functions where a utility function controls the contribution of each margin term to the overall score. The results obtained by margin-based feature selection technique were compared to a support vector machine-based recurring feature elimination approach. Two different hyperspectral data sets, one consisting of 65 bands (DAIS data) and other with 185 bands (AVIRIS data) were used. With digital airborne imaging spectrometer (DAIS) data, the classification accuracy by greedy feature flip algorithm and sigmoid utility function was 93.02% using a total of 24 selected features in comparison to an accuracy of 91.76% with full set of 65 features. The results suggest a significant increase in classification accuracy with 24 selected features. The classification accuracy (93.4%) achieved by the iterative search margin-based algorithm with 20 selected features using sigmoid utility function is also significantly more accurate than that achieved with 65 features. To judge the usefulness of margin-based feature selection approaches, another hyperspectral data set consisting of 185 features was used. A total of 65 selected features were used to evaluate the performance of margin-based feature selection approach. The results suggest a significantly improved performance by greedy feature flip-based feature selection technique with this data set also. This study also suggest that margin-based feature selection algorithms provide a comparable performance to support vector machine-based recurring feature elimination approach.

A Novel Feature Selection Algorithm Based on Hypothesis-Margin

Iterative search margin based algorithm(Simba) has been proven effective for feature selection. However, it still has the following disadvantages: (1) the previously proposed model still lacks enough robust to noises; and (2) the given model does not use any global information, in this way some useful discrimination information may be lost and the convergence speed is also influenced in some cases. In this paper, by incorporating global information, a novel margin based feature selection framework is introduced. According to the newly designed model, an improved margin based feature selection algorithm(Isimba) is proposed. By effectively adjusting the contribution of the global information, Isimba can efficiently reduce the computational cost and at the same time obtain more effective feature subsets as compared to Simba. The experiments on 6 artificial and 8 real-life benchmark datasets show that Isimba is effective and efficient.