Feature Subset Selection Problem Research Articles

Parallel fractional dominance MOEAs for feature subset selection in big data

In this paper, we solve the feature subset selection (FSS) problem with three objective functions namely, cardinality, area under receiver operating characteristic curve (AUC) and Matthews correlation coefficient (MCC) using novel multi-objective evolutionary algorithms (MOEAs). MOEAs often encounter poor convergence due to the increase in non-dominated solutions and getting entrapped in the local optima. This situation worsens when dealing with large, voluminous big and high-dimensional datasets. To address these challenges, we propose parallel, fractional dominance-based MOEAs for FSS under Spark. Further, to improve the exploitation of MOEAs, we introduce a novel batch opposition-based learning (BOP) along with a cardinality constraint on the opposite solution. Accordingly, we propose two variants, namely, BOP1 and BOP2. In BOP1, a single neighbour is randomly chosen in the opposite solution space, whereas in BOP2, a group of randomly chosen neighbours in the opposite solution space. In either case, the opposite solutions are evaluated to improve the exploitation capability of the underlying MOEAs. We observe that in terms of mean optimal objective function values and across all datasets, the proposed BOP2 variant of parallel fractional dominance-based algorithms emerges as the top performer in obtaining efficient solutions. Further, we introduce a novel metric, namely the ratio of hypervolume (HV) and inverted generated distance (IGD), HV/IGD, that combines both diversity and convergence. With respect to the mean HV/IGD computed over 20 runs and Formula 1 racing, the BOP1 variants of fractional dominance-based MOEAs outperformed other algorithms.

Feature subset selection using multimodal multiobjective differential evolution

The main aim of feature subset selection is to find the minimum number of required features to perform classification without affecting the accuracy. It is one of the useful real-world applications for different types of classification datasets. Different feature subsets may achieve similar classification accuracy, which can help the user to select the optimal features. There are two main objectives involved in selecting a feature subset: minimizing the number of features and maximizing the accuracy. However, most of the existing studies do not consider multiple feature subsets of the same size. In this paper, we have proposed an algorithm for multimodal multiobjective optimization based on differential evolution with respect to the feature subset selection problem. We have proposed the probability initialization method to identify the selected features with equal distribution in the search space. We have also proposed a niching technique to explore the search space and exploit the nearby solutions. Further, we have proposed a convergence archive to locate and store the optimal feature subsets. Exhaustive experimentation has been conducted on different datasets with varying characteristics to identify multiple feature subsets. We have also proposed an evaluation metric for the quantitative comparison of the proposed algorithm with the existing algorithms. Results have also been compared with existing algorithms in the objective space and in terms of classification accuracy, which shows the effectiveness of the proposed algorithm.

A mixed-integer exponential cone programming formulation for feature subset selection in logistic regression

Logistic regression is one of the widely-used classification tools to construct prediction models. For datasets with a large number of features, feature subset selection methods are considered to obtain accurate and interpretable prediction models, in which irrelevant and redundant features are removed. In this paper, we address the problem of feature subset selection in logistic regression using modern optimization techniques. To this end, we formulate this problem as a mixed-integer exponential cone program (MIEXP). To the best of our knowledge, this is the first time both nonlinear and discrete aspects of the underlying problem are fully considered within an exact optimization framework. We derive different versions of the MIEXP model by the means of regularization and goodness of fit measures including Akaike Information Criterion and Bayesian Information Criterion. Finally, we solve our MIEXP models using the solver MOSEK and evaluate the performance of our different versions over a set of toy examples and benchmark datasets. The results show that our approach is quite successful in obtaining accurate and interpretable prediction models compared to other methods from the literature.

A Method for Feature Subset Selection in Software Product Lines

Software product line (SPL) represents methods, tools, and techniques for creating a group of related software systems. Each product is a combination of multiple features. So, the task of production can be mapped to a feature subset selection problem, which is an NP-hard problem. This issue is very significant when the number of features in a software product line is huge. This chapter is aimed to address the feature subset selection in software product lines. Furthermore, the authors aim at studying the performance of a proposed multi-objective method in solving this NP-hard problem. Here, a multi-objective method (MOBAFS) is presented for feature selection in SPLs. The MOBAFS is a an optimization algorithm, which is inspired by the foraging behavior of honeybees. This technique is evaluated on five large-scale real-world software product lines in the range of 1,244 to 6,888 features. The proposed method is compared with the SATIBEA. According to the results of three solution quality indicators and two diversity metrics, the proposed method, in most cases, surpasses the other algorithm.

Internet financing credit risk evaluation using multiple structural interacting elastic net feature selection

Internet financing is an important alternative to banks where individuals or SMEs borrow money using online trading platforms. A central problem for internet financing is how to identify the most influential factors that are closely related to the credit risks. This problem is inherently challenging because the raw data of internet financing is often associated with complex structural correlations and usually contains many irrelevant and redundant features. To effectively identify the most salient features for credit risk evaluation in internet financing, we develop a new multiple structural interacting elastic net model for feature selection (MSIEN). Our idea is based on converting the original vectorial features into structure-based feature graph representations to encapsulate structural relationship between pairwise samples, and defining two new information theoretic criteria. One criterion maximizes joint relevance of different pairwise feature combinations in relation to the target feature graph and the other minimizes the redundancy between pairwise features. Then two structural interaction matrices are obtained with the elements representing the proposed information theoretic measures. To identify the most informative features, we formulate a new optimization model which combines the interaction matrices and an elastic net regularization model for the feature subset selection problem. We exploit an efficient iterative optimization algorithm to solve the proposed problem and also provide the theoretical analyses on its convergence property and computational complexity. Finally, experimental results on datasets of internet financing demonstrate the effectiveness of the proposed MSIEN method.

Multi-objectives TLBO hybrid method to select the related risk features with rheumatism disease

Features subset selection was commonly used in data mining and artificial intelligence techniques to produce a model with a minimal set of features that enhances the performance of the classifier. The essential motive for selecting features is to avoid the problem of a number of dimensions trap. This paper introduces a new technique of selection of features dependent on the modified of binary teaching–learning-based optimization and the suggested method called MBTLBO. This algorithm (teaching learning-based optimization TLBO) is one of the present metaheuristic that is been widely utilized to a several of intractable optimization issues in recent times. Such algorithm has been combined with supervised data mining technique (support vector machine) for the implementation of feature subset selection problem in binary identification. The collection of specific risk features with the rheumatic disease was implemented. The findings revealed that the new approach (MBTLBO) increases the accuracy of classification.

A new approach of attribute reduction of rough sets based on soft metric

Attribute reduction is considered as an important processing step for pattern recognition, machine learning and data mining. In this paper, we combine soft set and rough set to use them in applications. We generalize rough set model and introduce a soft metric rough set model to deal with the problem of heterogeneous numerical feature subset selection. We construct a soft metric on the family of knowledge structures based on the soft distance between attributes. The proposed model will degrade to the classical one if we specify a zero soft real number. We also provide a systematic study of attribute reduction of rough sets based on soft metric. Based on the constructed metric, we define co-information systems and consistent co-decision systems, and we provide a new method of attribute reductions of each system. Furthermore, we present a judgement theorem and discernibility matrix associated with attribute of each type of system. As an application, we present a case study from Zoo data set to verify our theoretical results.

Analysis of various transfer functions for binary owl search algorithm in feature selection problem

ABSTRACT Owl Search Algorithm (OSA) is a recently proposed nature-inspired meta-heuristic algorithm which is easily implementable and exhibits great potential for solving continuous optimization problems. In our earlier work, a binary version of owl search algorithm (BOSA), with transfer functions for mapping the continuous solution space into a binary one, has been developed and applied in optimal feature subset selection problem. In our preliminary simulation experiments, it was found that the performance of the solution depends on the type of transfer function used. In this work, an extensive analysis of various types of transfer functions and their respective effects on the selection of optimal feature subset has been studied by simulation experiments with multiple benchmark datasets. Transfer functions of three different families, S-shaped, V-shaped and quadratic, are used for designing eleven BOSA models, each of which is made by combining individual transfer function. The performances of the proposed wrapper based feature subset selection algorithm based on several BOSA models have been evaluated by simulation experiments with twenty datasets for finding out the best model. The best observed BOSA model has also been compared with other similar meta-heuristics algorithms for feature subset selection. Experimental results show that the feature subset selected by BOSA with quadratic transfer function produces the highest classification accuracy with the minimum number of selected features compared to other algorithms.

A Locally Searched Binary Artificial Bee Colony Algorithm Based on Hamming Distance for Binary Optimization

Artificial Bee Colony is a population based, bio-inspired optimization algorithm that developed for continues problems. The aim of this study is to develop a binary version of the Artificial Bee Colony (ABC) Algorithm to solve feature subset selection problem on bigger data. ABC Algorithm, has good global search capability but there is a lack of local search in the algorithm. To overcome this problem, the neighbor selection mechanism in the employed bee phase is improved by changing the new source generation formula that has hamming distance based local search capacity. With a re-population strategy, the diversity of the population is increased and premature convergence is prevented. To measure the effectiveness of the proposed algorithm, fourteen datasets which have more than 100 features were selected from UCI Machine Learning Repository and processed by the proposed algorithm. The performance of the proposed algorithm was compared to three well-known algorithms in terms of classification error, feature size and computation time. The results proved that the increased local search ability improves the performance of the algorithm for all criteria.

An hybrid method for feature selection based on multiobjective optimization and mutual information

In this paper we propose a hybrid approach using mutual information and multi-objective optimization for feature subset selection problem. The hybrid aspect is due to the sequence of a filter method and a wrapper method in order to take advantages of both. The filter method reduces the exploration space by keeping subsets having good internal properties and the wrapper method chooses among the remaining subsets with a classification performances criterion. In the filter step, the subsets are evaluated in a multi-objective way to ensure diversity within the subsets. The evaluation is based on the mutual information to estimate the dependency between features and classes and the redundancy between features within the same subset. We kept the non-dominated (Pareto optimal) subsets for the second step. In the wrapper step, the selection is made according to the stability of the subsets regarding classification performances during learning stage on a set of classifiers to avoid the specialization of the selected subsets for a given classifiers. The proposed hybrid approach is experimented on a variety of reference data sets and compared to the classical feature selection methods FSDD and mRMR. The resulting algorithm outperforms these algorithms and the computation complexity remains acceptable even if it increases with regards to these two fast selection methods.

A new binary grasshopper optimization algorithm for feature selection problem

The grasshopper optimization algorithm is one of the recently population-based optimization techniques inspired by the behaviours of grasshoppers in nature. It is an efficient optimization algorithm and since demonstrates excellent performance in solving continuous problems, but cannot resolve directly binary optimization problems. Many optimization problems have been modelled as binary problems since their decision variables varied in binary space such as feature selection in data classification. The main goal of feature selection is to find a small size subset of feature from a sizeable original set of features that optimize the classification accuracy. In this paper, a new binary variant of the grasshopper optimization algorithm is proposed and used for the feature subset selection problem. This proposed new binary grasshopper optimization algorithm is tested and compared to five well-known swarm-based algorithms used in feature selection problem. All these algorithms are implemented and experimented assessed on twenty data sets with various sizes. The results demonstrated that the proposed approach could outperform the other tested methods.

Ensemble Based Classification of Sentiments Using Forest Optimization Algorithm

Feature subset selection is a process to choose a set of relevant features from a high dimensionality dataset to improve the performance of classifiers. The meaningful words extracted from data forms a set of features for sentiment analysis. Many evolutionary algorithms, like the Genetic Algorithm (GA) and Particle Swarm Optimization (PSO), have been applied to feature subset selection problem and computational performance can still be improved. This research presents a solution to feature subset selection problem for classification of sentiments using ensemble-based classifiers. It consists of a hybrid technique of minimum redundancy and maximum relevance (mRMR) and Forest Optimization Algorithm (FOA)-based feature selection. Ensemble-based classification is implemented to optimize the results of individual classifiers. The Forest Optimization Algorithm as a feature selection technique has been applied to various classification datasets from the UCI machine learning repository. The classifiers used for ensemble methods for UCI repository datasets are the k-Nearest Neighbor (k-NN) and Naïve Bayes (NB). For the classification of sentiments, 15–20% improvement has been recorded. The dataset used for classification of sentiments is Blitzer’s dataset consisting of reviews of electronic products. The results are further improved by ensemble of k-NN, NB, and Support Vector Machine (SVM) with an accuracy of 95% for the classification of sentiment tasks.

Identifying the most informative features using a structurally interacting elastic net

Feature selection can efficiently identify the most informative features with respect to the target feature used in training. However, state-of-the-art vector-based methods are unable to encapsulate the relationships between feature samples into the feature selection process, thus leading to significant information loss. To address this problem, we propose a new graph-based structurally interacting elastic net method for feature selection. Specifically, we commence by constructing feature graphs that can incorporate pairwise relationship between samples. With the feature graphs to hand, we propose a new information theoretic criterion to measure the joint relevance of different pairwise feature combinations with respect to the target feature graph representation. This measure is used to obtain a structural interaction matrix where the elements represent the proposed information theoretic measure between feature pairs. We then formulate a new optimization model through the combination of the structural interaction matrix and an elastic net regression model for the feature subset selection problem. This allows us to (a) preserve the information of the original vectorial space, (b) remedy the information loss of the original feature space caused by using graph representation, and (c) promote a sparse solution and also encourage correlated features to be selected. Because the proposed optimization problem is non-convex, we develop an efficient alternating direction multiplier method (ADMM) to locate the optimal solutions. Extensive experiments on various datasets demonstrate the effectiveness of the proposed method.

F-WSS $$^{++}$$ + + : incremental wrapper subset selection algorithm for fuzzy extreme learning machine

Fuzzy extreme learning machine (F-ELM) is a hybrid combination made to get the benefits of fuzzy system and extreme learning machine (ELM). F-ELM randomly initializes the weights between input layer to the hidden layer and analytically tunes the weights between hidden layer to the output layer. Due to this random initialization and the availability of redundant and irrelevant features, F-ELM may degrade the overall (generalization) performance. To solve the mentioned problem in this paper, an advanced classification algorithm $$\hbox {F-WSS}^{++}$$ (incremental wrapper subset selection algorithm for F-ELM) is designed for multiclass and binary class classification problems. The merits of the proposed algorithm are analyzed theoretically and experimentally. The test results are cross checked for $$\hbox {F-WSS}^{++}$$ and $$\hbox {E-WSS}^{++}$$ (incremental wrapper subset selection algorithm for ELM) for the clinical dataset. The effectiveness of the $$\hbox {F-WSS}^{++}$$ algorithm is verified by statistical methods. It is observed that $$\hbox {F-WSS}^{++}$$ has the capability to handle weighted classification problem, feature subset selection problem and optimization problem. It also improves 9–10% classification accuracy by using only 50% features.

Novel multiobjective TLBO algorithms for the feature subset selection problem

Abstract Teaching Learning Based Optimization (TLBO) is a new metaheuristic that has been successfully applied to several intractable optimization problems in recent years. In this study, we propose a set of novel multiobjective TLBO algorithms combined with supervised machine learning techniques for the solution of Feature Subset Selection (FSS) in Binary Classification Problems (FSS-BCP). Selecting the minimum number of features while not compromising the accuracy of the results in FSS-BCP is a multiobjective optimization problem. We propose TLBO as a FSS mechanism and utilize its algorithm-specific parameterless concept that does not require any parameters to be tuned during the optimization. Most of the classical metaheuristics such as Genetic and Particle Swarm Optimization algorithms need additional efforts for tuning their parameters (crossover ratio, mutation ratio, velocity of particle, inertia weight, etc.), which may have an adverse influence on their performance. Comprehensive experiments are carried out on the well-known machine learning datasets of UCI Machine Learning Repository and significant improvements have been observed when the proposed multiobjective TLBO algorithms are compared with state-of-the-art NSGA-II, Particle Swarm Optimization, Tabu Search, Greedy Search, and Scatter Search algorithms.

Data analytics: feature extraction for application with small sample in classification algorithms

This paper focuses on improving the classification accuracy for supervised learning in areas of application with very few training data and with extremely available high dimensionality. This paper proposes a framework which acts as a decision support system incorporating both feature selection and feature extraction to improvise the classification accuracy. The feature selection technique comprises redundancy elimination and relevance analysis. Feature subset selection problems eliminate features which are redundant by using correlation-based maximum spanning tree. But, the eliminated features may contain useful information which may contribute in determining the target or class labels. The principal components are extracted from the eliminated features and they are complemented with the selected features to perform classification. The superiority of the proposed method over other feature selection methods, in terms of computational complexity and classification accuracy, is established extensively on various datasets.

PIECEWISE-LINEAR APPROXIMATION FOR FEATURE SUBSET SELECTION IN A SEQUENTIAL LOGIT MODEL

This paper concerns a method of selecting a subset of features for a sequential logit model. Tanaka and Nakagawa (2014) proposed a mixed integer quadratic optimization formulation for solving the problem based on a quadratic approximation of the logistic loss function. However, since there is a significant gap between the logistic loss function and its quadratic approximation, their formulation may fail to find a good subset of features. To overcome this drawback, we apply a piecewise-linear approximation to the logistic loss function. Accordingly, we frame the feature subset selection problem of minimizing an information criterion as a mixed integer linear optimization problem. The computational results demonstrate that our piecewise-linear approximation approach found a better subset of features than the quadratic approximation approach.

APPLICATIONS OF GRAVITATIONAL SEARCH ALGORITHM IN ENGINEERING

Gravitational search algorithm (GSA) is a nature-inspired conceptual framework with roots in gravitational kinematics, a branch of physics that models the motion of masses moving under the influence of gravity. In a recent article the authors reviewed the principles of GSA. This article presents a review of applications of GSA in engineering including combinatorial optimization problems, economic load dispatch problem, economic and emission dispatch problem, optimal power flow problem, optimal reactive power dispatch problem, energy management system problem, clustering and classification problem, feature subset selection problem, parameter identification, training neural networks, traveling salesman problem, filter design and communication systems, unit commitment problem and multiobjective optimization problems.

Optimal feature selection using distance-based discrete firefly algorithm with mutual information criterion

In this paper, we investigate feature subset selection problem by a new self-adaptive firefly algorithm (FA), which is denoted as DbFAFS. In classical FA, it uses constant control parameters to solve different problems, which results in the premature of FA and the fireflies to be trapped in local regions without potential ability to explore new search space. To conquer the drawbacks of FA, we introduce two novel parameter selection strategies involving the dynamical regulation of the light absorption coefficient and the randomization control parameter. Additionally, as an important issue of feature subset selection problem, the objective function has a great effect on the selection of features. In this paper, we propose a criterion based on mutual information, and the criterion can not only measure the correlation between two features selected by a firefly but also determine the emendation of features among the achieved feature subset. The proposed approach is compared with differential evolution, genetic algorithm, and two versions of particle swarm optimization algorithm on several benchmark datasets. The results demonstrate that the proposed DbFAFS is efficient and competitive in both classification accuracy and computational performance.

Angle Modulated Artificial Bee Colony Algorithms for Feature Selection

Optimal feature subset selection is an important and a difficult task for pattern classification, data mining, and machine intelligence applications. The objective of the feature subset selection is to eliminate the irrelevant and noisy feature in order to select optimum feature subsets and increase accuracy. The large number of features in a dataset increases the computational complexity thus leading to performance degradation. In this paper, to overcome this problem, angle modulation technique is used to reduce feature subset selection problem to four-dimensional continuous optimization problem instead of presenting the problem as a high-dimensional bit vector. To present the effectiveness of the problem presentation with angle modulation and to determine the efficiency of the proposed method, six variants of Artificial Bee Colony (ABC) algorithms employ angle modulation for feature selection. Experimental results on six high-dimensional datasets show that Angle Modulated ABC algorithms improved the classification accuracy with fewer feature subsets.