Cost-sensitive Feature Selection Research Articles

Neighborhood multigranulation rough sets for cost-sensitive feature selection on hybrid data

Feature selection is a vital preprocessing step in real applications of data mining and machine learning. With the prevalence of high-dimensional hybrid data sets in real-world scenarios, along with the presence of test costs and misclassification costs, the need for effective feature selection methods has become more prominent. However, existing feature selection approaches mainly focus on cost-sensitive data from a single granularity perspective, and they are primarily applicable to single-typed data sets. To address these limitations, this paper presents a novel feature selection approach specifically designed for hybrid data, considering variable test costs and misclassification costs. The proposed method is based on neighborhood multigranulation rough sets (NMRS), which provides more comprehensive and multi-angle data analysis for cost-sensitive hybrid data. First, a novel multigranulation model is developed to effectively process cost-sensitive hybrid data. Building upon this model, a cost-based multi-criteria measure is proposed to evaluate the significance of features. This measure takes into account the comprehensive information of candidate features, including their power in algebraic view, information view, and associated costs. Furthermore, a heuristic feature selection algorithm based on NMRS is proposed to handle hybrid data with the test costs and misclassification costs. This algorithm leverages the benefits of the proposed multigranulation model and cost-based measure to identify the most discriminative features efficiently. Finally, the experimental results on twelve different datasets show that the proposed heuristic algorithm outperforms other compared algorithms, especially in total cost and classification accuracy.

Multi-Objective Gray Wolf Optimizer with Cost-Sensitive Feature Selection for Predicting Students’ Academic Performance in College English

Feature selection is a widely utilized technique in educational data mining that aims to simplify and reduce the computational burden associated with data analysis. However, previous studies have overlooked the high costs involved in acquiring certain types of educational data. In this study, we investigate the application of a multi-objective gray wolf optimizer (GWO) with cost-sensitive feature selection to predict students’ academic performance in college English, while minimizing both prediction error and feature cost. To improve the performance of the multi-objective binary GWO, a novel position update method and a selection mechanism for a, b, and d are proposed. Additionally, the adaptive mutation of Pareto optimal solutions improves convergence and avoids falling into local traps. The repairing technique of duplicate solutions expands population diversity and reduces feature cost. Experiments using UCI datasets demonstrate that the proposed algorithm outperforms existing state-of-the-art algorithms in hypervolume (HV), inverted generational distance (IGD), and Pareto optimal solutions. Finally, when predicting the academic performance of students in college English, the superiority of the proposed algorithm is again confirmed, as well as its acquisition of key features that impact cost-sensitive feature selection.

Search-based cost-sensitive hypergraph learning for anomaly detection

Detecting anomaly plays a vital role in ensuring the reliability and safety of many applications. However, this task is challenging due to the unclear correlation underneath the testing data and the class imbalanced issue between anomalous and normal data. Moreover, it is noted that the misclassification of anomalous and normal data is usually associated with different costs in real applications. To solve these issues, we propose a search-based cost-sensitive hypergraph learning method (SCSHL) for anomaly detection. More specifically, to deal with the imbalanced issue, we first construct a target-specific subset of training data according to the testing samples and conduct a cost-sensitive feature selection method to select the effective features. To explore the unclear high-order correlation underneath the data, we employ the hypergraph structure to formulate the complex relationship among the testing data and further combine the cost information into hypergraph learning and conduct label propagation on the cost-sensitive hypergraph structure. To evaluate the effectiveness of the proposed method, we have conducted experiments on industry anomaly detection datasets, software defect prediction (SDP) datasets and outlier detection datasets (ODDS). Experimental results and comparisons with state-of-the-art methods have shown the superiority of our method.

Multiple Cost Optimisation for Alzheimer's Disease Diagnosis.

Current machine learning techniques for dementia diagnosis often do not take into account real-world practical constraints, which may include, for example, the cost of diagnostic assessment time and financial budgets. In this work, we built on previous cost-sensitive feature selection approaches by generalising to multiple cost types, while taking into consideration that stakeholders attempting to optimise the dementia care pathway might face multiple non-fungible budget constraints. Our new optimisation algorithm involved the searching of cost-weighting hyperparameters while constrained by total budgets. We then provided a proof of concept using both assessment time cost and financial budget cost. We showed that budget constraints could control the feature selection process in an intuitive and practical manner, while adjusting the hyperparameter increased the range of solutions selected by feature selection. We further showed that our budget-constrained cost optimisation framework could be implemented in a user-friendly graphical user interface sandbox tool to encourage non-technical users and stakeholders to adopt and to further explore and audit the model - a humans-in-the-loop approach. Overall, we suggest that setting budget constraints initially and then fine tuning the cost-weighting hyperparameters can be an effective way to perform feature selection where multiple cost constraints exist, which will in turn lead to more realistic optimising and redesigning of dementia diagnostic assessments. Clinical Relevance-By optimising diagnostic accuracy against various costs (e.g. assessment administration time and financial budget) predictive yet practical dementia diagnostic assessments can be redesigned to suit clinical use.

Fuzzy filter cost-sensitive feature selection with differential evolution

In recent years, a variety of feature selection methods based on evolutionary computation (EC) techniques have been developed for classification due to their robustness and search ability. However, the previous EC-based feature selection research mostly focuses on enhancing the prediction accuracy without taking the costs associated with the learning process into consideration. In other words, the impact of using EC-based feature selection to improve the cost-sensitive classification performance has not been well studied. Further, it is not possible to find a EC-based filter cost-sensitive feature selection method in the literature, to the best of our knowledge. We therefore design a cost-sensitive evaluation criterion using the principles of fuzzy mutual estimator and then adopt the criterion in differential evolution framework. According to a variety of experiments conducted on various benchmarks, the proposed filter method can effectively minimize both the classification error rate and the feature cost by removing irrelevant and distracting features from the dataset in a reasonable time.

Alzheimer's Disease Assessments Optimized for Diagnostic Accuracy and Administration Time.

Objective: Despite the potential of machine learning techniques to improve dementia diagnostic processes, research outcomes are often not readily translated to or adopted in clinical practice. Importantly, the time taken to administer diagnostic assessment has yet to be taken into account in feature-selection based optimisation for dementia diagnosis. We address these issues by considering the impact of assessment time as a practical constraint for feature selection of cognitive and functional assessments in Alzheimer’s disease diagnosis. Methods: We use three different feature selection algorithms to select informative subsets of dementia assessment items from a large open-source dementia dataset. We use cost-sensitive feature selection to optimise our feature selection results for assessment time as well as diagnostic accuracy. To encourage clinical adoption and further evaluation of our proposed accuracy-vs-cost optimisation algorithms, we also implement a sandbox-like toolbox with graphical user interface to evaluate user-chosen subsets of assessment items. Results: We find that there are subsets of accuracy-cost optimised assessment items that can perform better in terms of diagnostic accuracy and/or total assessment time than most other standard assessments. Discussion: Overall, our analysis and accompanying sandbox tool can facilitate clinical users and other stakeholders to apply their own domain knowledge to analyse and decide which dementia diagnostic assessment items are useful, and aid the redesigning of dementia diagnostic assessments. Clinical Impact (Clinical Research): By optimising diagnostic accuracy and assessment time, we redesign predictive and efficient dementia diagnostic assessments and develop a sandbox interface to facilitate evaluation and testing by clinicians and non-specialists.

Integration of aggressive bound tightening and Mixed Integer Programming for Cost-sensitive feature selection in medical diagnosis

Silent diseases is an umbrella term that captures a spectrum of chronic illnesses that produce no clinically obvious signs and are diagnosed at advanced stages when the damage is irreversible. Current diagnostic strategies of silent diseases depend on self-reported symptoms and observed behavior through extended periods of time, and until now there are no specific clinical tests to diagnose silent diseases. Scientific research suggests the importance of early diagnosis to restore the functionality and reduce diseases-related complications. Previous studies primarily focused on feature selection methods to aid in medical diagnosis. Traditional feature selection methods are primarily focused on correct classification and often ignore features’ costs; the cost of clinical tests required to acquire the feature value. However, in medical diagnosis, features have different associated costs. Because ignoring features’ costs may result in a high cost diagnostic strategy that cannot be used in practice, developing a low-cost diagnostic strategy remains a subject of much interest. In this paper, new Mixed Integer Programming (MIP) models, namely, Cost-sensitive Support Vector Machine (CS-SVM) and Cost-sensitive Multi-surface Method Tree (CS-MSMT) that allow for simultaneous selection of low-cost and informative features are proposed. The CS-SVM and CS-MSMT are superior because they have the ability to account for shared costs. The CS-SVM and CS-MSMT were modified to embed shared costs across feature groups, and are termed Discounted CS-SVM (dCS-SVM) and Discounted CS-MSMT (dCS-MSMT), respectively. Computationally effective algorithm that integrates aggressive bound tightening with the MIP formulation is proposed. To demonstrate the effectiveness of the proposed models, different analysis paradigms are conducted on six UCI medical datasets; Chronic Kidney Disease, Hepatitis, Heart Disease, Thyroid, Diabetes and Leukemia. The results demonstrate the efficiency and robustness of the CS-SVM and CS-MSMT (and consequently the dCS-SVM and dCS-MSMT) under various conditions. The CS-SVM and CS-MSMT improved accuracy by 10.3% and 3.4% and reduced costs by 94.3% and 72.4% in the leukemia dataset, respectively.

A Robust Cost-Sensitive Feature Selection Via Self-Paced Learning Regularization

Feature selection is a useful and important process, which has a widely use in high-dimensional data processing and artificial intelligence. Its goal is to select a relatively small and representative subset of data from the original data space so that can obtain a better learning performance. Many existing feature selection algorithms simply pursue high accuracy and ignore the cost of feature acquisition and error classification. In this paper, a novel cost-sensitive feature selection method via self-paced learning is proposed. The $$\sigma $$ -norm is introduced to constrain the model and enhance its robustness. Then, we combine the self-paced learning framework. It can control the number of samples involved in training process, which can reduce the impact of noise points. The proposed method can obtain a better classification accuracy while maintaining the lowest total cost. It also has better interpretability and practicability than the traditional feature selection algorithm due to the consideration of the collection and misclassification cost among various features. Extensive experiments have been conducted on eight data sets with six comparison algorithms. The proposed method achieves a good performance.

Cost-sensitive feature selection on multi-label data via neighborhood granularity and label enhancement

Multi-label feature selection, which is an efficient and effective pre-processing step in machine learning and data mining, can select a feature subset that contains more contributions for multi-label classification while improving the performance of the classifiers. In real-world applications, an instance may be associated with multiple related labels with different relative importances, and the process of obtaining different features usually requires different costs, containing money, and time, etc. However, most existing works with regard to multi-label feature selection do not take into consideration the above two critical issues simultaneously. Therefore, in this paper, we exploit the idea of neighborhood granularity to enhance the traditional logical labels into label distribution forms to excavate the deeper supervised information hidden in multi-label data, and further consider the effect of the test cost under three different distributions, simultaneously. Motivated by these issues, a novel test cost multi-label feature selection algorithm with label enhancement and neighborhood granularity is designed. Moreover, the proposed algorithm is tested upon ten publicly available benchmark multi-label datasets with six widely-used metrics from two different aspects. Finally, two groups of experimental results demonstrate that the proposed algorithm achieves the satisfactory and superior performance over other four state-of-the-art comparing algorithms, and it is effective for improving the learning performance and decreasing the total test costs of the selected feature subset.

G-Forest: An ensemble method for cost-sensitive feature selection in gene expression microarrays.

Microarray gene expression profiling has emerged as an efficient technique for cancer diagnosis, prognosis, and treatment. One of the major drawbacks of gene expression microarrays is the "curse of dimensionality", which hinders the usefulness of information in datasets and leads to computational instability. In recent years, feature selection techniques have emerged as effective tools to identify disease biomarkers to aid in medical screening and diagnosis. However, the existing feature selection techniques, first, do not suit the rare variance exists in genomic data; and second, do not consider the feature cost (i.e. gene cost). Because ignoring features' costs may result in high cost gene profiling, this study proposes a new algorithm, called G-Forest, for cost-sensitive feature selection in gene expression microarrays. G-Forest is an ensemble cost-sensitive feature selection algorithm that develops a population of biases for a Random Forest induction algorithm. The G-Forest embeds the feature cost in the feature selection process and allows for simultaneous selection of low-cost and most informative features. In particular, when constructing the initial population, the feature is randomly selected with a probability inversely proportional to its associated cost. The G-Forest was compared with multiple state-of-the-art algorithms. Experimental results showed the effectiveness and robustness of the G-Forest in selecting the least cost and most informative genes. The G-Forest improved accuracy up to 14 % and decreased costs up to 56 % - on average - when compared with the other approaches tested in this article.

Using Cost-Sensitive Learning and Feature Selection Algorithms to Improve the Performance of Imbalanced Classification

Imbalanced data problem is widely present in network intrusion detection, spam filtering, biomedical engineering, finance, science, being a challenge in many real-life data-intensive applications. Classifier bias occurs when traditional classification algorithms are used to deal with imbalanced data. As already known, the General Vector Machine (GVM) algorithm has good generalization ability, though it does not work well for the imbalanced classification. Additionally, the state-of-the-art Binary Ant Lion Optimizer (BALO) algorithm has high exploitability and fast convergence rate. Based on these facts, we have proposed in this paper a Cost-sensitive Feature selection General Vector Machine (CFGVM) algorithm based on GVM and BALO algorithms to tackle the imbalanced classification problem, delivering different cost weights to different classes of samples. In our method, the BALO algorithm determines the cost weights and extract more significant features to improve the classification performance. Experiments conducted on eleven imbalanced data sets have shown that the CFGVM algorithm significantly improves the classification performance of minority class samples. By comparing with similar algorithms and state-of-the-art algorithms, the proposed algorithm significantly outperforms in performance and produces better classification results.

Spam detection on social networks using cost-sensitive feature selection and ensemble-based regularized deep neural networks

Spam detection on social networks is increasingly important owing to the rapid growth of social network user base. Sophisticated spam filters must be developed to deal with this complex problem. Traditional machine learning approaches such as neural networks, support vector machines and Naïve Bayes classifiers are not effective enough to process and utilize complex features present in high-dimensional data on social network spam. Moreover, the traditional objective criteria of social network spam filters cannot cope with different costs assigned to type I and type II errors. To overcome these problems, here we propose a novel cost-sensitive approach to social network spam filtering. The proposed approach is composed of two stages. In the first stage, multi-objective evolutionary feature selection is used to minimize both the misclassification cost of the proposed model and the number of attributes necessary for spam filtering. Then, the approach uses cost-sensitive ensemble learning techniques with regularized deep neural networks as base learners. We demonstrate that this approach is effective for social network spam filtering on two benchmark datasets. We also show that the proposed approach outperforms other popular algorithms used in social network spam filtering, such as random forest, Naïve Bayes or support vector machines.

Cost-sensitive feature selection using two-archive multi-objective artificial bee colony algorithm

Since different features may require different costs, the cost-sensitive feature selection problem become more and more important in real-world applications. Generally, it includes two main conflicting objectives, i.e., maximizing the classification performance and minimizing the feature cost. However, most existing approaches treat this task as a single-objective optimization problem. To satisfy various requirements of decision-makers, this paper studies a multi-objective feature selection approach, called two-archive multi-objective artificial bee colony algorithm (TMABC-FS). Two new operators, i.e., convergence-guiding search for employed bees and diversity-guiding search for onlooker bees, are proposed for obtaining a group of non-dominated feature subsets with good distribution and convergence. And two archives, i.e., the leader archive and the external archive are employed to enhance the search capability of different kinds of bees. The proposed TMABC-FS is validated on several datasets from UCI, and is compared with two traditional algorithms and three multi-objective methods. Results have shown that TMABC-FS is an efficient and robust optimization method for solving cost-sensitive feature selection problems.

Feature–granularity selection with variable costs for hybrid data

In recent years, cost-sensitive feature selection has drawn much attention. However, some issues still remain to be investigated. Particularly, most existing work deals with single-typed data, while only a few studies deal with hybrid data; moreover, both the test cost of a feature and the misclassification cost of an object are often assumed to be fixed, but in fact they are usually variable with the error range of the data, or equivalently the data granularity. In view of these facts, a feature–granularity selection approach is proposed to select the optimal feature subset and the optimal data granularity simultaneously to minimize the total cost for processing hybrid data. In the approach, firstly an adaptive neighborhood model is constructed, in which the neighborhood granules are generated adaptively according to the types of features. Then, multiple kinds of variable cost setting are discussed according to reality, and finally, an optimal feature–granularity selection algorithm is designed. Experimental results on sixteen UCI datasets show that a good trade-off among feature dimension reduction, data granularity selection and total cost minimization could be achieved by the proposed algorithm. In particular, the influences of different cost settings to the feature–granularity selection are also discussed thoroughly in the paper, which would provide some feasible schemes for decision making.

Cost-sensitive feature selection via the ℓ2,1-norm

An essential step in data mining and machine learning is selecting a useful feature subset from the high-dimensional feature space. Many existing feature selection algorithms only consider precision, but do not consider error types and test cost. In this paper, we use the ℓ2,1-norm to propose a cost-sensitive embedded feature selection algorithm that minimizes the total cost rather than maximizing accuracy. The algorithm is a cost-sensitive feature selection algorithm with joint ℓ2,1-norm minimization of the loss function with misclassification costs. The ℓ2,1-norm based loss function with misclassification costs is robust to outliers. We also add an orthogonal constraint term to guarantee that each selected feature is independent. The proposed algorithm simultaneously takes into account both test costs and misclassification costs. Finally, an iterative updating algorithm is provided using the objective function that makes cost-sensitive feature selection more efficient. The cost-sensitive feature selection algorithm is more realistic than existing feature selection algorithms. Extensive experimental results on publicly available datasets demonstrate that the proposed algorithm is effective, can select a low-cost subset and achieve better performance than other feature selection algorithms in real-world applications.

Rough sets and Laplacian score based cost-sensitive feature selection

Cost-sensitive feature selection learning is an important preprocessing step in machine learning and data mining. Recently, most existing cost-sensitive feature selection algorithms are heuristic algorithms, which evaluate the importance of each feature individually and select features one by one. Obviously, these algorithms do not consider the relationship among features. In this paper, we propose a new algorithm for minimal cost feature selection called the rough sets and Laplacian score based cost-sensitive feature selection. The importance of each feature is evaluated by both rough sets and Laplacian score. Compared with heuristic algorithms, the proposed algorithm takes into consideration the relationship among features with locality preservation of Laplacian score. We select a feature subset with maximal feature importance and minimal cost when cost is undertaken in parallel, where the cost is given by three different distributions to simulate different applications. Different from existing cost-sensitive feature selection algorithms, our algorithm simultaneously selects out a predetermined number of “good” features. Extensive experimental results show that the approach is efficient and able to effectively obtain the minimum cost subset. In addition, the results of our method are more promising than the results of other cost-sensitive feature selection algorithms.

Multi-granularity feature selection on cost-sensitive data with measurement errors and variable costs

In real applications of data mining, machine learning and granular computing, measurement errors, test costs and misclassification costs often occur. Furthermore, the test cost of a feature is usually variable with the error range, and the variability of the misclassification cost is related to the object considered. Recently, some approaches based on rough sets have been introduced to study the error-based cost-sensitive feature selection problem. However, most of them consider only single-granularity cases, thus are not feasible for the case where the granularity diversity between different features should be taken into account. Motivated by this problem, we propose a multi-granularity feature selection approach which considers measurement errors and variable costs in terms of feature-value granularities. For a given feature, the feature-value granularity is evaluated by the error confidence level of the feature values. In this way, we build a theoretic framework called confidence-level-vector-based neighborhood rough set, and present a so-called heuristic feature-granularity selection algorithm, and a relevant competition strategy which can select both features and their respective feature-value granularities effectively and efficiently. Experiment results show that a satisfactory trade-off among feature dimension reduction, feature-value granularity selection and total cost minimization can be achieved by the proposed approach. This work would provide a new insight into the cost-sensitive feature selection problem from the multi-granularity perspective.

Cost-sensitive Feature Selection for Support Vector Machines

Feature Selection is a crucial procedure in Data Science tasks such as Classification, since it identifies the relevant variables, making thus the classification procedures more interpretable, cheaper in terms of measurement and more effective by reducing noise and data overfit. The relevance of features in a classification procedure is linked to the fact that misclassifications costs are frequently asymmetric, since false positive and false negative cases may have very different consequences. However, off-the-shelf Feature Selection procedures seldom take into account such cost-sensitivity of errors.In this paper we propose a mathematical-optimization-based Feature Selection procedure embedded in one of the most popular classification procedures, namely, Support Vector Machines, accommodating asymmetric misclassification costs. The key idea is to replace the traditional margin maximization by minimizing the number of features selected, but imposing upper bounds on the false positive and negative rates. The problem is written as an integer linear problem plus a quadratic convex problem for Support Vector Machines with both linear and radial kernels.The reported numerical experience demonstrates the usefulness of the proposed Feature Selection procedure. Indeed, our results on benchmark data sets show that a substantial decrease of the number of features is obtained, whilst the desired trade-off between false positive and false negative rates is achieved.

Cost-Sensitive Feature Selection by Optimizing F-Measures.

Feature selection is beneficial for improving the performance of general machine learning tasks by extracting an informative subset from the high-dimensional features. Conventional feature selection methods usually ignore the class imbalance problem, thus the selected features will be biased towards the majority class. Considering that F-measure is a more reasonable performance measure than accuracy for imbalanced data, this paper presents an effective feature selection algorithm that explores the class imbalance issue by optimizing F-measures. Since F-measure optimization can be decomposed into a series of cost-sensitive classification problems, we investigate the cost-sensitive feature selection by generating and assigning different costs to each class with rigorous theory guidance. After solving a series of cost-sensitive feature selection problems, features corresponding to the best F-measure will be selected. In this way, the selected features will fully represent the properties of all classes. Experimental results on popular benchmarks and challenging real-world data sets demonstrate the significance of cost-sensitive feature selection for the imbalanced data setting and validate the effectiveness of the proposed method.

Cost-Sensitive Feature Selection via F-Measure Optimization Reduction

Feature selection aims to select a small subset from the high-dimensional features which can lead to better learning performance, lower computational complexity, and better model readability. The class imbalance problem has been neglected by traditional feature selection methods, therefore the selected features will be biased towards the majority classes. Because of the superiority of F-measure to accuracy for imbalanced data, we propose to use F-measure as the performance measure for feature selection algorithms. As a pseudo-linear function, the optimization of F-measure can be achieved by minimizing the total costs. In this paper, we present a novel cost-sensitive feature selection (CSFS) method which optimizes F-measure instead of accuracy to take class imbalance issue into account. The features will be selected according to optimal F-measure classifier after solving a series of cost-sensitive feature selection sub-problems. The features selected by our method will fully represent the characteristics of not only majority classes, but also minority classes. Extensive experimental results conducted on synthetic, multi-class and multi-label datasets validate the efficiency and significance of our feature selection method.