Wrapper Approach Research Articles

Investigating machine learning techniques for MRI-based classification of brain neoplasms

Diagnosis and characterization of brain neoplasms appears of utmost importance for therapeutic management. The emerging of imaging techniques, such as Magnetic Resonance (MR) imaging, gives insight into pathology, while the combination of several sequences from conventional and advanced protocols (such as perfusion imaging) increases the diagnostic information. To optimally combine the multiple sources and summarize the information into a distinctive set of variables however remains difficult. The purpose of this study is to investigate machine learning algorithms that automatically identify the relevant attributes and are optimal for brain tumor differentiation. Different machine learning techniques are studied for brain tumor classification based on attributes extracted from conventional and perfusion MRI. The attributes, calculated from neoplastic, necrotic, and edematous regions of interest, include shape and intensity characteristics. Attributes subset selection is performed aiming to remove redundant attributes using two filtering methods and a wrapper approach, in combination with three different search algorithms (Best First, Greedy Stepwise and Scatter). The classification frameworks are implemented using the WEKA software. The highest average classification accuracy assessed by leave-one-out (LOO) cross-validation on 101 brain neoplasms was achieved using the wrapper evaluator in combination with the Best First search algorithm and the KNN classifier and reached 96.9% when discriminating metastases from gliomas and 94.5% when discriminating high-grade from low-grade neoplasms. A computer-assisted classification framework is developed and used for differential diagnosis of brain neoplasms based on MRI. The framework can achieve higher accuracy than most reported studies using MRI.

Predicting business failure using support vector machines with straightforward wrapper: A re-sampling study

Business failure prediction (BFP) is an effective tool to help financial institutions and relevant people to make the right decision in investments, especially in the current competitive environment. This topic belongs to a classification-type task, one of whose aims is to generate more accurate hit ratios. Support vector machine (SVM) is a statistical learning technique, whose advantage is its high generalization performance. The objective of this context is threefold. Firstly, SVM is used to predict business failure by utilizing a straightforward wrapper approach to help the model produce more accurate prediction. The wrapper approach is fulfilled by employing a forward feature selection method, composed of feature ranking and feature selection. Meanwhile, this work attempts to investigate the feasibility of using linear SVMs to select features for all SVMs in the wrapper since non-linear SVMs yield to over-fit the data. Finally, a robust re-sampling approach is used to evaluate model performances for the task of BFP in China. In the empirical research, performances of linear SVM, polynomial SVM, Gaussian SVM, and sigmoid SVM with the best filter of stepwise MDA, and wrappers respectively using linear SVM and non-linear SVMs as evaluating functions are to be compared. The results indicate that the non-linear SVM with radial basis function kernel and features selected by linear SVM compare significantly superiorly to all the other SVMs. Meanwhile, all SVMs with features selected by linear SVM produce at least as good performances as SVMs with other optimal features.

L p -Norm Multiple Kernel Learning

Learning linear combinations of multiple kernels is an appealing strategy when the right choice of features is unknown. Previous approaches to multiple kernel learning (MKL) promote sparse kernel combinations to support interpretability and scalability. Unfortunately, this l1-norm MKL is rarely observed to outperform trivial baselines in practical applications. To allow for robust kernel mixtures that generalize well, we extend MKL to arbitrary norms. We devise new insights on the connection between several existing MKL formulations and develop two efficient interleaved optimization strategies for arbitrary norms, that is lp-norms with p ≥ 1. This interleaved optimization is much faster than the commonly used wrapper approaches, as demonstrated on several data sets. A theoretical analysis and an experiment on controlled artificial data shed light on the appropriateness of sparse, non-sparse and l∞-norm MKL in various scenarios. Importantly, empirical applications of lp-norm MKL to three real-world problems from computational biology show that non-sparse MKL achieves accuracies that surpass the state-of-the-art. Data sets, source code to reproduce the experiments, implementations of the algorithms, and further information are available at http://doc.ml.tu-berlin.de/nonsparse_mkl/.

Discrimination between Upstairs and Downstairs Based on Accelerometer

An algorithm for the discrimination between human upstairs and downstairs using a tri-axial accelerometer is presented in this paper, which consists of vertical acceleration calibration, extraction of two kinds of features (Interquartile Range and Wavelet Energy), effective feature subset selection with the wrapper approach, and SVM classification. The proposed algorithm can recognize upstairs and downstairs with 95.64% average accuracy for different sensor locations, i.e. located on the subject's waist belt, in the trousers pocket, and in the shirt pocket. Even for the mixed data from all sensor locations, the average recognition accuracy can reach 94.84%. Experimental results have successfully validated the effectiveness of the proposed method.

Applying electromagnetism-like mechanism for feature selection

In recent years, methods of feature selection have been increasingly emphasized as venues for reducing cost and shortening the length of time required for computation in data mining. This study utilizes electromagnetism-like mechanism as a wrapper approach to feature selection. Birbil and Fang proposed EM in 2003. EM uses the attraction–repulsion mechanism of the electromagnetism theory to ascertain the optimal solution. Although EM has been applied to the topic of optimization in continuous space and a small number of studies on discrete problems, it has not been applied to the subject of feature selection. In this study, EM combined with 1-nearest-neighbor (1NN) was applied for feature selection and classification. This study utilized the total force exerted on a particle and evaluated this force to determine which features are to be selected. The most crucial features were selected according to the proposed method based on the minimum miss-classification rate, which was attained through 1NN. An unknown datum was classified by 1NN based on the chosen reduced model. To estimate the effectiveness of the proposed method, a numerical experiment was conducted using several data sets with diverse sizes, features, separability, and classes. Experimental results indicated that the proposed method outperformed other well-known algorithms in not only balanced classification accuracy but also efficiency of feature selection. Lastly, this study used an actual case concerning gestational diabetes mellitus to demonstrate the workability of the proposed method.

Predicting business failure using forward ranking-order case-based reasoning

With the rapid development of business computing for Chinese listed companies, it is focused on to use case-based reasoning (CBR) in business failure prediction (BFP). Ranking-order case-based reasoning (RCBR) uses ranking-order information among cases to calculate similarity in the framework of k-nearest neighbor. RCBR is sensitive to the choice of features, meaning that optimal features can help it produce better performance. In this research, we attempt to use wrapper approach to find the optimal feature subset for RCBR in BFP. Forward feature selection method and RCBR are combined to construct a new method, namely forward RCBR (FRCBR). The combination is implemented by combining forward feature selection with RCBR as a wrapper module. Hold out method is used to assessing the performance of the classifier. Empirical data were collected from Chinese listed companies in the Shenzhen Stock Exchange and Shanghai Stock Exchange. We employed the standalone RCBR, the classical CBR with Euclidean metric as its heart, the inductive CBR, the two statistical methods of logistic regression and multivariate discriminate analysis (MDA), and support vector machines to make comparisons. For comparative methods, stepwise MDA was employed to select optimal feature subset. Empirical results indicated that FRCBR can produce dominating performance in short-term BFP of Chinese listed companies.

A Hybrid Attribute Selection Approach for Text Classification

The application of text mining in organizations is growing. Text classification, an important type of text mining problem, is characterized by a large attribute space and entails an efficient and effective attribute selection procedure. There are two general attribute selection approaches: the filter approach and the wrapper approach. While the wrapper approach is potentially more effective in finding the best attribute subset, it is cost-prohibitive in most text classification applications. In this paper, we propose a hybrid attribute selection approach that is both efficient and effective for text classification problems. We apply the proposed approach to detect and prevent Internet abuse in the workplace, which is becoming a major problem in modern organizations. The empirical evaluations we conducted using a variety of classification algorithms, indexing schemes, and attribute selection methods demonstrate the utility of the proposed approach. We found that combining the filter and wrapper approaches not only boosts the accuracies of text classifiers but also brings down the computational costs significantly.

새로운 얼굴 특징공간을 이용한 모델 기반 얼굴 표정 인식

본 연구에서는 얼굴 그리드 각도를 특징공간으로 하는 새로운 모델 기반 얼굴 표정 인식 방법을 제안한다. 제안 방식은 6가지 얼굴 대표 표정을 인식하기 위해 표정 그리드를 이용하여 그리드의 각 간선과 정점이 형성하는 각도를 기반으로 얼굴 특징 공간을 구성한다. 이 방법은 다른 표정 인식 알고리즘의 정확도를 낮추는 원인인 변환, 회전, 크기변화와 같은 어파인 변환에 강건한 특징을 보인다. 또한, 본 연구에서는 각도로 특징공간을 구성하고 이 공간 내에서 Wrapper 방식으로 특징 부분집합을 선택하는 과정을 설명한다. 선택한 특징들은 SVM, 3-NN 분류기를 이용해 분류하고 분류 결과는 2중 교차검증을 통해 검증하도록 한다. 본 연구가 제안한 방법에서는 94%의 표정 인식 결과를 보였으며 특히 특징 부분집합 선택 알고리즘을 적용한 결과 전체 특징을 이용한 경우보다 약 10%의 인식율 개선 효과를 보인다. This paper introduces a new model based method for facial expression recognition that uses facial grid angles as feature space. In order to be able to recognize the six main facial expression, proposed method uses a grid approach and therefore it establishes a new feature space based on the angles that each gird's edge and vertex form. The way taken in the paper is robust against several affine transformations such as translation, rotation, and scaling which in other approaches are considered very harmful in the overall accuracy of a facial expression recognition algorithm. Also, this paper demonstrates the process that the feature space is created using angles and how a selection process of feature subset within this space is applied with Wrapper approach. Selected features are classified by SVM, 3-NN classifier and classification results are validated with two-tier cross validation. Proposed method shows 94% classification result and feature selection algorithm improves results by up to 10% over the full set of feature.

Prediction of intrinsic solubility of generic drugs using MLR, ANN and SVM analyses

The machine learning methods artificial neural network (ANN) and support vector machine (SVM) techniques were used to model intrinsic solubility of 74 generic drugs. The models obtained were compared with those obtained using multiple linear regression (MLR) analysis. Cluster analysis was used to split the data into a training set and test set. The appropriate descriptors were selected using a wrapper approach with multiple linear regressions as target learning algorithm. The descriptor selection and model building were performed with 10 fold cross validation using the training data set. The linear model fits the training set ( n = 60) with R 2 = 0.814, while ANN and SVM higher values of R 2 = 0.823 and 0.835, respectively. Though the SVM model shows improvement of training set fitting, the ANN model was slightly superior to SVM and MLR in predicting the test set. The quantitative structure–property relationship study suggests that the theoretically calculated descriptors log P, first-order valence connectivity index ( 1χ v), delta chi (Δ 2 χ ) and information content ( 2IC) have relevant relationships with intrinsic solubility of generic drugs studied.

Selecting the Most Relevant Variables for Anaerobic Digestion Imbalances: Two Case Studies

In this study, a wrapper approach was applied to objectively select the most important variables related to two different anaerobic digestion imbalances, acidogenic states and foaming. This feature selection method, implemented in artificial neural networks (ANN), was performed using input and output data from a fully instrumented pilot plant (1 m3 upflow fixed bed digester). Results for acidogenic states showed that pH, volatile fatty acids, and inflow rate were the most relevant variables. Results for foaming showed that inflow rate and total organic carbon were among the relevant variables, both of which were related to the feed loading of the digester. Because there is not a complete agreement on the causes of foaming, these results highlight the role of digester feeding patterns in the development of foaming.

A new wrapper feature selection approach using neural network

This paper presents a new feature selection (FS) algorithm based on the wrapper approach using neural networks (NNs). The vital aspect of this algorithm is the automatic determination of NN architectures during the FS process. Our algorithm uses a constructive approach involving correlation information in selecting features and determining NN architectures. We call this algorithm as constructive approach for FS (CAFS). The aim of using correlation information in CAFS is to encourage the search strategy for selecting less correlated (distinct) features if they enhance accuracy of NNs. Such an encouragement will reduce redundancy of information resulting in compact NN architectures. We evaluate the performance of CAFS on eight benchmark classification problems. The experimental results show the essence of CAFS in selecting features with compact NN architectures.

Simulated annealing for supervised gene selection

Genomic data, and more generally biomedical data, are often characterized by high dimensionality. An input selection procedure can attain the two objectives of highlighting the relevant variables (genes) and possibly improving classification results. In this paper, we propose a wrapper approach to gene selection in classification of gene expression data using simulated annealing along with supervised classification. The proposed approach can perform global combinatorial searches through the space of all possible input subsets, can handle cases with numerical, categorical or mixed inputs, and is able to find (sub-)optimal subsets of inputs giving low classification errors. The method has been tested on publicly available bioinformatics data sets using support vector machines and on a mixed type data set using classification trees. We also propose some heuristics able to speed up the convergence. The experimental results highlight the ability of the method to select minimal sets of relevant features.

Feature selection for time series prediction – A combined filter and wrapper approach for neural networks

Modelling artificial neural networks for accurate time series prediction poses multiple challenges, in particular specifying the network architecture in accordance with the underlying structure of the time series. The data generating processes may exhibit a variety of stochastic or deterministic time series patterns of single or multiple seasonality, trends and cycles, overlaid with pulses, level shifts and structural breaks, all depending on the discrete time frequency in which it is observed. For heterogeneous datasets of time series, such as the 2008 ESTSP competition, a universal methodology is required for automatic network specification across varying data patterns and time frequencies. We propose a fully data driven forecasting methodology that combines filter and wrapper approaches for feature selection, including automatic feature evaluation, construction and transformation. The methodology identifies time series patterns, creates and transforms explanatory variables and specifies multilayer perceptrons for heterogeneous sets of time series without expert intervention. Examples of the valid and reliable performance in comparison to established benchmark methods are shown for a set of synthetic time series and for the ESTSP’08 competition dataset, where the proposed methodology obtained second place.

Feature Subset Selection Problem using Wrapper Approach in Supervised Learning

Feature subset selection is of immense importance in the field of data mining. The increased dimensionality of data makes testing and training of general classification method difficult. Mining on the reduced set of attributes reduces computation time and also helps to make the patterns easier to understand. In this paper a wrapper approach for feature selection is proposed. As a part of feature selection step we used wrapper approach with Genetic algorithm as random search technique for subset generation ,wrapped with different classifiers/ induction algorithm namely decision tree C4.5, NaiveBayes, Bayes networks and Radial basis function as subset evaluating mechanism on four standard datasets namely Pima Indians Diabetes Dataset, Breast Cancer, Heart Stat log and Wisconsin Breast Cancer. Further the relevant attributes identified by proposed wrapper are validated using classifiers. Experimental results illustrate, employing feature subset selection using proposed wrapper approach has enhanced classification accuracy.

On sensitivity of case-based reasoning to optimal feature subsets in business failure prediction

Case-based reasoning (CBR) was firstly introduced into the area of business failure prediction (BFP) in 1996. The conclusion drawn out in its first application in this area is that CBR is not more applicable than multiple discriminant analysis (MDA) and Logit. On the contrary, there are some arguments which claim that CBR with k-nearest neighbor ( k-NN) as its heart is not surely outranked by those machine learning techniques. In this research, we attempt to investigate whether or not CBR is sensitive to the so-called optimal feature subsets in BFP, since feature subset is an important factor that accounts for CBR’s performance. When CBR is used to solve such classification problem, the retrieval process of its life-cycle is mainly used. We use the classical Euclidean metric technique to calculate case similarity. Empirical data two years prior to failure are collected from Shanghai Stock Exchange and Shenzhen Stock Exchange in China. Four filters, i.e. MDA stepwise method, Logit stepwise method, One-way ANOVA, independent-samples t-test, and the wrapper approach of genetic algorithm are employed to generate five optimal feature subsets after data normalization. Thirty-times hold-out method is used as assessment of predictive performances by combining leave-one-out cross-validation and hold-out method. The two statistical baseline models, i.e. MDA and Logit, and the new model of support vector machine are employed as comparative models. Empirical results indicate that CBR is truly sensitive to optimal feature subsets with data for medium-term BFP. The stepwise method of MDA, a filter approach, is the first choice for CBR to select optimal feature subsets, followed by the stepwise method of Logit and the wrapper. The two filter approaches of ANOVA and t-test are the fourth choice. If MDA stepwise method is employed to select optimal feature subset for the CBR system, there are no significant difference on predictive performance of medium-term BFP between CBR and the other three models, i.e. MDA, Logit, SVM. On the contrary, CBR is outperformed by the three models at the significant level of 1%, if ANOVA or t-test is used as feature selection method for CBR.

Feature extraction and selection from acoustic emission signals with an application in grinding wheel condition monitoring

Feature extraction and feature selection are two important issues in sensor-based condition monitoring of any engineering systems. In this study, acoustic emission signals were first collected during grinding operations, next processed by autoregressive modeling or discrete wavelet decomposition for feature extraction, and then the best feature subsets are found by three different feature selection methods, including two proposed ant colony optimization (ACO)-based method and the famous sequential forward floating selection method. Posing monitoring as a classification problem, the evaluation is carried out by the wrapper approach with four different algorithms serving as the classifier. Empirical test results were shown to illustrate the effectiveness of feature extraction and feature selection methods.

A New Wrapper Feature Selection Approach using Neural Network

This paper presents a new feature selection (FS) algorithm based on the wrapper approach using neural networks (NNs). The vital aspect of this algorithm is the automatic determination of NN architectures during the FS process. Our algorithm uses a constructive approach involving correlation information in selecting features and determining NN architectures. We call this algorithm as constructive approach for FS (CAFS). The aim of using correlation information in CAFS is to encourage the search strategy for selecting less correlated (distinct) features if they enhance accuracy of NNs. Such an encouragement will reduce redundancy of information resulting in compact NN architectures. We evaluate the performance of CAFS on eight benchmark classification problems. The experimental results show the essence of CAFS in selecting features with compact NN architectures.

An Evolutionary Method for Combining Different Feature Selection Criteria in Microarray Data Classification

The classification of cancers from gene expression profiles is a challenging research area in bioinformatics since the high dimensionality of microarray data results in irrelevant and redundant information that affects the performance of classification. This paper proposes using an evolutionary algorithm to select relevant gene subsets in order to further use them for the classification task. This is achieved by combining valuable results from different feature ranking methods into feature pools whose dimensionality is reduced by a wrapper approach involving a genetic algorithm and SVM classifier. Specifically, the GA explores the space defined by each feature pool looking for solutions that balance the size of the feature subsets and their classification accuracy. Experiments demonstrate that the proposed method provide good results in comparison to different state of art methods for the classification of microarray data.

Feature Selection Using Stochastic Search: An Application to System Identification

System identification using multiple-model strategies may involve thousands of models with several parameters. However, only a few models are close to the correct model. A key task involves finding which parameters are important for explaining candidate models. The application of feature selection to system identification is studied in this paper. A new feature selection algorithm is proposed. It is based on the wrapper approach and combines two algorithms. The search is performed using stochastic sampling and the classification uses a support vector machine strategy. This approach is found to be better than genetic algorithm-based strategies for feature selection on several benchmark data sets. Applied to system identification, the algorithm supports subsequent decision making.

Building lightweight intrusion detection system using wrapper-based feature selection mechanisms

Intrusion Detection System (IDS) is an important and necessary component in ensuring network security and protecting network resources and network infrastructures. How to build a lightweight IDS is a hot topic in network security. Moreover, feature selection is a classic research topic in data mining and it has attracted much interest from researchers in many fields such as network security, pattern recognition and data mining. In this paper, we effectively introduced feature selection methods to intrusion detection domain. We propose a wrapper-based feature selection algorithm aiming at building lightweight intrusion detection system by using modified random mutation hill climbing (RMHC) as search strategy to specify a candidate subset for evaluation, as well as using modified linear Support Vector Machines (SVMs) iterative procedure as wrapper approach to obtain the optimum feature subset. We verify the effectiveness and the feasibility of our feature selection algorithm by several experiments on KDD Cup 1999 intrusion detection dataset. The experimental results strongly show that our approach is not only able to speed up the process of selecting important features but also to yield high detection rates. Furthermore, our experimental results indicate that intrusion detection system with feature selection algorithm has better performance than that without feature selection algorithm both in detection performance and computational cost.