Non-informative Features Research Articles

Using Feature-Based Models with Complexity Penalization for Selecting Features

Feature selection and inference through modeling are combined into one method based on a network that can be used to point out irrelevant, redundant and dependent features in the data. It is shown that this network method is efficient in terms of reducing the number of calculations for estimating the probabilities under different model assumptions by breaking the data into fractions. We prove that the probability estimations within the network method lead to the detection of non-informative features with probability one if the data is sufficiently large. The proposed method’s accuracy in detecting complex relations between features, selecting informative features and classifying data-sets with different dimensions is assessed through experiments using both synthetic and real data. The results from the network method compare favorably with those from the well-known and powerful feature selection algorithms. It is further shown that the network method can handle complex relations between the features that are intractable for other algorithms.

Ensemble of a subset of kNN classifiers.

Combining multiple classifiers, known as ensemble methods, can give substantial improvement in prediction performance of learning algorithms especially in the presence of non-informative features in the data sets. We propose an ensemble of subset of kNN classifiers, ESkNN, for classification task in two steps. Firstly, we choose classifiers based upon their individual performance using the out-of-sample accuracy. The selected classifiers are then combined sequentially starting from the best model and assessed for collective performance on a validation data set. We use bench mark data sets with their original and some added non-informative features for the evaluation of our method. The results are compared with usual kNN, bagged kNN, random kNN, multiple feature subset method, random forest and support vector machines. Our experimental comparisons on benchmark classification problems and simulated data sets reveal that the proposed ensemble gives better classification performance than the usual kNN and its ensembles, and performs comparable to random forest and support vector machines.

The impact of configural superiority on the processing of spatial information.

The impact of context on perception has been well documented for over a century. In some cases, the introduction of context to a set of target features may produce a unified percept, leading to a quicker and more accurate classification; a configural superiority effect (Pomerantz, Sager, & Stoever, 1977). Although this effect has been well characterized in terms of the stimulus features that produce the effect, the specific impact context has on the spatial strategies adopted by observers when making perceptual judgments remains unclear. Here, we sought to address this question by using the methods of response classification and ideal observer analysis. In our main experiment, we used a stimulus set known to produce the configural superiority effect and found that although observers were faster in the presence of context, they were actually less efficient at extracting stimulus information. This surprising result was attributable to the use of a spatial strategy in which observers relied on redundant, noninformative features in the presence of context. A control experiment ruled out the possibility that the mere presence of added context led to these strategic shifts. Our results support previous notions about the nature of the perceptual shifts that are induced by the configural superiority effect. However, they also show that configural processing is more nuanced than originally thought: Although observers may be faster at making judgments when context induces the percept of a configural whole, there appears to be a hidden cost in terms of the efficiency with which information is used. (PsycINFO Database Record

Improving Classification Performance with PCA-CFS and PCA-SA Feature Selector

Data mining techniques are increasingly being used in Health care analytics to understand the probability of predicted outcomes. High dimensional data the identification of most relevant features that will best determine the target is a challenging task. Feature selection is aims to remove non-informative features from data mining models. This work is focused on improving the detection performance of classification algorithms by the use of Principal Component Analysis (PCA) based feature extraction and Correlation-based Feature Selection (CFS) and Shapley Values Analysis (SA) based feature ranking approach. The performance of projected hybrid method was evaluated through experiments on six medical datasets with various classifiers. The examined result projected the proposed hybrid feature selectors using the PCA-CFS and PCA-SA improves the detection performance of individual classification algorithms over the other methods in the literature.

Dominant Rotated Local Binary Patterns (DRLBP) for texture classification

In this paper, we present a novel rotation-invariant and computationally efficient texture descriptor called Dominant Rotated Local Binary Pattern (DRLBP). A rotation invariance is achieved by computing the descriptor with respect to a reference in a local neighborhood. A reference is fast to compute maintaining the computational simplicity of the Local Binary Patterns (LBP). The proposed approach not only retains the complete structural information extracted by LBP, but it also captures the complimentary information by utilizing the magnitude information, thereby achieving more discriminative power. For feature selection, we learn a dictionary of the most frequently occurring patterns from the training images, and discard redundant and non-informative features. To evaluate the performance we conduct experiments on three standard texture datasets: Outex12, Outex 10 and KTH-TIPS. The performance is compared with the state-of-the-art rotation invariant texture descriptors and results show that the proposed method is superior to other approaches.

A Study on Multi-Scale Kernel Optimisation via Centered Kernel-Target Alignment

Kernel mapping is one of the most widespread approaches to intrinsically deriving nonlinear classifiers. With the aim of better suiting a given dataset, different kernels have been proposed and different bounds and methodologies have been studied to optimise them. We focus on the optimisation of a multi-scale kernel, where a different width is chosen for each feature. This idea has been barely studied in the literature, although it has been shown to achieve better performance in the presence of heterogeneous attributes. The large number of parameters in multi-scale kernels makes it computationally unaffordable to optimise them by applying traditional cross-validation. Instead, an analytical measure known as centered kernel-target alignment (CKTA) can be used to align the kernel to the so-called ideal kernel matrix. This paper analyses and compares this and other alternatives, providing a review of the literature in kernel optimisation and some insights into the usefulness of multi-scale kernel optimisation via CKTA. When applied to the binary support vector machine paradigm (SVM), the results using 24 datasets show that CKTA with a multi-scale kernel leads to the construction of a well-defined feature space and simpler SVM models, provides an implicit filtering of non-informative features and achieves robust and comparable performance to other methods even when using random initialisations. Finally, we derive some considerations about when a multi-scale approach could be, in general, useful and propose a distance-based initialisation technique for the gradient-ascent method, which shows promising results.

Financial fraud detection using vocal, linguistic and financial cues

Corporate financial fraud has a severe negative impact on investors and the capital market in general. The current resources committed to financial fraud detection (FFD), however, are insufficient to identify all occurrences in a timely fashion. Methods for automating FFD have mainly relied on financial statistics, although some recent research has suggested that linguistic or vocal cues may also be useful indicators of deception. Tools based on financial numbers, linguistic behavior, and non-verbal vocal cues have each demonstrated the potential for detecting financial fraud. However, the performance of these tools continues to be poorer than desired, limiting their use on a stand-alone basis to help identify companies for further investigation. The hypothesis investigated in this study is that an improved tool could be developed if specific attributes from these feature categories were analyzed concurrently. Combining features across categories provided better fraud detection than was achieved by any of the feature categories alone. However, performance improvements were only observed if feature selection was used suggesting that it is important to discard non-informative features.

INFUSE: Interactive Feature Selection for Predictive Modeling of High Dimensional Data.

Predictive modeling techniques are increasingly being used by data scientists to understand the probability of predicted outcomes. However, for data that is high-dimensional, a critical step in predictive modeling is determining which features should be included in the models. Feature selection algorithms are often used to remove non-informative features from models. However, there are many different classes of feature selection algorithms. Deciding which one to use is problematic as the algorithmic output is often not amenable to user interpretation. This limits the ability for users to utilize their domain expertise during the modeling process. To improve on this limitation, we developed INFUSE, a novel visual analytics system designed to help analysts understand how predictive features are being ranked across feature selection algorithms, cross-validation folds, and classifiers. We demonstrate how our system can lead to important insights in a case study involving clinical researchers predicting patient outcomes from electronic medical records.

Enhanced real-time quality prediction model based on feature selected nonlinear calibration techniques

This paper evaluates multivariate statistical calibration models for predicting final quality values from process variables. The use of a feature selection technique for multivariate calibration is also provided. Instead of using a full set of process variables, some process variables selected can be used. The objective of such feature selection schemes is to eliminate non-informative variables producing better prediction performance. In this work, genetic algorithm is used as an optimization tool. The performance of the proposed calibration model is demonstrated using real process data. The quality of the final products from the plant is not measured in a real-time basis. Due to the time delay related to measuring final quality values, reliable and timely prediction of the quality characteristics is quite important for safe and efficient operation. By adopting a feature selection scheme along with a filtering step, the prediction performance improved because of the exclusion of non-informative features. The nonlinear calibration models with feature selection and a preprocessing step were shown to produce better performance than those without these two steps. In addition, most of the calibration models considered here benefits from the use of a feature selection step in this case study.

Scale-Based Local Feature Selection for Scene Text Recognition

Scene text recognition has drawn increasing concerns from the OCR community in recent years. Among numerous methods that have been proposed, local feature based methods represented by bag-of-features (BoFs) show notable robustness and efficiency. However, as the existing detectors are based on assumptions about local saliency, a vast number of non-informative local features would be detected in the feature detection stage. In this paper, we propose to remove non-informative local features by integrating feature scales with stroke width information.Experiments taken both on synthetic data and real scene data show that the proposed feature selection method could effectively filter non-informative features and improve the recognition accuracy.

An optimal symmetrical null space criterion of Fisher discriminant for feature extraction and recognition

Linear discriminant analysis (LDA) is one of the most effective feature extraction methods in statistical pattern recognition, which extracts the discriminant features by maximizing the so-called Fisher’s criterion that is defined as the ratio of between-class scatter matrix to within-class scatter matrix. However, classification of high-dimensional statistical data is usually not amenable to standard pattern recognition techniques because of an underlying small sample size (SSS) problem. A popular approach to the SSS problem is the removal of non-informative features via subspace-based decomposition techniques. Motivated by this viewpoint, many elaborate subspace decomposition methods including Fisherface, direct LDA (D-LDA), complete PCA plus LDA (C-LDA), random discriminant analysis (RDA) and multilinear discriminant analysis (MDA), etc., have been developed, especially in the context of face recognition. Nevertheless, how to search a set of complete optimal subspaces for discriminant analysis is still a hot topic of research in area of LDA. In this paper, we propose a novel discriminant criterion, called optimal symmetrical null space (OSNS) criterion that can be used to compute the Fisher’s maximal discriminant criterion combined with the minimal one. Meanwhile, by the reformed criterion, the complete symmetrical subspaces based on the within-class and between-class scatter matrices are constructed, respectively. Different from the traditional subspace learning criterion that derives only one principal subspace, in our approach two null subspaces and their orthogonal complements were all obtained through the optimization of OSNS criterion. Therefore, the algorithm based on OSNS has the potential to outperform the traditional LDA algorithms, especially in the cases of small sample size. Experimental results conducted on the ORL, FERET, XM2VTS and NUST603 face image databases demonstrate the effectiveness of the proposed method.

Feature selection in the Laplacian support vector machine

Traditional classifiers including support vector machines use only labeled data in training. However, labeled instances are often difficult, costly, or time consuming to obtain while unlabeled instances are relatively easy to collect. The goal of semi-supervised learning is to improve the classification accuracy by using unlabeled data together with a few labeled data in training classifiers. Recently, the Laplacian support vector machine has been proposed as an extension of the support vector machine to semi-supervised learning. The Laplacian support vector machine has drawbacks in its interpretability as the support vector machine has. Also it performs poorly when there are many non-informative features in the training data because the final classifier is expressed as a linear combination of informative as well as non-informative features. We introduce a variant of the Laplacian support vector machine that is capable of feature selection based on functional analysis of variance decomposition. Through synthetic and benchmark data analysis, we illustrate that our method can be a useful tool in semi-supervised learning.

Class information predicts activation by object fragments in human object areas

Object-related areas in the ventral visual system in humans are known from imaging studies to be preferentially activated by object images compared with noise or texture patterns. It is unknown, however, which features of the object images are extracted and represented in these areas. Here we tested the extent to which the representation of visual classes used object fragments selected by maximizing the information delivered about the class. We tested functional magnetic resonance imaging blood oxygenation level-dependent activation of highly informative object features in low-and high-level visual areas, compared with noninformative object features matched for low-level image properties. Activation in V1 was similar, but in the lateral occipital area and in the posterior fusiform gyrus, activation by “informative” fragments was significantly higher for three object classes. Behavioral studies also revealed high correlation between performance and fragments information. The results show that an objective class-information measure can predict classification performance and activation in human object-related areas.

A simulation-approximation approach to sample size planning for high-dimensional classification studies

Classification studies with high-dimensional measurements and relatively small sample sizes are increasingly common. Prospective analysis of the role of sample sizes in the performance of such studies is important for study design and interpretation of results, but the complexity of typical pattern discovery methods makes this problem challenging. The approach developed here combines Monte Carlo methods and new approximations for linear discriminant analysis, assuming multivariate normal distributions. Monte Carlo methods are used to sample the distribution of which features are selected for a classifier and the mean and variance of features given that they are selected. Given selected features, the linear discriminant problem involves different distributions of training data and generalization data, for which 2 approximations are compared: one based on Taylor series approximation of the generalization error and the other on approximating the discriminant scores as normally distributed. Combining the Monte Carlo and approximation approaches to different aspects of the problem allows efficient estimation of expected generalization error without full simulations of the entire sampling and analysis process. To evaluate the method and investigate realistic study design questions, full simulations are used to ask how validation error rate depends on the strength and number of informative features, the number of noninformative features, the sample size, and the number of features allowed into the pattern. Both approximation methods perform well for most cases but only the normal discriminant score approximation performs well for cases of very many weakly informative or uninformative dimensions. The simulated cases show that many realistic study designs will typically estimate substantially suboptimal patterns and may have low probability of statistically significant validation results.

Enriched random forests

Although the random forest classification procedure works well in datasets with many features, when the number of features is huge and the percentage of truly informative features is small, such as with DNA microarray data, its performance tends to decline significantly. In such instances, the procedure can be improved by reducing the contribution of trees whose nodes are populated by non-informative features. To some extent, this can be achieved by prefiltering, but we propose a novel, yet simple, adjustment that has demonstrably superior performance: choose the eligible subsets at each node by weighted random sampling instead of simple random sampling, with the weights tilted in favor of the informative features. This results in an 'enriched random forest'. We illustrate the superior performance of this procedure in several actual microarray datasets.