Feature Set Size Research Articles

Application of complex discrete wavelet transform in classification of Doppler signals using complex-valued artificial neural network

In biomedical signal classification, due to the huge amount of data, to compress the biomedical waveform data is vital. This paper presents two different structures formed using feature extraction algorithms to decrease size of feature set in training and test data. The proposed structures, named as wavelet transform-complex-valued artificial neural network (WT-CVANN) and complex wavelet transform-complex-valued artificial neural network (CWT-CVANN), use real and complex discrete wavelet transform for feature extraction. The aim of using wavelet transform is to compress data and to reduce training time of network without decreasing accuracy rate. In this study, the presented structures were applied to the problem of classification in carotid arterial Doppler ultrasound signals. Carotid arterial Doppler ultrasound signals were acquired from left carotid arteries of 38 patients and 40 healthy volunteers. The patient group included 22 males and 16 females with an established diagnosis of the early phase of atherosclerosis through coronary or aortofemoropopliteal (lower extremity) angiographies (mean age, 59 years; range, 48-72 years). Healthy volunteers were young non-smokers who seem to not bear any risk of atherosclerosis, including 28 males and 12 females (mean age, 23 years; range, 19-27 years). Sensitivity, specificity and average detection rate were calculated for comparison, after training and test phases of all structures finished. These parameters have demonstrated that training times of CVANN and real-valued artificial neural network (RVANN) were reduced using feature extraction algorithms without decreasing accuracy rate in accordance to our aim.

Identifying core sets of discriminatory features using particle swarm optimization

Forming an efficient feature space for classification problems is a grand challenge in pattern recognition. New optimization techniques emerging in areas such as Computational Intelligence have been investigated in the context of feature selection. Here, we propose an original two-phase feature selection method that uses particle swarm optimization (PSO), a biologically inspired optimization technique, which forms an initial core set of discriminatory features from the original feature space. This core set is then successively expanded by searching for additional discriminatory features. The performance of the proposed PSO feature selection method is evaluated using a nearest neighbor classifier. The design of the optimally reduced feature space is investigated in a parametric setting by varying the size of the core feature set and the training set. Numerical experiments, using data from the Machine Learning Repository, show that a substantial reduction of the feature space is accomplished. A thorough comparative analysis of results reported in the literature also reveals improvement in classification accuracy.

The peaking phenomenon in the presence of feature-selection

For a fixed sample size, a common phenomenon is that the error of a designed classifier decreases and then increases as the number of features grows. This peaking phenomenon has been recognized for forty years and depends on the classification rule and feature-label distribution. Historically, the peaking phenomenon has been treated by assuming a fixed ordering of the features, usually beginning with the strongest individual feature and proceeding with features of decreasing individual classification capability. This does not take into account feature-selection, which is commonplace in high-dimensional and small sample settings. This paper revisits the peaking phenomenon in the presence of feature-selection. Using massive simulation in a high-performance computing environment, the paper considers various combinations of feature-label models, feature-selection algorithms, and classifier models to produce a large library of error versus feature size curves. Owing to the prevalence of feature-selection in genomic classification, we also consider gene-expression-based classification of breast-cancer patient prognosis. Results vary widely and are strongly dependent on the combination. The error curves tend to fall into three categories: peaking, settling into a plateau, or falling very slowly over a long range of feature set sizes. It can be concluded that one should be wary of applying peaking results found in the absence of feature-selection to settings in which feature-selection is employed.

Modified linear discriminant analysis approaches for classification of high-dimensional microarray data

Linear discriminant analysis (LDA) is one of the most popular methods of classification. For high-dimensional microarray data classification, due to the small number of samples and large number of features, classical LDA has sub-optimal performance corresponding to the singularity and instability of the within-group covariance matrix. Two modified LDA approaches (MLDA and NLDA) were applied for microarray classification and their performance criteria were compared with other popular classification algorithms across a range of feature set sizes (number of genes) using both simulated and real datasets. The results showed that the overall performance of the two modified LDA approaches was as competitive as support vector machines and other regularized LDA approaches and better than diagonal linear discriminant analysis, k -nearest neighbor, and classical LDA. It was concluded that the modified LDA approaches can be used as an effective classification tool in limited sample size and high-dimensional microarray classification problems.

CROSS-GENRE FEATURE COMPARISONS FOR SPOKEN SENTENCE SEGMENTATION

Automatic sentence segmentation of spoken language is an important precursor to downstream natural language processing. Previous studies combine lexical and prosodic features, but can impose significant computational challenges because of the large size of feature sets. Little is understood about which features most benefit performance, particularly for speech data from different speaking styles. We compare sentence segmentation for speech from broadcast news versus natural multi-party meetings, using identical lexical and prosodic feature sets across genres. Results based on boosting and forward selection for this task show that (1) features sets can be reduced with little or no loss in performance, and (2) the contribution of different feature types differs significantly by genre. We conclude that more efficient approaches to sentence segmentation and similar tasks can be achieved, especially if genre differences are taken into account.

Novel Unsupervised Feature Filtering of Biological Data

Many methods have been developed for selecting small informative feature subsets in large noisy data. However, unsupervised methods are scarce. Examples are using the variance of data collected for each feature, or the projection of the feature on the first principal component. We propose a novel unsupervised criterion, based on SVD-entropy, selecting a feature according to its contribution to the entropy (CE) calculated on a leave-one-out basis. This can be implemented in four ways: simple ranking according to CE values (SR); forward selection by accumulating features according to which set produces highest entropy (FS1); forward selection by accumulating features through the choice of the best CE out of the remaining ones (FS2); backward elimination (BE) of features with the lowest CE. We apply our methods to different benchmarks. In each case we evaluate the success of clustering the data in the selected feature spaces, by measuring Jaccard scores with respect to known classifications. We demonstrate that feature filtering according to CE outperforms the variance method and gene-shaving. There are cases where the analysis, based on a small set of selected features, outperforms the best score reported when all information was used. Our method calls for an optimal size of the relevant feature set. This turns out to be just a few percents of the number of genes in the two Leukemia datasets that we have analyzed. Moreover, the most favored selected genes turn out to have significant GO enrichment in relevant cellular processes.

An evaluation of statistical spam filtering techniques

This paper evaluates five supervised learning methods in the context of statistical spam filtering. We study the impact of different feature pruning methods and feature set sizes on each learner's performance using cost-sensitive measures. It is observed that the significance of feature selection varies greatly from classifier to classifier. In particular, we found support vector machine, AdaBoost, and maximum entropy model are top performers in this evaluation, sharing similar characteristics: not sensitive to feature selection strategy, easily scalable to very high feature dimension, and good performances across different datasets. In contrast, naive Bayes, a commonly used classifier in spam filtering, is found to be sensitive to feature selection methods on small feature set, and fails to function well in scenarios where false positives are penalized heavily. The experiments also suggest that aggressive feature pruning should be avoided when building filters to be used in applications where legitimate mails are assigned a cost much higher than spams (such as λ = 999), so as to maintain a better-than-baseline performance. An interesting finding is the effect of mail headers on spam filtering, which is often ignored in previous studies. Experiments show that classifiers using features from message header alone can achieve comparable or better performance than filters utilizing body features only. This implies that message headers can be reliable and powerfully discriminative feature sources for spam filtering.

Optimal number of features as a function of sample size for various classification rules

Given the joint feature-label distribution, increasing the number of features always results in decreased classification error; however, this is not the case when a classifier is designed via a classification rule from sample data. Typically (but not always), for fixed sample size, the error of a designed classifier decreases and then increases as the number of features grows. The potential downside of using too many features is most critical for small samples, which are commonplace for gene-expression-based classifiers for phenotype discrimination. For fixed sample size and feature-label distribution, the issue is to find an optimal number of features. Since only in rare cases is there a known distribution of the error as a function of the number of features and sample size, this study employs simulation for various feature-label distributions and classification rules, and across a wide range of sample and feature-set sizes. To achieve the desired end, finding the optimal number of features as a function of sample size, it employs massively parallel computation. Seven classifiers are treated: 3-nearest-neighbor, Gaussian kernel, linear support vector machine, polynomial support vector machine, perceptron, regular histogram and linear discriminant analysis. Three Gaussian-based models are considered: linear, nonlinear and bimodal. In addition, real patient data from a large breast-cancer study is considered. To mitigate the combinatorial search for finding optimal feature sets, and to model the situation in which subsets of genes are co-regulated and correlation is internal to these subsets, we assume that the covariance matrix of the features is blocked, with each block corresponding to a group of correlated features. Altogether there are a large number of error surfaces for the many cases. These are provided in full on a companion website, which is meant to serve as resource for those working with small-sample classification. For the companion website, please visit http://public.tgen.org/tamu/ofs/ e-dougherty@ee.tamu.edu.

Determination of the optimal number of features for quadratic discriminant analysis via the normal approximation to the discriminant distribution

Given the joint feature-label distribution, increasing the number of features always results in decreased classification error; however, this is not the case when a classifier is designed via a classification rule from sample data. Typically, for fixed sample size, the error of a designed classifier decreases and then increases as the number of features grows. The problem is especially acute when sample sizes are very small and the potential number of features is very large. To obtain a general understanding of the kinds of feature-set sizes that provide good performance for a particular classification rule, performance must be evaluated based on accurate error estimation, and hence a model-based setting for optimizing the number of features is needed. This paper treats quadratic discriminant analysis (QDA) in the case of unequal covariance matrices. For two normal class-conditional distributions, the QDA classifier is determined according to a discriminant. The standard plug-in rule estimates the discriminant from a feature-label sample to obtain an estimate of the discriminant by replacing the means and covariance matrices by their respective sample means and sample covariance matrices. The unbiasedness of these estimators assures good estimation for large samples, but not for small samples. Our goal is to find an essentially analytic method to produce an error curve as a function of the number of features so that the curve can be minimized to determine an optimal number of features. We use a normal approximation to the distribution of the estimated discriminant. Since the mean and variance of the estimated discriminant will be exact, these provide insight into how the covariance matrices affect the optimal number of features. We derive the mean and variance of the estimated discriminant and compare feature-size optimization using the normal approximation to the estimated discriminant with optimization obtained by simulating the true distribution of the estimated discriminant. Optimization via the normal approximation to the estimated discriminant provides huge computational savings in comparison to optimization via simulation of the true distribution. Feature-size optimization via the normal approximation is very accurate when the covariance matrices differ modestly. The optimal number of features based on the normal approximation will exceed the actual optimal number when there is large disagreement between the covariance matrices; however, this difference is not important because the true misclassification error using the number of features obtained from the normal approximation and the number obtained from the true distribution differ only slightly, even for significantly different covariance matrices.

Fast branch & bound algorithms for optimal feature selection

A novel search principle for optimal feature subset selection using the Branch & Bound method is introduced. Thanks to a simple mechanism for predicting criterion values, a considerable amount of time can be saved by avoiding many slow criterion evaluations. We propose two implementations of the proposed prediction mechanism that are suitable for use with nonrecursive and recursive criterion forms, respectively. Both algorithms find the optimum usually several times faster than any other known Branch & Bound algorithm. As the algorithm computational efficiency is crucial, due to the exponential nature of the search problem, we also investigate other factors that affect the search performance of all Branch & Bound algorithms. Using a set of synthetic criteria, we show that the speed of the Branch & Bound algorithms strongly depends on the diversity among features, feature stability with respect to different subsets, and criterion function dependence on feature set size. We identify the scenarios where the search is accelerated the most dramatically (finish in linear time), as well as the worst conditions. We verify our conclusions experimentally on three real data sets using traditional probabilistic distance criteria.

A Feature Selection and Classification Technique for Text Categorization

Text categorization is the automated assigning of documents to predefined categories based on their contents. It involves two main tasks — feature selection and document classification. This paper discusses the weak points of the text categorization technique developed by Maron and modified by Lewis. Then, it introduces a technique for text categorization that uses new formulas for feature selection and document classification. These formulas have been formulated to overcome the weak points of Maron's and Lewis' techniques. Also, the paper describes the design of an experimental text categorization system that is composed of the same set of processes as the MAXCAT system developed by Lewis. The paper presents and analyses the results of applying the system on a set of training and test documents by using Lewis' and the proposed formulas. In addition, a method for separately evaluating the effectiveness of feature selection is given. Finally, the impact of the feature set size on the effectiveness of the classification system is investigated, using the system and applying one of the proposed classification formulas with different feature set sizes.

A new face authentication system for memory-constrained devices

Though biometrics to authenticate a person is a convenient tool, typical authentication algorithms by using biometrics may not be executable on the memory-constrained devices such as smart cards. We present a solution of a face authentication algorithm for open issue. Our achievement is two-fold. One is to present a face authentication algorithm with low memory requirement, which uses support vector machines (SVM) with the feature set extracted by genetic algorithms (GA). The other contribution is to suggest a method to reduce further, if needed, the amount of memory required in the authentication at the expense of verification rate by changing a controllable system parameter for a feature set size. Given a pre-defined amount of memory, this capability is quite effective to mount our algorithm on memory-constrained devices. Our experimental results show that the proposed method provides good performance in terms of accuracy and memory requirement.

Acoustic classification: An overview of theory and reality

Historically, Bayes’ decision theory has formed the statistical foundation for the development of acoustic classification techniques for problems that contain large numbers of data classes. The successful realization of this approach depends directly on mitigating two important mismatches that are introduced using this classification model. First, the underlying probabilistic structure in the problem is not typically known and therefore must be estimated as part of a classifier training phase. Second, due to the high dimensionality of time series data, a feature extraction stage is added as a preprocessor to most classifiers. Attempting to extract the sufficient information in the problem while limiting the feature set size to a dimension supported by the training data is the ultimate designer’s challenge. This is particularly true in the case of finite data sample sizes. This talk will trace the development evolution of a low false alarm rate multiclass operational algorithm that will provide an overview of the fundamental challenges and resulting mitigation approaches. Examples using real data will be used to demonstrate the points being presented.

Multi-Channel Filtering Approach for Unsupervised Segmentation of Subsurface Radar Images

The volume of image data generated in ground-penetrating radar surveys can severely restrict the practicality of this site investigation technique. This is particularly true in situations where automatic analysis or interpretation is required, as segmentation and classification tasks that utilise multivariate data are critically affected by the volume and dimensionality of the data. A general-purpose unsupervised image segmentation system is presented here for the automatic detection of image regions exhibiting different visual texture properties. A bank of Gabor filters is used to achieve multi-channel filtering analogous to the processing of information in the visual cortex. A suboptimal feature-selection procedure is proposed to automatically select the set of texture features best suited for the particular application. The reduction in the size of the feature set both reduces the computation time and improves the accuracy of the final classification. Unsupervised neural networks are employed to segment and classify the image data into categories. Results presented for two typical radargrams match closely the interpretation of a human expert, in the fraction of the time. Applications of this technique can be found in disciplines as diverse as medical image analysis and ultrasonic non-destructive testing.

Informativity and Asymmetry in Comparisons

We propose an account of comparison asymmetries based on viewing comparison as a process of structural alignment and mapping. Specifically, we hypothesize that (1) comparison asymmetries result from directional differences in informativity, and that (2) asymmetries can therefore be predicted from the relative degree of systematicity or conceptual coherence of the items being compared. In Experiment 1, we found a clear preference for placing the more systematic of two passages in the base position of a comparison. Experiments 1 and 2 further showed that structural alignability is crucial in obtaining such asymmetries. In Experiment 3, we found that asymmetries are predicted by the relative systematicity of the comparisons items rather than by the relative size of the distinctive feature sets. These results are inconsistent with accounts of asymmetry based on feature contrast or stimulus bias. In Experiments 4 through 6, we tested the functional implications of our account by examining inference projection and perceived informativity across asymmetric comparisons. Critically, comparisons having the more systematic item as the base were more likely to result in inference projection and other forms of target modification and were rated as more informative than reverse comparisons. We conclude by demonstrating that this account can explain comparison asymmetries without positing underlying asymmetries in subjective similarity, and that it offers a unified approach to the directionality of literal comparisons, analogies, and metaphors.

Theoretical and methodological approaches to apparent movement of short-range process and pattern perception

A variable raster and vector display processor interfaced with a PDP-11/23 computer is briefly described. Several experiments using random-dot cinematograms (RDCs) (raster display) and line segments (vector display) are discussed. The experiments on cooperative processes of apparent movement of RDCs and spatial-frequency-filtered RDCs showed that the average cooperative neighborhood of neural interaction is about 15 min arc of visual angle. In pattern perception, based on Julesz’s texton theory (1981) and Shaw’s attention model (1980), this study showed that the aperture of attention in serial and parallel processing is a function of feature gradient, set size, and eccentricity.