Small Number Of Features Research Articles

Effective Feature Selection Method for Deep Learning-Based Automatic Modulation Classification Scheme Using Higher-Order Statistics

Recently, in order to satisfy the requirements of commercial communication systems and military communication systems, automatic modulation classification (AMC) schemes have been considered. As a result, various artificial intelligence algorithms such as a deep neural network (DNN), a convolutional neural network (CNN), and a recurrent neural network (RNN) have been studied to improve the AMC performance. However, since the AMC process should be operated in real time, the computational complexity must be considered low enough. Furthermore, there is a lack of research to consider the complexity of the AMC process using the data-mining method. In this paper, we propose a correlation coefficient-based effective feature selection method that can maintain the classification performance while reducing the computational complexity of the AMC process. The proposed method calculates the correlation coefficients of second, fourth, and sixth-order cumulants with the proposed formula and selects an effective feature according to the calculated values. In the proposed method, the deep learning-based AMC method is used to measure and compare the classification performance. From the simulation results, it is indicated that the AMC performance of the proposed method is superior to the conventional methods even though it uses a small number of features.

Optimal Feature Search for Vigilance Estimation Using Deep Reinforcement Learning

A low level of vigilance is one of the main reasons for traffic and industrial accidents. We conducted experiments to evoke the low level of vigilance and record physiological data through single-channel electroencephalogram (EEG) and electrocardiogram (ECG) measurements. In this study, a deep Q-network (DQN) algorithm was designed, using conventional feature engineering and deep convolutional neural network (CNN) methods, to extract the optimal features. The DQN yielded the optimal features: two CNN features from ECG and two conventional features from EEG. The ECG features were more significant for tracking the transitions within the alertness continuum with the DQN. The classification was performed with a small number of features, and the results were similar to those from using all of the features. This suggests that the DQN could be applied to investigating biomarkers for physiological responses and optimizing the classification system to reduce the input resources.

Self-Supervised Discovery of Anatomical Shape Landmarks.

Statistical shape analysis is a very useful tool in a wide range of medical and biological applications. However, it typically relies on the ability to produce a relatively small number of features that can capture the relevant variability in a population. State-of-the-art methods for obtaining such anatomical features rely on either extensive preprocessing or segmentation and/or significant tuning and post-processing. These shortcomings limit the widespread use of shape statistics. We propose that effective shape representations should provide sufficient information to align/register images. Using this assumption we propose a self-supervised, neural network approach for automatically positioning and detecting landmarks in images that can be used for subsequent analysis. The network discovers the landmarks corresponding to anatomical shape features that promote good image registration in the context of a particular class of transformations. In addition, we also propose a regularization for the proposed network which allows for a uniform distribution of these discovered landmarks. In this paper, we present a complete framework, which only takes a set of input images and produces landmarks that are immediately usable for statistical shape analysis. We evaluate the performance on a phantom dataset as well as 2D and 3D images.

An Efficient Binary Clonal Selection Algorithm with Optimum Path Forest for Feature Selection

Feature selection is an important step in different applications such as data mining, classification, pattern recognition, and optimization. Until now, finding the most informative set of features among a large dataset is still an open problem. In computer science, a lot of metaphors are imported from nature and biology and proved to be efficient when applying them in an artificial way to solve a lot of problems. Examples include Neural Networks, Human Genetics, Flower Pollination, and Human Immune system. Clonal selection is one of the processes that happens in the human immune system while recognizing new infections. Mimicking this process in an artificial way resulted in a powerful algorithm, which is the Clonal Selection Algorithm. In this paper, we tried to explore the power of the Clonal Selection Algorithm in its binary form for solving the feature selection problem, we used the accuracy of the Optimum-Path Forest classifier, which is much faster than other classifiers, as a fitness function to be optimized. Experiments on three public benchmark datasets are conducted to compare the proposed Binary Clonal Selection Algorithm in conjunction with the Optimum Path Forest classifier with other four powerful algorithms. The four algorithms are Binary Flower Pollination Algorithm, Binary Bat Algorithm, Binary Cuckoo Search, and Binary Differential Evolution Algorithm. In terms of classification accuracy, experiments revealed that the proposed method outperformed the other four algorithms and moreover with a smaller number of features. Also, the proposed method took less average execution time in comparison with the other algorithms, except for Binary Cuckoo Search. The statistical analysis showed that our proposal has a significant difference in accuracy compared with the Binary Bat Algorithm and the Binary Differential Evolution Algorithm.

The Answer is in the Text: Multi-Stage Methods for Phishing Detection Based on Feature Engineering

A phishing attack is a threat based on fraudulent communication, usually by e-mail, where the cybercriminals, impersonating a trusted person or organization, try to lure and coax a target. Phishing detection approaches that obtain highly representational features from the text of these e-mails are a suitable strategy to counter these threats since these features can be used to train machine learning algorithms, thus generating models able to classify mail samples as phishing or legitimate messages. This paper proposes a multi-stage approach to detect phishing e-mail attacks using natural language processing and machine learning. The proposed multi-stage approach consists of feature engineering within natural language processing, lemmatization, feature selection, feature extraction, improved learning techniques for resampling and cross-validation, and the configuration of hyperparameters. We present two methods of the proposed approach, the first one exploiting the Chi-Square statistics and the Mutual Information to improve the dimensionality reduction, while the second method associates Principal Component Analysis (PCA) and Latent Semantic Analysis (LSA). Both methods handle the problems of the “curse of dimensionality”, the sparsity, and the amount of information that must be obtained from the context in the Vector Space Model (VSM) representation. These methods yield reduced feature sets that, combined with the XGBoost and Random Forest machine learning algorithms, lead to an F1-measure of 100% success rate, for validation tests with the SpamAssassin Public Corpus and the Nazario Phishing Corpus datasets. Even considering just the text in e-mail bodies, the proposed multi-stage phishing detection approach outperforms state-of-the-art schemes for an accredited data set, requiring a much smaller number of features and presenting lower computational cost.

Effective Technique to Reduce the Dimension of Text Data

In this article, features are selected using feature clustering and ranking of features for imbalanced text data. Initially the text documents are represented in lower dimension using the term class relevance (TCR) method. The class wise clustering is recommended to balance the documents in each class. Subsequently, the clusters are treated as classes and the documents of each cluster are represented in the lower dimensional form using the TCR again. The features are clustered and for each feature cluster the cluster representative is selected and these representatives are used as selected features of the documents. Hence, this proposed model reduces the dimension to a smaller number of features. For selecting the cluster representative, four feature evaluation methods are used and classification is done by using SVM classifier. The performance of the method is compared with the global feature ranking method. The experiment is conducted on two benchmark datasets the Reuters-21578 and the TDT2 dataset. The experimental results show that this method performs well when compared to the other existing works.

A Solution to the High-Dimensional Classification Problem Using an Improved Hybrid Feature Selection Algorithm Guided by Interaction Information

This paper addresses the high-dimensional classification problem, which is very important in machine learning. When the number of features of the data is very high, the classification performance of a given classifier can degrade because there are not enough samples for training. One of the solutions to cope with this problem is to perform feature selection to reduce the number of features. We propose a new hybrid feature selection algorithm based on interaction information that improves upon the previous one. Our improved method employs interaction information to select candidate features to be added to the current feature subset. Cohen’s $d$ is used as the significance testing to decide whether a new feature is permanently added to the subset. We adopt new stopping criteria to allow intensive search. Our search method is efficient and is able to find excellent solutions. Experiments results on eleven high-dimensional data sets show that compared to other hybrid feature selection algorithms, our proposed algorithm provides high classification accuracy and requires a small number of features for classification.

A Machine-Learning-Based Prediction Method for Hypertension Outcomes Based on Medical Data.

The outcomes of hypertension refer to the death or serious complications (such as myocardial infarction or stroke) that may occur in patients with hypertension. The outcomes of hypertension are very concerning for patients and doctors, and are ideally avoided. However, there is no satisfactory method for predicting the outcomes of hypertension. Therefore, this paper proposes a prediction method for outcomes based on physical examination indicators of hypertension patients. In this work, we divide the patients’ outcome prediction into two steps. The first step is to extract the key features from the patients’ many physical examination indicators. The second step is to use the key features extracted from the first step to predict the patients’ outcomes. To this end, we propose a model combining recursive feature elimination with a cross-validation method and classification algorithm. In the first step, we use the recursive feature elimination algorithm to rank the importance of all features, and then extract the optimal features subset using cross-validation. In the second step, we use four classification algorithms (support vector machine (SVM), C4.5 decision tree, random forest (RF), and extreme gradient boosting (XGBoost)) to accurately predict patient outcomes by using their optimal features subset. The selected model prediction performance evaluation metrics are accuracy, F1 measure, and area under receiver operating characteristic curve. The 10-fold cross-validation shows that C4.5, RF, and XGBoost can achieve very good prediction results with a small number of features, and the classifier after recursive feature elimination with cross-validation feature selection has better prediction performance. Among the four classifiers, XGBoost has the best prediction performance, and its accuracy, F1, and area under receiver operating characteristic curve (AUC) values are 94.36%, 0.875, and 0.927, respectively, using the optimal features subset. This article’s prediction of hypertension outcomes contributes to the in-depth study of hypertension complications and has strong practical significance.

A new multi-objective differential evolution approach for simultaneous clustering and feature selection

Today’s real-world data mostly involves incomplete, inconsistent, and/or irrelevant information that causes many drawbacks to transform it into an understandable format. In order to deal with such issues, data preprocessing is a proven discipline in data mining. One of the typical tasks in data preprocessing, feature selection aims to reduce the dimensionality in the data and thereby contributes to further processing. Feature selection is widely used to enhance the performance of a supervised learning algorithm (e.g., classification) but is rarely used in unsupervised tasks (e.g., clustering). This paper introduces a new multi-objective differential evolution approach in order to find relatively homogeneous clusters without the prior knowledge of cluster number using a smaller number of features from all available features in the data. To analyze the goodness of the introduced approach, several experiments are conducted on a various number of real-world and synthetic benchmarks using a variety of clustering approaches. From the analyzes through several different criteria, it is suggested that our method can significantly improve the clustering performance while reducing the dimensionality at the same time.

Human Activities and Postures Recognition: From Inertial Measurements to Quaternion-Based Approaches.

This paper presents two approaches to assess the effect of the number of inertial sensors and their location placements on recognition of human postures and activities. Inertial and Magnetic Measurement Units (IMMUs)—which consist of a triad of three-axis accelerometer, three-axis gyroscope, and three-axis magnetometer sensors—are used in this work. Five IMMUs are initially used and attached to different body segments. Placements of up to three IMMUs are then considered: back, left foot, and left thigh. The subspace k-nearest neighbors (KNN) classifier is used to achieve the supervised learning process and the recognition task. In a first approach, we feed raw data from three-axis accelerometer and three-axis gyroscope into the classifier without any filtering or pre-processing, unlike what is usually reported in the state-of-the-art where statistical features were computed instead. Results show the efficiency of this method for the recognition of the studied activities and postures. With the proposed algorithm, more than 80% of the activities and postures are correctly classified using one IMMU, placed on the lower back, left thigh, or left foot location, and more than 90% when combining all three placements. In a second approach, we extract attitude, in term of quaternion, from IMMUs in order to more precisely achieve the recognition process. The obtained accuracy results are compared to those obtained when only raw data is exploited. Results show that the use of attitude significantly improves the performance of the classifier, especially for certain specific activities. In that case, it was further shown that using a smaller number of features, with quaternion, in the recognition process leads to a lower computation time and better accuracy.

Decoding task engagement from distributed network electrophysiology in humans

Objective. Here, our objective was to develop a binary decoder to detect task engagement in humans during two distinct, conflict-based behavioral tasks. Effortful, goal-directed decision-making requires the coordinated action of multiple cognitive processes, including attention, working memory and action selection. That type of mental effort is often dysfunctional in mental disorders, e.g. when a patient attempts to overcome a depression or anxiety-driven habit but feels unable. If the onset of engagement in this type of focused mental activity could be reliably detected, decisional function might be augmented, e.g. through neurostimulation. However, there are no known algorithms for detecting task engagement with rapid time resolution. Approach. We defined a new network measure, fixed canonical correlation (FCCA), specifically suited for neural decoding applications. We extracted FCCA features from local field potential recordings in human volunteers to give a temporally continuous estimate of mental effort, defined by engagement in experimental conflict tasks. Main results. Using a small number of features per participant, we accurately decoded and distinguished task engagement from other mental activities. Further, the decoder distinguished between engagement in two different conflict-based tasks within seconds of their onset. Significance. These results demonstrate that network-level brain activity can detect specific types of mental efforts. This could form the basis of a responsive intervention strategy for decision-making deficits.

Beta Distribution-Based Cross-Entropy for Feature Selection.

Analysis of high-dimensional data is a challenge in machine learning and data mining. Feature selection plays an important role in dealing with high-dimensional data for improvement of predictive accuracy, as well as better interpretation of the data. Frequently used evaluation functions for feature selection include resampling methods such as cross-validation, which show an advantage in predictive accuracy. However, these conventional methods are not only computationally expensive, but also tend to be over-optimistic. We propose a novel cross-entropy which is based on beta distribution for feature selection. In beta distribution-based cross-entropy (BetaDCE) for feature selection, the probability density is estimated by beta distribution and the cross-entropy is computed by the expected value of beta distribution, so that the generalization ability can be estimated more precisely than conventional methods where the probability density is learnt from data. Analysis of the generalization ability of BetaDCE revealed that it was a trade-off between bias and variance. The robustness of BetaDCE was demonstrated by experiments on three types of data. In the exclusive or-like (XOR-like) dataset, the false discovery rate of BetaDCE was significantly smaller than that of other methods. For the leukemia dataset, the area under the curve (AUC) of BetaDCE on the test set was 0.93 with only four selected features, which indicated that BetaDCE not only detected the irrelevant and redundant features precisely, but also more accurately predicted the class labels with a smaller number of features than the original method, whose AUC was 0.83 with 50 features. In the metabonomic dataset, the overall AUC of prediction with features selected by BetaDCE was significantly larger than that by the original reported method. Therefore, BetaDCE can be used as a general and efficient framework for feature selection.

New efficient initialization and updating mechanisms in PSO for feature selection and classification

Feature selection is one of the important and difficult issues in classification. Particle swarm optimization (PSO) is an efficient evolutionary computing technique that has been widely used to deal with feature selection problem. However, it has been observed that the traditional initialization and personal best and global best updating mechanisms in PSO often limit its performance for feature selection and has to be further explored to see the full potential of PSO for the same. This paper proposes two new efficient initialization and updating mechanisms in PSO with the goal of minimizing the number of features and maximizing the classification performance in less computational time. The proposed algorithms are compared with six existing feature selection methods, including two traditional PSO-based feature selection methods and four PSO with different initialization strategy and updating mechanism-based feature selection methods. Experiments on eight benchmark dataset show that the proposed algorithms can automatically evolve a feature subset with a smaller number of features with higher classification performance than using all features. The proposed algorithms also outperform the eight existing feature selection algorithms in terms of the classification accuracy, the number of features, and the computational cost.

Heuristic Search Algorithm for Dimensionality Reduction Optimally Combining Feature Selection and Feature Extraction

The following are two classical approaches to dimensionality reduction: 1. Approximating the data with a small number of features that exist in the data (feature selection). 2. Approximating the data with a small number of arbitrary features (feature extraction). We study a generalization that approximates the data with both selected and extracted features. We show that an optimal solution to this hybrid problem involves a combinatorial search, and cannot be trivially obtained even if one can solve optimally the separate problems of selection and extraction. Our approach that gives optimal and approximate solutions uses a “best first” heuristic search. The algorithm comes with both an a priori and an a posteriori optimality guarantee similar to those that can be obtained for the classical weighted A* algorithm. Experimental results show the effectiveness of the proposed approach.

Analyzing Depthwise Convolution Based Neural Network: Study Case in Ship Detection and Land Cover Classification

Various methods are available to perform feature extraction on satellite images. Among the available alternatives, deep convolutional neural network (ConvNet) is the state of the art method. Although previous studies have reported successful attempts on developing and implementing ConvNet on remote sensing application, several issues are not well explored, such as the use of depthwise convolution, final pooling layer size, and comparison between grayscale and RGB settings. The objective of this study is to perform analysis to address these issues. Two feature learning algorithms were proposed, namely ConvNet as the current state of the art for satellite image classification and Gray Level Co-occurence Matrix (GLCM) which represents a classic unsupervised feature extraction method. The experiment demonstrated consistent result with previous studies that ConvNet is superior in most cases compared to GLCM, especially with 3x3xn final pooling. The performance of the learning algorithms are much higher on features from RGB channels, except for ConvNet with relatively small number of features.

Classification of mammogram images using the energy probability in frequency domain and most discriminative power coefficients

AbstractThe purpose of this work is to develop a computer‐aided diagnosis (CAD) system to assist radiologists in the classification of mammogram images. The CAD system is composed of three main steps. The first step is image preprocessing and segmentation with the seeded region growing algorithm applied on a localized triangular region to remove only the muscle. In the second step of the CAD system, we proposed a novel features extraction method, which consists of three stages. In the first, the discrete cosine transform (DCT) is applied on all obtained regions of interest and then only the upper left corner (ULC) of DCT coefficients is retained. Second, we have applied the energy probability to the ULCs that is used as a criterion for selecting discriminant information. At the last stage, a new Most Discriminative power coefficient algorithm has been proposed to select the most significant features. In the final step of the CAD, the support vector machines, Naive Bayes, and artificial neural network (ANN) classifiers are used to make an effective classification. The evaluation of the proposed algorithm on the mini‐Mammographic Image Analysis Society database shows its efficiency over other recently proposed CAD systems in the literature, whereas an accuracy of 100% can be achieved using ANN with a small number of features.

FEATURE FILTERING AND SELECTION FOR DRY MATTER ESTIMATION ON PERENNIAL RYEGRASS: A CASE STUDY OF VEGETATION INDICES

Abstract. Vegetation indices (VIs) have been extensively employed as a feature for dry matter (DM) estimation. During the past five decades more than a hundred vegetation indices have been proposed. Inevitably, the selection of the optimal index or subset of indices is not trivial nor obvious. This study, performed on a year-round observation of perennial ryegrass (n = 900), indicates that for this response variable (i.e. kg.DM.ha−1), more than 80% of indices present a high degree of collinearity (correlation > |0.8|.) Additionally, the absence of an established workflow for feature selection and modelling is a handicap when trying to establish meaningful relations between spectral data and biophysical/biochemical features. Within this case study, an unsupervised and supervised filtering process is proposed to an initial dataset of 97 VIs. This research analyses the effects of the proposed filtering and feature selection process to the overall stability of final models. Consequently, this analysis provides a straightforward framework to filter and select VIs. This approach was able to provide a reduced feature set for a robust model and to quantify trade-offs between optimal models (i.e. lowest root mean square error – RMSE = 412.27 kg.DM.ha−1) and tolerable models (with a smaller number of features – 4 VIs and within 10% of the lowest RMSE.)

Feature selection for intrusion detection using new multi-objective estimation of distribution algorithms

The manipulation of a large number of features has become a critical problem in Intrusion Detection Systems(IDS). Therefore, Feature Selection (FS) is integrated to select the significant features, in order to avoid the computational complexity, and improve the classification performance. In this paper, we present a new multi-objective feature selection algorithm MOEDAFS (Multi-Objective Estimation of Distribution Algorithms (EDA) for Feature Selection). The MOEDAFS is based on EDA and Mutual Information (MI). EDA is used to explore the search space and MI is integrated as a probabilistic model to guide the search by modeling the redundancy and relevance relations between features. Therefore, we propose four probabilistic models for MOEDAFS. MOEDAFS selects the better feature subsets (non-dominated solutions) that have a better detection accuracy and smaller number of features. MOEDAFS uses two objective functions (minimizing classification Error Rate (ER) and minimizing the Number of Features(NF)). In order to demonstrate the performance of MOEDAFS, a comparative study is designed by internal and external comparison on NSL-KDD dataset. Internal comparison is performed between the four versions of MOEDAFS. External comparison is organized against some well-known deterministic, metaheuristic, and multi-objective feature selection algorithms that have a single and Multi-solution. Experimental results demonstrate that MOEDAFS outperforms recent algorithms.

Selection of PolSAR Observables for Crop Biophysical Variable Estimation With Global Sensitivity Analysis

The role of global sensitivity analysis (GSA) is to quantify and rank the most influential features for biophysical variable estimation. In this letter, an approximation model, called high-dimensional model representation (HDMR), is utilized to develop a regression method in conjunction with a GSA in the context of determining key input drivers in the estimation of crop biophysical variables from polarimetric synthetic aperture radar data. A multitemporal Radarsat-2 data set is used for the retrieval of three biophysical variables of barley: leaf area index, normalized difference vegetation index, and Biologische Bundesanstalt, Bundessortenamt und CHemische Industrie stage. The HDMR technique is first adopted to estimate a regression model with all available polarimetric features for each biophysical parameter, and sensitivity indices of each feature are then derived to explain the original space with a smaller number of features in which a final regression model is established. To evaluate the applicability of this methodology, root-mean square and coefficient of determination were performed under different amounts of samples. Results highlight that HDMR can be used effectively in biophysical variable estimation for not only reducing computational cost but also for providing a robust regression.

Fine‐tuning restricted Boltzmann machines using quaternions and its application for spam detection

Restricted Boltzmann Machines (RBMs) have been used in a number of applications, but only a few works have addressed them in the context of information security. However, such models have their performance severely affected by some hyperparameters that are usually hand-tuned. In this work, the authors consider learning features in an unsupervised fashion by means of RBMs fine-tuned by hypercomplex-based metaheuristic techniques in the context of malicious content detection. Experiments are conducted over three public datasets and six metaheuristic techniques, which are used to fine-tune RBM hyperparameters such that RBM extracts features that best represent malicious content present in spam e-mail messages, and generates a dataset to be used as input to classification through the Optimum Path Forest supervised algorithm. Experimental results demonstrate that a small number of features generated through RBM can achieve a competitive accuracy in relation to the original dataset, however, with lower computational cost. Furthermore, this study presents the power of quaternions for RBMs parameter optimisation, comparing it against the well-known Harmonic Search, as well as its variants Improved Harmonic Search and Parameter Setting-Free Harmonic Search. It was concluded that RBM-based learning techniques are suitable for features extraction in the context of this work.