Mutual Information Feature Selection Research Articles

Automated Classification of Fatty and Normal Liver Ultrasound Images Based on Mutual Information Feature Selection

BackgroundFatty Liver Disease (FLD) is one of the most critical diseases that should be detected and cured at the earlier stage in order to decrease the mortality rate. To identify the FLD, ultrasound images have been widely used by the radiologists. However, due to poor quality of ultrasound images, they found difficulties in recognizing FLD. To resolve this problem, many researchers have developed various Computer Aided Diagnosis (CAD) systems for the classification of fatty and normal liver ultrasound images. However, the performance of existing CAD systems is not good in terms of sensitivity while classifying the FLD. MethodsIn this paper, an attempt has been made to present a CAD system for the classification of liver ultrasound images. For this purpose, texture features are extracted by using seven different texture models to represent the texture of Region of Interest (ROI). Highly discriminating features are selected by using Mutual Information (MI) feature selection method. ResultsExtensive experiments have been carried out with four different classifiers, and for carrying out this study, 90 liver ultrasound images have been taken. From the experimental results, it has been found that the proposed CAD system is able to give 95.55% accuracy and sensitivity of 97.77% with the 20 best features selected by the MI feature selection technique. ConclusionThe experimental results show that the proposed system can be used for the classification of fatty and normal liver ultrasound images with higher accuracy.

Local difference ternary sequences descriptor based on unsupervised min redundancy mutual information feature selection

Texture feature description research have received significant attention in recent years. It is widely known that the local texture feature descriptor can achieve good performance under various image conditions, such as geometric size variation, different poses, complex illumination and partial occlusion. Although Local Difference Binary is an acknowledged excellent feature descriptor, it only computes the intensity and gradient difference on pairwise grid cells and ignores the image grid texture intensity and gradient. This paper proposes a novel local texture descriptor, named as Local Difference Ternary (LDT), which can not only represent difference and texture information of the grid cells intensity and gradient simultaneously, but also capture richer detailed texture information. In addition, the Unsupervised Min Redundancy Mutual Information (UMRMI) for feature selection is proposed to select the optimal subset of LDT features for achieving more powerfully discriminative ability. For the purpose of further improving the efficiency and effectiveness of UMRMI, we extend UMRMI to k-means space, namely k-UMRMI. Furthermore, a multi-degree scheme is adopted to achieve richer texture description. Finally, Radial Function Neural Network is employed for classification, which is an excellent classifier, especially for larger samples. Several experimental results on certain benchmark face databases demonstrate that our proposed method is remarkably superior to some other state-of-the-art approaches under various image conditions.

Intrusion detection mechanism with machine learning process A case study with FMIFSSVM, FLCFSSVM, misuses SVM, anomaly SVM and Bayesian methods

Today, there is a far reaching of Internet benefits everywhere throughout the world, numerous sorts and vast number of security dangers are expanding. Since it isn't in fact possible to assemble a framework without any vulnerability, Intrusion Detection System (IDS), which can successfully distinguish Intrusion, gets to have pulled in consideration. Intrusion detection can be characterized as the way toward distinguishing irregular, unauthorized or unapproved action that objective is to target a system and its assets. IDS plays a very important role for analyzing the network passage, also it assumes a key part to analyze the system activity log and each log is portrayed by huge arrangement of highlights and it requires tremendous computational preparing force and time for the characterization procedure. This work proposes filter based feature selection methods to predict intrusion with Feature based Mutual Information Feature Selection Support Vector Machine (FMIFSSVM), Feature based Liner Correlation Feature Selection Support Vector Machine (FLCFSSVM), misuses SVM, anomaly SVM and Bayesian methods. The performances of these methods are considered by using the intrusion detection calculation data set called Knowledge Discovery in Databases (KDD) cup 99. Detection Rate (DR), False Alarm Rate (FAR) and Percentage of Successful Prediction (PSP) are the major performance measures studied in this work.

Fuzzy Mutual Information Feature Selection Based on Representative Samples

Building classification models from real-world datasets became a difficult task, especially in datasets with high dimensional features. Unfortunately, these datasets may include irrelevant or redundant features which have a negative effect on the classification performance. Selecting the significant features and eliminating undesirable features can improve the classification models. Fuzzy mutual information is widely used feature selection to find the best feature subset before classification process. However, it requires more computation and storage space. To overcome these limitations, this paper proposes an improved fuzzy mutual information feature selection based on representative samples. Based on benchmark datasets, the experiments show that the proposed method achieved better results in the terms of classification accuracy, selected feature subset size, storage, and stability.

Differential evolution for filter feature selection based on information theory and feature ranking

Feature selection is an essential step in various tasks, where filter feature selection algorithms are increasingly attractive due to their simplicity and fast speed. A common filter is to use mutual information to estimate the relationships between each feature and the class labels (mutual relevancy), and between each pair of features (mutual redundancy). This strategy has gained popularity resulting a variety of criteria based on mutual information. Other well-known strategies are to order each feature based on the nearest neighbor distance as in ReliefF, and based on the between-class variance and the within-class variance as in Fisher Score. However, each strategy comes with its own advantages and disadvantages. This paper proposes a new filter criterion inspired by the concepts of mutual information, ReliefF and Fisher Score. Instead of using mutual redundancy, the proposed criterion tries to choose the highest ranked features determined by ReliefF and Fisher Score while providing the mutual relevance between features and the class labels. Based on the proposed criterion, two new differential evolution (DE) based filter approaches are developed. While the former uses the proposed criterion as a single objective problem in a weighted manner, the latter considers the proposed criterion in a multi-objective design. Moreover, a well known mutual information feature selection approach (MIFS) based on maximum-relevance and minimum-redundancy is also adopted in single-objective and multi-objective DE algorithms for feature selection. The results show that the proposed criterion outperforms MIFS in both single objective and multi-objective DE frameworks. The results also indicate that considering feature selection as a multi-objective problem can generally provide better performance in terms of the feature subset size and the classification accuracy.

An Anomaly Detection Method Based on Normalized Mutual Information Feature Selection and Quantum Wavelet Neural Network

This paper presents an anomaly detection model based on normalized mutual information feature selection (NMIFS) and quantum wavelet neural network (QWNN). The goal of the proposed model is to address the problem of determining the feature subset used to detect an anomaly in a machine learning task. In order to achieve an effective reduction for high-dimensional feature data, the NMIFS method is used to select the best feature combination from a given set of sample features. Then, the best combination of feature vectors are sent to the QWNN classifier for learning and training in the training phase, and the anomaly detection model is obtained. At the detection stage, the data is fed into the detection model and ultimately generates accurate detection results. The learning algorithm of structural risk minimization extreme learning machine is employed by the QWNN classifier to account for empirical and confidence risk. The experimental results on real abnormal data demonstrate that the NMIFS–QWNN method has higher detection accuracy and a lower false negative rate than the existing common anomaly detection methods. Furthermore, the complexity of the algorithm is low and the detection accuracy can reach up to 95.8%.

Maximum relevance minimum common redundancy feature selection for nonlinear data

In recent years, feature selection based on relevance redundancy trade-off criteria has become a very promising and popular approach in the field of machine learning. However, the existing algorithmic frameworks of mutual information feature selection have certain limitations for the common feature selection problems in practice. To overcome these limitations, the idea of a new framework is developed by introducing a novel maximum relevance and minimum common redundancy criterion and a minimax nonlinear optimization approach. In particular, a novel mutual information feature selection method based on the normalization of the maximum relevance and minimum common redundancy (N-MRMCR-MI) is presented, which produces a normalized value in the range [0, 1] and results in a regression problem. We perform extensive experimental comparisons over numerous state-of-art algorithms using different forecasts (Bayesian Additive Regression tree, treed Gaussian process, k-NN, and SVM) and different data sets (two simulated and five real datasets). The results show that the proposed algorithm outperforms the others in terms of feature selection and forecasting accuracy.

Quadratic Mutual Information Feature Selection

We propose a novel feature selection method based on quadratic mutual information which has its roots in Cauchy–Schwarz divergence and Renyi entropy. The method uses the direct estimation of quadratic mutual information from data samples using Gaussian kernel functions, and can detect second order non-linear relations. Its main advantages are: (i) unified analysis of discrete and continuous data, excluding any discretization; and (ii) its parameter-free design. The effectiveness of the proposed method is demonstrated through an extensive comparison with mutual information feature selection (MIFS), minimum redundancy maximum relevance (MRMR), and joint mutual information (JMI) on classification and regression problem domains. The experiments show that proposed method performs comparably to the other methods when applied to classification problems, except it is considerably faster. In the case of regression, it compares favourably to the others, but is slower.

A Computer-Aided Diagnosis System With EEG Based on the P3b Wave During an Auditory Odd-Ball Task in Schizophrenia

To design a Computer-aided diagnosis (CAD) system using an optimized methodology over the P3b wave in order to objectively and accurately discriminate between healthy controls (HC) and schizophrenic subjects (SZ). We train, test, analyze, and compare various machine learning classification approaches optimized in terms of the correct classification rate (CCR), the degenerated Youden's index (DYI) and the area under the receiver operating curve (AUC). CAD system comprises five stages: electroencephalography (EEG) preprocessing, feature extraction, seven electrode groupings, discriminant feature selection, and binary classification. With two optimal combinations of electrode grouping, filtering, feature selection algorithm, and classification machine, we get either a mean CCR = 93.42%, specificity = 0.9673, sensitivity = 0.8727, DYI = 0.9188, and AUC = 0.9567 (total-15Hz-J5-MLP), or a mean CCR = 92.23%, specificity = 0.9499, sensitivity = 0.8838, DYI = 0.9162, and AUC = 0.9807 (right hemisphere-35Hz-J5-SVM), which to our knowledge are higher than those available to date. We have verified that a more restrictive low-pass filtering achieves higher CCR as compared to others at higher frequencies in the P3b wave. In addition, results validate previous hypothesis about the importance of the parietal-temporal region, associated with memory processing, allowing us to identify powerful {feature,electrode} pairs in the diagnosis of schizophrenia, achieving higher CCR and AUC in classification of both right and left Hemispheres, and parietal-temporal EEG signals, like, for instance, the {PSE, P4} pair (J5 and mutual information feature selection). Diagnosis of schizophrenia is made thoroughly by psychiatrists but as any human-based decision that has a subjective component. This CAD system provides the human expert with an objective complimentary measure to help him in diagnosing schizophrenia.

Classifying syndromes in Chinese medicine using multi-label learning algorithm with relevant features for each label.

To develop an effective Chinese Medicine (CM) diagnostic model of coronary heart disease (CHD) and to confifirm the scientifific validity of CM theoretical basis from an algorithmic viewpoint. Four types of objective diagnostic data were collected from 835 CHD patients by using a self-developed CM inquiry scale for the diagnosis of heart problems, a tongue diagnosis instrument, a ZBOX-I pulse digital collection instrument, and the sound of an attending acquisition system. These diagnostic data was analyzed and a CM diagnostic model was established using a multi-label learning algorithm (REAL). REAL was employed to establish a Xin (Heart) qi defificiency, Xin yang defificiency, Xin yin defificiency, blood stasis, and phlegm fifive-card CM diagnostic model, which had recognition rates of 80.32%, 89.77%, 84.93%, 85.37%, and 69.90%, respectively. The multi-label learning method established using four diagnostic models based on mutual information feature selection yielded good recognition results. The characteristic model parameters were selected by maximizing the mutual information for each card type. The four diagnostic methods used to obtain information in CM, i.e., observation, auscultation and olfaction, inquiry, and pulse diagnosis, can be characterized by these parameters, which is consistent with CM theory.

Mutual information for feature selection: estimation or counting?

In classification, feature selection is an important pre-processing step to simplify the dataset and improve the data representation quality, which makes classifiers become better, easier to train, and understand. Because of an ability to analyse non-linear interactions between features, mutual information has been widely applied to feature selection. Along with counting approaches, a traditional way to calculate mutual information, many mutual information estimations have been proposed to allow mutual information to work directly on continuous datasets. This work focuses on comparing the effect of counting approach and kernel density estimation (KDE) approach in feature selection using particle swarm optimisation as a search mechanism. The experimental results on 15 different datasets show that KDE can work well on both continuous and discrete datasets. In addition, feature subsets evolved by KDE achieves similar or better classification performance than the counting approach. Furthermore, the results on artificial datasets with various interactions show that KDE is able to capture correctly the interaction between features, in both relevance and redundancy, which can not be achieved by using the counting approach.

Band selection by mutual information for hyper-spectral image classification

Band selection is one of the most important problems in hyper-spectral image classification. Indeed, the presence of irrelevant and/or redundant bands can harm the performance of classification accuracy. This paper investigates the effectiveness of four mutual information feature selector MIFS algorithms to select the informative bands for hyper-spectral image classification. These algorithms are: MIFS, MIFS-U, MIFS-U2 and NMIFS. Our main motivation behind the study of this family algorithm is due to the fact that mutual information MI is an effective indicator to measure the overall statistical dependency between variables and it has proved its efficiency in many pattern recognition problems, especially in remote sensing. The experimental results have been made on two AVIRIS hyper-spectral datasets Indian Pines and Salinas and prove that MIFS algorithm and its variants give promising performances, in terms of dimensionality reduction and classification accuracy than MI-est method Guo et al., 2006, specially for high dimensional data with many irrelevant and/or redundant bands.

Scalable Information Gain Variant on Spark Cluster for Rapid Quantification of Microarray

Microarray technology is one of the emerging technologies in the field of genetic research, which many researchers often use to monitor expression levels of genes in a given organism. Microarray experiments have wide range of applications in health care sector. The colossal amount of raw gene expression data often leads to computational and analytical challenges including feature selection and classification of the dataset into correct group or class. In this paper, mutual information feature selection method based on spark framework (sf-MIFS) is proposed to determine the pertinent features. After completion of feature selection process, various classifiers i.e., Logistic Regression (sf-LoR) and Naive Bayes (sf-NB) based on Spark framework has been applied to classify the microarray datasets. A detailed comparative analysis in terms of execution time and accuracy is enumerated on the proposed feature selection and classifier methodologies, based on Spark framework and conventional system respectively.

Forecasting electricity load with advanced wavelet neural networks

Electricity load forecasting is a key task in the planning and operation of power systems and electricity markets, and its importance increases with the advent of smart grids. In this paper, we present AWNN, a new approach for very short-term load forecasting. AWNN decomposes the complex electricity load data into components with different frequencies that are predicted separately. It uses an advanced wavelet transform with entropy cost function to select the best wavelet basis for data decomposition, mutual information for feature selection and neural networks for prediction. The performance of AWNN is comprehensively evaluated using Australian and Spanish electricity load data for one-step and multi-step ahead predictions, and compared with a number of benchmark algorithms and baselines.

Radiomic Machine-Learning Classifiers for Prognostic Biomarkers of Head and Neck Cancer.

"Radiomics" extracts and mines a large number of medical imaging features in a non-invasive and cost-effective way. The underlying assumption of radiomics is that these imaging features quantify phenotypic characteristics of an entire tumor. In order to enhance applicability of radiomics in clinical oncology, highly accurate and reliable machine-learning approaches are required. In this radiomic study, 13 feature selection methods and 11 machine-learning classification methods were evaluated in terms of their performance and stability for predicting overall survival in head and neck cancer patients. Two independent head and neck cancer cohorts were investigated. Training cohort HN1 consisted of 101 head and neck cancer patients. Cohort HN2 (n = 95) was used for validation. A total of 440 radiomic features were extracted from the segmented tumor regions in CT images. Feature selection and classification methods were compared using an unbiased evaluation framework. We observed that the three feature selection methods minimum redundancy maximum relevance (AUC = 0.69, Stability = 0.66), mutual information feature selection (AUC = 0.66, Stability = 0.69), and conditional infomax feature extraction (AUC = 0.68, Stability = 0.7) had high prognostic performance and stability. The three classifiers BY (AUC = 0.67, RSD = 11.28), RF (AUC = 0.61, RSD = 7.36), and NN (AUC = 0.62, RSD = 10.52) also showed high prognostic performance and stability. Analysis investigating performance variability indicated that the choice of classification method is the major factor driving the performance variation (29.02% of total variance). Our study identified prognostic and reliable machine-learning methods for the prediction of overall survival of head and neck cancer patients. Identification of optimal machine-learning methods for radiomics-based prognostic analyses could broaden the scope of radiomics in precision oncology and cancer care.

Hybrid of binary gravitational search algorithm and mutual information for feature selection in intrusion detection systems

Intrusion detection systems (IDSs) play an important role in the security of computer networks. One of the main challenges in IDSs is the high-dimensional input data analysis. Feature selection is a solution to overcoming this problem. This paper presents a hybrid feature selection method using binary gravitational search algorithm (BGSA) and mutual information (MI) for improving the efficiency of standard BGSA as a feature selection algorithm. The proposed method, called MI-BGSA, used BGSA as a wrapper-based feature selection method for performing global search. Moreover, MI approach was integrated into the BGSA, as a filter-based method, to compute the feature---feature and the feature---class mutual information with the aim of pruning the subset of features. This strategy found the features considering the least redundancy to the selected features and also the most relevance to the target class. A two-objective function based on maximizing the detection rate and minimizing the false positive rate was defined as a fitness function to control the search direction of the standard BGSA. The experimental results on the NSL-KDD dataset showed that the proposed method can reduce the feature space dramatically. Moreover, the proposed algorithm found better subset of features and achieved higher accuracy and detection rate as compared to the some standard wrapper-based and filter-based feature selection methods.

Classification of radiology reports for falls in an HIV study cohort.

To identify patients in a human immunodeficiency virus (HIV) study cohort who have fallen by applying supervised machine learning methods to radiology reports of the cohort. We used the Veterans Aging Cohort Study Virtual Cohort (VACS-VC), an electronic health record-based cohort of 146 530 veterans for whom radiology reports were available (N=2 977 739). We created a reference standard of radiology reports, represented each report by a feature set of words and Unified Medical Language System concepts, and then developed several support vector machine (SVM) classifiers for falls. We compared mutual information (MI) ranking and embedded feature selection approaches. The SVM classifier with MI feature selection was chosen to classify all radiology reports in VACS-VC. Our SVM classifier with MI feature selection achieved an area under the curve score of 97.04 on the test set. When applied to all the radiology reports in VACS-VC, 80 416 of these reports were classified as positive for a fall. Of these, 11 484 were associated with a fall-related external cause of injury code (E-code) and 68 932 were not, corresponding to 29 280 patients with potential fall-related injuries who could not have been found using E-codes. Feature selection was crucial to improving the classifier's performance. Feature selection with MI allowed us to select the number of discriminative features to use for classification, in contrast to the embedded feature selection method, in which the number of features is chosen automatically. Machine learning is an effective method of identifying patients who have suffered a fall. The development of this classifier supplements the clinical researcher's toolkit and reduces dependence on under-coded structured electronic health record data.

Nondestructive detection of chilling injury in cucumber fruit using hyperspectral imaging with feature selection and supervised classification

Chilling injury, as a physiological disorder in cucumbers, occurs after the fruit has been subjected to low temperatures. It is thus desirable to detect chilling injury at early stages and/or remove chilling injured cucumbers during sorting and grading. This research was aimed to apply hyperspectral imaging technique, combined with feature selection methods and supervised classification algorithms, to detect chilling injury in cucumbers. Hyperspectral reflectance (500–675nm) and transmittance (675–1000nm) images for normal and chilling injured cucumbers were acquired, using an in-house developed online hyperspectral imaging system. Three feature selection methods including mutual information feature selection (MIFS), max-relevance min-redundancy (MRMR), and sequential forward selection (SFS) were used and compared for optimal wavebands selection. Supervised classifications with naïve Bayes (NB), support vector machine (SVM), and k-nearest neighbor (KNN) were then implemented for the two-class (i.e., normal and chilling) and three-class (i.e., normal, lightly chilling, and severely chilling) classifications based on the spectral and image analysis at selected two-band ratios. It was found that the majority of the optimal wavebands selected by MIFS, MRMR, and SFS for both two-class and three-class classifications were from the spectral transmittance images in the short-near infrared region. The SFS feature selection method together with the SVM classifier resulted in the best overall classification accuracy of 100%, and the overall accuracy of 90.5% for the three-class classification, based on the spectral analysis. The classification results based on the textural features (first-order statistics and second-order statistics features) extracted from the optimal two-band ratio images were comparable to those achieved using the spectral features, with the best overall accuracies of 100% and 91.6% for the two-class and the three class classifications, respectively. These results demonstrated the potential of hyperspectral imaging technique for online detection of chilling injury in cucumbers.

Railcar Bogie Performance Monitoring using Mutual Information and Support Vector Machines


 
 
 Railcar condition monitoring is an area of high importance and global relevance. The economic and safety concerns of equipment maintenance in North America mandate efforts in prognostics and health management. This paper presents the results from the development of a vibration based condition monitoring algorithm for freight rail, utilizing mutual information feature selection and support vector machine classification of bogie component faults. The algorithm is an implementation of a previously proposed railcar condition monitoring solution by the authors. The proposed monitoring solution is a data-driven method which was developed with measurements taken at a railroad test laboratory under controlled conditions. Vibration data was collected from multiple locations on a railcar over several test runs, each utilizing wheelsets with different levels of wear. The input of controlled wheel wear levels was aimed at varying the system outputs to resemble those of cars with different levels of mileage in revenue service. The generated data sets were processed and a feature set was extracted from the acceleration signals. The data was divided into training and validation partitions using a cross validation scheme to preserve the sequence for both sets. A mutual information (MI) estimation algorithm was used to rank the features based on their similarity to the classified fault state. Both the optimized feature set from the MI feature selection algorithm as well as the full, non-discriminate feature set were used as inputs to the support vector machine to assess classification accuracy. The results of this assessment are presented in the paper along with a presentation of the methods. The paper concludes with a proposal for a monitoring strategy aimed at specifically detecting faulty components and practicing predictive maintenance.
 
 

Borsa Istanbul (BIST) daily prediction using financial news and balanced feature selection

In this paper, a novel method is proposed to predict the direction of Borsa Istanbul (BIST) 100 Index (BIST100) open prices using the news articles released, as well as the price data, from the day before. Although English news articles have been used for market-prediction before, to the best of our knowledge, Turkish news articles together with prices have not yet been used to predict the Turkish markets. Turkish text mining techniques are applied on news articles to form feature vectors for each trading day. The feature vectors are assigned three labels based on the direction of the price change from the closing price of the day before and whether the change is significant. News articles are represented using high dimensional features, some of which could be noisy or irrelevant for prediction. There is also the scarcity of training data. Therefore, this study incorporates feature selection methods to select features that could improve classification performance. By its nature, significant positive or negative changes in stock price happen much less than non-significant changes, resulting in an imbalanced data set. Most feature selection methods in literature aim to reduce the classification accuracy. However, for imbalanced datasets, other measures, such as macro-averaged F-measure need to be considered. The paper proposes a feature selection methods that is able to deal with the class imbalance problem through oversampling of the minority classes and consideration of an ensemble of selected features. In order to decide on importance of features, as the relevance criterion for each feature, the proposed methodology uses mutual information which can detect nonlinear dependencies between variables. Therefore, the proposed feature selection method is called Balanced Mutual Information (BMI) feature selection method. Experiments were performed based on news articles provided by two different news sources: Public Disclosure Platform of BIST and financial news websites. It was shown that, using Balanced Mutual Information feature selection method, the significant changes in the BIST100 Index were predicted with an accuracy of 0.74 and a macro-averaged F-measure of 0.68. The BMI feature selection method was compared with Mutual Information and Chi-square based feature selection methods and it was found out that BMI method results in higher performance using a smaller number of features.