Mutual Information-based Feature Selection Method Research Articles

Accuracy of machine learning techniques to predict stress echocardiography results using clinical variables

Abstract Background Stress echocardiography (SE) is a diagnostic prognostic imaging modality for suspected coronary artery disease (CAD). Cardiovascular risk factors (CVRF) are used in the assessment of the probability of CAD. The link between the outcome of SE and patients’ variables including CVRF, current medication, and anthropometric variables has not been widely investigated.Objective: The aim of this study is to understand whether a machine learned algorithm can be used to predict significant CAD defined by inducible ischemia on SE in patients with suspected CAD based on anthropometrics, CVRF, and medication as variables. Methods A machine learning (ML) framework was generated to automate the prediction of inducible ischemia on stress echocardiography results. A mutual information-based feature selection methods (support vector machine (SVM) and random forest (RF) classifiers) were used to investigate the amount of information that each feature carried to define the positive outcome of SE.Data from 2201 patients were used to train and validate the framework. Patients mean age was 62 (SD 11) years. The data consists of anthropological data and classical CVRF, atrial fibrillation, prior diagnosis of CAD, stroke, obstructive airways disease, chronic inflammatory disease, chronic kidney disease (CKD>3) and prescribed medications at the time of the test. There were 394 positive and 1807 negative SE cases. Five-fold cross-validation was used to validate the performance of the framework. Results The feature selection methods showed that prescribed medications such as Weight, Diabetes, Antiplatelet drugs and ACE- inhibitor/ARB, were the features that shared the most information about the outcome of SE (sensitivity and specificity of 81% and 51% and 81% and 61% for SVM and RF, respectively) with the accuracy of 82%. Conclusion This study shows that ML can predict the outcome of SE based on only 4 clinical features. Further research correlating the clinical variable and SE result to cardiovascular outcome will improve the model’s clinical utility and facilitate patient selection for early treatment avoiding unnecessary downstream testing.

Local causal structure learning with missing data

Local causal structure learning aims to discover and distinguish the direct causes and direct effects of a target variable. However, the state-of-the-art algorithms for local causal structure learning fail to perform well when dealing with missing data. The general approach is to fill in the missing data using imputation techniques before learning the local causal structure, but this method suffers from problems such as low accuracy, low efficiency, and instability. To address these issues, we propose a novel method for local causal structure learning with missing data, named misLCS. Firstly, we design an iterative data imputation method to obtain the complete and correct data from the missing data. Then, misLCS adopts a data subset strategy to get a data subset that variables are closely related to the target variable. Thirdly, within this data subset, misLCS constructs the local causal skeleton of the target variable using a mutual information-based feature selection method and orients the direction of edges using conditional independence tests and Meek rules. Finally, misLCS updates the missing data in preparation for the next iteration. This procedure continues until the direct causes and direct effects of the target variable have been identified. Our experiments on seven benchmark Bayesian networks and a real-world bioinformatics dataset, with a number of variables from 11 to 801, demonstrate that our algorithm achieves better accuracy than the existing local causal structure learning algorithms.

Classification and feature selection methods based on fitting logistic regression to PU data

In our work, we examine the classification methods where the positive and unlabeled data are considered and where the conditional distribution of the true class label given the feature vector is governed by the model of logistic regression. Our first objective is to compute and compare the selected metrics allowing for the quality assessment of these methods. In this context, we investigate four methods of the posterior probability estimation, where the risk of logistic loss function is optimized: the naive approach, the weighted likelihood approach, as well as the quite recently proposed methods – the joint approach, and the LassoJoint method. The corresponding evaluations are basically performed for 13 machine learning models on some chosen – both low- and high-dimensional – datasets. Some of the mentioned machine learning model schemes have been directly borrowed from literature and some have been obtained through some modifications in the existing procedures. Our second goal is to establish the most stable and efficient approach for the posterior probability estimation. Moreover, we use the AdaSampling scheme for comparison of the considered classification methods. We also conducted comparisons of feature selection procedures – the Mutual Information-Based feature selection method and the LassoJoint approach. The current article is an enhancement of the conference paper Furmańczyk et al. (2022).

Identify Best Learning Method for Heart Diseases Prediction Under impact of Different Datasets Characteristics

This paper introduces an experimental study of the heart disease datasets characteristics impact on the performance of classification algorithms in the aim of identifying the best algorithm for each dataset under its characteristics. The performance of five machine learning algorithms (logistic regression (LR), K-Nearest Neighbor (KNN), Decision tree (DT), Random Forest (RF), and support vector machine (SVM)), single layer neural network (ANN), and deep neural network (DNN), has been evaluated using five heart disease datasets under four data complexity measurement: number of samples (dataset size), number of features (dimension of dataset), Data sparsity measures, and correlation of features. All datasets have been processed and normalized then the mutual information-based feature selection method was used to solve the overfitting problem. The results show that in general, the machine learning especially the Random Forest algorithm achieves high classification accuracy than deep learning network. In other hand, the high sparsity and less mutual information of dataset has large impact on degradation of the performance of classification algorithms than other characteristics of data.

Detection of Eucalyptus Leaf Disease with UAV Multispectral Imagery

Forest disease is one of the most important factors affecting tree growth and product quality, reducing economic values of forest ecosystem goods and services. In order to prevent and control forest diseases, accurate detection in a timely manner is essential. Unmanned aerial vehicles (UAVs) are becoming an important tool for acquiring multispectral imagery, but have not been extensively used for detection of forest diseases. This research project selected a eucalyptus forest as a case study to explore the performance of leaf disease detection using high spatial resolution multispectral imagery that had been acquired by UAVs. The key variables sensitive to eucalyptus leaf diseases, including spectral bands and vegetation indices, were identified by using a mutual information–based feature selection method, then distinguishing disease levels using random forest and spectral angle mapper approaches. The results show that green, red edge, and near-infrared wavelengths, nitrogen reflectance index, and greenness index are sensitive to forest diseases. The random forest classifier, based on a combination of sensitive spectral bands (green, red edge, and near-infrared wavelengths) and a nitrogen reflectance index, provided the best differentiation results for healthy and three disease severity levels (mild, moderate, and severe) with overall accuracy of 90.1% and kappa coefficient of 0.87. This research provides a new way to detect eucalyptus leaf diseases, and the proposed method may be suitable for other forest types.

Chemoinformatics and Machine Learning Approaches for Identifying Antiviral Compounds

Current pandemics propelled research efforts in unprecedented fashion, primarily triggering computational efforts towards new vaccine and drug development as well as drug repurposing. There is an urgent need to design novel drugs with targeted biological activity and minimum adverse reactions that may be useful to manage viral outbreaks. Hence an attempt has been made to develop Machine Learning based predictive models that can be used to assess whether a compound has the potency to be antiviral or not. To this end, a set of 2358 antiviral compounds were compiled from the CAS COVID-19 antiviral SAR dataset whose activity was reported based on IC50 value. A total 1157 two-dimensional molecular descriptors were computed among which, the most highly correlated descriptors were selected using Tree-based, Correlation-based and Mutual information-based feature selection methods. Seven Machine Learning algorithms i. e., Random Forest, XGBoost, Support Vector Machine, KNN, Decision Tree, MLP Classifier and Logistic Regression were benchmarked. The best performance was achieved by the models developed using Random Forest and XGBoost algorithms in all the feature selection methods. The maximum predictive accuracy of both these models was 88 % with internal validation. Whereas, with an external dataset, a maximum accuracy of 93.10 % for XGBoost and 100 % for Random Forest based model was achievable. Furthermore, the study demonstrated scaffold analysis of the molecules as a pragmatic approach to explore the importance of structurally diverse compounds in data driven studies.

Hand Gesture Recognition Using Multiple Acoustic Measurements at Wrist

This article investigates the use of acoustic signals recorded at the human wrist for hand gesture recognition. The prototype consists of 40 microphones to be worn at the wrist. The gesture recognition performance is evaluated through the identification of 36 gestures in American sign language (ASL), including 26 ASL alphabetical characters and 10 ASL numbers. The optimal area for sensor band placement (distal/proximal) is examined to reveal the location of the highest discrimination accuracy. Ten subjects are recruited to perform over ten trials for each set of hand gestures. Using mutual information-based feature selection methods, two time-domain features: difference absolute mean value and log-energy entropy, is selected from 35 commonly used time-domain features for the hand gesture classification. As a proof-of-concept, two common classification methods, linear discriminant analysis and support vector machine, is used to classify hand gestures from the acoustic measurements recorded by each subject. Results show the intrasubject average classification accuracy above 90% using the two features with all 40 microphones, while the average classification accuracy exceeding 84% is obtained using ten microphones. These results indicate that acoustic signatures from the human wrist can be utilized for hand gesture recognition, while the use of few, simple features, with low computational requirements is sufficient to characterize some hand gestures. The proposed technique demonstrates a promising means for developing a low-cost wearable hand gesture recognition device using microphones.

Minimal Patient Clinical Variables to Accurately Predict Stress Echocardiography Outcome: Validation Study Using Machine Learning Techniques.

BackgroundStress echocardiography is a well-established diagnostic tool for suspected coronary artery disease (CAD). Cardiovascular risk factors are used in the assessment of the probability of CAD. The link between the outcome of stress echocardiography and patients’ variables including risk factors, current medication, and anthropometric variables has not been widely investigated.ObjectiveThis study aimed to use machine learning to predict significant CAD defined by positive stress echocardiography results in patients with chest pain based on anthropometrics, cardiovascular risk factors, and medication as variables. This could allow clinical prioritization of patients with likely prediction of CAD, thus saving clinician time and improving outcomes.MethodsA machine learning framework was proposed to automate the prediction of stress echocardiography results. The framework consisted of four stages: feature extraction, preprocessing, feature selection, and classification stage. A mutual information–based feature selection method was used to investigate the amount of information that each feature carried to define the positive outcome of stress echocardiography. Two classification algorithms, support vector machine (SVM) and random forest classifiers, have been deployed. Data from 529 patients were used to train and validate the framework. Patient mean age was 61 (SD 12) years. The data consists of anthropological data and cardiovascular risk factors such as gender, age, weight, family history, diabetes, smoking history, hypertension, hypercholesterolemia, prior diagnosis of CAD, and prescribed medications at the time of the test. There were 82 positive (abnormal) and 447 negative (normal) stress echocardiography results. The framework was evaluated using the whole dataset including cases with prior diagnosis of CAD. Five-fold cross-validation was used to validate the performance of the framework. We also investigated the model in the subset of patients with no prior CAD.ResultsThe feature selection methods showed that prior diagnosis of CAD, sex, and prescribed medications such as angiotensin-converting enzyme inhibitor/angiotensin receptor blocker were the features that shared the most information about the outcome of stress echocardiography. SVM classifiers showed the best trade-off between sensitivity and specificity and was achieved with three features. Using only these three features, we achieved an accuracy of 67.63% with sensitivity and specificity 72.87% and 66.67% respectively. However, for patients with no prior diagnosis of CAD, only two features (sex and angiotensin-converting enzyme inhibitor/angiotensin receptor blocker use) were needed to achieve accuracy of 70.32% with sensitivity and specificity at 70.24%.ConclusionsThis study shows that machine learning can predict the outcome of stress echocardiography based on only a few features: patient prior cardiac history, gender, and prescribed medication. Further research recruiting higher number of patients who underwent stress echocardiography could further improve the performance of the proposed algorithm with the potential of facilitating patient selection for early treatment/intervention avoiding unnecessary downstream testing.

Using Vehicle Interior Noise Classification for Monitoring Urban Rail Transit Infrastructure.

This study developed a multi-classification model for vehicle interior noise from the subway system, collected on smartphones. The proposed model has the potential to be used to analyze the causes of abnormal noise using statistical methods and evaluate the effect of rail maintenance work. To this end, first, we developed a multi-source data (audio, acceleration, and angle rate) collection framework via smartphone built-in sensors. Then, considering the Shannon entropy, a 1-second window was selected to segment the time-series signals. This study extracted 45 features from the time- and frequency-domains to establish the classifier. Next, we investigated the effects of balancing the training dataset with the Synthetic Minority Oversampling Technique (SMOTE). By comparing and analyzing the classification results of importance-based and mutual information-based feature selection methods, the study employed a feature set consisting of the top 10 features by importance score. Comparisons with other classifiers indicated that the proposed XGBoost-based classifier runs fast while maintaining good accuracy. Finally, case studies were provided to extend the applications of this classifier to the analysis of abnormal vehicle interior noise events and evaluate the effects of rail grinding.

Predicting Lung Cancer Patients' Survival Time via Logistic Regression-based Models in a Quantitative Radiomic Framework.

Background: Selection of the best treatment modalities for lung cancer depends on many factors, like survival time, which are usually determined by imaging.Objectives: To predict the survival time of lung cancer patients using the advantages of both radiomics and logistic regression-based classification models.Material and Methods: Fifty-nine patients with primary lung adenocarcinoma were included in this retrospective study and pre-treatment contrast-enhanced CT images were acquired. The patients lived more than 2 years were classified as the ‘Alive’ class and otherwise as the ‘Dead’ class. In our proposed quantitative radiomic framework, we first extracted the associated regions of each lung lesion from pre-treatment CT images for each patient via grow cut segmentation algorithm. Then, 40 radiomic features were extracted from the segmented lung lesions. In order to enhance the generalizability of the classification models, the mutual information-based feature selection method was applied to each feature vector. We investigated the performance of six logistic regression-based classification models.Results: It was observed that the mutual information feature selection method can help the classifier to achieve better predictive results. In our study, the Logistic regression (LR) and Dual Coordinate Descent method for Logistic Regression (DCD-LR) models achieved the best results indicating that these classification models have strong potential for classifying the more important class (i.e., the ‘Alive’ class). Conclusion: The proposed quantitative radiomic framework yielded promising results, which can guide physicians to make better and more precise decisions and increase the chance of treatment success.

An isomiR expression panel based novel breast cancer classification approach using improved mutual information

BackgroundGene expression-based profiling has been used to identify biomarkers for different breast cancer subtypes. However, this technique has many limitations. IsomiRs are isoforms of miRNAs that have critical roles in many biological processes and have been successfully used to distinguish various cancer types. Biomarker isomiRs for identifying different breast cancer subtypes has not been investigated. For the first time, we aim to show that isomiRs are better performing biomarkers and use them to explain molecular differences between breast cancer subtypes.ResultsIn this study, a novel method is proposed to identify specific isomiRs that faithfully classify breast cancer subtypes. First, as a null hypothesis method we removed the lowly expressed isomiRs from small sequencing data generated from diverse breast cancers types. Second, we developed an improved mutual information-based feature selection method to calculate the weight of each isomiR expression. The weight of isomiR measures the importance of a given isomiR in classifying breast cancer subtypes. The improved mutual information enables to apply the dataset in which the feature is continuous data and label is discrete data; whereby, the traditional mutual information cannot be applied in this dataset. Finally, the support vector machine (SVM) classifier is applied to find isomiR biomarkers for subtyping.ConclusionsHere we demonstrate that isomiRs can be used as biomarkers in the identification of different breast cancer subtypes, and in addition, they may provide new insights into the diverse molecular mechanisms of breast cancers. We have also shown that the classification of different subtypes of breast cancer based on isomiRs expression is more effective than using published gene expression profiling. The proposed method provides a better performance outcome than Fisher method and Hellinger method for discovering biomarkers to distinguish different breast cancer subtypes. This novel technique could be directly applied to identify biomarkers in other diseases.

A Data Fusion-Based Hybrid Sensory System for Older People’s Daily Activity and Daily Routine Recognition

Sensor-based human activity recognition (HAR) has received considerable attention due to its wide applications in health care. Each sensor modality has its advantages and limitations. Single sensor modalities sometimes may not cope with complex situations in practice. To resolve this challenge, we design and develop a practical hybrid sensory HAR system for older people. To enhance the performance of the system, we propose a unique data fusion method through combining both wearable sensors and ambient sensors. The wearable sensors in this paper are used for identifying the specific daily activities. The ambient sensors delivering the occupant’s room-level daily routine provide a more comprehensive surveillance with the wearable sensors together; meanwhile, the captured room-level location information is also used in the data fusion to trigger the sub classification models pretrained by wearable data. We also explore a new feature set extracted from wearable sensors to improve the system performance. We experimentally evaluate our system by applying four typical mutual information-based feature selection methods and the support vector machines classification algorithm instead of other complex algorithms, with the aim of exploring a practical way to improve recognition accuracy. The ground-truth data are gathered from 21 subjects, including 17 daily activities with the sample size of 2,142,000. The experimental results demonstrate the effectiveness of our method. The new feature set help improve the accuracy to 96.82% ± 0.15 from 89.81% ± 0.54 using wearable data only; and the data fusion with ambient information achieves a further increased accuracy of 98.32%.

An Improved Combination of Spectral and Spatial Features for Vegetation Classification in Hyperspectral Images

Due to the advances in hyperspectral sensor technology, hyperspectral images have gained great attention in precision agriculture. In practical applications, vegetation classification is usually required to be conducted first and then the vegetation of interest is discriminated from the others. This study proposes an integrated scheme (SpeSpaVS_ClassPair_ScatterMatrix) for vegetation classification by simultaneously exploiting image spectral and spatial information to improve vegetation classification accuracy. In the scheme, spectral features are selected by the proposed scatter-matrix-based feature selection method (ClassPair_ScatterMatrix). In this method, the scatter-matrix-based class separability measure is calculated for each pair of classes and then averaged as final selection criterion to alleviate the problem of mutual redundancy among the selected features, based on the conventional scatter-matrix-based class separability measure (AllClass_ScatterMatrix). The feature subset search is performed by the sequential floating forward search method. Considering the high spectral similarity among different green vegetation types, Gabor features are extracted from the top two principal components to provide complementary spatial features for spectral features. The spectral features and Gabor features are stacked into a feature vector and then the ClassPair_ScatterMatrix method is used on the formed vector to overcome the over-dimensionality problem and select discriminative features for vegetation classification. The final features are fed into support vector machine classifier for classification. To verify whether the ClassPair_ScatterMatrix method could well avoid selecting mutually redundant features, the mean square correlation coefficients were calculated for the ClassPair_ScatterMatrix method and AllClass_ScatterMatrix method. The experiments were conducted on a widely used agricultural hyperspectral image. The experimental results showed that (1) the The proposed ClassPair_ScatterMatrix method could better alleviate the problem of selecting mutually redundant features, compared to the AllClass_ScatterMatrix method; (2) compared with the representative mutual information-based feature selection methods, the scatter-matrix-based feature selection methods generally achieved higher classification accuracies, and the ClassPair_ScatterMatrix method especially, produced the highest classification accuracies with respect to both data sets (87.2% and 90.1%); and (3) the proposed integrated scheme produced higher classification accuracy, compared with the decision fusion of spectral and spatial features and the methods only involving spectral features or spatial features. The comparative experiments demonstrate the effectiveness of the proposed scheme.

A Fuzzy Mutual Information-based Feature Selection Method for Classification

In this paper, we present a feature selection method called Fuzzy Mutual Information-based Feature Selection with Non-Dominated solution (FMIFS-ND) using a fuzzy mutual information measure which selects features based on feature-class fuzzy mutual information and feature-feature fuzzy mutual information. To evaluate classification accuracy of the proposed method, a modification of the k-nearest neighbor (KNN) classifier is also presented in this paper to classify instances based on the distance or similarity between individual features. The performance of both methods is evaluated on multiple UCI datasets by using four classifiers. We compare the accuracy of our feature selection method with existing feature selection methods and validate accuracy of the proposed classifier with decision trees, random forests, naive Bayes, KNN and support vector machines (SVM). Experimental results show that the feature selection method gives high classification accuracy in most high dimensional datasets as well as the accuracy of proposed classifiers outperforms the traditional KNN classifier.

Genetic differences among ethnic groups.

BackgroundMany differences between different ethnic groups have been observed, such as skin color, eye color, height, susceptibility to some diseases, and response to certain drugs. However, the genetic bases of such differences have been under-investigated. Since the HapMap project, large-scale genotype data from Caucasian, African and Asian population samples have been available. The project found that these populations were located in different areas of the PCA (Principal Component Analysis) plot. However, as an unsupervised method, PCA does not measure the differences in each single nucleotide polymorphism (SNP) among populations.ResultsWe applied an advanced mutual information-based feature selection method to detect associations between SNP status and ethnic groups using the latest HapMap Phase 3 release version 3, which included more sub-populations. A total of 299 SNPs were identified, and they can accurately predicted the ethnicity of all HapMap populations. The 10-fold cross validation accuracy of the SMO (sequential minimal optimization) model on training dataset was 0.901, and the accuracy on independent test dataset was 0.895.ConclusionsIn-depth functional analysis of these SNPs and their nearby genes revealed the genetic bases of skin and eye color differences among populations.Electronic supplementary materialThe online version of this article (doi:10.1186/s12864-015-2328-0) contains supplementary material, which is available to authorized users.

Can high-order dependencies improve mutual information based feature selection?

Mutual information (MI) based approaches are a popular paradigm for feature selection. Most previous methods have made use of low-dimensional MI quantities that are only effective at detecting low-order dependencies between variables. Several works have considered the use of higher dimensional mutual information, but the theoretical underpinning of these approaches is not yet comprehensive. To fill this gap, in this paper, we systematically investigate the issues of employing high-order dependencies for mutual information based feature selection. We first identify a set of assumptions under which the original high-dimensional mutual information based criterion can be decomposed into a set of low-dimensional MI quantities. By relaxing these assumptions, we arrive at a principled approach for constructing higher dimensional MI based feature selection methods that takes into account higher order feature interactions. Our extensive experimental evaluation on real data sets provides concrete evidence that methodological inclusion of high-order dependencies improve MI based feature selection.

Modeling Difficult-to-Measure Process Parameters Based on Intrinsic Mode Functions Frequency Spectral Features of Mechanical Vibration and Acoustical Signals

Heavy mechanical devices of complex industrial process produce soundly mechanical vibration and acoustical signals. Some difficult-to-measure key process parameters have direct relationship with these signals. A newly ensemble empirical mode decomposition (EEMD), Fast Fourier Transform (FFT), Mutual information (MI), and Kernel partial least squares (KPLS) based modeling approach is proposed to measure these process parameters. At first, different scale intrinsic mode functions (IMFs) of mechanical vibration and acoustical signals are obtained using EEMD technology. Then, FFT transforms these multi-scale IMFs into frequency domain, and MI based feature selection method selects interesting frequency spectral features. Finally, KPLS constructs the final soft sensor models using the selected features. Experimental results based on vibration and acoustical signals of ball mill demonstrate this approach is more effective than other exist multi-scale decomposition based methods.

MIFS-ND: A mutual information-based feature selection method

Feature selection is used to choose a subset of relevant features for effective classification of data. In high dimensional data classification, the performance of a classifier often depends on the feature subset used for classification. In this paper, we introduce a greedy feature selection method using mutual information. This method combines both feature–feature mutual information and feature–class mutual information to find an optimal subset of features to minimize redundancy and to maximize relevance among features. The effectiveness of the selected feature subset is evaluated using multiple classifiers on multiple datasets. The performance of our method both in terms of classification accuracy and execution time performance, has been found significantly high for twelve real-life datasets of varied dimensionality and number of instances when compared with several competing feature selection techniques.

Mutual information-based method for selecting informative feature sets

Feature selection is one of the fundamental problems in pattern recognition and data mining. A popular and effective approach to feature selection is based on information theory, namely the mutual information of features and class variable. In this paper we compare eight different mutual information-based feature selection methods. Based on the analysis of the comparison results, we propose a new mutual information-based feature selection method. By taking into account both the class-dependent and class-independent correlation among features, the proposed method selects a less redundant and more informative set of features. The advantage of the proposed method over other methods is demonstrated by the results of experiments on UCI datasets (Asuncion and Newman, 2010 [1]) and object recognition.

Mutual information-based feature selection approach for software defect prediction

Predicting defect-prone software modules accurately and effectively is an important way to control the quality of a software system during software development.Feature selection can highly improve the accuracy and efficiency of the software defect prediction model.A mutual information-based feature selection method for software defect prediction was proposed.The optimal feature subsets generated by the proposed approach were applied to train and test various prediction models.Forward search was introduced into the proposed approach,and nonlinear equilibrium coefficient was also added to the evaluation function.The experimental results show that all the classifiers achieve higher accuracy and performance by using the feature subset provided by the proposed approach.