Mutual Information Based Feature Selection Research Articles

Accuracy of machine learning techniques to predict stress echocardiography results using clinical variables

Abstract Background Stress echocardiography (SE) is a diagnostic prognostic imaging modality for suspected coronary artery disease (CAD). Cardiovascular risk factors (CVRF) are used in the assessment of the probability of CAD. The link between the outcome of SE and patients’ variables including CVRF, current medication, and anthropometric variables has not been widely investigated.Objective: The aim of this study is to understand whether a machine learned algorithm can be used to predict significant CAD defined by inducible ischemia on SE in patients with suspected CAD based on anthropometrics, CVRF, and medication as variables. Methods A machine learning (ML) framework was generated to automate the prediction of inducible ischemia on stress echocardiography results. A mutual information-based feature selection methods (support vector machine (SVM) and random forest (RF) classifiers) were used to investigate the amount of information that each feature carried to define the positive outcome of SE.Data from 2201 patients were used to train and validate the framework. Patients mean age was 62 (SD 11) years. The data consists of anthropological data and classical CVRF, atrial fibrillation, prior diagnosis of CAD, stroke, obstructive airways disease, chronic inflammatory disease, chronic kidney disease (CKD>3) and prescribed medications at the time of the test. There were 394 positive and 1807 negative SE cases. Five-fold cross-validation was used to validate the performance of the framework. Results The feature selection methods showed that prescribed medications such as Weight, Diabetes, Antiplatelet drugs and ACE- inhibitor/ARB, were the features that shared the most information about the outcome of SE (sensitivity and specificity of 81% and 51% and 81% and 61% for SVM and RF, respectively) with the accuracy of 82%. Conclusion This study shows that ML can predict the outcome of SE based on only 4 clinical features. Further research correlating the clinical variable and SE result to cardiovascular outcome will improve the model’s clinical utility and facilitate patient selection for early treatment avoiding unnecessary downstream testing.

Power Quality Disturbances Classification Through Optimal Feature Selective Mechanism

In the Detection and classification of Power Quality Disturbances, feature selection is very important because the more number of features causes more complexity and less number of feature impacts the accuracy. Hence, this paper proposed a new feature selection mechanism called as Flexible Mutual Information based Feature Selection (FMIFS) to represent the Power Quality Disturbances with an effective set of features. FMIFS computes the Mutual Redundancy between PQDs and removes redundant features between them. Once the optimal features are obtained for each PQD, then they are fed to Multi-class Support Vector Machine (MC-SVM) for classification. MC-SVM classifies each PQD based on the feature trained to the system. At experimental analysis, we applied our method on totally 11 types of PQDs for classification and the performance is measured through recall, precision, F1-score, and False Alarm Rate (FAR).

Application of machine learning algorithms and feature selection methods for better prediction of sludge production in a real advanced biological wastewater treatment plant

Although the management of sewage sludge is an important and challenging task of wastewater treatment, there is a scarcity of studies on the prediction of waste sludge. To overcome this deficiency, the present work aims to develop an appropriate model providing accurate and fast prediction of sewage sludge. With this aim, different machine learning (ML) algorithms were tested by data obtained from a real advanced biological wastewater treatment plant located in Kocaeli, Turkey. In modelling studies, a data set from January 2022 to December 2022 composed of 208 daily measurements was considered. The flow rate of the plant (Q), polyelectrolyte dosage (PD) and removed amounts of total suspended solids (TSS), chemical oxygen demand (COD), biological oxygen demand (BOD), total phosphorous (TP), total nitrogen (TN) were assigned as input parameters to predict sludge production (SP). The precision of the models was evaluated in terms of Mean Square Error (MSE), Mean Absolute Percentage Error (MAPE), Root Mean Square Error (RMSE), Mean Absolute Error (MAE), and correlation coefficient (R2). Among the various tested models Kernel Ridge Regression provided the best accuracy with R2 value of 0.94 and MAE value of 3.25. Mutual information-based feature selection (MIFS) and correlation-based feature selection (CFS) algorithms were also used in the study in order to enhance the model performance. Thus, higher prediction accuracies were achieved using the selected subset of features. Furthermore, importance contribution of features were calculated and visualized by SHapley Additive exPlanations (SHAP) technique. The overall results of the work indicate the feasibility of ML models for describing the dynamic and complex nature of SP. The process operators may benefit from this modelling approach since it enables accurate and fast estimation of sewage sludge by using fewer and easily measurable parameters.

Optimizing Multimodal Scene Recognition through Mutual Information-Based Feature Selection in Deep Learning Models

The field of scene recognition, which lies at the crossroads of computer vision and artificial intelligence, has experienced notable progress because of scholarly pursuits. This article introduces a novel methodology for scene recognition by combining convolutional neural networks (CNNs) with feature selection techniques based on mutual information (MI). The main goal of our study is to address the limitations inherent in conventional unimodal methods, with the aim of improving the precision and dependability of scene classification. The focus of our research is around the formulation of a comprehensive approach for scene detection, utilizing multimodal deep learning methodologies implemented on a solitary input image. Our work distinguishes itself by the innovative amalgamation of CNN- and MI-based feature selection. This integration provides distinct advantages and enhanced capabilities when compared to prevailing methodologies. In order to assess the effectiveness of our methodology, we performed tests on two openly accessible datasets, namely, the scene categorization dataset and the AID dataset. The results of these studies exhibited notable levels of precision, with accuracies of 100% and 98.83% achieved for the corresponding datasets. These findings surpass the performance of other established techniques. The primary objective of our end-to-end approach is to reduce complexity and resource requirements, hence creating a robust framework for the task of scene categorization. This work significantly advances the practical application of computer vision in various real-world scenarios, leading to a large improvement in the accuracy of scene recognition and interpretation.

Local causal structure learning with missing data

Local causal structure learning aims to discover and distinguish the direct causes and direct effects of a target variable. However, the state-of-the-art algorithms for local causal structure learning fail to perform well when dealing with missing data. The general approach is to fill in the missing data using imputation techniques before learning the local causal structure, but this method suffers from problems such as low accuracy, low efficiency, and instability. To address these issues, we propose a novel method for local causal structure learning with missing data, named misLCS. Firstly, we design an iterative data imputation method to obtain the complete and correct data from the missing data. Then, misLCS adopts a data subset strategy to get a data subset that variables are closely related to the target variable. Thirdly, within this data subset, misLCS constructs the local causal skeleton of the target variable using a mutual information-based feature selection method and orients the direction of edges using conditional independence tests and Meek rules. Finally, misLCS updates the missing data in preparation for the next iteration. This procedure continues until the direct causes and direct effects of the target variable have been identified. Our experiments on seven benchmark Bayesian networks and a real-world bioinformatics dataset, with a number of variables from 11 to 801, demonstrate that our algorithm achieves better accuracy than the existing local causal structure learning algorithms.

Identification and screening of key traffic violations: based on the perspective of expressing driver’s accident risk

Drawing on the core idea of Propensity Score Matching, this study proposes a new concept named Historical Traffic Violation Propensity to describe the driver’s historical traffic violations, and combines the new concept with an improved mutual information-based feature selection algorithm to construct a method for screening key traffic violations from the perspective of expressing driver’s accident risk. The validation analysis based on the real data collected in Shenzhen demonstrated that drivers’ state of Historical Traffic Violation Propensity on 19 key traffic violations screened have a stronger predictive ability of their subsequent accidents compared to the level in existing research. The positive state of Historical Traffic Violation Propensity on ‘Drinking’, ‘Parking in dangerous areas’, ‘Wrong use of turn lights’, ‘Violating prohibited and restricted traffic regulations’, and ‘Disobeying prohibition sign’ will increase the probability of a driver’s subsequent accident by more than 1.7 times. The research provides directions to more efficiently and accurately capture the driver’s accident risk through historical traffic violations, which is valuable for identifying high-risk drivers as well as the key psychological or physical risk factors that manifest in daily driving activities and lead to subsequent accidents.

Dynamic mutual information-based feature selection for multi-label learning

In classification problems, feature selection is used to identify important input features to reduce the dimensionality of the input space while improving or maintaining classification performance. Traditional feature selection algorithms are designed to handle single-label learning, but classification problems have recently emerged in multi-label domain. In this study, we propose a novel feature selection algorithm for classifying multi-label data. This proposed method is based on dynamic mutual information, which can handle redundancy among features controlling the input space. We compare the proposed method with some existing problem transformation and algorithm adaptation methods applied to real multi-label datasets using the metrics of multi-label accuracy and hamming loss. The results show that the proposed method demonstrates more stable and better performance for nearly all multi-label datasets.

Graph based heterogeneous feature extraction for enhanced hardware Trojan detection at gate-level using optimized XGBoost algorithm

The requirement of reduced production costs in the domain of integrated circuit(IC) manufacturing and constrained time-to-market are met at the cost of a plethora of security concerns in chip design. In the fourth industrial revolution, where system-on-chip forms the backbone of various electronics, malicious hardware modules known as hardware Trojans within the third-party intellectual property impose a significant security threat. The flexibility and ease of intrusion provided in the earlier stages of design, known as the pre-silicon stage, make it vulnerable to these hardware attacks. Such a scenario further emphasizes the need for effective pre-silicon detection schemes. Since this stage is more vulnerable to hardware attacks. Most existing pre-silicon schemes rely on domain knowledge for feature extraction to develop knowledge-driven but exhibit limited Trojan space exploration. Hence, a proposal that comprises four major stages, viz., multifaceted heterogeneous feature extraction, dynamic hybrid sampling, mutual information-based feature selection, and genetic algorithm-based optimization, has been attempted to handle the challenges. Experimentation on TRIT-TC benchmark circuits resulted in a better predictive value for the chosen test circuits, which proves that the proposal outperforms existing schemes.

Classification and feature selection methods based on fitting logistic regression to PU data

In our work, we examine the classification methods where the positive and unlabeled data are considered and where the conditional distribution of the true class label given the feature vector is governed by the model of logistic regression. Our first objective is to compute and compare the selected metrics allowing for the quality assessment of these methods. In this context, we investigate four methods of the posterior probability estimation, where the risk of logistic loss function is optimized: the naive approach, the weighted likelihood approach, as well as the quite recently proposed methods – the joint approach, and the LassoJoint method. The corresponding evaluations are basically performed for 13 machine learning models on some chosen – both low- and high-dimensional – datasets. Some of the mentioned machine learning model schemes have been directly borrowed from literature and some have been obtained through some modifications in the existing procedures. Our second goal is to establish the most stable and efficient approach for the posterior probability estimation. Moreover, we use the AdaSampling scheme for comparison of the considered classification methods. We also conducted comparisons of feature selection procedures – the Mutual Information-Based feature selection method and the LassoJoint approach. The current article is an enhancement of the conference paper Furmańczyk et al. (2022).

A Clustering based Feature Selection Approach using Maximum Spanning Tree

Mutual information (MI) based feature selection methods are getting popular as its ability to capture the nonlinear and linear relationship among random variables and thus it performs better in different fields of machine learning. Traditional MI based feature selection algorithms use different techniques to find out the joint performance of features and select the relevant features among them. However, to do this, in many cases, they might incorporate redundant features. To solve these issues, we propose a feature selection method, namely Clustering based Feature Selection (CbFS), to cluster the features in such a way so that redundant and complementary features are grouped in the same cluster. Then, a subset of representative features is selected from each cluster. Experimental results of CbFS and four state-of-the-art methods are reported to measure the excellency of CbFS over twenty benchmark UCI datasets and three renowned network intrusion datasets. It shows that CbFS performs better than the comparative methods in terms of accuracy and performs better in identifying attack or normal instances in security datasets. DUJASE Vol. 7 (2) 47-55, 2022 (July)

Abstract 2467: Novel biomarkers of metastasis-prone breast cancer revealed by high-throughput proteomics assay

Abstract Relatively high cost of gene expression-based prognostic tools in breast cancer (BC) hinders the public accessibility. Therefore, there is a need for development of robust prognostic biomarker for prediction of early and late metastasis. We designed high-throughput proteomics assay to compare protein expression profiles of BC with no metastasis (NM), late metastasis (LM), and early metastasis (M) on 29 patients with BC at Seoul National University Hospital (SNUH). We defined the LM as the development of metastasis after 5 years from the diagnosis. Mass spectrometry (MS) data were collected in data-dependent acquisition (DDA) mode and quantified by Perseus (version 1.5.8.5). ANOVA was used to determine differentially expressed proteins (DEP). Mutual information-based feature selection was used to pick features for metastasis. We used METABRIC dataset to validate prognostic significance. Cell migration and invasion assays were further performed using TNBC cell line after transfecting the cells with siRNA. Mean metastasis free survival time was 8.4 years with LM, and 1.6 years in the patients in M group. A total of 9,455 proteins were identified across the discovery set, and 6,639 proteins were quantified with label-free quantification intensities. 830 proteins were found to be differentially expressed among the groups with an FDR-adjusted p-value < 0.05. Mutual information selected TUBB as the most highly expressed in M group (median, 36.4), least in NM group (median, 36.0), and intermediate in LM groups (median, 36.1). The expression of VWA5A sequentially decreased from NM group (median, 29.6), LM group (median, 29.2), and M group (median, 28.2). In METABRIC cohort, we found that high TUBB expression is strong, significant poor prognostic factor for DFS in all cases with BC (p < 0.0001), ER+ subgroup (p = 0.0001), HER2+ subgroup (p = 0.0085), but not in TNBC (p = 0.1017). In similar manner, lower expression of VWA5A was significantly correlated with shorter DFS in all (p = 0.0003) and ER+ subtype (p < 0.0001), but not in HER2+ and TNBC subsets (p = 0.1350 and 0.2314, respectively). Discrepant prognostic significance of two candidate biomarkers in certain subtypes of BCs indicate that these molecules may behave differently according to intrinsic subtypes. Therefore, we performed functional experiments using TNBC cell lines. siRNA induced silencing of TUBB in MDA-MB-231 and Hs378T resulted in significantly decreased invasion and migration, while siVWA5A transfected BT20 and HCC70 showed stronger propensity toward invasion and migration. Though impacts on survival were not ascertained, our proteomics and in vitro analyses suggest TUBB and VWA5A as the feasible biomarkers representing the biological aggressiveness of BCs including TNBCs. Further investigations on TUBB and VWA5A are warranted to gather deeper understandings how these two molecules results in different biological behaviors of BCs. Citation Format: Jiwon Koh, Dabin Jeong, Dohyun Han, Sohyeon Yang, Sun Kim, Han Suk Ryu. Novel biomarkers of metastasis-prone breast cancer revealed by high-throughput proteomics assay [abstract]. In: Proceedings of the American Association for Cancer Research Annual Meeting 2023; Part 1 (Regular and Invited Abstracts); 2023 Apr 14-19; Orlando, FL. Philadelphia (PA): AACR; Cancer Res 2023;83(7_Suppl):Abstract nr 2467.

Identify Best Learning Method for Heart Diseases Prediction Under impact of Different Datasets Characteristics

This paper introduces an experimental study of the heart disease datasets characteristics impact on the performance of classification algorithms in the aim of identifying the best algorithm for each dataset under its characteristics. The performance of five machine learning algorithms (logistic regression (LR), K-Nearest Neighbor (KNN), Decision tree (DT), Random Forest (RF), and support vector machine (SVM)), single layer neural network (ANN), and deep neural network (DNN), has been evaluated using five heart disease datasets under four data complexity measurement: number of samples (dataset size), number of features (dimension of dataset), Data sparsity measures, and correlation of features. All datasets have been processed and normalized then the mutual information-based feature selection method was used to solve the overfitting problem. The results show that in general, the machine learning especially the Random Forest algorithm achieves high classification accuracy than deep learning network. In other hand, the high sparsity and less mutual information of dataset has large impact on degradation of the performance of classification algorithms than other characteristics of data.

Phishing URL Detection using LSTM Based Ensemble Learning Approaches

Increasing incidents of phishing attacks tempt a significant challenge for cybersecurity personals. Phishing is a deceitful venture with an intention to steal confidential information of an organization or an individual. Many works have been performed to build anti-phishing solutions over the years, but attackers are coming with new manoeuvres from time to time. Many of the existing techniques are experimented based on limited set of URLs and dependent on other software to collect domain related information of the URLs. In this paper, with an aim to build a more accurate and effective phishing attack detection system, we used the concept of ensemble learning using Long Short-Term Memory (LSTM) models. We proposed ensemble of LSTM models using bagging approach and stacking approach. For performing classification using LSTM method, no separate feature extraction is done. Ensemble models are built integrating the predictions of multiple LSTM models. Performances of proposed ensemble LSTM methods are compared with five different machine learning classification methods. To implement these machine learning algorithms, different URL based lexical features are extracted. Mutual Information based feature selection algorithm is used to select more relevant features to perform classifications. Both the bagging and the stacking approaches of ensemble learning using LSTM models outperform other machine learning techniques. The results are compared with other anti-phishing solutions implemented using deep learning methods. Our approaches have proved to be the more accurate one with a low false positive rate of less than 0.15% performed comparatively on a larger dataset.

An integrated decision support system for heart failure prediction based on feature transformation using grid of stacked autoencoders

Heart failure (HF) is a life-threatening disease. Therefore, prediction of risks associated with HF is essential at the early stages of the disease in order to ensure safety and reduce the mortality rate associated with it. To meet this objective, numerous medical decision support systems (MDSS) have been developed for HF risk prediction. Most of these MDSS focus on feature transformation-based methods and their integration with machine learning models. However, careful investigation of these MDSS reveals that they improved the accuracy on the testing data while the performance on the training data was either compromised or not taken into account. Such an improvement is not pertinent from machine learning perspective. Furthermore, the produced HF prediction accuracy also needs considerable amount of improvement. To tackle such issues, this study proposes a more robust approach that integrates a grid of stacked autoencoders with neural network model. The stacked autoencoders transform features by yielding more robust set of features and the neural network classifies the newly extracted set of features. The strength of the proposed integrated MDSS is evaluated by using an online HF benchmark database namely Cleveland database. The performance of the proposed feature transformation method is also compared with other well known methods namely Principal Component Analysis (PCA), Kernel PCA (KPCA) and feature selection methods i.e., chi-square, Gini index, F-score and mutual information based feature selection. The proposed approach outperformed other state-of-the-art feature transformation methods. The proposed integrated MDSS achieves classification accuracy of 95.55%, specificity of 97.95%, and sensitivity of 92.68%.

Efficient approximate top-k mutual information based feature selection

Efficient approximate top-k mutual information based feature selection

Detection of Eucalyptus Leaf Disease with UAV Multispectral Imagery

Forest disease is one of the most important factors affecting tree growth and product quality, reducing economic values of forest ecosystem goods and services. In order to prevent and control forest diseases, accurate detection in a timely manner is essential. Unmanned aerial vehicles (UAVs) are becoming an important tool for acquiring multispectral imagery, but have not been extensively used for detection of forest diseases. This research project selected a eucalyptus forest as a case study to explore the performance of leaf disease detection using high spatial resolution multispectral imagery that had been acquired by UAVs. The key variables sensitive to eucalyptus leaf diseases, including spectral bands and vegetation indices, were identified by using a mutual information–based feature selection method, then distinguishing disease levels using random forest and spectral angle mapper approaches. The results show that green, red edge, and near-infrared wavelengths, nitrogen reflectance index, and greenness index are sensitive to forest diseases. The random forest classifier, based on a combination of sensitive spectral bands (green, red edge, and near-infrared wavelengths) and a nitrogen reflectance index, provided the best differentiation results for healthy and three disease severity levels (mild, moderate, and severe) with overall accuracy of 90.1% and kappa coefficient of 0.87. This research provides a new way to detect eucalyptus leaf diseases, and the proposed method may be suitable for other forest types.

Assessment of submentalis muscle activity for sleep-wake classification of healthy individuals and patients with sleep disorders.

This work proposes a method utilizing only the submentalis EMG channel for the classification of sleep and wake states among the healthy individuals and patients with various sleep disorders such as sleep apnea hypopnea syndrome, dyssomnia, etc. We extracted autoregressive model parameters, discrete wavelet transform coefficients, Hjorth's complexity and mobility, relative bandpowers, Poincaré plot descriptors and statistical features from the EMG signal. We also used the energy of each epoch as a feature to distinguish between the sleep and wake states. Mutual information based feature selection approach was considered to obtain the top 25 features which provided maximum accuracy. For classification, we employed an ensemble of decision trees with random undersampling and boosting technique to deal with the class-imbalance problem in the sleep data. We achieved an overall accuracy of about 85% for the healthy population and about 70% on an average across different pathological groups. This work shows the potential of EMG chin activity for sleep analysis. Clinical Relevance- Automatic and reliable sleep-wake classification can reduce the burden of sleep experts in analyzing overnight sleep data (~ 8 hours) and also assist them to diagnose various neurological disorders at an early stage. Utilizing EMG channel provides an easier and convenient long-term recording of data without causing much disturbance in sleepunlike EEG which is inconvenient and hampers the natural sleep.

PKNN-MIFS: A Parallel KNN Classifier over an Optimal Subset of Features

Feature selection help select an optimal subset of features from a large feature space to achieve better classification performance. The performance of KNN classifier can be improved significantly using an appropriate subset of features from a large feature space. Recent development in General Purpose Graphics Processing Units (GPGPU) has provided us a low cost yet high performance computing support for wide range of applications. This paper presents a parallel KNN classifier powered by a mutual information based feature selection called PKNN-MIFS for effective classification of real life data. It selects an optimal subset of features from the original feature set by exploiting the mutual information concept for the estimation of feature-class and feature-feature relevance. It selects a non-redundant feature by giving higher priority on feature-class relevance. The performance of the proposed PKNN-MIFS has been evaluated over several datasets and has been found to be superior to its closed counterpart.

Low-precision feature selection on microarray data: an information theoretic approach

The number of interconnected devices, such as personal wearables, cars, and smart-homes, surrounding us every day has recently increased. The Internet of Things devices monitor many processes, and have the capacity of using machine learning models for pattern recognition, and even making decisions, with the added advantage of diminishing network congestion by allowing computations near to the data sources. The main restriction is the low computation capacity of these devices. Thus, machine learning algorithms capable of maintaining accuracy while using mechanisms that exploit certain characteristics, such as low-precision versions, are needed. In this paper, low-precision mutual information-based feature selection algorithms are employed over DNA microarray datasets, showing that 16-bit and some times even 8-bit representations of these algorithms can be used without significant variations in the final classification results achieved.Graphical Graphical abstract

Evaluation of hybrid forecasting methods for organic Rankine cycle: Unsupervised learning-based outlier removal and partial mutual information-based feature selection

The construction of organic Rankine cycle (ORC) system model is the key to system performance analysis and prediction. However, traditional analysis methods have obvious limitations in constructing strong coupling relationship between operating parameters and performance due to the complex thermal power conversion process of ORC system. First, this study systematically analyzes the nonlinear relationship between ORC system operating parameters and performance by using unsupervised learning and bilinear interpolation algorithm. Compared with the traditional thermodynamic modeling method, the artificial neural network (ANN) has obvious advantages in constructing the mapping relationship of ORC system. However, the ORC system prediction model still has the defects of low accuracy, poor robustness, and high time cost due to the absence of outlier removal and feature dimensionality reduction. A hybrid algorithm for ORC system prediction model construction is proposed on the basis of the data characteristics, information theory and unsupervised learning. This algorithm can remove outliers and reduce the dimensionality of features in ORC system simultaneously. Then, the effectiveness of outlier removal, feature dimensionality reduction, and overall performance of the hybrid algorithm is verified. The mean squared error and mean absolute percentage error of the model is 1.64 × 10−11 and 5.1 × 10−3%. Compared with other algorithms, the hybrid algorithm suitable for ORC system has improved in accuracy and time cost. The accuracy of the hybrid algorithm is improved by 5.56% at least. The time cost of the hybrid algorithm is reduced by at least 17.05%. The hybrid algorithm can provide direct guidance for constructing ANN model of ORC system.