Mutual Information Feature Selection Research Articles

Redundancy Coefficient Gradual Up-weighting-based Mutual Information Feature Selection technique for Crypto-ransomware early detection

Crypto-ransomware is a type of malware whose effect is irreversible even after detection and removal. Thus, early detection is crucial to protect user files from being encrypted and held to ransom. Several studies have proposed early detection solutions based on the data acquired during the pre-encryption phase of the attacks. However, the lack of sufficient data in the early phases of the attack adversely affects the ability of feature selection techniques in these models to perceive the common characteristics of the attack features, which makes it challenging to reduce the redundant features, consequently decreasing the detection accuracy. Therefore, this study proposes a novel Redundancy Coefficient Gradual Upweighting (RCGU) technique that makes better redundancy–relevancy trade-offs during feature selection. Unlike existing feature significance estimation techniques that rely on the comparison between the candidate feature and the common characteristics of the already-selected features, RCGU compares the mutual information between the candidate feature and each feature in the selected set individually. Therefore, RCGU increases the weight of the redundancy term proportional to the number of already selected features. By integrating the RCGU into the Mutual Information Feature Selection (MIFS) technique, the Enhanced MIFS (EMIFS) was developed. Further improvement was achieved by proposing MM-EMIFS which incorporates the MaxMin approximation with EMIFS to prevent the redundancy overestimation that RCGU could cause when the number of features in the already-selected set increases. The experimental evaluation shows that the proposed techniques achieved accuracy higher than that in related works, which confirms the ability of RCGU to make better redundancy–relevancy trade-offs and select more discriminative pre-encryption attack features compared to existing solutions.

Differentiating patients with schizophrenia from healthy controls by hippocampal subfields using radiomics

BackgroundAccurately diagnosing schizophrenia is still challenging due to the lack of validated biomarkers. Here, we aimed to investigate whether radiomic features in bilateral hippocampal subfields from magnetic resonance images (MRIs) can differentiate patients with schizophrenia from healthy controls (HCs). MethodsA total of 152 participants with MRI (86 schizophrenia and 66 HCs) were allocated to training (n = 106) and test (n = 46) sets. Radiomic features (n = 642) from the bilateral hippocampal subfields processed with automatic segmentation techniques were extracted from T1-weighted MRIs. After feature selection, various combinations of classifiers (logistic regression, extra-trees, AdaBoost, XGBoost, or support vector machine) and subsampling were trained. The performance of the classifier was validated in the test set by determining the area under the curve (AUC). Furthermore, the association between selected radiomic features and clinical symptoms in schizophrenia was assessed. ResultsThirty radiomic features were identified to differentiate participants with schizophrenia from HCs. In the training set, the AUC exhibited poor to good performance (range: 0.683–0.861). The best performing radiomics model in the test set was achieved by the mutual information feature selection and logistic regression with an AUC, accuracy, sensitivity, and specificity of 0.821 (95% confidence interval 0.681–0.961), 82.1%, 76.9%, and 70%, respectively. Greater maximum values in the left cornu ammonis 1–3 subfield were associated with a higher severity of positive symptoms and general psychopathology in participants with schizophrenia. ConclusionRadiomic features from hippocampal subfields may be useful biomarkers for identifying schizophrenia.

Wind turbine condition monitoring based on a novel multivariate state estimation technique

With the development of the wind energy industry, condition monitoring (CM) has become one of the important ways to improve the reliability of wind turbines (WTs). A data driven WTCM method based on a novel multivariate state estimation technique (MSET) is proposed, which can realize the fault early warning of WT components. In order to reduce the information redundancy of operational parameters, the maximum conditional mutual information (CMI) feature selection algorithm is applied to the training data of MSET. To improve the performance and flexibility of MSET, a dynamic memory matrix (MM) construction method based on k-nearest neighbor algorithm is proposed, which can provide a dynamic MM that varies in real-time with the current operational condition. A fast calculation method of the dynamic MM is also proposed, which can reduce the computation time by 15–26%. Two residual-based CM methods are proposed to realize fault early warning. The real-time method is based on the real-time residuals and if several consecutive residuals exceed the threshold, the fault alerts will be issued. The long-term method divides the historical residuals into day-level and analyzes them based on control charts. The proposed method is applied to two real cases of the gearbox and generator bearing overheating faults. The experimental results show that the maximum CMI algorithm and the dynamic MM significantly improve the estimation error of MSET. Compared with the recorded fault information, the proposed method realizes the fault early warning about 2–20 days in advance.

Fuzzy and Real-Coded Chemical Reaction Optimization for Intrusion Detection in Industrial Big Data Environment

Analysis and modeling of the intrusion detection system is an important phenomenon for any communication network, which helps to monitor the network traffic and avoid suspicious activity in the Big Data environment. The machine learning approach for modeling the intrusion detection system requires analysis of large network data, which may include some irrelevant features resulting in unnecessary computational and analytical burden. In this article, a fuzzy and real coded chemical reaction optimization-based cluster analysis approach with feature selection is proposed for the intrusion detection system in a Big Data platform. The proposed cluster analysis model is achieved through Fuzzy C-Mean (FCM) with real-coded chemical reaction optimization, which boosts FCM to start with optimized cluster centers. Also, the use of the Flexible Mutual Information Feature Selection approach helps this model to avoid the processing of a large number of features, which drastically affects processing elements.

Improved Chicken Swarm Optimized Recurrent Neural Networks for Big Data Intrusion Detection System

Cyber security threats are an ever increasing, frequent and complex issue in the modern information era. With the advent of big data, incremental increase of huge amounts of data has further increased the security problems. Intrusion Detection Systems (IDS) were been developed to monitor and secure the cyber data systems and networks from any intrusions. However, the intrusion detection is difficult due to the rapid evolution of security attacks and the high volume, variety and speed of big data. In addition, the shallow architectures of existing IDS models lead to high computation cost and high memory requirements, thus further diminishing the efficiency of intrusion detection. The recent studies have suggested the use of data analytics and the deep learning algorithms can be effective in improving the IDS. An efficient IDS model is developed in this study by using the improved Elman-type Recurrent Neural Networks (RNN) in which the Improved Chicken Swarm Optimization (ICSO) optimally determines RNN parameters. RNN is an efficient method for classifying network traffic data but its traditional training algorithms are slow in convergence and faces local optimum problem. The introduction of ICSO with enhanced global search ability significantly avoids those limitations and improves the training process of RNN. This optimized deep learning algorithm of RNN, named as ICSO-RNN, is employed in the IDS with Intuitionistic Fuzzy Mutual Information feature selection to analyze larger network traffic datasets. The proposed IDS model using ICSO-RNN is tested on UNSW NB15 dataset. The final outcomes suggested that ICSO-RNN model has high performance in intrusion detection, with minimum training time and is proficient for big data.

Development of a Novel Soft Sensor with Long Short-Term Memory Network and Normalized Mutual Information Feature Selection

In this paper, a novel soft sensor is developed by combining long short-term memory (LSTM) network with normalized mutual information feature selection (NMIFS). In the proposed algorithm, LSTM is designed to handle time series with high nonlinearity and dynamics of industrial processes. NMIFS is conducted to perform the input variable selection for LSTM to simplify the excessive complexity of the model. The developed soft sensor combines the excellent dynamic modelling of LSTM and precise variable selection of NMIFS. Simulations on two actual production datasets are used to demonstrate the performance of the proposed algorithm. The developed soft sensor could precisely predict the objective variables and has better performance than other methods.

Towards Modified Entropy Mutual Information Feature Selection to Forecast Medium-Term Load Using a Deep Learning Model in Smart Homes.

Over the last decades, load forecasting is used by power companies to balance energy demand and supply. Among the several load forecasting methods, medium-term load forecasting is necessary for grid’s maintenance planning, settings of electricity prices, and harmonizing energy sharing arrangement. The forecasting of the month ahead electrical loads provides the information required for the interchange of energy among power companies. For accurate load forecasting, this paper proposes a model for medium-term load forecasting that uses hourly electrical load and temperature data to predict month ahead hourly electrical loads. For data preprocessing, modified entropy mutual information-based feature selection is used. It eliminates the redundancy and irrelevancy of features from the data. We employ the conditional restricted Boltzmann machine (CRBM) for the load forecasting. A meta-heuristic optimization algorithm Jaya is used to improve the CRBM’s accuracy rate and convergence. In addition, the consumers’ dynamic consumption behaviors are also investigated using a discrete-time Markov chain and an adaptive k-means is used to group their behaviors into clusters. We evaluated the proposed model using GEFCom2012 US utility dataset. Simulation results confirm that the proposed model achieves better accuracy, fast convergence, and low execution time as compared to other existing models in the literature.

Heart Disease Identification Method Using Machine Learning Classification in E-Healthcare

Heart disease is one of the complex diseases and globally many people suffered from this disease. On time and efficient identification of heart disease plays a key role in healthcare, particularly in the field of cardiology. In this article, we proposed an efficient and accurate system to diagnosis heart disease and the system is based on machine learning techniques. The system is developed based on classification algorithms includes Support vector machine, Logistic regression, Artificial neural network, K-nearest neighbor, Naive bays, and Decision tree while standard features selection algorithms have been used such as Relief, Minimal redundancy maximal relevance, Least absolute shrinkage selection operator and Local learning for removing irrelevant and redundant features. We also proposed novel fast conditional mutual information feature selection algorithm to solve feature selection problem. The features selection algorithms are used for features selection to increase the classification accuracy and reduce the execution time of classification system. Furthermore, the leave one subject out cross-validation method has been used for learning the best practices of model assessment and for hyperparameter tuning. The performance measuring metrics are used for assessment of the performances of the classifiers. The performances of the classifiers have been checked on the selected features as selected by features selection algorithms. The experimental results show that the proposed feature selection algorithm (FCMIM) is feasible with classifier support vector machine for designing a high-level intelligent system to identify heart disease. The suggested diagnosis system (FCMIM-SVM) achieved good accuracy as compared to previously proposed methods. Additionally, the proposed system can easily be implemented in healthcare for the identification of heart disease.

An Improved Normalized Mutual Information Variable Selection Algorithm for Neural Network-Based Soft Sensors.

In this paper, normalized mutual information feature selection (NMIFS) and tabu search (TS) are integrated to develop a new variable selection algorithm for soft sensors. NMIFS is applied to select influential variables contributing to the output variable and avoids selecting redundant variables by calculating mutual information (MI). A TS based strategy is designed to prevent NMIFS from falling into a local optimal solution. The proposed algorithm performs the variable selection by combining the entropy information and MI and validating error information of artificial neural networks (ANNs); therefore, it has advantages over previous MI-based variable selection algorithms. Several simulation datasets with different scales, correlations and noise parameters are implemented to demonstrate the performance of the proposed algorithm. A set of actual production data from a power plant is also used to check the performance of these algorithms. The experiments showed that the developed variable selection algorithm presents better model accuracy with fewer selected variables, compared with other state-of-the-art methods. The application of this algorithm to soft sensors can achieve reliable results.

A Feature Selection Algorithm Based on Equal Interval Division and Minimal-Redundancy–Maximal-Relevance

Minimal-redundancy–maximal-relevance (mRMR) algorithm is a typical feature selection algorithm. To select the feature which has minimal redundancy with the selected features and maximal relevance with the class label, the objective function of mRMR subtracts the average value of mutual information between features from mutual information between features and the class label, and selects the feature with the maximum difference. However, the problem is that the feature with the maximum difference is not always the feature with minimal redundancy maximal relevance. To solve the problem, the objective function of mRMR is first analyzed and a constraint condition that determines whether the objective function can guarantee the effectiveness of the selected features is achieved. Then, for the case where the objective function is not accurate, an idea of equal interval division is proposed and combined with ranking to process the interval of mutual information between features and the class label, and that of the average value of mutual information between features. Finally, a feature selection algorithm based on equal interval division and minimal-redundancy–maximal-relevance (EID–mRMR) is proposed. To validate the performance of EID–mRMR, we compare it with several incremental feature selection algorithms based on mutual information and other feature selection algorithms. Experimental results demonstrate that the EID–mRMR algorithm can achieve better feature selection performance.

Predicting Lung Cancer Patients' Survival Time via Logistic Regression-based Models in a Quantitative Radiomic Framework.

Background: Selection of the best treatment modalities for lung cancer depends on many factors, like survival time, which are usually determined by imaging.Objectives: To predict the survival time of lung cancer patients using the advantages of both radiomics and logistic regression-based classification models.Material and Methods: Fifty-nine patients with primary lung adenocarcinoma were included in this retrospective study and pre-treatment contrast-enhanced CT images were acquired. The patients lived more than 2 years were classified as the ‘Alive’ class and otherwise as the ‘Dead’ class. In our proposed quantitative radiomic framework, we first extracted the associated regions of each lung lesion from pre-treatment CT images for each patient via grow cut segmentation algorithm. Then, 40 radiomic features were extracted from the segmented lung lesions. In order to enhance the generalizability of the classification models, the mutual information-based feature selection method was applied to each feature vector. We investigated the performance of six logistic regression-based classification models.Results: It was observed that the mutual information feature selection method can help the classifier to achieve better predictive results. In our study, the Logistic regression (LR) and Dual Coordinate Descent method for Logistic Regression (DCD-LR) models achieved the best results indicating that these classification models have strong potential for classifying the more important class (i.e., the ‘Alive’ class). Conclusion: The proposed quantitative radiomic framework yielded promising results, which can guide physicians to make better and more precise decisions and increase the chance of treatment success.

Infrared and radar fusion detection method based on heterogeneous data preprocessing

The difficulty in infrared and radar fusion target detection lies in the pre-processing steps such as sample reduction and feature reduction of heterogeneous sensors. For the problem of data volume redundancy, HVS sample reduction is proposed to reduce the amount of data in the infrared data set. Infrared and radar joint feature vectors are constructed through nearest neighbor data association. For the problem of feature redundancy, a weighted mutual information feature selection method based on prior information is proposed according to the difference of feature sources. The fusion data set is then used for classification. The experiments show that the HVS-based sample reduction and the prior weighted feature selection achieved a higher detection probability and a shorter detection time, and improved the robustness in the case of large differences between the training set and the test set.

Comprehensive feature selection for classifying the treatment outcome of high-intensity ultrasound therapy in uterine fibroids

The study aim was to utilise multiple feature selection methods in order to select the most important parameters from clinical patient data for high-intensity focused ultrasound (HIFU) treatment outcome classification in uterine fibroids. The study was retrospective using patient data from 66 HIFU treatments with 89 uterine fibroids. A total of 39 features were extracted from the patient data and 14 different filter-based feature selection methods were used to select the most informative features. The selected features were then used in a support vector classification (SVC) model to evaluate the performance of these parameters in predicting HIFU therapy outcome. The therapy outcome was defined as non-perfused volume (NPV) ratio in three classes: <30%, 30–80% or >80%. The ten most highly ranked features in order were: fibroid diameter, subcutaneous fat thickness, fibroid volume, fibroid distance, Funaki type I, fundus location, gravidity, Funaki type III, submucosal fibroid type and urinary symptoms. The maximum F1-micro classification score was 0.63 using the top ten features from Mutual Information Maximisation (MIM) and Joint Mutual Information (JMI) feature selection methods. Classification performance of HIFU therapy outcome prediction in uterine fibroids is highly dependent on the chosen feature set which should be determined prior using different classifiers.

An Accurate Data Preparation Approach for the Prediction of Mortality in ACLF Patients using the CANONIC Dataset.

The incidence of chronic liver disease has increased in Europe and can lead to Acute on Chronic Liver Failure (ACLF) which is associated with high levels of mortality due to multisystem organ failure. The characteristics of the ACLF patients can change very rapidly within a short period of time. Continuous assessment of their recovery status is critical for clinicians to adjust and deliver effective treatment. The aim of this paper is to validate the usefulness of a data preparation approach by combining different criteria to replace missing values, balance target-class variables, select useful patient characteristics and optimise hyperparameters of machine learning models for the prediction of ACLF associated mortality rates. A key step in the data preparation is a feature selection Mutual Information (MI) based multivariate approach to build smaller, and yet equally and in some cases more informative, subsets of patient characteristics than those frequently proposed for the prediction of mortality, from patients with ACLF in the CANONIC dataset. The usefulness of the data preparation approach proposed to predict mortality was evaluated by training the XGBoost and Logistic Regression models with the prepared data. Evaluations of the models trained using a test set provided evidence of an overall high accuracy in the prediction of the mortality rates of patients for days after their diagnosis, and in some cases even higher when reduced and more informative subsets of patient characteristics were found.

Designing a general type-2 fuzzy expert system for diagnosis of depression

Depression is a common and important mental disorder that affects the quality of human life. Since people with depression are not aware of their disorder and sometimes suffer from physical symptoms such as chronic pain, refer to a physician instead of a psychologist. Hence, physician’s diagnosis is not always correct in all patients. In the other words, misdiagnosis may occur by mislabeling their mental disorder as physical diseases. Delay in depression diagnosis may have irrecoverable outcomes such as suicide. Therefore, the most challenging aspect of depression diagnosis is to limit time loss and preserve accuracy. In this paper, a novel general type-2 fuzzy expert system for depression diagnosis, considering two main objectives, was developed. These objectives include accuracy of the system and diagnosis time. The proposed system might be a helpful guideline for the physician to lead patients toward psychologist by asking 15 questions from patients. The proposed general Type-2 expert system has five steps. In the first step, we generate general type-2 membership function by using zSlices method and interval agreement approach (IAA). Then fuzzy rules are extracted out of data gathered from hospital and we extend Mendel method briefly in the second step. Approximate reasoning is applied in the third step. In the fourth step, we solve a multi-objective problem to minimize time and maximize accuracy by using MOEA/D method. Accordingly, in order to minimize time, feature selection is applied. In this process, we use MIFS (Mutual Information Feature Selection) method and briefly, we extend it. In the final step, we choose an appropriate solution from achieved Pareto Front (PF). The proposed general type-2 expert system has been tested and evaluated to show its performance. This Intelligent system is able to diagnose depression accurately at a suitable time.

Estimation of Solar Irradiance at Weather Stations in Korea Using Regionally Trained Artificial Neural Network Models

The ability to estimate solar irradiance is necessary for efficient installation and management of solar photovoltaic systems. Weather observation data and Communication, Ocean and Meteorological Satellite (COMS) images were used to estimate solar irradiance at 94 Korea Meteorological Administration (KMA) weather stations. First, Korea was divided into 13 regions by applying K-means clustering to cloud indices derived from visible channel images of COMS, and artificial neural network (ANN) models were trained in 10 regions. Ten variables were selected as inputs for ANN by mutual information feature selection to build multi-layer perceptron networks in MATLAB. The cross validation resulted in a 0.96 correlation coefficient and 80.87 W/m2 (22.62%) root mean square error (RMSE). The regionally trained ANN models could reduce the relative RMSE by 2%p compared to non-regionally trained models. The solar irradiances at the 94 KMA weather stations were estimated by applying these models. The solar irradiance of points that have no measured data from a pyranometer can be estimated using these regional ANN models.

Feature Selection Algorithms for Wind Turbine Failure Prediction

It is well known that each year the wind sector has profit losses due to wind turbine failures and operation and maintenance costs. Therefore, operations related to these actions are crucial for wind farm operators and linked companies. One of the key points for failure prediction on wind turbine using SCADA data is to select the optimal or near optimal set of inputs that can feed the failure prediction (prognosis) algorithm. Due to a high number of possible predictors (from tens to hundreds), the optimal set of inputs obtained by exhaustive-search algorithms is not viable in the majority of cases. In order to tackle this issue, show the viability of prognosis and select the best set of variables from more than 200 analogous variables recorded at intervals of 5 or 10 min by the wind farm’s SCADA, in this paper a thorough study of automatic input selection algorithms for wind turbine failure prediction is presented and an exhaustive-search-based quasi-optimal (QO) algorithm, which has been used as a reference, is proposed. In order to evaluate the performance, a k-NN classification algorithm is used. Results showed that the best automatic feature selection method in our case-study is the conditional mutual information (CMI), while the worst one is the mutual information feature selection (MIFS). Furthermore, the effect of the number of neighbours (k) is tested. Experiments demonstrate that k = 1 is the best option if the number of features is higher than 3. The experiments carried out in this work have been extracted from measures taken along an entire year and corresponding to gearbox and transmission systems of Fuhrländer wind turbines.

Various Feature Selection Techniques in Type 2 Diabetic Patients for the Prediction of Cardiovascular Disease

Cardiovascular disease (CVD) is a serious but preventable complication of type 2 diabetes mellitus (T2DM) that results in substantial disease burden, increased health services use, and higher risk of premature mortality [10]. People with diabetes are also at a greatly increased risk of cardiovascular which results in sudden death, which increases year by year. Data mining is the search for relationships and global patterns that exist in large databases but are `hidden' among the vast amount of data, such as a relationship between patient data and their medical diagnosis. Usually medical databases of type 2 diabetic patients are high dimensional in nature. If a training dataset contains irrelevant and redundant features (i.e., attributes), classification analysis may produce less accurate results. In order for data mining algorithms to perform efficiently and effectively on high-dimensional data, it is imperative to remove irrelevant and redundant features. Feature selection is one of the important and frequently used data preprocessing techniques for data mining applications in medicine. Many of the research area in data mining has improved the predictive accuracy of the classifiers by applying the various techniques of feature selection This paper illustrates, the application of feature selection technique in medical databases, will enable to find small number of informative features leading to potential improvement in medical diagnosis. It is proposed to find an optimal feature subset of the PIMA Indian Diabetes Dataset using Artificial Bee Colony technique with Differential Evolution, Symmetrical Uncertainty Attribute set Evaluator and Fast Correlation-Based Filter (FCBF). Then Mutual information based feature selection is done by introducing normalized mutual information feature selection (NMIFS). And valid classes of input features are selected by applying Hybrid Fuzzy C Means algorithm (HFCM).

Facial Expression Recognition Based on Discrete Separable Shearlet Transform and Feature Selection

In this paper, a novel approach to facial expression recognition based on the discrete separable shearlet transform (DSST) and normalized mutual information feature selection is proposed. The approach can be divided into five steps. First, all test and training images are preprocessed. Second, DSST is applied to the preprocessed facial expression images, and all the transformation coefficients are obtained as the original feature set. Third, an improved normalized mutual information feature selection is proposed to find the optimal feature subset of the original feature set, thus we can retain the key classification information of the original data. Fourth, the feature extraction and selection of the feature space is reduced by employing linear discriminant analysis. Finally, a support vector machine is used to recognize the expressions. In this study, experimental verification was carried out on four open facial expression databases. The results show that this method can not only improve the recognition rate of facial expressions, but also significantly reduce the computational complexity and improve the system efficiency.

Rolling Bearing Diagnosis Based on Composite Multiscale Weighted Permutation Entropy.

In this paper, composite multiscale weighted permutation entropy (CMWPE) is proposed to evaluate the complexity of nonlinear time series, and the advantage of the CMWPE method is verified through analyzing the simulated signal. Meanwhile, considering the complex nonlinear dynamic characteristics of fault rolling bearing signal, a rolling bearing fault diagnosis approach based on CMWPE, joint mutual information (JMI) feature selection, and k-nearest-neighbor (KNN) classifier (CMWPE-JMI-KNN) is proposed. For CMWPE-JMI-KNN, CMWPE is utilized to extract the fault rolling bearing features, JMI is applied for sensitive features selection, and KNN classifier is employed for identifying different rolling bearing conditions. Finally, the proposed CMWPE-JMI-KNN approach is used to analyze the experimental dataset, the analysis results indicate the proposed approach could effectively identify different fault rolling bearing conditions.