Joint Mutual Information Research Articles

Identification of potential biomarkers on microarray data using distributed gene selection approach

In recent times, several feature selection (FS) methods have introduced to identify the biomarkers from gene expression datasets. It has gained extensive attention to solve cancer classification problem, but they have some limitations. First, the majority of FS approaches increases the computational cost due to the centralized data structure. Second, an irrelevant ranked gene that could perform well regarding classification accuracy with suitable subset of genes will be left out of the selection. To resolve these problems, we introduce a novel two-stage FS approach by combining Spearman's Correlation (SC) and distributed filter FS methods which can select the highly discriminative genes for distinguishing samples from high dimensional datasets. Concerning distributed FS, data is distributed by features according to vertical distribution and then performs a merging procedure which updates the feature subset along with improved classification accuracy. Moreover, it is used to quantify the relation between gene-gene and the gene-class and simultaneously detect subsets of essential genes. The proposed method is verified on six gene datasets with the help of four well-known classifiers namely, support vector machine, naïve Bayes, k-nearest neighbor, and decision tree. The performance of the proposed method is compared with traditional filter techniques such as Relief-F, Information gain, minimum redundancy maximum relevance, joint mutual information, Chi-square, and t-test. The experimental results demonstrate that the proposed method has significantly improved the performance regarding computational time and classification accuracy in comparison to standard algorithms when applied to the non-partitioned dataset.

Mutual Information Based Feature Selection for Fingerprint Identification

In the field of fingerprint identification, local histograms coding is one of the most popular techniques used for fingerprint representation, due to its simplicity. This technique is based on the concatenation of the local histograms resulting in a high dimension histogram, which causes two problems. First, long computing time and big memory capacities are required with databases growing. Second, the recognition rate may be degraded due to the curse of dimensionality phenomenon. In order to resolve these problems, we propose to reduce the dimensionality of histograms by choosing only the pertinent bins from them using a feature selection approach based on the mutual information computation. For fingerprint features extraction we use four descriptors: Local Binary Patterns (LBP), Histogram of Gradients (HoG), Local Phase Quantization (LPQ) and Binarized Statistical Image Features (BSIF). As mutual information based selection methods, we use four strategies: Maximization of Mutual Information (MIFS), minimum Redundancy and Maximal Relevance (mRMR), Conditional Info max Feature Extraction (CIFE) and Joint Mutual Information (JMI). We compare results in terms of recognition rates and number of selected features for the investigated descriptors and selection strategies. Our results are conducted on the four FVC 2002 datasets which present different image qualities. We show that the combination of mRMR or CIFE feature selection methods with HoG features gives the best results. We also show that the selection of useful fingerprint features can surely improve the recognition rate and reduce the complexity of the system in terms of computation cost. The feature selection algorithms may reach 98% of time reduction by considering only 20% of the total number of features while also improving the recognition rate of about 2% by avoiding the curse of dimensionality phenomena.

A Machine Learning Approach for the Identification of a Biomarker of Human Pain using fNIRS

Pain is a highly unpleasant sensory and emotional experience, and no objective diagnosis test exists to assess it. In clinical practice there are two main methods for the estimation of pain, a patient’s self-report and clinical judgement. However, these methods are highly subjective and the need of biomarkers to measure pain is important to improve pain management, reduce risk factors, and contribute to a more objective, valid, and reliable diagnosis. Therefore, in this study we propose the use of functional near-infrared spectroscopy (fNIRS) and machine learning for the identification of a possible biomarker of pain. We collected pain information from 18 volunteers using the thermal test of the quantitative sensory testing (QST) protocol, according to temperature level (cold and hot) and pain intensity (low and high). Feature extraction was completed in three different domains (time, frequency, and wavelet), and a total of 69 features were obtained. Feature selection was carried out according to three criteria, information gain (IG), joint mutual information (JMI), and Chi-squared (χ2). The significance of each feature ranking was evaluated using three learning models separately, linear discriminant analysis (LDA), the K-nearest neighbour (K-NN) and support vector machines (SVM) using the linear and Gaussian and polynomial kernels. The results showed that the Gaussian SVM presented the highest accuracy (94.17%) using only 25 features to identify the four types of pain in our database. In addition, we propose the use of the top 13 features according to the JMI criteria, which exhibited an accuracy of 89.44%, as promising biomarker of pain. This study contributes to the idea of developing an objective assessment of pain and proposes a potential biomarker of human pain using fNIRS.

A novel approach for selective feature mechanism for two-phase intrusion detection system

Intrusion Detection is an important aspect to secure the computing systems from different intrusions. To improve the accuracy and to reduce the computational time, this paper proposes a two-phase hybrid method based on the SVM and RNN. In addition, this paper also had a proposal to obtain a few sets of features with a feature selection technique in which the detection performance increases. For the two-phase system, two different feature selection techniques were proposed which solves both the linear dependency and non-linear dependency between the features. In the first phase, the RNN combines with the proposed Joint Mutual Information Maximization (JMIM) based feature selection and in the second phase, the Support Vector Machine (SVM) combines with correlation based feature selection. Extensive simulations are carried out over the proposed system using two different datasets, NSL-KDD and Kyoto2006+. The performance is measured through the performance metrics such as Detection Rate (DR), Precision, False Alarm Rate (FAR), Accuracy and F-Score. Furthermore, a comparative analysis with few recent hybrid frameworks is also enumerated. The obtained results signify the effectiveness of proposed method.

Motor Imagery EEG Classification Based on Decision Tree Framework and Riemannian Geometry.

This paper proposes a novel classification framework and a novel data reduction method to distinguish multiclass motor imagery (MI) electroencephalography (EEG) for brain computer interface (BCI) based on the manifold of covariance matrices in a Riemannian perspective. For method 1, a subject-specific decision tree (SSDT) framework with filter geodesic minimum distance to Riemannian mean (FGMDRM) is designed to identify MI tasks and reduce the classification error in the nonseparable region of FGMDRM. Method 2 includes a feature extraction algorithm and a classification algorithm. The feature extraction algorithm combines semisupervised joint mutual information (semi-JMI) with general discriminate analysis (GDA), namely, SJGDA, to reduce the dimension of vectors in the Riemannian tangent plane. And the classification algorithm replaces the FGMDRM in method 1 with k-nearest neighbor (KNN), named SSDT-KNN. By applying method 2 on BCI competition IV dataset 2a, the kappa value has been improved from 0.57 to 0.607 compared to the winner of dataset 2a. And method 2 also obtains high recognition rate on the other two datasets.

Feature selection based on feature interactions with application to text categorization

Feature selection is an import preprocessing approach for machine learning and text mining. It reduces the dimensions of high-dimensional data. A popular approach is based on information theoretic measures. Most of the existing methods used two- and three-dimensional mutual information terms that are ineffective in detecting higher-order feature interactions. To fill this gap, we employ two- through five-way interactions for feature selection. We first identify a relaxed assumption to decompose the mutual information-based feature selection problem into a sum of low-order interactions. A direct calculation of the decomposed interaction terms is computationally expensive. We employ five-dimensional joint mutual information, a computationally efficient measure, to estimate the interaction terms. We use the ‘maximum of the minimum’ nonlinear approach to avoid the overestimation of the feature significance. We also apply the proposed method to text categorization. To evaluate the performance of the proposed method, we compare it with eleven popular feature selection methods, eighteen benchmark data and seven text categorization data. Experimental results with four different types of classifiers provide concrete evidence that higher-order interactions are effective in improving feature selection methods.

Rolling Bearing Diagnosis Based on Composite Multiscale Weighted Permutation Entropy.

In this paper, composite multiscale weighted permutation entropy (CMWPE) is proposed to evaluate the complexity of nonlinear time series, and the advantage of the CMWPE method is verified through analyzing the simulated signal. Meanwhile, considering the complex nonlinear dynamic characteristics of fault rolling bearing signal, a rolling bearing fault diagnosis approach based on CMWPE, joint mutual information (JMI) feature selection, and k-nearest-neighbor (KNN) classifier (CMWPE-JMI-KNN) is proposed. For CMWPE-JMI-KNN, CMWPE is utilized to extract the fault rolling bearing features, JMI is applied for sensitive features selection, and KNN classifier is employed for identifying different rolling bearing conditions. Finally, the proposed CMWPE-JMI-KNN approach is used to analyze the experimental dataset, the analysis results indicate the proposed approach could effectively identify different fault rolling bearing conditions.

The feasibility and importance of considering climate change impacts in building retrofit analysis

Current building energy use is projected to increase by 1.7% annually until 2025, and the great potential for energy reduction in existing buildings has created opportunities in building energy retrofit projects. In this research, a framework and method are proposed to evaluate the impacts of different retrofit options to existing building under climate change. A Python retrofit tool is developed to perform parametric study by running EnergyPlus under different retrofit scenarios for existing buildings. With the help of Latin-hypercube sampling (LHS) method and a joint mutual information maximization (JMIM)-based feature selection method, the energy conservation measure (ECM) that may have the most potential in reducing the energy use or the lifecycle net present value (NPV) of a target existing building can be selected. A validated data-driven model is used to predict the building’s future hourly energy use based on EnergyPlus simulation results generated by future extreme year weather data. It is demonstrated that global climate change will alter the optimal solution of future ECM combination and its influence varies from building to building, location to location. The optimal retrofit strategy of selecting the best ECM combinations under current climate condition will be subject to change in the future climate condition.

Non-destructive identification of native egg by near-infrared spectroscopy and data driven-based class-modeling.

Eggs are very important parts of human diets worldwide. It is very common to pass feed eggs off as native ones of high commercial values in Chinese markets. One urgent and challenging work is to develop a non-destructive method for verifying the authenticity of native eggs. The present work focuses on exploring the feasibility of combining near-infrared (NIR) spectroscopy with data driven-based class-modeling (DDCM) and model-independent variable selection, i.e., joint mutual information (JMI). A total of 122 eggs of three types were collected. Principal component analysis (PCA) was utilized for exploratory analysis. The JMI algorithm selected only 20 informative variables out of 1557 original variables for class-modeling. DDCM constructed a class-model for each kind of eggs by optimizing parameters such as degrees of freedom (DoF) and the number of principal components (NPC). All class-models and the decision rules were validated on the corresponding test sets. In short, these models achieved an acceptable performance and are also more consistent with actual needs than classification models. The results show that NIR spectroscopy combined with class-modeling is a potential tool for detecting the authenticity of a specific kind of native eggs.

Feature selection considering weighted relevancy

Feature selection plays an important role in pattern recognition and machine learning. Feature selection based on information theory intends to preserve the feature relevancy between features and class labels while eliminating irrelevant and redundant features. Previous feature selection methods have offered various explanations for feature relevancy, but they ignored the relationships between candidate feature relevancy and selected feature relevancy. To fill this gap, we propose a feature selection method named Feature Selection based on Weighted Relevancy (WRFS). In WRFS, we introduce two weight coefficients that use mutual information and joint mutual information to balance the importance between the two kinds of feature relevancy terms. To evaluate the classification performance of our method, WRFS is compared to three competing feature selection methods and three state-of-the-art methods by two different classifiers on 18 benchmark data sets. The experimental results indicate that WRFS outperforms the other baselines in terms of the classification accuracy, AUC and F1 score.

Weighted General Group Lasso for Gene Selection in Cancer Classification.

Relevant gene selection is crucial for analyzing cancer gene expression datasets including two types of tumors in cancer classification. Intrinsic interactions among selected genes cannot be fully identified by most existing gene selection methods. In this paper, we propose a weighted general group lasso (WGGL) model to select cancer genes in groups. A gene grouping heuristic method is presented based on weighted gene co-expression network analysis. To determine the importance of genes and groups, a method for calculating gene and group weights is presented in terms of joint mutual information. To implement the complex calculation process of WGGL, a gene selection algorithm is developed. Experimental results on both random and three cancer gene expression datasets demonstrate that the proposed model achieves better classification performance than two existing state-of-the-art gene selection methods.

Untargeted Identification of Black Rice by Near-Infrared Spectroscopy and One-Class Models

ABSTRACTBlack rice is one of the famous rare rice varieties in China. It is common to sell inferior black rice intentionally declared as famous brands due to economical motivation. There is an urgent need to develop an analytical method for untargeted identification of black rice. The present work focuses on exploring the feasibility of the untargeted identification of black rice by the combination of near-infrared (NIR) spectroscopy and data driven-based class modeling and variable selection. A total of 142 samples of three brands were collected and used for measurements. The samples of a specific class were used as the target class. Principal component analysis was applied for the preliminary analysis. The model-independent variable selection method, i.e., joint mutual information, was used for spectral compression. Only the 10 most informative variables were picked from original variables based on which an optimal class-model for the target class was constructed and validated by means of an external test set. As a result, the model achieved 100% of sensitivity and specificity. It can be concluded that NIR spectroscopy combined with one-class modeling is a feasible tool for the untargeted identification of black rice.

Nonuniform State Space Reconstruction for Multivariate Chaotic Time Series.

State space reconstruction is the foundation of chaotic system modeling. Selection of reconstructed variables is essential to the analysis and prediction of multivariate chaotic time series. As most existing state space reconstruction theorems deal with univariate time series, we have presented a novel nonuniform state space reconstruction method using information criterion for multivariate chaotic time series. We derived a new criterion based on low dimensional approximation of joint mutual information for time delay selection, which can be solved efficiently through the use of an intelligent optimization algorithm with low computation complexity. The embedding dimension is determined by conditional entropy, after which the reconstructed variables have relatively strong independence and low redundancy. The scheme, which integrates nonuniform embedding and feature selection, results in better reconstructions for multivariate chaotic systems. Moreover, the proposed nonuniform state space reconstruction method shows good performance in forecasting benchmark and actual multivariate chaotic time series.

Material identification based on machine-learning algorithms for hybrid workpieces during cylindrical operations

New design concepts for high-performance components are part of the current research. Because of various material properties and chemical composition, the cutting characteristics and chip formation mechanisms change during the machining process. Thus, it can be mandatory to identify the material and adapt the process parameters during machining. As a result, the workpiece quality is optimized while increasing the tool life. Therefore, this paper investigates a new approach to determine the machined material in-process by machine-learning. A cylindrical turning process is performed for friction welded EN-AW6082/20MnCr5 and C22/41Cr4 shafts. Acceleration and process force signals as well as control signals are measured and monitoring features are generated. These features are ranked and selected based on the information value by the joint mutual information method. Afterwards, four machine-learning models are trained to identify the machined material based on the signal features. The monitoring quality is evaluated during various cylindrical turning processes and the most appropriate machine-learning algorithm is determined. Thus, a new methodology for in-process material identification in CNC turning machines based on signal analysis and machine-learning algorithm is proposed.

Photoanthropometric face iridial proportions for age estimation: An investigation using features selected via a joint mutual information criterion

Age assessment from images is of high interest in the forensic community because of the necessity to establish formal protocols to identify child pornography, child missing and abuses where visual evidences are the mostly admissible. Recently, photoanthropometric methods have been found useful for age estimation correlating facial proportions in image databases with samples of some age groups. Notwithstanding the advances, newer facial features and further analysis are needed to improve accuracy and establish larger applicability. In this investigation, frontal images of 1000 individuals (500 females, 500 males), equally distributed in five age groups (6, 10, 14, 18, 22 years old) were used in a 10 fold cross-validated experiment for three age thresholds classifications (<10, <14, <18 years old). A set of novel 40 features, based on a relation between landmark distances and the iris diameter, is proposed and joint mutual information is used to select the most relevant and complementary features for the classification task. In a civil image identification database with diverse ancestry, receiver operating characteristic (ROC) curves were plotted to verify accuracy, and the resultant AUCs achieved 0.971, 0.969, and 0.903 for the age classifications (<10, <14, <18 years old), respectively. These results add support to continuing research in age assessment from images using the metric approach. Still, larger samples are necessary to evaluate reliability in extensive conditions.

A binning formula of bi-histogram for joint entropy estimation using mean square error minimization

Histograms have extensively been used as a simple tool for nonparametric probability density function estimation. However, practically, the accuracy of some histogram-based derived quantities, such as the marginal entropy (ME), the joint entropy (JE), or the mutual information (MI) depends on the number of bins chosen for the histogram. In this paper, we investigate the binning problem of bi-histogram for the estimation of JE. By minimizing a theoretical mean square error (MSE) of JE estimation, we derive a new formula for the optimal number of bins of bi-histogram for continuous random variables. This novel JE estimation has been used in the MI estimation to avoid the error accumulation of joint MI between the class variable and feature subset in the feature selection. In a synthetic Gaussian feature selection problem, only the proposed method permits to retrieve the exact number of relevant features that explain the class variable when compared to a concurrent univariate estimator based on binning formula that has been proposed for ME estimation. In speech and speaker recognition applications, the proposed method permits to select a limited number of features which guaranties approximately the same or an even better recognition rate than using the total number of features.

Nondestructive Discrimination of Pharmaceutical Preparations Using Near-Infrared Spectroscopy and Partial Least-Squares Discriminant Analysis

ABSTRACTMetronidazole is a widely used antibacterial and amoebicide drug. The feasibility of the classification of metronidazole samples with respect to their brands was investigated by near-infrared (NIR) spectroscopy along with chemometrics. A total of 92 samples of different lots and four brands were collected for measurements. First, principal component analysis was conducted to visualize the difference between metronidazole samples of different brands. Then, based on an effective classifier-independent method, i.e., joint mutual information, only the 30 most important variables were selected for modeling. From the independent test set, the partial least-squares discriminant analysis model based on the reduced variable set was compared with the corresponding full-spectrum model using all variables, which indicates the model based on the reduced variable set outperforms the full-spectrum model. It appears that the combination of NIR spectroscopy, joint mutual information, and partial least-squares discriminant analysis is a potential method for the classification of metronidazole from different brands and can, therefore, be used in the screening of counterfeit pharmaceutical products.

Feature selection considering two types of feature relevancy and feature interdependency

Feature selection based on information theory, which is used to select a group of the most informative features, has extensive application fields such as machine learning, data mining and natural language processing. However, numerous previous methods suffer from two common defects. (1) Feature relevancy is defined without distinguishing candidate feature relevancy and selected feature relevancy. (2) Some interdependent features may be misinterpreted as redundant features. In this study, we propose a feature selection method named Dynamic Relevance and Joint Mutual Information Maximization (DRJMIM) to address these two defects. DRJMIM includes four stages. First, the relevancy is divided into two categories: candidate feature relevancy and selected feature relevancy. Second, according to candidate feature relevancy that is joint mutual information, some redundant features are selected. Third, the redundant features are combined with a dynamic weight to reduce the selection possibility of true redundant features while increasing the false ones. Finally, the most informative and interdependent features are selected and true redundant features are eliminated simultaneously. Furthermore, our method is compared with five competitive feature selection methods on 12 publicly available data sets. The classification results show that DRJMIM performs better than other five methods. Its statistical significance is verified by a paired two-tailed t-test. Meanwhile, DRJMIM obtains few number of selected features when it achieves the highest classification accuracy.

Quadratic Mutual Information Feature Selection

We propose a novel feature selection method based on quadratic mutual information which has its roots in Cauchy–Schwarz divergence and Renyi entropy. The method uses the direct estimation of quadratic mutual information from data samples using Gaussian kernel functions, and can detect second order non-linear relations. Its main advantages are: (i) unified analysis of discrete and continuous data, excluding any discretization; and (ii) its parameter-free design. The effectiveness of the proposed method is demonstrated through an extensive comparison with mutual information feature selection (MIFS), minimum redundancy maximum relevance (MRMR), and joint mutual information (JMI) on classification and regression problem domains. The experiments show that proposed method performs comparably to the other methods when applied to classification problems, except it is considerably faster. In the case of regression, it compares favourably to the others, but is slower.

RJMIM: A New Feature Selection Method Based On Joint Mutual Information

Feature selection based on information theory plays an important role in classification algorithm due to its computational efficiency and independent from classification method. It is widely used in many application areas like datamining, bioinformatics and machine learning. But drawbacks of these methods are the neglect of the feature interaction and overestimation of features significance due to the limitations of goal functions criterion. To address this problem, we proposed a new feature goal function RJMIM. The method employed joint mutual information and information interaction,which alleviates the shortcomings of overestimation of the feature significance as demonstrated both theoretically and experimentally. The experiments conducted to verify the performance of the proposed method, it compared with four well-known feature selection methods use three publically available datasets from UCI.The average classification accuracy and C4.5 classifier is used to assess the effectiveness of RJMIM method.