Mutual Information Gain Research Articles

Analysis of feature selection measures for text categorisation

The curse of dimensionality has made dimension reduction an essential step in text categorisation. Feature selection is an approach for dimension reduction. In this paper an analysis on feature selection measures for text categorisation is performed. Under the unsupervised approach document frequency and under the supervised approach chi-square, odds ratio, mutual information, and information gain are considered for analysis. They are considered here because they are the widely used and effective measures. Analysis of these measures is performed using the 20 newsgroups dataset. Twenty newsgroups dataset consists of closely related categories as well as highly unrelated categories. Certain categories of 20 newsgroups dataset are selected and organised into three groups of overlapping (highly related) classes, non-overlapping (highly unrelated) classes and combination of overlapping and non-overlapping classes. Feature selection and subsequent classification is applied to the three groups separately and the classification performance is studied based on the feature selection measures. The noticeable behaviour was with odds ratio measure in that it performed well for non-overlapping group and overlapping groups considered separately and was poorer in performance for the group containing both overlapping and non-overlapping categories. Remaining measures showed consistent behaviour for all the three groups. Classification was achieved using support vector machine classifier. The performance comparisons of different measures on different groups are presented in terms of micro-F1 and macro-F1.

Mutual information analysis for digital circuits}{Mutual information analysis for digital circuits

Information theory is an effective tool to study information flow systems. The information entropy is a measure of the average uncertainty in the random variable, and the mutual information is a measure of the dependence between two random variables. As the mutual information, which is often used to characterize the processing capability in the communication system, is closely related to the channel capacity and circuit power, we use the information theory to analyze the digital circuits in this paper. We discussed variation trend of mutual information and its influence factors for four typical circuit models, symmetric, asymmetric, feedback and non-feedback circuits. We also proved that the circuit with feedback has fault tolerance and mutual information gain. The analysis method is suitable for other circuits, and it can provide a theoretical basis for circuit design.

Senti‐CS: Building a lexical resource for sentiment analysis using subjective feature selection and normalized Chi‐Square‐based feature weight generation

AbstractSentiment analysis involves the detection of sentiment content of text using natural language processing. Natural language processing is a very challenging task due to syntactic ambiguities, named entity recognition, use of slangs, jargons, sarcasm, abbreviations and contextual sensitivity. Sentiment analysis can be performed using supervised as well as unsupervised approaches. As the amount of data grows, unsupervised approaches become vital as they cut down on the learning time and the requirements for availability of a labelled dataset. Sentiment lexicons provide an easy application of unsupervised algorithms for text classification. SentiWordNet is a lexical resource widely employed by many researchers for sentiment analysis and polarity classification. However, the reported performance levels need improvement. The proposed research is focused on raising the performance of SentiWordNet3.0 by using it as a labelled corpus to build another sentiment lexicon, named Senti‐CS. The part of speech information, usage based ranks and sentiment scores are used to calculate Chi‐Square‐based feature weight for each unique subjective term/part‐of‐speech pair extracted from SentiWordNet3.0. This weight is then normalized in a range of −1 to +1 using min–max normalization. Senti‐CS based sentiment analysis framework is presented and applied on a large dataset of 50000 movie reviews. These results are then compared with baseline SentiWordNet, Mutual Information and Information Gain techniques. State of the art comparison is performed for the Cornell movie review dataset. The analyses of results indicate that the proposed approach outperforms state‐of‐the‐art classifiers.

HICC: an entropy splitting-based framework for hierarchical co-clustering

Two-dimensional contingency tables or co-occurrence matrices arise frequently in various important applications such as text analysis and web-log mining. As a fundamental research topic, co-clustering aims to generate a meaningful partition of the contingency table to reveal hidden relationships between rows and columns. Traditional co-clustering algorithms usually produce a predefined number of flat partition of both rows and columns, which do not reveal relationship among clusters. To address this limitation, hierarchical co-clustering algorithms have attracted a lot of research interests recently. Although successful in various applications, the existing hierarchical co-clustering algorithms are usually based on certain heuristics and do not have solid theoretical background. In this paper, we present a new co-clustering algorithm, HICC, with solid theoretical background. It simultaneously constructs a hierarchical structure of both row and column clusters, which retains sufficient mutual information between rows and columns of the contingency table. An efficient and effective greedy algorithm is developed, which grows a co-cluster hierarchy by successively performing row-wise or column-wise splits that lead to the maximal mutual information gain. Extensive experiments on both synthetic and real datasets demonstrate that our algorithm can reveal essential relationships of row (and column) clusters and has better clustering precision than existing algorithms. Moreover, the experiments on real dataset show that HICC can effectively reveal hidden relationships between rows and columns in the contingency table.

Bayesian Sequential Parameter Estimation by Cognitive Radar With Multiantenna Arrays

In this paper we consider the problem of Bayesian sequential parameter estimation of extended targets for cognitive radar with multi-antenna arrays using adaptive waveforms. The target is modeled as a complex Gaussian random process. Using iterative waveform transmission, the cognitive radar estimates the target's characteristic parameters and updates its probabilistic model based on new measurements. The adaptive waveform is designed by minimizing the conditional entropy from the posterior density of the model parameters. We analyze the performance of the developed Bayesian sequential estimation algorithm and derive expressions for the signal-to-noise ratio gain, the asymptotical posterior Cramer Rao bound, and the mutual information gain. The analysis and numerical simulations demonstrate that the adaptive sequential Bayesian estimator yields accelerated convergence of the estimate towards its true value and a smaller estimation error compared with the conventional Bayesian estimator that uses fixed waveform transmission under Gaussian or non-Gaussian noise.

Evaluation of Features on Sentimental Analysis

Sentimental analysis is the method of finding sentiment such as positive or negative from a text data. In this paper we are using some feature selection techniques such as Mutual information, Chi-Square, Information gain and TF-idf to select features from high dimensionality of feature set. These methods are evaluated over the dataset which contains 2000 review data about MOVIES. The classification is performed using support vector machine provided by weka9 tool. We also investigate that which is best feature to extract sentiments from the reviews. We are considering unigram, bigram, POS tags of words and function words as our feature set.

Summarizing Newspaper Comments

This work investigates summarizing the conversations that occur in the comments section of the UK newspaper the Guardian. In the comment summarization task comments are clustered and ranked within the cluster. The top comments from each cluster are used to give an overview of that cluster. It was found that topic model clustering gave the most agreement when evaluated against a human gold standard. This approach is compared to cosine distance clustering and k-means clustering. PageRank was found to be the prefered ranking system when compared with TF-IDF, Mutual Information gain and Maximal Marginal Relevance and evaluated against sets of comments summarized by a journalist for the Guardian letters page.

Recovering directed networks in neuroimaging datasets using partially conditioned Granger causality

We address the computational and conceptual problems arising when conditional Granger causality, necessary to disambiguate direct and mediated influences, is used on short and noisy datasets of many variables, as it is typically the case in some EEG protocols and in fMRI. We show that considering Granger causality in the framework of information theory we can limit the conditioning to a limited number of variables chosen as the most informative, obtaining more stable and reliable results on fMRI datasets both at region and voxel level. The results on a dynamical model simulated on the structural connectome matrix show that PCGC performs significantly better than fully conditioned and pairwise GC, in particular when the signals are short. We then apply PCGC to publicly available fMRI datasets, with different TRs and divided in two sessions. It is shown that the series with shorter TR have highest content. The curve of the mutual information gain when an additional variable is used for conditioning starts to become flat around 10 variables, indicating the optimal number of variables to condition to. These variables are located not only in proximity of the target variable, but distributed across what we could call an informational resting state network across the whole brain. This behavior is stable across sections and TRs. With the additional step of pre-selecting the variables according to the community structure, PCGC is applied as well to time series from individual voxels. This allows to extend the concept of functional connectivity density to effective connectivity, revealing hubs for incoming and outgoing information transfer.

Rotated Multi-D Constellations in Rayleigh Fading: Mutual Information Improvement and Pragmatic Approach for Near-Capacity Performance in High-Rate Regions

This paper studies the mutual information improvement attained by rotated multidimensional (multi-D) constellations via a unitary precoder G in Rayleigh fading. At first, based on the symmetric cut-off rate of the N-D signal space, we develop a design criterion with regard to the precoder G. It is then demonstrated that the use of rotated constellations in only a reasonably low dimensional signal space can significantly increase the mutual information in high-rate regimes. Based on parameterizations of unitary matrices, we then construct good unitary precoder G in 4-D signal space using a simple optimization problem, which involves only four real variables and it is applicable to any modulation scheme. To further illustrate the potential of multi-D constellation and to show the practical use of mutual information improvement, we propose a simple yet powerful bit-interleaved coded modulation (BICM) scheme in which a (multi-D) mapping technique employed in a multi-D rotated constellation is concatenated with a short-memory high-rate convolutional code. By using extrinsic information transfer (EXIT) charts, it is shown that the proposed technique provides an exceptionally good error performance. In particular, both EXIT chart analysis and simulation results indicate that a turbo pinch-off and a bit error rate around 10-6 happen at a signal-to-noise ratio that is well below the coded modulation and BICM capacities using traditional signal sets. For example, with code rates ranging from 2/3 to 7/8, the proposed system can operate 0.82 dB-2.93 dB lower than the BICM capacity with QPSK and Gray labeling. The mutual information gain offered by rotated constellations can be therefore utilized to design simple yet near Shannon limit systems in the high-rate regions.

Linear Precoding for Relay Networks: A Perspective on Finite-Alphabet Inputs

This paper considers the precoder design for dual-hop amplify-and-forward relay networks and formulates the design from the standpoint of finite-alphabet inputs. In particular, the mutual information is employed as the utility function, which, however, results in a nonlinear and nonconcave problem. This paper exploits the structure of the optimal precoder that maximizes the mutual information and develops a two-step algorithm based on convex optimization and optimization on the Stiefel manifold. By doing so, the proposed algorithm is insensitive to initial point selection and able to achieve a near global optimal precoder solution. Besides, it converges fast and offers high mutual information gain. These advantages are verified by numerical examples, which also show the large performance gain in mutual information also represents the large gain in the coded bit-error rate.

Automatic Mapping Between Large-Scale Heterogeneous Language Resources for NLP Applications: A Case of Sejong Semantic Classes and KorLexNoun for Korean

This paper proposes a statistical-based linguistic methodology for automatic mapping between large-scale heterogeneous languages resources for NLP applications in general. As a particular case, it treats automatic mapping between two large-scale heterogeneous Korean language resources: Sejong Semantic Classes (SJSC) in the Sejong Electronic Dictionary (SJD) and nouns in KorLex. KorLex is a large-scale Korean WordNet, but it lacks syntactic information. SJD contains refined semantic-syntactic information, with semantic labels depending on SJSC, but the list of its entry words is much smaller than that of KorLex. The goal of our study is to build a rich language resource by integrating useful information within SJD into KorLex. In this paper, we use both linguistic and statistical methods for constructing an automatic mapping methodology. The linguistic aspect of the methodology focuses on the following three linguistic clues: monosemy/polysemy of word forms, instances (example words), and semantically related words. The statistical aspect of the methodology uses the three statistical formulae X2, Mutual Information and Information Gain to obtain candidate synsets. Compared with the performance of manual mapping, the automatic mapping based on our proposed statistical linguistic methods shows good performance rates in terms of correctness, specifically giving recall 0.838, precision 0.718 , and F1 0.774. (Pusan National University)

Multiresolution image registration based on tree data structures

This paper presents a new approach for image registration based on the partitioning of the two source images in binary-space and quadtree structures, obtained with a maximum mutual information gain algorithm. Two different implementation approaches that differ in the level at which information is considered have been studied. The first works at pixel level using the simplified images directly, while the second works at node level dealing with the tree data structure. The obtained results show an outstanding accuracy and robustness of the proposed methods. In particular, the use of binary-space partitioned images drastically reduces the grid effects in comparison with regular downsampled images. An important advantage of our approach comes from the reduced size of the data structures corresponding to the simplified images, which makes this method appropriate to be applied in a multiresolution framework and telemedicine applications.

Active Testing for Face Detection and Localization

We provide a novel search technique which uses a hierarchical model and a mutual information gain heuristic to efficiently prune the search space when localizing faces in images. We show exponential gains in computation over traditional sliding window approaches, while keeping similar performance levels.

EXIT-Chart-Aided Hybrid Multiuser Detector for Multicarrier Interleave-Division Multiple Access

A generically applicable hybrid multiuser detector (MUD) concept is proposed by appropriately activating different MUDs in consecutive turbo iterations based on the mutual information (MI) gain. It is demonstrated that the proposed hybrid MUD is capable of approaching the optimal Bayesian MUD's performance despite its reduced complexity, which is at a modestly increased complexity in comparison with that of the suboptimum soft interference cancellation (SoIC) MUD.

Feature selection combining new document frequency with binary discernibility matrix

Feature selection is a core research topic in text categorization. Several classic feature selection methods were analyzed and their deficiencies were summarized. A new document frequency was proposed, and Rough Set (RS) theory was adopted to provide an attribute reduction algorithm based on binary discernibility matrix. Based on the attribute reduction algorithm and the new document frequency, a comprehensive feature selection method was given. The comprehensive method firstly used the new document frequency to select features to filter out some terms, and then employed the attribute reduction algorithm to eliminate redundancy. The experimental results on data of 8 classes, 300 documents each class from http://www.people.com.cn show that the comprehensive method has higher accuracy and recall rate compared with Mutual Information (MI), CHI value and Information Gain (IG) methods.

Image Segmentation Using Information Bottleneck Method

In image processing, segmentation algorithms constitute one of the main focuses of research. In this paper, new image segmentation algorithms based on a hard version of the information bottleneck method are presented. The objective of this method is to extract a compact representation of a variable, considered the input, with minimal loss of mutual information with respect to another variable, considered the output. First, we introduce a split-and-merge algorithm based on the definition of an information channel between a set of regions (input) of the image and the intensity histogram bins (output). From this channel, the maximization of the mutual information gain is used to optimize the image partitioning. Then, the merging process of the regions obtained in the previous phase is carried out by minimizing the loss of mutual information. From the inversion of the above channel, we also present a new histogram clustering algorithm based on the minimization of the mutual information loss, where now the input variable represents the histogram bins and the output is given by the set of regions obtained from the above split-and-merge algorithm. Finally, we introduce two new clustering algorithms which show how the information bottleneck method can be applied to the registration channel obtained when two multimodal images are correctly aligned. Different experiments on 2-D and 3-D images show the behavior of the proposed algorithms.

Hierarchical Classifier Design Using Mutual Information

A nonparametric algorithm is presented for the hierarchical partitioning of the feature space. The algorithm is based on the concept of average mutual information, and is suitable for multifeature multicategory pattern recognition problems. The algorithm generates an efficient partitioning tree for specified probability of error by maximizing the amount of average mutual information gain at each partitioning step. A confidence bound expression is presented for the resulting classifier. Three examples, including one of handprinted numeral recognition, are presented to demonstrate the effectiveness of the algorithm.