Information Theoretic Learning Research Articles

Information theoretic and machine learning approaches to quantify non-linear visual feature interaction underlying visual object recognition

Understanding what object features the visual system relies upon, when engaged in visual object recognition, is a longstanding challenge in psychophysical studies of visual perception. Although successful approaches have been developed to address this issue (e.g., image classification methods [1]), they have a major limitation: they can estimate the relationship between input visual images and output behavioral responses only under the assumption of a linear observer model (e.g., an observer performing a weighted sum of the information carried by individual pixels within an image). For instance, one popular image classification approach (named the Bubbles method [2]) recovers the salient features used by an observer to identify an object, by presenting the target object partially occluded by opaque masks punctured by transparent windows, and then averaging the trials yielding to correct object identification. However, this method cannot tell whether multiple salient features in an object interact non-linearly (e.g., whether those features need to be simultaneously visible for the object to be correctly identified). Here we propose two approaches to overcome this limitation. As a preliminary step, we simulated observers with various recognition strategies. Each observer had to discriminate bubbles-masked input images of two different objects, by comparing them with either linear or non-linear object templates. One of the object templates contained two features, whose evidence in any given input image was computed as the dot-product between the image and the features themselves. Finally, the overall template evidence was computed either as: 1) the product of the two feature evidences (AND-like feature interaction); 2) the maximum of the feature evidences (OR-like feature interaction); or 3) the sum of the feature evidences (linear interaction). The simulated observers served as benchmarks to validate two different feature-interaction analysis approaches. The first approach consisted of measuring the mutual information between the product/max/sum of every pair of pixels in the object image and the simulated observers’ responses. This approach successfully recovered the simulated feature interaction strategy, which was shown to convey significantly higher information about the observer’s response, as compared to the two other alternative strategies (permutation test; p < 106). The second approach consisted in fitting a model to the AND-like, non-linear observer’s responses, which could predict responses to novel input images. As a model we chose a regularized logistic regression classifier, which was fed with both individual pixel values and the product of pixel pairs. The classifier successfully generalized to unseen input images (training performance=100%, test performance=87%) and its maximal weights matched/captured the simulated feature interaction (e.g., pixels’ products had maximal weights). Finally, we also started applying sparse classifiers[3], which, by relying on diluted discrete weights, could potentially yield a cleaner (i.e., more parsimonious) estimate of salient object features and their interaction. In summary, our simulations show that both information theoretic and classifiers-based approaches can recover the type and the strength of the interaction among the salient features of an object. We are now working to apply these methods to analyze data obtained in a Bubbles-based behavioral study of rat visual object recognition.

Algorithms for maximum-likelihood bandwidth selection in kernel density estimators

In machine learning and statistics, kernel density estimators are rarely used on multivariate data due to the difficulty of finding an appropriate kernel bandwidth to overcome overfitting. However, the recent advances on information-theoretic learning have revived the interest on these models. With this motivation, in this paper we revisit the classical statistical problem of data-driven bandwidth selection by cross-validation maximum likelihood for Gaussian kernels. We find a solution to the optimization problem under both the spherical and the general case where a full covariance matrix is considered for the kernel. The fixed-point algorithms proposed in this paper obtain the maximum likelihood bandwidth in few iterations, without performing an exhaustive bandwidth search, which is unfeasible in the multivariate case. The convergence of the methods proposed is proved. A set of classification experiments are performed to prove the usefulness of the obtained models in pattern recognition.

랜덤 심볼열과 결정 궤환을 사용한 자력 등화 알고리듬

Abstract Non-linear equalization techniques using decision feedback structure are highly demanded for cancellation of intersymbol interferences occurred in severe channel environments. In this paper decision feedback structure is applied to the linear blind equalizer algorithm that is based on information theoretic learning and a randomly generated symbol set. At the decision feedback equalizer (DFE) the random symbols are generated to have the same probability density function (PDF) as that of the transmitted symbols. By minimizing difference between the PDF of blind DFE output and that of randomly generated symbols, the proposed DFE algorithm produces equalized output signal. From the simulation results, the proposed method has shown enhanced convergence and error performance compared to its linear counterpart. Key Words : Decision feedback, Blind equalization, PDF, Random symbols, ITL. * 교신저자 : 김남용(namyong@kangwon.ac.kr)접수일 11년 11월 02일 수정일 (1차 11년 12월 26일, 2차 11년 12월 29일) 게재확정일 12년 01월 05일

Constrained co-clustering with non-negative matrix factorisation

Co-clustering refers to the problem of deriving sub-matrices of the data matrix by simultaneously clustering the rows (data instances) and columns (features) of the matrix. While very effective in discovering useful knowledge, many of the co-clustering algorithms adopt a completely unsupervised approach. Integration of domain knowledge can guide the co-clustering process and greatly enhance the overall performance. We propose a semi-supervised Non-negative Matrix-factorisation (SS-NMF) based framework to integrate domain knowledge in the form of must-link and cannot-link constraints. Specifically, we augment the data matrix by integrating the constraints using metric learning and then perform NMF to obtain co-clustering. Under the proposed framework, we present two approaches to integrate domain knowledge, viz. a distance metric learning approach and an information theoretic metric learning approach. Through experiments performed on real-world web service data and publicly available text datasets, we demonstrate the performance of the proposed SS-NMF based approach for data co-clustering.

An entropy approach to filtering for networked control systems

This paper is concerned with minimum error entropy filter design for non-linear Networked Control Systems (NCSs) with multiple-packet transmission mechanism. Since the error of the filter in NCSs is generally non-Gaussian, the filter is designed under the information theoretic learning frame. The convergent condition of the proposed filter is established. A simulation example shows that the proposed approach is feasible and effective.

Relative information maximization and its application to the extraction of explicit class structure in SOM

In this paper, we propose a new type of information theoretic method to determine the appropriate quantity of information to be contained in neural networks. Though information theoretic methods have been extensively applied to neural networks, they have been concerned with information maximization and minimization. In the present paper, we point out the necessity of paying due attention to the content of the obtained information, or the quality of the information content. We should explore more exactly what kinds of information should be obtained in learning. We applied this idea to information theoretic competitive learning in which mutual information between competitive units and input patterns can be used to realize competitive processes. We do not maximize simply the mutual information but the relative information, namely, the ratio of mutual information between competitive units and input patterns to the total information in networks. By maximizing the relative information, we can produce total information in which the maximum mutual information is included. We applied this method to two data sets from the machine learning database, namely, the glass data and the musk problem. The experimental results are summarized by the following three points. First, the relative information could be maximized, meaning that the peak values of relative information could be obtained for both sets of data. Second, improved quantization and topographic errors were obtained by maximizing the relative information. Third, when the relative information was maximized, clearer class structures could be obtained in terms of the U-matrix and conditional mutual information.

A Unified Framework for Quadratic Measures of Independence

This paper proposes a unified framework for several available measures of independence by generalizing the concept of information theoretic learning (ITL). The key component of ITL is the use of inner product between two density functions as a measure of similarity between two random variables. We show that by generalizing the inner product using a symmetric strictly positive-definite kernel and by choosing appropriate kernels, it is possible to reproduce a number of popular measures of independence. This unified framework also allows the design of new strictly positive-definite kernels and corresponding measures of independence. Following this framework we explore a new measure of independence and apply it in the context of linear independent component analysis (ICA). An attractive property of the proposed method is that it does not involve any free parameter and we demonstrate that it performs equally well compared to the existing methods for ICA.

Information Theoretic Learning: Reny's Entropy and Kernel Perspectives (Principe, J.; 2010) [Book Review

This book, derived from Jose Principe and his group's 10 years' research in information theory and statistical learning, gives a comprehensive introduction, analysis and demonstration of almost all the major components required for understanding and developing the new theme of information-theoretical learning. The basic strategy utilized by the author is to apply information theory descriptors (namely entropy and divergence, in contrast to the statistical measures of mean and covariance)as nonparametric cost functions for the design of adaptive systems, thus creating a new paradigm of information theoretic learning. And like in statistical learning, unsupervised or supervised training modes are also fully explored.

A study of performance on microarray data sets for a classifier based on information theoretic learning

Gene-expression microarray is a novel technology that allows the examination of tens of thousands of genes at a time. For this reason, manual observation is not feasible and machine learning methods are progressing to face these new data. Specifically, since the number of genes is very high, feature selection methods have proven valuable to deal with these unbalanced-high dimensionality and low cardinality-data sets. In this work, the FVQIT (Frontier Vector Quantization using Information Theory) classifier is employed to classify twelve DNA gene-expression microarray data sets of different kinds of cancer. A comparative study with other well-known classifiers is performed. The proposed approach shows competitive results outperforming all other classifiers.

Correntropy based feature selection using binary projection

Most feature selection algorithms based on information-theoretic learning (ITL) adopt ranking process or greedy search as their searching strategies. The former selects features individually so that it ignores feature interaction and dependencies. The latter heavily relies on the search paths, as only one path will be explored with no possible back-track. In addition, both strategies typically lead to heuristic algorithms. To cope with these problems, this article proposes a novel feature selection framework based on correntropy in ITL, namely correntropy based feature selection using binary projection (BPFS). Our framework selects features by projecting the original high-dimensional data to a low-dimensional space through a special binary projection matrix. The formulated objective function aims at maximizing the correntropy between selected features and class labels. And this function can be efficiently optimized via standard mathematical tools. We apply the half-quadratic method to optimize the objective function in an iterative manner, where each iteration reduces to an assignment subproblem which can be highly efficiently solved with some off-the-shelf toolboxes. Comparative experiments on six real-world datasets indicate that our framework is effective and efficient.

Discriminant Independent Component Analysis

A conventional linear model based on Negentropy maximization extracts statistically independent latent variables which may not be optimal to give a discriminant model with good classification performance. In this paper, a single-stage linear semisupervised extraction of discriminative independent features is proposed. Discriminant independent component analysis (dICA) presents a framework of linearly projecting multivariate data to a lower dimension where the features are maximally discriminant with minimal redundancy. The optimization problem is formulated as the maximization of linear summation of Negentropy and weighted functional measure of classification. Motivated by independence among extracted features, Fisher linear discriminant is used as the functional measure of classification. Experimental results show improved classification performance when dICA features are used for recognition tasks in comparison to unsupervised (principal component analysis and ICA) and supervised feature extraction techniques like linear discriminant analysis (LDA), conditional ICA, and those based on information theoretic learning approaches. dICA features also give reduced data reconstruction error in comparison to LDA and ICA method based on Negentropy maximization.

자가 발생 심볼열과 커널 사이즈 조절을 통한 유클리드 거리 알고리듬의 복소 채널 블라인드 등화

The complex-valued blind algorithm based on a set of randomly generated symbols and Euclidean distance can take advantage of information theoretic learning and cope with the channel phase rotation problems. On the algorithm, in this paper, the effect of kernel size has been studied and a kernel-modified version of the algorithm that rearranges the forces between the information potentials by kernel-modification has been proposed. In simulation results for 16 QAM and complex-channel models, the proposed algorithm show significantly enhanced performance of symbol-point concentration and no phase rotation problems caused by the complex channel models.

Data Mining in Earth System Science (DMESS 2011)

From field-scale measurements to global climate simulations and remote sensing, the growing body of very large and long time series Earth science data are increasingly difficult to analyze, visualize, and interpret. Data mining, information theoretic, and machine learning techniques—such as cluster analysis, singular value decomposition, block entropy, Fourier and wavelet analysis, phase-space reconstruction, and artificial neural networks—are being applied to problems of segmentation, feature extraction, change detection, model-data comparison, and model validation. The size and complexity of Earth science data exceed the limits of most analysis tools and the capacities of desktop computers. New scalable analysis and visualization tools, running on parallel cluster computers and supercomputers, are required to analyze data of this magnitude. This workshop will demonstrate how data mining techniques are applied in the Earth sciences and describe innovative computer science methods that support analysis and discovery in the Earth sciences.

‘Good’ or ‘bad’ wind power forecasts: a relative concept

AbstractThis paper reports a study on the importance of the training criteria for wind power forecasting and calls into question the generally assumed neutrality of the ‘goodness’ of particular forecasts. The study, focused on the Spanish Electricity Market as a representative example, combines different training criteria and different users of the forecasts to compare them in terms of the benefits obtained. In addition to more classical criteria, an information theoretic learning training criterion, called parametric correntropy, is introduced as a means to correct problems detected in other criteria and achieve more satisfactory compromises among conflicting criteria, namely forecasting value and quality. We show that the interests of wind farm owners may lead to a preference for biased forecasts, which may be in conflict with the larger needs of secure operating policies. The ideas and conclusions are supported by results from three real wind farms. Copyright © 2010 John Wiley &amp; Sons, Ltd.

Maximization of zero-error probability for adaptive channel equalization

A new blind equalization algorithm that is based on maximizing the probability that the constant modulus errors concentrate near zero is proposed. The cost function of the proposed algorithm is to maximize the probability that the equalizer output power is equal to the constant modulus of the transmitted symbols. Two blind information-theoretic learning (ITL) algorithms based on constant modulus error signals are also introduced: One for minimizing the Euclidean probability density function distance and the other for minimizing the constant modulus error entropy. The relations between the algorithms and their characteristics are investigated, and their performance is compared and analyzed through simulations in multi-path channel environments. The proposed algorithm has a lower computational complexity and a faster convergence speed than the other ITL algorithms that are based on a constant modulus error. The error samples of the proposed blind algorithm exhibit more concentrated density functions and superior error rate performance in severe multi-path channel environments when compared with the other algorithms.

Learning Low-Rank Kernel Matrices with Column-Based Methods

We consider the problem of computing a low-rank approximation for large kernel matrix. In this article, a novel strategy called two-stage low-rank kernel matrix selection is proposed for computational efficiency enhancement. Firstly, two permutation sets are obtained by a proposed hybrid column-based selection method, which leads to significant reduction of kernel matrix in size. Secondly, entries of the resultant matrix are selected using information theoretic learning. Then this matrix is used for classification. Experimental results on real data sets have shown the superiority of the proposed method in terms of computational efficiency and classification accuracy, especially when training samples size is large.

Information theoretic learning with adaptive kernels

This paper presents an online algorithm for adapting the kernel width that is a free parameter in information theoretic cost functions using Renyi's entropy. This kernel computes the interactions between the error samples and essentially controls the nature of the performance surface over which the parameters of the system adapt. Since the error in an adaptive system is non-stationary during training, a fixed value of the kernel width may affect the adaptation dynamics and even compromise the location of the global optimum in parameter space. The proposed online algorithm for adapting the kernel width is derived from first principles and minimizes the Kullback–Leibler divergence between the estimated error density and the true density. We characterize the performance of this novel approach with simulations of linear and nonlinear systems training, using the minimum error entropy criterion with the proposed adaptive kernel algorithm. We conclude that adapting the kernel width improves the rate of convergence of the parameters, and decouples the convergence rate and misadjustment of the filter weights.

The Equivalency of Minimum Error Entropy Criterion and Minimum Dispersion Criterion for Symmetric Stable Signal Processing

The minimum error entropy (MEE) criterion in information theoretic learning is an efficient way to deal with non-Gaussian signal processing. And the minimum dispersion (MD) criterion has been widely applied in stable signal processing. In this letter, we show that there exists an equivalence between the MD criterion and the MEE criterion where symmetric $\alpha$ -stable $(S\alpha S)$ random variables are considered as the errors of the adaptive signal processing. As an application, we propose an algorithm with the MEE criterion for the time delay estimation (TDE) problem which was solved by the MD criterion.

Learning explicit and implicit visual manifolds by information projection

Natural images have a vast amount of visual patterns distributed in a wide spectrum of subspaces of varying complexities and dimensions. Understanding the characteristics of these subspaces and their compositional structures is of fundamental importance for pattern modeling, learning and recognition. In this paper, we start with small image patches and define two types of atomic subspaces: explicit manifolds of low dimensions for structural primitives and implicit manifolds of high dimensions for stochastic textures. Then we present an information theoretical learning framework that derives common models for these manifolds through information projection, and study a manifold pursuit algorithm that clusters image patches into those atomic subspaces and ranks them according to their information gains. We further show how those atomic subspaces change over an image scaling process and how they are composed to form larger and more complex image patterns. Finally, we integrate the implicit and explicit manifolds to form a primal sketch model as a generic representation in early vision and to generate a hybrid image template representation for object category recognition in high level vision. The study of the mathematical structures in the image space sheds lights on some basic questions in human vision, such as atomic elements in visual perception, the perceptual metrics in various manifolds, and the perceptual transitions over image scales. This paper is based on the J.K. Aggarwal Prize lecture by the first author at the International Conference on Pattern Recognition, Tempa, FL. 2008.

Generalized Risk Zone: Selecting Observations for Classification

In this paper, we extend the risk zone concept by creating the Generalized Risk Zone. The Generalized Risk Zone is a model-independent scheme to select key observations in a sample set. The observations belonging to the Generalized Risk Zone have shown comparable, in some experiments even better, classification performance when compared to the use of the whole sample. The main tool that allows this extension is the Cauchy-Schwartz divergence, used as a measure of dissimilarity between probability densities. To overcome the setback concerning pdf's estimation, we used the ideas provided by the Information Theoretic Learning, allowing the calculation to be performed on the available observations only. We used the proposed methodology with Learning Vector Quantization, feedforward Neural Networks, Support Vector Machines, and Nearest Neighbors.