Information Maximization Principle Research Articles

AID-purifier: A light auxiliary network for boosting adversarial defense

In this study, we propose AID-purifier that can boost the robustness of adversarially-trained networks by purifying their inputs. AID-purifier is an auxiliary network that works as an add-on to an already trained main classifier. To keep it computationally light, it is trained as a discriminator with a binary cross-entropy loss. To obtain additionally useful information from the adversarial examples, the architecture design is closely related to the information maximization principle where two layers of the main classification network are piped into the auxiliary network. To assist the iterative optimization procedure of purification, the auxiliary network is trained with AVmixup. AID-purifier can be also used together with other purifiers such as PixelDefend for an extra enhancement. Because input purification has been studied relative less when compared to adversarial training or gradient masking, we conduct extensive attack experiments to validate AID-purifier’s robustness. The overall results indicate that the best performing adversarially-trained networks can be enhanced further with AID-purifier. The code is available in https://github.com/yelobean/AIDPurifier.

Information-Theoretic Representation Learning for Positive-Unlabeled Classification.

Recent advances in weakly supervised classification allow us to train a classifier from only positive and unlabeled (PU) data. However, existing PU classification methods typically require an accurate estimate of the class-prior probability, a critical bottleneck particularly for high-dimensional data. This problem has been commonly addressed by applying principal component analysis in advance, but such unsupervised dimension reduction can collapse the underlying class structure. In this letter, we propose a novel representation learning method from PU data based on the information-maximization principle. Our method does not require class-prior estimation and thus can be used as a preprocessing method for PU classification. Through experiments, we demonstrate that our method, combined with deep neural networks, highly improves the accuracy of PU class-prior estimation, leading to state-of-the-art PU classification performance.

HodgeRank With Information Maximization for Crowdsourced Pairwise Ranking Aggregation

Recently, crowdsourcing has emerged as an effective paradigm for human-powered large scale problem solving in various domains. However, task requester usually has a limited amount of budget, thus it is desirable to have a policy to wisely allocate the budget to achieve better quality. In this paper, we study the principle of information maximization for active sampling strategies in the framework of HodgeRank, an approach based on Hodge Decomposition of pairwise ranking data with multiple workers. The principle exhibits two scenarios of active sampling: Fisher information maximization that leads to unsupervised sampling based on a sequential maximization of graph algebraic connectivity without considering labels; and Bayesian information maximization that selects samples with the largest information gain from prior to posterior, which gives a supervised sampling involving the labels collected. Experiments show that the proposed methods boost the sampling efficiency as compared to traditional sampling schemes and are thus valuable to practical crowdsourcing experiments.

Learning in neural networks based on a generalized fluctuation theorem.

Information maximization has been investigated as a possible mechanism of learning governing the self-organization that occurs within the neural systems of animals. Within the general context of models of neural systems bidirectionally interacting with environments, however, the role of information maximization remains to be elucidated. For bidirectionally interacting physical systems, universal laws describing the fluctuation they exhibit and the information they possess have recently been discovered. These laws are termed fluctuation theorems. In the present study, we formulate a theory of learning in neural networks bidirectionally interacting with environments based on the principle of information maximization. Our formulation begins with the introduction of a generalized fluctuation theorem, employing an interpretation appropriate for the present application, which differs from the original thermodynamic interpretation. We analytically and numerically demonstrate that the learning mechanism presented in our theory allows neural networks to efficiently explore their environments and optimally encode information about them.

Information-Theoretic Dictionary Learning for Image Classification

We present a two-stage approach for learning dictionaries for object classification tasks based on the principle of information maximization. The proposed method seeks a dictionary that is compact, discriminative, and generative. In the first stage, dictionary atoms are selected from an initial dictionary by maximizing the mutual information measure on dictionary compactness, discrimination and reconstruction. In the second stage, the selected dictionary atoms are updated for improved reconstructive and discriminative power using a simple gradient ascent algorithm on mutual information. Experiments using real data sets demonstrate the effectiveness of our approach for image classification tasks.

Semi-supervised information-maximization clustering

Semi-supervised clustering aims to introduce prior knowledge in the decision process of a clustering algorithm. In this paper, we propose a novel semi-supervised clustering algorithm based on the information-maximization principle. The proposed method is an extension of a previous unsupervised information-maximization clustering algorithm based on squared-loss mutual information to effectively incorporate must-links and cannot-links. The proposed method is computationally efficient because the clustering solution can be obtained analytically via eigendecomposition. Furthermore, the proposed method allows systematic optimization of tuning parameters such as the kernel width, given the degree of belief in the must-links and cannot-links. The usefulness of the proposed method is demonstrated through experiments.

Spatiotemporal receptive fields of direction selective cells self-organized by an Infomax-based learning model

Spatiotemporal receptive fields of direction selective cells, or cells whose responses are selective to direction of motion of visual stimuli, were investigated theoretically. A learning algorithm for a spatiotemporal receptive field of a model visual cell was derived according to an information maximization principle. It was assumed that the model visual cell receives input from nonlagged and lagged model LGN neurons. According to the algorithm, model cells were trained by using computer-generated moving images as training data. After the training, cells tuned to various directions of motion were generated. Also, generated synaptic weight patterns of the cells were similar to Gabor wavelets and receptive fields of simple cells in the visual cortex. Thus, they were orientation and spatial frequency selective as well as direction selective.

Blind-source separation of seismic signals based on information maximization

Prediction methods for seismic multiples are never ideal in practice and an adaptive subtraction process is needed to account for mismatches between the predicted and the actual multiples. We are interested in the problem of separating primary and multiple seismic signals based on their statistical properties. We link recent advances in the blind-source separation problem to the multiple removal problem, and present a novel adaptive subtraction method based on an information maximization principle. Compared with previous methods, our proposed method uses higher-order statistics of the data and incorporates the filtering nature of the adaptive subtraction problem into our algorithm formulation. We use simulations to show that our proposed adaptive subtraction method outperforms the popular least-squares adaptive subtraction and the independent component analysis methods quantitatively, as measured by the mean-squared error, and qualitatively, as evaluated by the visual quality of the image reconstruction.

Information maximization principle explains the emergence of complex cell-like neurons.

We propose models and a method to qualitatively explain the receptive field properties of complex cells in the primary visual cortex. We apply a learning method based on the information maximization principle in a feedforward network, which comprises an input layer of image patches, simple cell-like first-output-layer neurons, and second-output-layer neurons (Model 1). The information maximization results in the emergence of the complex cell-like receptive field properties in the second-output-layer neurons. After learning, second-output-layer neurons receive connection weights having the same size from two first-output-layer neurons with sign-inverted receptive fields. The second-output-layer neurons replicate the phase invariance and iso-orientation suppression. Furthermore, on the basis of these results, we examine a simplified model showing the emergence of complex cell-like receptive fields (Model 2). We show that after learning, the output neurons of this model exhibit iso-orientation suppression, cross-orientation facilitation, and end stopping, which are similar to those found in complex cells. These properties of model neurons suggest that complex cells in the primary visual cortex become selective to features composed of edges to increase the variability of the output.

Online reservoir adaptation by intrinsic plasticity for backpropagation–decorrelation and echo state learning

We propose to use a biologically motivated learning rule based on neural intrinsic plasticity to optimize reservoirs of analog neurons. This rule is based on an information maximization principle, it is local in time and space and thus computationally efficient. We show experimentally that it can drive the neurons' output activities to approximate exponential distributions. Thereby it implements sparse codes in the reservoir. Because of its incremental nature, the intrinsic plasticity learning is well suited for joint application with the online backpropagation-decorrelation or the least mean squares reservoir learning, whose performance can be strongly improved. We further show that classical echo state regression can also benefit from reservoirs, which are pre-trained on the given input signal with the implicit plasticity rule.

An Algorithm for the Detection of Faces on the Basis of Gabor Features and Information Maximization

We propose an algorithm for the detection of facial regions within input images. The characteristics of this algorithm are (1) a vast number of Gabor-type features (196,800) in various orientations, and with various frequencies and central positions, which are used as feature candidates in representing the patterns of an image, and (2) an information maximization principle, which is used to select several hundred features that are suitable for the detection of faces from among these candidates. Using only the selected features in face detection leads to reduced computational cost and is also expected to reduce generalization error. We applied the system, after training, to 42 input images with complex backgrounds (Test Set A from the Carnegie Mellon University face data set). The result was a high detection rate of 87.0%, with only six false detections. We compared the result with other published face detection algorithms.

Binocular disparity encoding cells generated through an Infomax based learning algorithm

A learning algorithm for a model binocular cell was derived according to an information maximization principle and by using a low signal-to-noise-ratio approximation. The algorithm updates cell's synaptic weights so that the information obtained from the cell's output is increased. According to the algorithm, model binocular cells were trained by using computer-generated stereo images as training data. As a result, cells tuned to various disparities were generated. Also, generated synaptic weight patterns of the cells were similar to Gabor-wavelets and receptive fields of simple cells in the visual cortex. Thus, they were orientation and spatial frequency selective as well as disparity selective. Gabor functions were used to fit the generated weight patterns. The fitting results indicated that the generated cells encode disparities in terms of phase disparity and/or position disparity. This result agrees with experimental findings by Anzai et al. [J Neurophys 82 (1999) 874] and is consistent with ICA-based theoretical results obtained [Network: Comput Neural Syst 11 (2000) 191].

Spike-timing-dependent plasticity and relevant mutual information maximization.

Synaptic plasticity was recently shown to depend on the relative timing of the pre- and postsynaptic spikes. This article analytically derives a spike-dependent learning rule based on the principle of information maximization for a single neuron with spiking inputs. This rule is then transformed into a biologically feasible rule, which is compared to the experimentally observed plasticity. This comparison reveals that the biological rule increases information to a near-optimal level and provides insights into the structure of biological plasticity. It shows that the time dependency of synaptic potentiation should be determined by the synaptic transfer function and membrane leak. Potentiation consists of weight-dependent and weight-independent components whose weights are of the same order of magnitude. It further suggests that synaptic depression should be triggered by rare and relevant inputs but at the same time serves to unlearn the baseline statistics of the network's inputs. The optimal depression curve is uniformly extended in time, but biological constraints that cause the cell to forget past events may lead to a different shape, which is not specified by our current model. The structure of the optimal rule thus suggests a computational account for several temporal characteristics of the biological spike-timing-dependent rules.

Visual features of intermediate complexity and their use in classification

The human visual system analyzes shapes and objects in a series of stages in which stimulus features of increasing complexity are extracted and analyzed. The first stages use simple local features, and the image is subsequently represented in terms of larger and more complex features. These include features of intermediate complexity and partial object views. The nature and use of these higher-order representations remains an open question in the study of visual processing by the primate cortex. Here we show that intermediate complexity (IC) features are optimal for the basic visual task of classification. Moderately complex features are more informative for classification than very simple or very complex ones, and so they emerge naturally by the simple coding principle of information maximization with respect to a class of images. Our findings suggest a specific role for IC features in visual processing and a principle for their extraction.

Artificial neural networks for noisy image super-resolution

Noisy incoherent objects, which are too close to be remotely separated by optically imaging beyond the Rayleigh diffraction limit, might be resolved by employing the artificial neural network (ANN) smart pixel post-processing and its mathematical framework, independent component analysis (ICA). It is shown that ICA ANN approach to super-resolution based on information maximization principle could be seen as a part of the general approach called space-bandwidth product adaptation method. Our success is perhaps due to the blind source separation smart-pixel detectors behind the imaging lens (inverse adaptation), while the Rayleigh diffraction limit remains valid for a single instance of the deterministic imaging systems' realization. The blindness is due to the unknown objects, and the unpredictable propagation effect on the net imaging point spread function. Such a software/firmware enhancement of imaging system may have a profound implication to the designs of the new (third) generation imaging systems as well as other non-optical imaging systems.

Implications of a sensory-motor theory for the representation and segregation of speech

Discussed is a computational model which attempts to simulate the principal brain structures involved in sensory-motor sequencing, including auditory cortex, posterior parietal cortex, cerebellum, motor cortex, and the motor output system. A particular focus in this presentation is on the auditory cortical component responsible for computing a 3-D spatio-temporal power spectrum of the auditory image flow [N. Todd, Proc. 16th ICA (1998)], which captures information topographically on three levels, roughly corresponding to timbre, pitch, and rhythm, and reflects the temporal resolution of the cochlea, inferior colliculus, and auditory cortex, respectively. By considering the spatio-temporal orientations of cortical receptive fields (RFs), it is possible to delineate a set of stationary and moving primitive features upon which the higher-order constructs of pitch and timbre are built. A separate model population of secondary auditory cortical neurons, which receive inputs from the population of primary RFs, provides a potential means of accounting for a number of important phonological and melodic processes. Implications of this model are discussed for the development of new algorithms for the representation and segregation of speech via self-organization of primary and secondary RFs by means of the principle of information maximization.