Ideas Of Information Theory Research Articles

Testing entropy-based search strategies for a visual classification task

There is experimental evidence that saccades during visual search preferentially target locations that contain task-relevant information [1]. However, the question remains what kind of strategy people use to decide what is relevant for a task. Do we use simple heuristics or complex algorithms based on the ideas of information theory? For example, for a shape-learning and -matching task sequential entropy-minimization was successfully used to predict human fixations [2]. Inspired by this fact, we test three entropy-based strategies for sequential location (image patches) selection during a visual classification task. In our experiments, we used a more constrained setup than eye-tracking, were the subject had to click on an image location explicitly. During the experiment, images of clock digits consisting of seven patches at different locations are presented. Initially all patches are covered and shown as a gray outline (Figure ​(Figure1A).1A). Patches can be uncovered by clicking at them. The task is to uncover a minimal number of patches that, in the subject's opinion, help to identify the digit. The presentation stops when the digit is unambiguously identified (Figure 1 A bottom). Figure 1 A. Examples of covered and partially uncovered digits. B. Log-likelihoods of the observed data given different strategies for a single subject. Determining the task-relevant parts of a visual scene is closely related to the problem of feature selection in machine learning. Correspondingly, we tested to what extend human behavior can be explained in terms of feature selection criteria. The first strategy (MI, Mutual Information), is considered as a heuristic. It simply ranks all locations according to the mutual information they provide about digit identity . The second strategy (CMI, Conditional Mutual Information) takes into account high-order dependencies between locations, i.e. selects those that are both informative and non-redundant with respect to the already selected locations. The third strategy (AMI, Adaptive conditional Mutual Information) is an adaptive version of CMI, which takes into account also observed values of the already attended locations, suggesting that every next decision depends on what one has seen on the previous steps. Thus, in contrast to MI and CMI, the optimal location sequence is different for each image. We compare these strategies with respect to their explanatory power of the observed behavioral data . For this, we turn them into generative models of patch sequences. Each model assumes that the next patch is chosen as softly maximizing the strategy-specific information about the image class. The softmax function is parametrized with β. For low values of β all sequences are equally likely, i.e. the subject acts randomly, whereas a high β concentrates the probability mass on sequences which select the most informative patches – according to each strategy – on each step. Preliminary results show that the behavior of most subjects is best explained by the AMI strategy. Our clicking experiment provide evidence that for a visual classification task people are able to employ quite complex entropy-based search strategies. We found in particular that, even though it is more computationally demanding, most people act adaptively, i.e. take into account image-specific information.

Information theory ‐ a psychological study in old and new concepts

The ideas of information theory which underlie cybernetics and its information‐processing aspects still has a solid reputation to be both extremely abstract and highly mathematical. Furthermore, the tract has been accused of a slow development and not accentuating the difference between information and communication theories. In this paper an attempt is made to demonstrate that principles and concepts used in information theory can be both understood and used from a psychological basis ‐ without the use of complicated mathematics. The difference between information and communication theory is considered as well as some new and revolutionary concepts. The study shows that the influences of the investigated area have grown considerably with the contributions from information physics.

A new version of playing battleships

A new version of playing battleships is presented that highlights some of the essential ideas of Information Theory and Decision Theory;

On the entropy of the liquid state

Using some of the basic ideas of information theory, an approximate expression is derived for the contribution to the entropy of a real monatomic liquid arising from the bond-length variation between nearest neighbouring atoms. Some further simplifying approximations enable an equation of state to be derived. This is compared with experimental results for liquid argon. When the equation of state is adjusted to fit one experimental isotherm, the calculated values of CP are found to lie within 20% of the experimental values.

Exponential Ensemble for Random Matrices

Using the ideas of information theory, it is pointed out that the Gaussian ensemble for random Hermitian matrices can be characterized as the ``most random'' ensemble of these matrices. Pursuing the same characterization for positive matrices, we are led in a natural way to the definition of the exponential ensemble. Transforming to a representation of eigenvectors and eigenvalues, the joint distribution function for the eigenvalues of positive Hermitian matrices is found for this ensemble. The asymptotic single-level density formula is derived, using a semiclassical model. It is found that the density is convex from below over most of the domain of eigenvalues. Since this is similar to the exponential dependence expected for nuclear spectra, the density is examined in a region near the level taken to correspond with the lowest nuclear level. It turns out, however, that the density is concave from below near this level, and that a large number of levels are contained in this concave region. Hence the exponential ensemble does not fairly represent nuclear energy levels, at least in this region. Various changes are made in the measure on the matrix ensemble to determine to what extent the level density depends on this measure. It is seen that the level density graph retains a characteristic shape for a wide variety of measures. The relationship of the limiting behavior of the level density for positive matrices to the semicircle law is noted.