Information-theoretic Account Research Articles

Word reuse and combination support efficient communication of emerging concepts

A key function of the lexicon is to express novel concepts as they emerge over time through a process known as lexicalization. The most common lexicalization strategies are the reuse and combination of existing words, but they have typically been studied separately in the areas of word meaning extension and word formation. Here, we offer an information-theoretic account of how both strategies are constrained by a fundamental tradeoff between competing communicative pressures: Word reuse tends to preserve the average length of word forms at the cost of less precision, while word combination tends to produce more informative words at the expense of greater word length. We test our proposal against a large dataset of reuse items and compounds that appeared in English, French, and Finnish over the past century. We find that these historically emerging items achieve higher levels of communicative efficiency than hypothetical ways of constructing the lexicon, and both literal reuse items and compounds tend to be more efficient than their nonliteral counterparts. These results suggest that reuse and combination are both consistent with a unified account of lexicalization grounded in the theory of efficient communication.

Causation Bridges the Two Times

Abstract The two-times problem, where time as experienced seems to have distinctive features different than those found in fundamental physics, appears to be more intractable than necessary, I argue, because the two times are marked out from the positions furthest apart: neuroscience and physics. I offer causation as exactly the kind of bridge between these two times that authors like Buonomano and Rovelli (forthcoming) are seeking. It is a historical contingency from philosophical discussions around phenomenology, and a methodological artefact from neuroscience, that most studies of temporal features of experience require subjects to be sufficiently still that their engagement with affordances in the environment can be at best tested in artificial and highly constrained ways. Physics does not offer an account of causation, but accounts of causation are tied to or grounded in physics in ways that can be clearly delineated. Causation then serves as a bridge that coordinates time as experienced, via interaction with affordances in the environment, with time in physics as it constrains causal relationships. I conclude by showing how an information-theoretic account of causation fits neatly into and extends the information gathering and utilizing system (IGUS) of Gruber et al. (Front. Psychol., 13, 718505).

The Information-Theoretic Account of Knowledge, Closure and the KK Thesis

Abstract One common objection to Dretske’s Information Theoretic Account of Knowledge (ITAK) is that it violates closure. I show that it does not, and that extant arguments attempting to establish that it does rely instead on the KK thesis. That thesis does fail for ITAK. I show moreover that an interesting consequence of ITAK obeying the closure principle after all is that on this view if skepticism is false, we can have a great deal of empirical knowledge, but it is in principle impossible to know that skepticism is false. In short, a proper understanding of how ITAK closes off the KK thesis shows that we can 1) take seriously the skeptic, we can 2) respond to her appropriately that we do have knowledge and we can 3) keep closure.

Predictable Words Are More Likely to Be Omitted in Fragments-Evidence From Production Data.

Instead of a full sentence like Bring me to the university (uttered by the passenger to a taxi driver) speakers often use fragments like To the university to get their message across. So far there is no comprehensive and empirically supported account of why and under which circumstances speakers sometimes prefer a fragment over the corresponding full sentence. We propose an information-theoretic account to model this choice: A speaker chooses the encoding that distributes information most uniformly across the utterance in order to make the most efficient use of the hearer's processing resources (Uniform Information Density, Levy and Jaeger, 2007). Since processing effort is related to the predictability of words (Hale, 2001) our account predicts two effects of word probability on omissions: First, omitting predictable words (which are more easily processed), avoids underutilizing processing resources. Second, inserting words before very unpredictable words distributes otherwise excessively high processing effort more uniformly. We test these predictions with a production study that supports both of these predictions. Our study makes two main contributions: First we develop an empirically motivated and supported account of fragment usage. Second, we extend previous evidence for information-theoretic processing constraints on language in two ways: We find predictability effects on omissions driven by extralinguistic context, whereas previous research mostly focused on effects of local linguistic context. Furthermore, we show that omissions of content words are also subject to information-theoretic well-formedness considerations. Previously, this has been shown mostly for the omission of function words.

Topic drop in German: Empirical support for an information-theoretic account to a long-known omission phenomenon

AbstractGerman allows fortopic drop(Fries 1988), the omission of a preverbal constituent from a V2 sentence. I address the underexplored question of why speakers use topic drop with a corpus study and two acceptability rating studies. I propose an information-theoretic explanation based on the Uniform Information Density hypothesis (Levy and Jaeger 2007) that accounts for the full picture of data. The information-theoretic approach predicts that topic drop is more felicitous when the omitted constituent is predictable in context and easy to recover. This leads to a more optimal use of the hearer’s processing capacities. The corpus study on the FraC corpus (Horch and Reich 2017) shows that grammatical person, verb probability and verbal inflection impact the frequency of topic drop. The two rating experiments indicate that these differences in frequency are also reflected in acceptability and additionally evidence an impact of topicality on topic drop. Taken together my studies constitute the first systematic empirical investigation of previously only sparsely researched observations from the literature. My information-theoretic account provides a unifying explanation of these isolated observations and is also able to account for the effect of verb probability that I find in my corpus study.

Smooth Signals and Syntactic Change

A large body of recent work argues that considerations of information density predict various phenomena in linguistic planning and production. However, the usefulness of an information theoretic account for explaining diachronic phenomena has remained under-explored. Here, we test the idea that speakers prefer informationally uniform utterances on diachronic data from historical English and Icelandic. Our results show that: (i) the information density approach allows us to predict that Subject and Object type will affect the frequencies of OV and VO in specific ways, creating a complex Constant Rate Effect, (ii) the bias towards information uniformity explains this CRE and may help to explain others, and (iii) communities of speakers are constant in their average target level of information uniformity over long periods of historical time. This finding is consistent with an understanding of this bias which places it deep in the human language faculty and the human faculty for communication.

Patterns Bit by Bit. An Entropy Model for Rule Induction

ABSTRACTFrom limited evidence, children track the regularities of their language impressively fast and they infer generalized rules that apply to novel instances. This study investigated what drives the inductive leap from memorizing specific items and statistical regularities to extracting abstract rules. We propose an innovative entropy model that offers one consistent information-theoretic account for both learning the regularities in the input and generalizing to new input. The model predicts that rule induction is an encoding mechanism gradually driven as a natural automatic reaction by the brain’s sensitivity to the input complexity (entropy) interacting with the finite encoding power of the human brain (channel capacity). In two artificial grammar experiments with adults we probed the effect of input complexity on rule induction. Results showed that as the input becomes more complex, the tendency to infer abstract rules increases gradually.

Collective ignorance: an information theoretic account

We are ignorant knowers. This paper proposes an information theoretic explanation of that fact. The explanation is a conjunction of three claims. First, that even in those dimensions where we are capable of picking up information, there is information that we don’t pick up. Second, that there can be dimensions of information for which we lack the capacity to pick up any information whatsoever. Third, that we don’t know whether the faculties and cognitive capacities we are endowed with process all the information that they pick up.

Developmental parsing and linguistic knowledge: Reexamining the role of cognitive control in the kindergarten path effect

During spoken language comprehension, young children have difficulties in revising incorrect predictions about the structural properties of sentences. Recent research on individual differences suggests that these errors may reflect immature cognitive control. However, this evidence overlooks challenges with interpreting cross-task correlations and additional effects of linguistic knowledge on developmental parsing. To account for within-individual variation in task performance, this study compared sentence comprehension across two samples: one where socioeconomic status (SES) background was related to global language knowledge (Diagnostic Evaluation of Language Variation–Screening Test; n = 60) and another where it was related to cognitive control abilities (Stroop task; n = 46). Children (3- to 6-year-olds) heard sentences with an agent-first bias (e.g., “The blicket will be quickly …”) that predicted late-arriving verb morphology (e.g., actives: “… eating the seal”) or conflicted with this cue (e.g., passives: “… eaten by the seal”). Consistent with prior work, final interpretations were less accurate when revision was needed for passives compared with actives. Critically, when SES was a proxy for global language variation, children from higher-SES backgrounds revised mispredictions more than their lower-SES peers on average. However, when SES tracked variation in cognitive control instead, SES effects on revision were absent. This suggests that variation in revising mispredictions during development may be related to linguistic knowledge rather than cognitive control. We discuss these results in light of known effects of linguistic knowledge on sentence comprehension and describe an information-theoretic account for why limited knowledge may lead children to favor predicted meanings over revised meanings during comprehension.

Noise, Economy, and the Emergence of Information Structure in a Laboratory Language.

The acceptability of sentences in natural language is constrained not only grammaticality, but also by the relationship between what is being conveyed and such factors as context and the beliefs of interlocutors. In many languages the critical element in a sentence (its focus) must be given grammatical prominence. There are different accounts of the nature of focus marking. Some researchers treat it as the grammatical realization of a potentially arbitrary feature of universal grammar and do not provide an explicit account of its origins; others have argued, however, that focus marking is a (grammaticalized) functional solution to the problem of efficiently transmitting information via a noisy channel. By adding redundancy to highlight critical elements in particular, focus protects key parts of the message from noise. If this information-theoretic account is true, then we should expect focus-like behavior to emerge even in non-linguistic communication systems given sufficient noise and pressures for efficiency. We tested this in an experiment in which participants played a simple communication game in which they had to click cells on a grid to communicate one of two line figures drawn across the grid. We manipulated the noise, available time, and required effort, and measured patterns of redundancy. Because the lines in many cases overlapped, meaning that only some parts of each line could be used to distinguish it from the other, we were able to compare the extent to which effort was expended on adding redundancy to critical (non-overlapping) and non-critical (overlapping) parts of the message. The results supported the information-theoretic account of focus and shed light on the emergence of information structure in language.

Efficient compression in color naming and its evolution

We derive a principled information-theoretic account of cross-language semantic variation. Specifically, we argue that languages efficiently compress ideas into words by optimizing the information bottleneck (IB) trade-off between the complexity and accuracy of the lexicon. We test this proposal in the domain of color naming and show that (i) color-naming systems across languages achieve near-optimal compression; (ii) small changes in a single trade-off parameter account to a large extent for observed cross-language variation; (iii) efficient IB color-naming systems exhibit soft rather than hard category boundaries and often leave large regions of color space inconsistently named, both of which phenomena are found empirically; and (iv) these IB systems evolve through a sequence of structural phase transitions, in a single process that captures key ideas associated with different accounts of color category evolution. These results suggest that a drive for information-theoretic efficiency may shape color-naming systems across languages. This principle is not specific to color, and so it may also apply to cross-language variation in other semantic domains.

Compositional clustering in task structure learning.

Humans are remarkably adept at generalizing knowledge between experiences in a way that can be difficult for computers. Often, this entails generalizing constituent pieces of experiences that do not fully overlap, but nonetheless share useful similarities with, previously acquired knowledge. However, it is often unclear how knowledge gained in one context should generalize to another. Previous computational models and data suggest that rather than learning about each individual context, humans build latent abstract structures and learn to link these structures to arbitrary contexts, facilitating generalization. In these models, task structures that are more popular across contexts are more likely to be revisited in new contexts. However, these models can only re-use policies as a whole and are unable to transfer knowledge about the transition structure of the environment even if only the goal has changed (or vice-versa). This contrasts with ecological settings, where some aspects of task structure, such as the transition function, will be shared between context separately from other aspects, such as the reward function. Here, we develop a novel non-parametric Bayesian agent that forms independent latent clusters for transition and reward functions, affording separable transfer of their constituent parts across contexts. We show that the relative performance of this agent compared to an agent that jointly clusters reward and transition functions depends environmental task statistics: the mutual information between transition and reward functions and the stochasticity of the observations. We formalize our analysis through an information theoretic account of the priors, and propose a meta learning agent that dynamically arbitrates between strategies across task domains to optimize a statistical tradeoff.

Expectancy-Violation and Information-Theoretic Models of Melodic Complexity

The present study assesses two types of models for melodic complexity: one based on expectancy violations and the other one related to an information-theoretic account of redundancy in music. Seven different datasets spanning artificial sequences, folk and pop songs were used to refine and assess the models. The refinement eliminated unnecessary components from both types of models. The final analysis pitted three variants of the two model types against each other and could explain from 46-74% of the variance in the ratings across the datasets. The most parsimonious models were identified with an information-theoretic criterion. This suggested that the simplified expectancy-violation models were the most efficient for these sets of data. However, the differences between all optimised models were subtle in terms both of performance and simplicity.

Safety, Closure, and the Flow of Information

In his earlier writings, Fred Dretske proposed an anti-skeptical strategy that is based on a rejection of the view that knowledge is closed under known entailment. This strategy is seemingly congenial with a sensitivity condition for knowledge, which is often associated with Dretske’s epistemology. However, it is not obvious how Dretske’s early account meshes with the information-theoretic view developed in Knowledge and the Flow of Information. One aim of this paper is to elucidate the connections between these accounts. First I argue that, contrary to an objection raised by Christoph Jager, the information-theoretic account is compatible with Dretske’s anti-skeptical strategy based on the rejection of closure. This strategy invokes the notion of channel conditions, which are roughly speaking those conditions that are necessary and jointly sufficient for a signal to carry information. I propose an interpretation of the account that is based on the idea that a signal’s carrying information requires that the channel conditions are stable. It is shown that the resulting account incorporates both a sensitivity condition and a safety condition for knowledge. Finally, I demonstrate how this proposal allows for knowledge of modally robust propositions without making its acquisition too easy, as simple safety accounts do. I end with a suggestion concerning the direction that future research should take, based on the fact that in its present form the information-theoretic account does not capture inferential knowledge.

The Givenness of the Human Learning Experience and Its Incompatibility with Information Analytics

The rise of learning analytics, the application of complex metrics developed to exploit the proliferation of ‘Big Data’ in educational work, raises important moral questions about the nature of what is measurable in education. Teachers, schools and nations are increasingly held to account based on metrics, exacerbating the tendency for fine-grained measurement of learning experiences. In this article, the origins of learning analytics ontology are explored, drawing upon core ideas in the philosophy of computing, such as the general definition of information and the information-theoretic account of knowledge. Drawing upon a reading of Descartes Meditatio II, which extends the phenomenology of Jean-Luc Marion into a pedagogy of intentionality, the article identifies a fundamental incompatibility between the subjective experience of learning and the information-theoretic account of knowledge. Human subjects experience and value their own information incommensurably with the ways in which computers measure and quantify information. The consequences of this finding for the design of online learning environments, and the necessary limitations of learning analytics and measurement are explored.

Propositional content in signalling systems

Skyrms, building on the work of Dretske, has recently developed a novel information-theoretic account of propositional content in simple signalling systems. Information-theoretic accounts of content traditionally struggle to accommodate the possibility of misrepresentation, and I show that Skyrms’s account is no exception. I proceed to argue, however, that a modified version of Skyrms’s account can overcome this problem. On my proposed account, the propositional content of a signal is determined not by the information that it actually carries, but by the information that it would carry at the nearest separating equilibrium of the underlying evolutionary dynamics. I show that this amended account yields reasonable ascriptions of false propositional content in a well-known formal model of the evolution of communication (the ‘Philip Sidney’ game), and close with a discussion of the serious but perhaps not insuperable difficulties we face in applying the account to examples of signalling in the real world.

Optimal Coding Predicts Attentional Modulation of Activity in Neural Systems

Neuronal activity in response to a fixed stimulus has been shown to change as a function of attentional state, implying that the neural code also changes with attention. We propose an information-theoretic account of such modulation: that the nervous system adapts to optimally encode sensory stimuli while taking into account the changing relevance of different features. We show using computer simulation that such modulation emerges in a coding system informed about the uneven relevance of the input features. We present a simple feedforward model that learns a covert attention mechanism, given input patterns and coding fidelity requirements. After optimization, the system gains the ability to reorganize its computational resources (and coding strategy) depending on the incoming attentional signal, without the need of multiplicative interaction or explicit gating mechanisms between units. The modulation of activity for different attentional states matches that observed in a variety of selective attention experiments. This model predicts that the shape of the attentional modulation function can be strongly stimulus dependent. The general principle presented here accounts for attentional modulation of neural activity without relying on special-purpose architectural mechanisms dedicated to attention. This principle applies to different attentional goals, and its implications are relevant for all modalities in which attentional phenomena are observed.

KNOWLEDGE EXTERNALISM

Abstract: A popular counterexample directed against externalist epistemological views is that of an agent (Lehrer's “Truetemp” for example) whose beliefs are clearly neither justified nor known but that were generated in the manner that the externalist requires, thereby demonstrating externalism to be insufficient. In this essay I develop and defend an externalist account of knowledge – essentially an elaboration of Fred Dreske's information-theoretic account – that is not susceptible to those criticisms. I then briefly discuss the relationship between knowledge and justification.

Information, Closure, And Knowledge: On Jäger’s Objection To Dretske

Christoph Jager (2004) argues that Dretske’s information theory of knowledge raises a serious problem for his denial of closure of knowledge under known entailment: Information is closed under known entailment (even under entailment simpliciter); given that Dretske explains the concept of knowledge in terms of “information”, it is hard to stick with his denial of closure for knowledge. Thus, one of the two basic claims of Dretske would have to go. Since giving up the denial of closure would commit Dretske to skepticism, it would most probably be better to rather give up the information-theoretic account of knowledge. But that means that one of the best externalist views of knowledge has to be given up. I argue here that Jager is mistaken and that there is no problem for Dretske. There is a rather easy way out of Jager’s problem.

Asymmetric dependencies, ideal conditions, and meaning

Abstract Jerry Fodor has proposed a causal theory of meaning based on the notion of a certain asymmetric dependency between the causes of a symbol's tokens. This theory is held to be an improvement on Dennis Stampe's causal theory of meaning and Fred Dretske's information theoretic account, because it allegedly solves what Fodor calls the “disjunction problem”, and does so without recourse to the kind of optimal (ideal) conditions to which Stampe and Dretske appeal. A series of counterexamples is proposed to Fodor's account, which, it is argued, can only be met by reintroducing that same appeal to optimal conditions that he had sought to eliminate. It is then argued that Fodor's notion of asymmetric dependence is not fundamental to the explanation of why a symbol means what it does: on the contrary, the symbol's meaning what it does is explanatorily prior to the obtaining of the asymmetry, so the asymmetry cannot be used to explain the symbol's meaning. Finally, it is argued that the “disjunction problem ...