Cloze Probability Research Articles

DERCo: A Dataset for Human Behaviour in Reading Comprehension Using EEG

This paper introduces the DERCo (Dublin EEG-based Reading Experiment Corpus), a language resource combining electroencephalography (EEG) and next-word prediction data obtained from participants reading narrative texts. The dataset comprises behavioral data collected from 500 participants recruited through the Amazon Mechanical Turk online crowd-sourcing platform, along with EEG recordings from 22 healthy adult native English speakers. The online experiment was designed to examine the context-based word prediction by a large sample of participants, while the EEG-based experiment was developed to extend the validation of behavioral next-word predictability. Online participants were instructed to predict upcoming words and complete entire stories. Cloze probabilities were then calculated for each word so that this predictability measure could be used to support various analyses pertaining to semantic context effects in the EEG recordings. EEG-based analyses revealed significant differences between high and low predictable words, demonstrating one important type of potential analysis that necessitates close integration of these two datasets. This material is a valuable resource for researchers in neurolinguistics due to the word-level EEG recordings in context.

Chinese EFL learners' processing of English binomials: the role of interlexical and intralexical factors.

Binomials have been relatively understudied compared to other types of multiword expressions (MWEs) in second language research, such as collocations and idioms. This study investigated English as a Foreign Language (EFL) learners' processing of English binomials and how it is influenced by interlexical factors (L1-L2 congruency and L1-lexicalization) and intralexical factors (word and binomial frequency, binomial reversibility, and binomial predictability). Forty Chinese EFL learners participated in a phrase acceptability judgment task of 64 target binomials (16 congruent L1-lexicalized, 16 congruent L1-nonlexicalized, and 32 incongruent) and 64 non-binomial controls. Results revealed that learners experienced difficulty judging the formulaicity of binomials. They processed binomial stimuli significantly faster than non-binomial baselines, demonstrating a binomial phrase effect. They also processed L1-L2 congruent items faster and more accurately than incongruent items, showing a robust congruency effect. The congruent items which are lexicalized in the L1 showed further processing advantage than the non-lexicalized items, indicating a graded congruency effect. Moreover, binomial reversibility and binomial predictability (measured with cloze probability) also showed significant effects. These findings highlight the need to distinguish and investigate different types of congruency, explore appropriate measures for MWE predictability, and to examine binomials focusing on their unique features.

Phonological prediction during comprehension: A review and meta-analysis of visual-world eye-tracking studies

Current theories of language prediction stipulate that people can predict various types of linguistic information, including the phonological form of a highly predictable word, and some theories posit that phonological prediction plays a pivotal role in prediction-driven learning. However, a review of studies investigating phonological prediction suggests that the effect is inconsistent and small, which raises a question about its role during everyday language comprehension and language learning. Here, I conduct a meta-analysis of visual-world eye-tracking studies investigating phonological prediction with the goal of revealing the robustness and the time-course of the phonological prediction effect. The combined analysis of 20 experiments revealed a small but reliable effect of the phonological prediction effect. This effect emerged rapidly but was not closely aligned to the predictable word onset. The size of this effect depended on the target word cloze probability and depended marginally on the experiment design and the type of visual stimuli. I discuss the implications for the theories of language prediction.

Prior Context and Individual Alpha Frequency Influence Predictive Processing during Language Comprehension.

The extent to which the brain predicts upcoming information during language processing remains controversial. To shed light on this debate, the present study reanalyzed Nieuwland and colleagues' (2018) [Nieuwland, M. S., Politzer-Ahles, S., Heyselaar, E., Segaert, K., Darley, E., Kazanina, N., et al. Large-scale replication study reveals a limit on probabilistic prediction in language comprehension. eLife, 7, e33468, 2018] replication of DeLong and colleagues (2015) [DeLong, K. A., Urbach, T. P., & Kutas, M. Probabilistic word pre-activation during language comprehension inferred from electrical brain activity. Nature Neuroscience, 8, 1117-1121, 2005]. Participants (n = 356) viewed sentences containing articles and nouns of varying predictability, while their EEG was recorded. We measured ERPs preceding the critical words (namely, the semantic prediction potential), in conjunction with postword N400 patterns and individual neural metrics. ERP activity was compared with two measures of word predictability: cloze probability and lexical surprisal. In contrast to prior literature, semantic prediction potential amplitudes did not increase as cloze probability increased, suggesting that the component may not reflect prediction during natural language processing. Initial N400 results at the article provided evidence against phonological prediction in language, in line with Nieuwland and colleagues' findings. Strikingly, however, when the surprisal of the prior words in the sentence was included in the analysis, increases in article surprisal were associated with increased N400 amplitudes, consistent with prediction accounts. This relationship between surprisal and N400 amplitude was not observed when the surprisal of the two prior words was low, suggesting that expectation violations at the article may be overlooked under highly predictable conditions. Individual alpha frequency also modulated the relationship between article surprisal and the N400, emphasizing the importance of individual neural factors for prediction. The present study extends upon existing neurocognitive models of language and prediction more generally, by illuminating the flexible and subject-specific nature of predictive processing.

Is predicting during language processing worth it? Effects of cloze probability and semantic similarity on failed predictions.

Prediction during language processing has been hypothesized to lead to processing benefits. These possible benefits have led to several prominent theories that center around prediction as an essential mechanism in language processing. Such theories typically assume predicting is better than not predicting at all, but do not always account for the potential processing costs from failed predictions. Predicting wrongly can be costly, but the cost may depend on how wrong the prediction was. Across three experiments, we manipulate cloze probability, semantic relatedness, and language modality (production vs. comprehension) to determine whether predicting almost correctly is better than predicting completely incorrectly, and if so, if predicting almost correctly is better than not predicting at all. Results showed that when a predicted ending is replaced with a related term, it is processed faster than when it is replaced with an unrelated term, but that related term is not named more quickly than when it appears after a low constraint sentence. This pattern held regardless of whether participants were asked to produce the sentence-final term by naming a picture (Experiments 1 and 2), or if they were asked to perform a semantic classification of the sentence-final word (Experiment 3). Thus, predicting almost correctly is better than predicting completely incorrectly, but it's not better than not predicting at all. This carries implications for current accounts that argue for processing benefits of prediction during language processing, and suggests that prediction may be used to fine-tune the language system rather than to speed language processing. (PsycInfo Database Record (c) 2024 APA, all rights reserved).

Predictive processing of music and language in autism: Evidence from Mandarin and English speakers.

Atypical predictive processing has been associated with autism across multiple domains, based mainly on artificial antecedents and consequents. As structured sequences where expectations derive from implicit learning of combinatorial principles, language and music provide naturalistic stimuli for investigating predictive processing. In this study, we matched melodic and sentence stimuli in cloze probabilities and examined musical and linguistic prediction in Mandarin- (Experiment 1) and English-speaking (Experiment 2) autistic and non-autistic individuals using both production and perception tasks. In the production tasks, participants listened to unfinished melodies/sentences and then produced the final notes/words to complete these items. In the perception tasks, participants provided expectedness ratings of the completed melodies/sentences based on the most frequent notes/words in the norms. While Experiment 1 showed intact musical prediction but atypical linguistic prediction in autism in the Mandarin sample that demonstrated imbalanced musical training experience and receptive vocabulary skills between groups, the group difference disappeared in a more closely matched sample of English speakers in Experiment 2. These findings suggest the importance of taking an individual differences approach when investigating predictive processing in music and language in autism, as the difficulty in prediction in autism may not be due to generalized problems with prediction in any type of complex sequence processing.

Strong Prediction: Language Model Surprisal Explains Multiple N400 Effects.

Theoretical accounts of the N400 are divided as to whether the amplitude of the N400 response to a stimulus reflects the extent to which the stimulus was predicted, the extent to which the stimulus is semantically similar to its preceding context, or both. We use state-of-the-art machine learning tools to investigate which of these three accounts is best supported by the evidence. GPT-3, a neural language model trained to compute the conditional probability of any word based on the words that precede it, was used to operationalize contextual predictability. In particular, we used an information-theoretic construct known as surprisal (the negative logarithm of the conditional probability). Contextual semantic similarity was operationalized by using two high-quality co-occurrence-derived vector-based meaning representations for words: GloVe and fastText. The cosine between the vector representation of the sentence frame and final word was used to derive contextual cosine similarity estimates. A series of regression models were constructed, where these variables, along with cloze probability and plausibility ratings, were used to predict single trial N400 amplitudes recorded from healthy adults as they read sentences whose final word varied in its predictability, plausibility, and semantic relationship to the likeliest sentence completion. Statistical model comparison indicated GPT-3 surprisal provided the best account of N400 amplitude and suggested that apparently disparate N400 effects of expectancy, plausibility, and contextual semantic similarity can be reduced to variation in the predictability of words. The results are argued to support predictive coding in the human language network.

Information entropy facilitates (not impedes) lexical processing during language comprehension.

It is well known that contextual predictability facilitates word identification, but it is less clear whether the uncertainty associated with the current context (i.e., its lexical entropy) influences sentence processing. On the one hand, high entropy contexts may lead to interference due to greater number of lexical competitors. On the other hand, predicting multiple lexical competitors may facilitate processing through the preactivation of shared semantic features. In this study, we examined whether entropy measured at the trial level (i.e., for each participant, for each item) corresponds to facilitatory or inhibitory effects. Trial-level entropy captures each individual's knowledge about specific contexts and is therefore a more valid and sensitive measure of entropy (relative to the commonly employed item-level entropy). Participants (N = 112) completed two experimental sessions (with counterbalanced orders) that were separated by a 3- to 14-day interval. In one session, they produced up to 10 completions for sentence fragments (N = 647). In another session, they read the same sentences including a target word (whose entropy value was calculated based on the produced completions) while reading times were measured. We observed a facilitatory (not inhibitory) effect of trial-level entropy on lexical processing over and above item-level measures of lexical predictability (including cloze probability, surprisal, and semantic constraint). Extra analyses revealed that greater semantic overlap between the target and the produced responses facilitated target processing. Thus, the results lend support to theories of lexical prediction maintaining that prediction involves broad activation of semantic features rather than activation of full lexical forms.

Are there independent effects of constraint and predictability on eye movements during reading?

Evidence of processing costs for unexpected words presented in place of a more expected completion remains elusive in the eye-movement literature. The current study investigated whether such prediction error costs depend on the source of constraint violation provided by the prior context. Participants' eye movements were recorded as they read predictable words and unpredictable alternatives that were either semantically related or unrelated in three-sentence passages. The passages differed in whether the source of constraint originated solely from the global context provided by the first two semantically rich sentences of the passage, from the local context provided by the final sentence of the passage, from both the global and local context, or from none of the three sentences of the passage. The results revealed the expected processing advantage for predictable completions in any constraining context, although the relative contributions of the different sources of constraint varied across the time course of word processing. Unpredictable completions, however, did not yield any processing costs when the context constrained toward a different word, instead producing immediate processing benefits in the presence of any constraining context. Moreover, the initial processing of related unpredictable completions was enhanced further by the provision of a supportive global context. Predictability effects therefore do not appear to be determined by cloze probability alone but also by the nature of the prior contextual constraint especially when they encourage the construction of higher-level discourse representations. The implications of these findings for understanding existing theoretical models of predictive processing are discussed. (PsycInfo Database Record (c) 2024 APA, all rights reserved).

Predictive Brain Activity Shows Congruent Semantic Specificity in Language Comprehension and Production.

Sentence fragments strongly predicting a specific subsequent meaningful word elicit larger preword slow waves, prediction potentials (PPs), than unpredictive contexts. To test the current predictive processing models, 128-channel EEG data were collected from both sexes to examine whether (1) different semantic PPs are elicited in language comprehension and production and (2) whether these PPs originate from the same specific "prediction area(s)" or rather from widely distributed category-specific neuronal circuits reflecting the meaning of the predicted item. Slow waves larger after predictable than unpredictable contexts were present both before subjects heard the sentence-final word in the comprehension experiment and before they pronounced the sentence-final word in the production experiment. Crucially, cortical sources underlying the semantic PP were distributed across several cortical areas and differed between the semantic categories of the expected words. In both production and comprehension, the anticipation of animal words was reflected by sources in posterior visual areas, whereas predictable tool words were preceded by sources in the frontocentral sensorimotor cortex. For both modalities, PP size increased with higher cloze probability, thus further confirming that it reflects semantic prediction, and with shorter latencies with which participants completed sentence fragments. These results sit well with theories viewing distributed semantic category-specific circuits as the mechanistic basis of semantic prediction in the two modalities.

Prediction in SVO and SOV languages: processing and typological considerations

Abstract In this study, we tested the possibility that different word orders engender different processing preferences. Our key hypothesis was that a head-initial language like English (SVO) allows more prediction compared to a head-final language like Japanese (SOV). In Experiment 1, English and Japanese native speakers completed a cloze task in which they heard a sentence fragment (SV_ in English and SO_ in Japanese) and had to complete it with the word they thought best. We assessed cloze probability of the words produced and voice onset times. Experiment 2 examined written completions in English, in which we compared the cloze probabilities in SV_ fragments versus OS_ fragments. Following the central hypothesis, this experiment allowed us to determine (within language) whether there is more prediction from a noun and verb compared to two nouns. Finally, in Experiment 3, we compared written completions in English and Japanese, in which the stimuli given to participants were identical (SV_ in English and S_V in Japanese). Results across all three experiments were consistent with greater prediction in English. We argue that prediction is one important factor in processing, that it is relied on more in English than in Japanese, and that prediction will be especially favored in languages like English in which the verb regularly precedes its direct object.

Frequency effects in the auditory grammatical decision task

ABSTRACT We investigated effects of phrase-frequency and the frequency of content words in two auditory grammatical decision experiments testing grammatically correct 4-word phrases intermixed with ungrammatical 4-word sequences. A significant phrase-frequency effect was found in Experiment 1 while controlling for syntactic structure (the sequence of parts-of-speech) and word-frequency. No effect of word-frequency was found in Experiment 2 when controlling for phrase-frequency and syntactic structure. A third experiment measured the cloze probability of the final words of the grammatically correct phrases tested in Experiments 1 and 2. Although entering cloze probability as a covariate rendered the effect of phrase-frequency non-significant, a re-analysis of a subset of phrases matched for cloze probability revealed a significant effect of phrase-frequency, while the effect of word-frequency remained non-significant.

Cloze probability, predictability ratings, and computational estimates for 205 English sentences, aligned with existing EEG and reading time data

We release a database of cloze probability values, predictability ratings, and computational estimates for a sample of 205 English sentences (1726 words), aligned with previously released word-by-word reading time data (both self-paced reading and eye-movement records; Frank et al., Behavior Research Methods, 45(4), 1182–1190. 2013) and EEG responses (Frank et al., Brain and Language, 140, 1–11. 2015). Our analyses show that predictability ratings are the best predictors of the EEG signal (N400, P600, LAN) self-paced reading times, and eye movement patterns, when spillover effects are taken into account. The computational estimates are particularly effective at explaining variance in the eye-tracking data without spillover. Cloze probability estimates have decent overall psychometric accuracy and are the best predictors of early fixation patterns (first fixation duration). Our results indicate that the choice of the best measurement of word predictability in context critically depends on the processing index being considered.

Discourse coherence modulates use of predictive processing during sentence comprehension

Context has been shown to be vitally important for comprehension. Lexical processing is facilitated when words are highly predictable given their local sentence context, suggesting that people pre-activate likely upcoming words to aid comprehension. However, this facilitation is affected by knowledge about the global context in which comprehension takes place: people predict less when in an environment where expectations are frequently violated. The current study investigated whether discourse coherence is an additional cue that comprehenders use to modulate lexical prediction. In a series of online, self-paced reading experiments, participants read target sentences preceded by short contextual preambles. Local facilitation effects were manipulated through the cloze probability of a critical word within the target sentence and discourse coherence was manipulated by varying the degree to which the target sentence was consistent with the information presented in the preamble. In the first two experiments, target sentences were read more slowly when they occurred in less coherent discourses, but no local facilitation effects were observed. In the third experiment, we strengthened the predictability manipulation by using semantically anomalous critical words. In this experiment, predictable words were processed more quickly and anomalous words more slowly when they occurred in highly coherent discourse. Our results suggest that comprehenders are sensitive to shifts in the topic of discourse and that they downregulate predictive processing when they encounter incoherence in the discourse. This is consistent with recent theoretical accounts suggesting that comprehenders flexibly engage in predictive processing, pre-activating semantic and lexical information less when their expectations are less likely to be reliable.

Filling the gap: Cloze probability and sentence constraint norms for 807 European Portuguese sentences

Sentence processing is affected by the sentence context and word expectancy. To investigate sentence comprehension experimentally, it is useful to have sentence completion norms with both context constraint and word expectancy measures. In this study, two experiments were conducted to collect norms for completion of 807 European Portuguese sentences. Context constraint was measured through type-token ratio and proportion idiosyncratic responses, while word expectancy was assessed by cloze probability. Besides establishing norms for a large sample of sentences, the study investigated the impact of the production procedure and method of analysis. In Experiment 1, a single-production procedure was used, i.e., participants completed each sentence fragment with only a single response, whereas in Experiment 2, a multiple-production procedure was used, i.e., participants have to provide up to three completion words for each sentence fragment. In Experiment 2, the analyses were obtained using two distinct methods: first-response analysis and combined-response analysis. The results showed that cloze and context measures are comparable between production paradigms and that the results from both analysis methods were correlated. The advantages of each production procedure and analysis method are discussed.

Predictability and decomposability separately contribute to compositional processing of idiomatic language.

When reading, comprehenders construct a message-level representation and integrate new information as it becomes available. Such compositional processing may differ for idioms, where the meanings of the individual words do not always relate to the figurative meaning. Here, we examined how predictability and idiom decomposability contribute to compositional processing. Participants' EEG was recorded while they read sentences containing idioms that varied in decomposability and phrase-final word cloze probability, or their literal match (break the ice/slip on the ice) with little context prior to the phrase, along with adjective insertion conditions (break the freezing ice/slip on the freezing ice). Cloze probability modulated N400 amplitudes to critical words for both idiomatic and literal phrases, whereas P600 amplitudes only elicited by idiomatic phrases were also modulated. Phrases with adjective insertions reduced acceptability judgments, particularly for idioms, and led to N400 amplitude differences compared to critical words, but only for idioms. N400 differences were also found between idiomatic and literal contexts at the point of adjective insertion. Additionally, both cloze probability and decomposability modulated gamma band activity, with greater gamma activity for more predictable and less decomposable idioms, but at different times and with different scalp topographies, suggesting dissociable components of processing. These results support a hybrid model of processing in which multiple linguistic factors determine the type of processing engaged by the brain to comprehend non-literal language. When prior context is minimal, compositional processing may still be engaged when reading idioms.

No Evidence for Semantic Prediction Deficits in Individuals With Cerebellar Degeneration.

Cerebellar involvement in language processing has received considerable attention in the neuroimaging and neuropsychology literatures. Building off the motor control literature, one account of this involvement centers on the idea of internal models. In the context of language, this hypothesis suggests that the cerebellum is essential for building semantic models that, in concert with the cerebral cortex, help anticipate or predict linguistic input. To date, supportive evidence has primarily come from neuroimaging studies showing that cerebellar activation increases in contexts in which semantic predictions are generated and violated. Taking a neuropsychological approach, we put the internal model hypothesis to the test, asking if individuals with cerebellar degeneration (n = 14) show reduced sensitivity to semantic prediction. Using a sentence verification task, we compare reaction time to sentences that vary in terms of cloze probability. We also evaluated a more constrained variant of the prediction hypothesis, asking if the cerebellum facilitates the generation of semantic predictions when the content of a sentence refers to a dynamic rather than static mental transformation. The results failed to support either hypothesis: Compared to matched control participants (n = 17), individuals with cerebellar degeneration showed a similar reduction in reaction time for sentences with high cloze probability and no selective impairment in predictions involving dynamic transformations. These results challenge current theorizing about the role of the cerebellum in language processing, pointing to a misalignment between neuroimaging and neuropsychology research on this topic.

Prediction in the maze: Evidence for probabilistic pre-activation from the English a/an contrast

The idea that comprehenders predict upcoming linguistic content has become core to many theories of language processing. Experimental studies exploiting morphosyntactic and phonotactic constraints on a word form preceding a high cloze target word have been key to underpinning predictive accounts of comprehension, but investigating these tight sequential contrasts with traditional behavioral methods is difficult. The maze task, with its more focal measure of incremental processing, may provide a cheap and easy methodology to study early cues to prediction. An experiment investigating the a/an contrast (DeLong, Urbach, & Kutas, 2005; Nieuwland, et al., 2018) using A-maze (Boyce, Futrell, & Levy, 2020) finds that unexpected articles, as well as nouns, elicit slower focal response times. Response times are also shown to be inversely related to noun cloze probabilities, with slower responders showing larger effects of expectation. This study demonstrates that the maze task can be sensitive to expectation and is a useful alternative methodology for investigating prediction in comprehension.

Evoking the N400 Event-related Potential (ERP) Component Using a Publicly Available Novel Set of Sentences with Semantically Incongruent or Congruent Eggplants (Endings)

During speech comprehension, the ongoing context of a sentence is used to predict sentence outcome by limiting subsequent word likelihood. Neurophysiologically, violations of context-dependent predictions result in amplitude modulations of the N400 event-related potential (ERP) component. While the N400 is widely used to measure semantic processing and integration, no publicly-available auditory stimulus set is available to standardize approaches across the field. Here, we developed an auditory stimulus set of 442 sentences that utilized the semantic anomaly paradigm, measured cloze probability for all stimuli, and was made for both children and adults. With 20 neurotypical adults, we validated that this set elicits robust N400′s, as well as two additional semantically-related ERP components: the recognition potential (∼ 250 ms) and the late positivity component (∼ 600 ms). This stimulus set (https://doi.org/10.5061/dryad.9ghx3ffkg) and the 20 high-density (128-channel) electrophysiological datasets (https://doi.org/10.5061/dryad.6wwpzgmx4) are made publicly available to promote data sharing and reuse. Future studies that use this stimulus set to investigate sentential semantic comprehension in both control and clinical populations may benefit from the increased comparability and reproducibility within this field of research.

Hedging Bets in Linguistic Prediction: Younger and Older Adults Vary in the Breadth of Predictive Processing

Language processing is predictive in nature, but it is unknown whether language users generate multiple predictions about upcoming content simultaneously or whether spreading activation from one pre-activated word facilitates other words downstream. Simultaneously, developmental accounts of predictive processing simultaneously highlight potential tension among spreading activation vs. multiple activation accounts. We used self-paced reading to investigate if younger and older readers of German generate (multiple) graded predictions about the grammatical gender of nouns. Gradedness in predictions was operationalized as the difference in cloze probability between the most likely and second-most likely continuation that could complete a sentence. Sentences with a greater probabilistic difference were considered as imbalanced and more biased towards one gender. Sentences with lower probabilistic differences were considered to be more balanced towards multiple genders. Both young and older adults engaged in predictive processing. However, only younger adults activated multiple predictions, with slower reading times (RTs) when gender representations were balanced, but facilitation when one gender was more likely than others. In contrast, older adults’ RTs did not pattern with imbalance but merely with predictability, showing that, while able to generate predictions based on context, older adults did not predict multiple gender continuations. Hence, our findings suggest that (younger) language users generate graded predictions about upcoming content, by weighing possible sentence continuations according to their difference in cloze probability. Compared to younger adults, older adults’ predictions are reduced in scope. The results provide novel theoretical insights into the developmental mechanisms involved in predictive processing.