Real-time Language Comprehension Research Articles

Comparing Functional Trend and Learning among Groups in Intensive Binary Longitudinal Eye-Tracking Data using By-Variable Smooth Functions of GAMM.

This paper presents a model specification for group comparisons regarding a functional trend over time within a trial and learning across a series of trials in intensive binary longitudinal eye-tracking data. The functional trend and learning effects are modeled using by-variable smooth functions. This model specification is formulated as a generalized additive mixed model, which allowed for the use of the freely available mgcv package (Wood in Package 'mgcv.' https://cran.r-project.org/web/packages/mgcv/mgcv.pdf , 2023) in R. The model specification was applied to intensive binary longitudinal eye-tracking data, where the questions of interest concern differences between individuals with and without brain injury in their real-time language comprehension and how this affects their learning over time. The results of the simulation study show that the model parameters are recovered well and the by-variable smooth functions are adequately predicted in the same condition as those found in the application.

Spontaneous perspective-taking in real-time language comprehension: evidence from eye-movements and grain of coordination

Linguistic communication requires interlocutors to consider differences in each other’s knowledge (perspective-taking). However, perspective-taking might either be spontaneous or strategic. We monitored listeners’ eye movements in a referential communication task. A virtual speaker gave temporally ambiguous instructions with scalar adjectives (“big” in “big cubic block”). Scalar adjectives assume a contrasting object (a small cubic block). We manipulated whether the contrasting object (a small triangle) for a competitor object (a big triangle) was in common ground (visible to both speaker and listener) or was occluded so it was in the listener’s privileged ground, in which case perspective-taking would allow earlier reference-resolution. We used a complex visual context with multiple objects, making strategic perspective-taking unlikely when all objects are in the listener’s referential domain. A turn-taking, puzzle-solving task manipulated whether participants could anticipate a more restricted referential domain. Pieces were either confined to a small area (requiring fine-grained coordination) or distributed across spatially distinct regions (requiring only coarse-grained coordination). Results strongly supported spontaneous perspective-taking: Although comprehension was less time-locked in the coarse-grained condition, participants in both conditions used perspective information to identify the target referent earlier when the competitor contrast was in privileged ground, even when participants believed instructions were computer-generated.

Large-scale evidence for logarithmic effects of word predictability on reading time

During real-time language comprehension, our minds rapidly decode complex meanings from sequences of words. The difficulty of doing so is known to be related to words' contextual predictability, but what cognitive processes do these predictability effects reflect? In one view, predictability effects reflect facilitation due to anticipatory processing of words that are predictable from context. This view predicts a linear effect of predictability on processing demand. In another view, predictability effects reflect the costs of probabilistic inference over sentence interpretations. This view predicts either a logarithmic or a superlogarithmic effect of predictability on processing demand, depending on whether it assumes pressures toward a uniform distribution of information over time. The empirical record is currently mixed. Here, we revisit this question at scale: We analyze six reading datasets, estimate next-word probabilities with diverse statistical language models, and model reading times using recent advances in nonlinear regression. Results support a logarithmic effect of word predictability on processing difficulty, which favors probabilistic inference as a key component of human language processing.

Processing of grammatical gender agreement morphemes in Polish: evidence from the Visual World Paradigm

This paper presents a psycholinguistic study of the processing of grammatical gender agreement morphemes in Polish, which has three gender categories (masculine, feminine, neuter), as well as what language-internal factors impact this processing. Results from an eye-tracking study using the Visual World Paradigm show that, during real-time language comprehension, adult monolingual speakers of Polish use cues from gender agreement on a prenominal adjective to anticipate the upcoming noun. An exploration of language-internal factors affecting this anticipatory processing finds this effect in all three genders, suggesting that encountering the relevant nominative-case agreement morpheme during language comprehension leads to automatic activation of a gender node in the mental lexicon, consistent with the literature on other languages with grammatical gender. These results hold true for the neuter agreement morpheme, despite the fact that this morpheme also instantiates default gender agreement in the language and is syncretic with the nominative plural agreement morpheme in all three genders. Further investigation finds that, while agreement morphemes for each gender prompt anticipatory processing, the reliability of a masculine agreement morpheme as a cue to gender is reduced in the presence of a neuter distractor, and vice versa. This raises questions regarding phonological proximity between the realized suffix and the suffix that would cue the distractor, with implications for the acquisition and processing of gender agreement morphology in Polish.

Five-year-olds’ sensitivity to knowledge discrepancies about object identity during online language comprehension

In everyday communication, children experience situations where their knowledge or perspectives differ from those of their communicative partner. The current study examined this issue in the context of real-time language comprehension, focusing on 5-year-old children’s ability to manage knowledge discrepancies about the identity of mutually visible objects. In Experiment 1, we examined 5-year-olds’ ability to manage privileged knowledge about an object’s identity. Using a referential communication task, we tested children (N = 60) in either a shared knowledge condition, where both the child and the speaker knew the identity of a visually misleading object (e.g., a candle that looks like an apple), or a privileged knowledge condition, where only the child knew the identity of the visually misleading object. Of interest was whether children could suppress private knowledge while processing a phonologically related word (e.g., “Look at the candy”). Results showed that children did not inhibit this knowledge during the early moments of referential interpretation. In Experiment 2 (N = 30), we contrasted the privileged knowledge condition in Experiment 1 with the more traditional scenario used to test common ground use, where the child knows the speaker cannot see certain display objects. Results confirmed a stronger ability to manage discrepancies in the latter case. Together, the findings demonstrate differences in children’s ability to manage distinct types of knowledge discrepancies during real-time language comprehension.

Sensitivity to Semantic Relationships in U.S. Monolingual English-Speaking Typical Talkers and Late Talkers.

Late talkers (LTs) are a group of children who exhibit delays in language development without a known cause. Although a hallmark of LTs is a reduced expressive vocabulary, little is known about LTs' processing of semantic relations among words in their emerging vocabularies. This study uses an eye-tracking task to compare 2-year-old LTs' and typical talkers' (TTs') sensitivity to semantic relationships among early acquired words. U.S. monolingual English-speaking LTs (n = 21) and TTs (n = 24) completed a looking-while-listening task in which they viewed two images on a screen (e.g., a shirt and a pizza), while they heard words that referred to one of the images (e.g., Look! Shirt!; target-present condition) or a semantically related item (e.g., Look! Hat!; target-absent condition). Children's eye movements (i.e., looks to the target) were monitored to assess their sensitivity to these semantic relationships. Both LTs and TTs looked longer at the semantically related image than the unrelated image on target-absent trials, demonstrating sensitivity to the taxonomic relationships used in the experiment. There was no significant group difference between LTs and TTs. Both groups also looked more to the target in the target-present condition than in the target-absent condition. These results reveal that, despite possessing smaller expressive vocabularies, LTs have encoded semantic relationships in their receptive vocabularies and activate these during real-time language comprehension. This study furthers our understanding of LTs' emerging linguistic systems and language processing skills. https://doi.org/10.23641/asha.23303987.

Temporary disruption in language processing reflected as multiscale temporal discoordination in a recurrent network

By juxtaposing time series analyses of activity measured from a fully recurrent network undergoing disrupted processing and of activity measured from a continuous meta-cognitive report of disruption in real-time language comprehension, we present an opportunity to compare the temporal statistics of the state-space trajectories inherent to both systems. Both the recurrent network and the human language comprehension process appear to exhibit long-range temporal correlations and low entropy when processing is undisrupted and coordinated. However, when processing is disrupted and discoordinated, they both exhibit more short-range temporal correlations and higher entropy. We conclude that by measuring human language comprehension in a dense-sampling manner similar to how we analyze the networks, and analyzing the resulting data stream with nonlinear time series analysis techniques, we can obtain more insight into the temporal character of these discoordination phases than by simply marking the points in time at which they peak.

Temporary disruption in language processing reflected as multiscale temporal discoordination in a recurrent network

By juxtaposing time series analyses of activity measured from a fully recurrent network undergoing disrupted processing and of activity measured from a continuous meta-cognitive report of disruption in real-time language comprehension, we present an opportunity to compare the temporal statistics of the state-space trajectories inherent to both systems. Both the recurrent network and the human language comprehension process appear to exhibit long-range temporal correlations and low entropy when processing is undisrupted and coordinated. However, when processing is disrupted and discoordinated, they both exhibit more short-range temporal correlations and higher entropy. We conclude that by measuring human language comprehension in a dense-sampling manner similar to how we analyze the networks, and analyzing the resulting data stream with nonlinear time series analysis techniques, we can obtain more insight into the temporal character of these discoordination phases than by simply marking the points in time at which they peak.

Wrong or right? Brain potentials reveal hemispheric asymmetries to semantic relations during word-by-word sentence reading as a function of (fictional) knowledge

How does individual-level variation in experience and knowledge influence neural mechanisms important for real-time language comprehension? We used event-related brain potentials (ERPs) combined with lateralized visual field (VF) presentation of sentence-final words to examine asymmetries in hemispheric processing as individuals who varied in their knowledge of the fictional world of Harry Potter (HP) read sentences about general topics and HP. HP sentence endings were either contextually supported, unrelated anomalies, or semantically related anomalies. Amongst HP experts, we observed contextual support effects with presentation to both left and right VFs, but related anomaly effects only with presentation to the left VF (right hemisphere). Our findings are in line with accounts in which the left hemisphere (LH) activates relatively narrow/specific semantic information and the RH activates a broader range of semantic relations, including those relating to the construction of mental/situation models, as individuals attempt to comprehend sentences, one word at a time. We suggest that RH-biased hemispheric asymmetries in processing related (but linguistically inappropriate) words in written sentences may emerge as a function of each individual's degree of relevant knowledge. We tentatively hypothesize that content experts may optimize hemispheric differences in scope of semantic activation to maximize both precision (in the LH) and flexibility (in the RH) during language comprehension.

Tracking Object-State Representations During Real-Time Language Comprehension by Native and Non-native Speakers of English.

The present “visual world” eye-tracking study examined the time-course of how native and non-native speakers keep track of implied object-state representations during real-time language processing. Fifty-two native speakers of English and 46 non-native speakers with advanced English proficiency joined this study. They heard short stories describing a target object (e.g., an onion) either having undergone a substantial change-of-state (e.g., chop the onion) or a minimal change-of-state (e.g., weigh the onion) while their eye movements toward competing object-states (e.g., a chopped onion vs. an intact onion) and two unrelated distractors were tracked. We found that both groups successfully directed their visual attention toward the end-state of the target object that was implied in the linguistic context. However, neither group showed anticipatory eye movements toward the implied object-state when hearing the critical verb (e.g., “weigh/chop”). Only native English speakers but not non-native speakers showed a bias in visual attention during the determiner (“the”) before the noun (e.g., “onion”). Our results suggested that although native and non-native speakers of English largely overlapped in their time-courses of keeping track of object-state representations during real-time language comprehension, non-native speakers showed a short delay in updating the implied object-state representations.

Listener\u2019s personality traits predict changes in pupil size during auditory language comprehension

Research suggests that listeners’ comprehension of spoken language is concurrently affected by linguistic and non-linguistic factors, including individual difference factors. However, there is no systematic research on whether general personality traits affect language processing. We correlated 88 native English-speaking participants’ Big-5 traits with their pupillary responses to spoken sentences that included grammatical errors, "He frequently have burgers for dinner"; semantic anomalies, "Dogs sometimes chase teas"; and statements incongruent with gender stereotyped expectations, such as "I sometimes buy my bras at Hudson's Bay", spoken by a male speaker. Generalized additive mixed models showed that the listener's Openness, Extraversion, Agreeableness, and Neuroticism traits modulated resource allocation to the three different types of unexpected stimuli. No personality trait affected changes in pupil size across the board: less open participants showed greater pupil dilation when processing sentences with grammatical errors; and more introverted listeners showed greater pupil dilation in response to both semantic anomalies and socio-cultural clashes. Our study is the first one demonstrating that personality traits systematically modulate listeners’ online language processing. Our results suggest that individuals with different personality profiles exhibit different patterns of the allocation of cognitive resources during real-time language comprehension.

Children flexibly seek visual information to support signed and spoken language comprehension.

During grounded language comprehension, listeners must link the incoming linguistic signal to the visual world despite uncertainty in the input. Information gathered through visual fixations can facilitate understanding. But do listeners flexibly seek supportive visual information? Here, we propose that even young children can adapt their gaze and actively gather information for the goal of language comprehension. We present 2 studies of eye movements during real-time language processing, where the value of fixating on a social partner varies across different contexts. First, compared with children learning spoken English (n = 80), young American Sign Language (ASL) learners (n = 30) delayed gaze shifts away from a language source and produced a higher proportion of language-consistent eye movements. This result provides evidence that ASL learners adapt their gaze to effectively divide attention between language and referents, which both compete for processing via the visual channel. Second, English-speaking preschoolers (n = 39) and adults (n = 31) fixated longer on a speaker's face while processing language in a noisy auditory environment. Critically, like the ASL learners in Experiment 1, this delay resulted in gathering more visual information and a higher proportion of language-consistent gaze shifts. Taken together, these studies suggest that young listeners can adapt their gaze to seek visual information from social partners to support real-time language comprehension. (PsycInfo Database Record (c) 2020 APA, all rights reserved).

An Extra Cue Is Beneficial for Native Speakers but Can Be Disruptive for Second Language Learners: Integration of Prosody and Visual Context in Syntactic Ambiguity Resolution.

It has long been debated whether non-native speakers can process sentences in the same way as native speakers do or they suffer from certain qualitative deficit in their ability of language comprehension. The current study examined the influence of prosodic and visual information in processing sentences with a temporarily ambiguous prepositional phrase (“Put the cake on the plate in the basket”) with native English speakers and Japanese learners of English. Specifically, we investigated (1) whether native speakers assign different pragmatic functions to the same prosodic cues used in different contexts and (2) whether L2 learners can reach the correct analysis by integrating prosodic cues with syntax with reference to the visually presented contextual information. The results from native speakers showed that contrastive accents helped to resolve the referential ambiguity when a contrastive pair was present in visual scenes. However, without a contrastive pair in the visual scene, native speakers were slower to reach the correct analysis with the contrastive accent, which supports the view that the pragmatic function of intonation categories are highly context dependent. The results from L2 learners showed that visually presented context alone helped L2 learners to reach the correct analysis. However, L2 learners were unable to assign contrastive meaning to the prosodic cues when there were two potential referents in the visual scene. The results suggest that L2 learners are not capable of integrating multiple sources of information in an interactive manner during real-time language comprehension.

Extending Situated Language Comprehension (Accounts) with Speaker and Comprehender Characteristics: Toward Socially Situated Interpretation.

More and more findings suggest a tight temporal coupling between (non-linguistic) socially interpreted context and language processing. Still, real-time language processing accounts remain largely elusive with respect to the influence of biological (e.g., age) and experiential (e.g., world and moral knowledge) comprehender characteristics and the influence of the ‘socially interpreted’ context, as for instance provided by the speaker. This context could include actions, facial expressions, a speaker’s voice or gaze, and gestures among others. We review findings from social psychology, sociolinguistics and psycholinguistics to highlight the relevance of (the interplay between) the socially interpreted context and comprehender characteristics for language processing. The review informs the extension of an extant real-time processing account (already featuring a coordinated interplay between language comprehension and the non-linguistic visual context) with a variable (‘ProCom’) that captures characteristics of the language user and with a first approximation of the comprehender’s speaker representation. Extending the CIA to the sCIA (social Coordinated Interplay Account) is the first step toward a real-time language comprehension account which might eventually accommodate the socially situated communicative interplay between comprehenders and speakers.

Who's who? The incremental processing of presuppositions (and beyond)

Who's who? The incremental processing of presuppositions (and beyond)

Mapping language to visual referents: Does the degree of image realism matter?

Studies of real-time spoken language comprehension have shown that listeners rapidly map unfolding speech to available referents in the immediate visual environment. This has been explored using various kinds of 2-dimensional (2D) stimuli, with convenience or availability typically motivating the choice of a particular image type. However, work in other areas has suggested that certain cognitive processes are sensitive to the level of realism in 2D representations. The present study examined the process of mapping language to depictions of objects that are more or less realistic, namely photographs versus clipart images. A custom stimulus set was first created by generating clipart images directly from photographs of real objects. Two visual world experiments were then conducted, varying whether referent identification was driven by noun or verb information. A modest benefit for clipart stimuli was observed during real-time processing, but only for noun-driving mappings. The results are discussed in terms of their implications for studies of visually situated language processing.

Do resource constraints affect lexical processing? Evidence from eye movements.

Human language is massively ambiguous, yet we are generally able to identify the intended meanings of the sentences we hear and read quickly and accurately. How we manage and resolve ambiguity incrementally during real-time language comprehension given our cognitive resources and constraints is a major question in human cognition. Previous research investigating resource constraints on lexical ambiguity resolution has yielded conflicting results. Here we present results from two experiments in which we recorded eye movements to test for evidence of resource constraints during lexical ambiguity resolution. We embedded moderately biased homographs in sentences with neutral prior context and either long or short regions of text before disambiguation to the dominant or subordinate interpretation. The length of intervening material had no effect on ease of disambiguation. Instead, we found only a main effect of meaning at disambiguation, such that disambiguating to the subordinate meaning of the homograph was more difficult-results consistent with the reordered access model and contemporary probabilistic models, but inconsistent with the capacity-constrained model.

Word learning in linguistic context: Processing and memory effects

During language acquisition, children exploit syntactic cues within sentences to learn the meanings of words. Yet, it remains unknown how this strategy develops alongside an ability to access cues during real-time language comprehension. This study investigates how on-line sensitivity to syntactic cues impacts off-line interpretation and recall of word meanings. Adults and 5-year-olds heard novel words embedded in sentences that were (1) consistent with an agent-first bias (e.g., “The blicket will be eating the seal”→“the blicket” is an agent), (2) required revision of this bias (e.g., “The blicket will be eaten by the seal”→“the blicket” is a theme), or (3) weakened this bias through a familiar NP1 (e.g., “The seal will be eating/eaten by the blicket”→“the seal” is an agent or theme). Across both ages, eye-movements during sentences revealed decreased sensitivity to syntactic cues in contexts that required syntactic revision. In children, the magnitude of on-line sensitivity was positively associated with the accuracy of learning after the sentence. Parsing challenges during the word-learning task also negatively impacted children’s later memory for word meanings during a recall task. Altogether, these results suggest that real-time demands impact word learning, through interpretive failures and memory interference.

A Hierarchical Generative Framework of Language Processing: Linking Language Perception, Interpretation, and Production Abnormalities in Schizophrenia.

Language and thought dysfunction are central to the schizophrenia syndrome. They are evident in the major symptoms of psychosis itself, particularly as disorganized language output (positive thought disorder) and auditory verbal hallucinations (AVHs), and they also manifest as abnormalities in both high-level semantic and contextual processing and low-level perception. However, the literatures characterizing these abnormalities have largely been separate and have sometimes provided mutually exclusive accounts of aberrant language in schizophrenia. In this review, we propose that recent generative probabilistic frameworks of language processing can provide crucial insights that link these four lines of research. We first outline neural and cognitive evidence that real-time language comprehension and production normally involve internal generative circuits that propagate probabilistic predictions to perceptual cortices — predictions that are incrementally updated based on prediction error signals as new inputs are encountered. We then explain how disruptions to these circuits may compromise communicative abilities in schizophrenia by reducing the efficiency and robustness of both high-level language processing and low-level speech perception. We also argue that such disruptions may contribute to the phenomenology of thought-disordered speech and false perceptual inferences in the language system (i.e., AVHs). This perspective suggests a number of productive avenues for future research that may elucidate not only the mechanisms of language abnormalities in schizophrenia, but also promising directions for cognitive rehabilitation.

Discovering prominence and its role in language processing: An individual (differences) approach

Abstract It has been suggested that, during real time language comprehension, the human language processing system attempts to identify the argument primarily responsible for the state of affairs (the “actor”) as quickly and unambiguously as possible. However, previous work on a prominence (e.g. animacy, definiteness, case marking) based heuristic for actor identification has suffered from underspecification of the relationship between different cue hierarchies. Qualitative work has yielded a partial ordering of many features (e.g. MacWhinney et al. 1984), but a precise quantification has remained elusive due to difficulties in exploring the full feature space in a particular language. Feature pairs tend to correlate strongly in individual languages for semantic-pragmatic reasons (e.g., animate arguments tend to be actors and actors tend to be morphosyntactically privileged), and it is thus difficult to create acceptable stimuli for a fully factorial design even for binary features. Moreover, the exponential function grows extremely rapidly and a fully crossed factorial design covering the entire feature space would be prohibitively long for a purely within-subjects design. Here, we demonstrate the feasibility of parameter estimation in a short experiment. We are able to estimate parameters at a single subject level for the parameters animacy, case and number. This opens the door for research into individual differences and population variation. Moreover, the framework we introduce here can be used in the field to measure more “exotic” languages and populations, even with small sample sizes. Finally, pooled single-subject results are used to reduce the number of free parameters in previous work based on the extended Argument Dependency Model (Bornkessel-Schlesewsky and Schlesewsky 2006, 2009, 2013, in press; Alday et al. 2014). Multimedia: OpenSesame experiment and Python support scripts, sample stimuli, R scripts for analysis