Visual-world Eye-tracking Research Articles

Prediction and Production of Simple Mathematical Equations: Evidence from Visual World Eye-Tracking.

The relationship between the production and the comprehension systems has recently become a topic of interest for many psycholinguists. It has been argued that these systems are tightly linked and in particular that listeners use the production system to predict upcoming content. In this study, we tested how similar production and prediction processes are in a novel version of the visual world paradigm. Dutch speaking participants (native speakers in Experiment 1; German-Dutch bilinguals in Experiment 2) listened to mathematical equations while looking at a clock face featuring the numbers 1 to 12. On alternating trials, they either heard a complete equation ("three plus eight is eleven") or they heard the first part ("three plus eight is") and had to produce the result ("eleven") themselves. Participants were encouraged to look at the relevant numbers throughout the trial. Their eye movements were recorded and analyzed. We found that the participants' eye movements in the two tasks were overall very similar. They fixated the first and second number of the equations shortly after they were mentioned, and fixated the result number well before they named it on production trials and well before the recorded speaker named it on comprehension trials. However, all fixation latencies were shorter on production than on comprehension trials. These findings suggest that the processes involved in planning to say a word and anticipating hearing a word are quite similar, but that people are more aroused or engaged when they intend to respond than when they merely listen to another person.

Referential precedents in spoken language comprehension: A review and meta-analysis

Abstract Listeners’ interpretations of referring expressions are influenced by referential precedents—temporary conventions established in a discourse that associate linguistic expressions with referents. A number of psycholinguistic studies have investigated how much precedent effects depend on beliefs about the speaker’s perspective versus more egocentric, domain-general processes. We review and provide a meta-analysis of visual-world eyetracking studies of precedent use, focusing on three principal effects: (1) a same speaker advantage for maintained precedents; (2) a different speaker advantage for broken precedents; and (3) an overall main effect of precedents. Despite inconsistent claims in the literature, our combined analysis reveals surprisingly consistent evidence supporting the existence of all three effects, but with different temporal profiles. These findings carry important implications for existing theoretical explanations of precedent use, and challenge explanations based solely on the use of information about speakers’ perspectives.

Presuppositions are Fast, whether Hard or Soft - Evidence from the visual world

One focus of work on the processing of linguistic meaning has been the relative processing speed of different aspects of meaning. While much early work has focused on implicatures in comparison to literal asserted content (e.g., Bott & Noveck 2004, Huang & Snedeker 2009, among many others), the present paper extends recent efforts to experimentally investigate another aspect of meaning, namely presuppositions. It investigates the triggers again and stop using the visual world eye tracking paradigm, and provides evidence for rapid processing of presupposed content. Our study finds no difference in timing for the two triggers, which is of theoretical relevance given proposals for distinguishing classes of triggers, such as hard vs. soft (Abusch 2010). Whatever differences between these there may be are apparently not affecting the online processing time course. As a further comparison, again was also compared to twice, which expresses essentially the same meaning without a presupposition. Shifts in eye movements for these two cases also appear to be entirely on par, further supporting the notion that presupposed and asserted content are available in parallel early on in online processing.

Rapid Linguistic Ambiguity Resolution in Young Children with Autism Spectrum Disorder: Eye Tracking Evidence for the Limits of Weak Central Coherence.

Individuals with autism spectrum disorders (ASD) have often been reported to have difficulty integrating information into its broader context, which has motivated the Weak Central Coherence theory of ASD. In the linguistic domain, evidence for this difficulty comes from reports of impaired use of linguistic context to resolve ambiguous words. However, recent work has suggested that impaired use of linguistic context may not be characteristic of ASD, and is instead better explained by co-occurring language impairments. Here, we provide a strong test of these claims, using the visual world eye tracking paradigm to examine the online mechanisms by which children with autism resolve linguistic ambiguity. To address concerns about both language impairments and compensatory strategies, we used a sample whose verbal skills were strong and whose average age (7; 6) was lower than previous work on lexical ambiguity resolution in ASD. Participants (40 with autism and 40 controls) heard sentences with ambiguous words in contexts that either strongly supported one reading or were consistent with both (John fed/saw the bat). We measured activation of the unintended meaning through implicit semantic priming of an associate (looks to a depicted baseball glove). Contrary to the predictions of weak central coherence, children with ASD, like controls, quickly used context to resolve ambiguity, selecting appropriate meanings within a second. We discuss how these results constrain the generality of weak central coherence.

Context-driven expectations about focus alternatives

What is conveyed by a sentence frequently depends not only on the descriptive content carried by its words, but also on implicit alternatives determined by the context of use. Four visual world eye-tracking experiments examined how alternatives are generated based on aspects of the discourse context and used in interpreting sentences containing the focus operators only and also. Experiment 1 builds on previous reading time studies showing that the interpretations of only sentences are constrained by alternatives explicitly mentioned in the preceding discourse, providing fine-grained time course information about the expectations triggered by only. Experiments 2 and 3 show that, in the absence of explicitly mentioned alternatives, lexical and situation-based categories evoked by the context are possible sources of alternatives. While Experiments 1–3 all demonstrate the discourse dependence of alternatives, only explicit mention triggered expectations about alternatives that were specific to sentences with only. By comparing only with also, Experiment 4 begins to disentangle expectations linked to the meanings of specific operators from those generalizable to the class of focus-sensitive operators. Together, these findings show that the interpretation of sentences with focus operators draws on both dedicated mechanisms for introducing alternatives into the discourse context and general mechanisms associated with discourse processing.

Unintentional activation of translation equivalents in bilinguals leads to attention capture in a cross-modal visual task.

Using a variant of the visual world eye tracking paradigm, we examined if language non- selective activation of translation equivalents leads to attention capture and distraction in a visual task in bilinguals. High and low proficient Hindi-English speaking bilinguals were instructed to programme a saccade towards a line drawing which changed colour among other distractor objects. A spoken word, irrelevant to the main task, was presented before the colour change. On critical trials, one of the line drawings was a phonologically related word of the translation equivalent of the spoken word. Results showed that saccade latency was significantly higher towards the target in the presence of this cross-linguistic translation competitor compared to when the display contained completely unrelated objects. Participants were also slower when the display contained the referent of the spoken word among the distractors. However, the bilingual groups did not differ with regard to the interference effect observed. These findings suggest that spoken words activates translation equivalent which bias attention leading to interference in goal directed action in the visual domain.

The time course of auditory and language-specific mechanisms in compensation for sibilant assimilation.

Models of spoken-word recognition differ on whether compensation for assimilation is language-specific or depends on general auditory processing. English and French participants were taught words that began or ended with the sibilants /s/ and /∫/. Both languages exhibit some assimilation in sibilant sequences (e.g., /s/ becomes like [∫] in dress shop and classe chargée), but they differ in the strength and predominance of anticipatory versus carryover assimilation. After training, participants were presented with novel words embedded in sentences, some of which contained an assimilatory context either preceding or following. A continuum of target sounds ranging from [s] to [∫] was spliced into the novel words, representing a range of possible assimilation strengths. Listeners' perceptions were examined using a visual-world eyetracking paradigm in which the listener clicked on pictures matching the novel words. We found two distinct language-general context effects: a contrastive effect when the assimilating context preceded the target, and flattening of the sibilant categorization function (increased ambiguity) when the assimilating context followed. Furthermore, we found that English but not French listeners were able to resolve the ambiguity created by the following assimilatory context, consistent with their greater experience with assimilation in this context. The combination of these mechanisms allows listeners to deal flexibly with variability in speech forms.

Activation of shape and semantic information during ambiguous homophone processing: eye tracking evidence from Hindi.

In two visual world eye tracking studies, we examined the activation of subordinate meanings of ambiguous homophones in Hindi and particularly when the sentence context is biased towards the dominant meaning. Participants listened to sentences that were either neutral or biased towards the dominant meaning of the homophone and saw a display containing four pictures. In experiment 1, the display had a shape competitor of the subordinate meaning of the homophone in both neutral and biased conditions along with three unrelated distractors. Experiment 2 had semantic competitors of the subordinate meaning of the homophones along with three distractors. Proportion of fixations to different objects overtime suggested that participants activated the subordinate meanings and oriented their attention to the shape and semantic competitors even when the prior context was biased towards the dominant meaning. Overall, these data from Hindi provide further support to those models of lexical access that assume exhaustive access of both the meanings of an ambiguous homophone. These data suggest even a dominant bias does not eliminate the activation of perceptual and conceptual features of the subordinate meaning.

Inferring Difficulty: Flexibility in the Real-time Processing of Disfluency

Upon hearing a disfluent referring expression, listeners expect the speaker to refer to an object that is previously unmentioned, an object that does not have a straightforward label, or an object that requires a longer description. Two visual-world eye-tracking experiments examined whether listeners directly associate disfluency with these properties of objects, or whether disfluency attribution is more flexible and involves situation-specific inferences. Since in natural situations reference to objects that do not have a straightforward label or that require a longer description is correlated with both production difficulty and with disfluency, we used a mini-artificial lexicon to dissociate difficulty from these properties, building on the fact that recently learned names take longer to produce than existing words in one's mental lexicon. The results demonstrate that disfluency attribution involves situation-specific inferences; we propose that in new situations listeners spontaneously infer what may cause production difficulty. However, the results show that these situation-specific inferences are limited in scope: listeners assessed difficulty relative to their own experience with the artificial names, and did not adapt to the assumed knowledge of the speaker.

Reliability of Visual-World Eyetracking for Lexical and Sentence Comprehension Tasks

Reliability of Visual-World Eyetracking for Lexical and Sentence Comprehension Tasks

Building Chinese relative clause structures with lexical and syntactic cues: evidence from visual world eye-tracking and reading times

Relative clauses (RCs) in Chinese are prenominal. In object-modifying, object-extracted RCs (e.g. Click on [RC the ball broke] window, meaning ‘Click on the window [RC that the ball broke]), the ambiguous status of the local noun ball and the long-distance attachment of the head noun window into the main verb appear to make online parsing of Chinese RCs particularly difficult. By interposing mismatching classifiers and the passive marker BEI into the RC sentences, we investigated whether the presence of incomplete heads would add storage costs, as predicted by the Dependency Locality Theory (DLT), or would serve as retrieval cues to help pre-build the RC structure, as predicted by the cue-based retrieval theory. Results from a visual world eye-tracking experiment and a self-paced reading showed that Chinese comprehenders are able to use BEI cues and the mismatching classifier (albeit to a less extent) to pre-build RC structure, providing support for the cue-based retrieval theory.

Effects of Speaker Emotional Facial Expression and Listener Age on Incremental Sentence Processing

We report two visual-world eye-tracking experiments that investigated how and with which time course emotional information from a speaker's face affects younger (N = 32, Mean age = 23) and older (N = 32, Mean age = 64) listeners’ visual attention and language comprehension as they processed emotional sentences in a visual context. The age manipulation tested predictions by socio-emotional selectivity theory of a positivity effect in older adults. After viewing the emotional face of a speaker (happy or sad) on a computer display, participants were presented simultaneously with two pictures depicting opposite-valence events (positive and negative; IAPS database) while they listened to a sentence referring to one of the events. Participants' eye fixations on the pictures while processing the sentence were increased when the speaker's face was (vs. wasn't) emotionally congruent with the sentence. The enhancement occurred from the early stages of referential disambiguation and was modulated by age. For the older adults it was more pronounced with positive faces, and for the younger ones with negative faces. These findings demonstrate for the first time that emotional facial expressions, similarly to previously-studied speaker cues such as eye gaze and gestures, are rapidly integrated into sentence processing. They also provide new evidence for positivity effects in older adults during situated sentence processing.

Young Toddlers’ Word Comprehension Is Flexible and Efficient

Much of what is known about word recognition in toddlers comes from eyetracking studies. Here we show that the speed and facility with which children recognize words, as revealed in such studies, cannot be attributed to a task-specific, closed-set strategy; rather, children’s gaze to referents of spoken nouns reflects successful search of the lexicon. Toddlers’ spoken word comprehension was examined in the context of pictures that had two possible names (such as a cup of juice which could be called “cup” or “juice”) and pictures that had only one likely name for toddlers (such as “apple”), using a visual world eye-tracking task and a picture-labeling task (n = 77, mean age, 21 months). Toddlers were just as fast and accurate in fixating named pictures with two likely names as pictures with one. If toddlers do name pictures to themselves, the name provides no apparent benefit in word recognition, because there is no cost to understanding an alternative lexical construal of the picture. In toddlers, as in adults, spoken words rapidly evoke their referents.

Information Structure Affects the Resolution of the Subject Pronouns Er and Der in Spoken German Discourse

Two visual-world eye-tracking experiments were designed to investigate the resolution of ambiguous German pronouns, the personal pronoun (er) and the d-pronoun (der) in spoken discourse. Specifically, the influence of the order of mention and the information status of the antecedent candidates on the resolution preferences following canonical and non-canonical antecedent structures was explored. The results suggest that the two pronominal forms have different coreference functions when they follow canonical topic-comment antecedent structures, in that personal pronouns prefer first-mentioned topical antecedents and d-pronouns second-mentioned non-topical antecedents. However, after non-canonically marked topic-focus antecedent structures, the pronouns had overlapping functions, namely an overall preference towards the second-mentioned focused entity. The findings suggest that pronoun resolution is affected by the information status of the antecedent candidates and that resolution preferences change across antecedent word orders.

Effects of verb meaning on lexical integration in agrammatic aphasia: Evidence from eyetracking

Relatively little is known about the time course of access to the lexical representations of verbs in agrammatic aphasia and its effects on the prediction and integration of the verb's arguments. The present study used visual-world eyetracking to test whether verb meaning can be used by agrammatic aphasic individuals to predict and facilitate the integration of a subsequent noun argument. Nine adults with agrammatic aphasia and ten age-matched controls participated in the study. In Experiment 1, participants viewed arrays of four objects (e.g., jar, plate, stick, pencil) while listening to sentences containing either a restrictive verb that was semantically compatible only with the target object or an unrestrictive verb compatible with all four objects (e.g., Susan will open/break the jar). For both participant groups, the restrictive condition elicited more fixations to the target object immediately after the verb. Experiment 2 differed from Experiment 1 in that the auditory sentences presented were incomplete (e.g., Susan will open/break the…). For controls, restrictive verbs elicited more target fixations immediately after the verb; however, the effects of verb type were noted downstream from the verb for the aphasic listeners. The results suggest that individuals with agrammatic aphasia have preserved ability to use verb information to facilitate integration of overt arguments, but prediction of upcoming arguments is impaired. Impaired lexical-semantic prediction processes may be caused by damage to the left inferior frontal gyrus, which has been argued to support higher-level lexical processes.

Grammatical gender in adult L2 acquisition: Relations between lexical and syntactic variability

In order to identify the causes of inflectional variability in adult second-language (L2) acquisition, this study investigates lexical and syntactic aspects of gender processing in real-time L2 production and comprehension. Twenty advanced to near-native adult first language (L1) English speakers of L2 German and 20 native controls were tested in a study comprising two experiments. In elicited production, we probe accuracy in lexical gender assignment. In a visual-world eye tracking task, we test the predictive processing of syntactic gender agreement between determiners and nouns. The findings show clear contingencies (1) between overall accuracy in lexical gender assignment in production and target predictive processing of syntactic gender agreement in comprehension and (2) between the speed of lexical access and predictive syntactic gender agreement. These findings support lexical and computational accounts of L2 inflectional variability and argue against models positing representational deficits in morphosyntax in late L2 acquisition and processing.

Delayed use of fundamental frequency (F0) rise in non-native speech segmentation

Research has shown that second/foreign language (L2) speech segmentation is less efficient than native language (L1) segmentation, because L2 learners cannot suppress L1 segmentation routines. This study uses visual-world eye-tracking to determine whether English learners of French can learn to use F0 for locating word-final boundaries in French. F0 rise is often word-initial in English but often word-final in French. Native French speakers and English learners of French heard sentences where lexical competitors were created between the target noun and the following adjective (stimulus: chat grincheux ‘cranky cat’; target: chat ‘cat’; competitor: chagrin ‘sorrow’). The target was either accented or unaccented, and the stimuli were either natural or resynthesized (swapped F0 between accented and unaccented targets). Participants selected the word they heard from given options (target, competitor, distracters), and fixations were recorded from target-word onset. Accuracy (word selection): learners, but not natives, were more accurate for target words with F0 rise than without it. Fixations: natives, but not learners, showed higher differential proportions of target and competitor fixations for target words with F0 rise than without it. Proficiency did not interact with the variables. This suggests that L2 learners’ use of F0 rise is delayed compared to that of natives.

Focusing on pronouns: Consequences of subjecthood, pronominalisation, and contrastive focus

We report two visual-world eye-tracking experiments that investigated the effects of subjecthood, pronominalisation, and contrastive focus on the interpretation of pronouns in subsequent discourse. By probing the effects of these factors on real-time pronoun interpretation, we aim to contribute to our understanding of how topicality-related factors (subjecthood, givenness) interact with contrastive focus effects, and to investigate whether the seemingly mixed results obtained in prior work on topicality and focusing could be related to effects of subjecthood. Our results indicate that structural and semantic prominence (specifically, agentive subjects) influence pronoun interpretation even when separated from information-structural notions, and thus need to be taken into account when investigating topicality and focusing. We discuss how our results allow us to reconcile the distinct findings of prior studies. More generally, this research contributes to our understanding of how the language comprehension system integrates different kinds of information during real-time reference resolution.

Activating Gender Stereotypes During Online Spoken Language Processing

This study used the visual world eye-tracking method to investigate activation of general world knowledge related to gender-stereotypical role names in online spoken language comprehension in Finnish. The results showed that listeners activated gender stereotypes elaboratively in story contexts where this information was not needed to build coherence. Furthermore, listeners made additional inferences based on gender stereotypes to revise an already established coherence relation. Both results are consistent with mental models theory (e.g., Garnham, 2001). They are harder to explain by the minimalist account (McKoon & Ratcliff, 1992) which suggests that people limit inferences to those needed to establish coherence in discourse.

Dynamics of activation of semantically similar concepts during spoken word recognition.

Semantic similarity effects provide critical insight into the organization of semantic knowledge and the nature of semantic processing. In the present study, we examined the dynamics of semantic similarity effects by using the visual world eyetracking paradigm. Four objects were shown on a computer monitor, and participants were instructed to click on a named object, during which time their gaze position was recorded. The likelihood of fixating competitor objects was predicted by the degree of semantic similarity to the target concept. We found reliable, graded competition that depended on degree of target-competitor similarity, even for distantly related items for which priming has not been found in previous priming studies. Time course measures revealed a consistently earlier fixation peak for near semantic neighbors relative to targets. Computational investigations with an attractor dynamical model, a spreading activation model, and a decision model revealed that a combination of excitatory and inhibitory mechanisms is required to obtain such peak timing, providing new constraints on models of semantic processing.