Visual-world Eye-tracking Experiment Research Articles

Informativity, information status and the accessibility of indefinite noun phrases

In discourse processing, speakers collaborate toward a shared mental model by introducing discourse referents and picking them up with the adequate linguistic forms. Discourse referents compete with each other with respect to their prominence and their accessibility for pronouns. This study focuses on transitive sentences with proper names as subjects and indefinite noun phrases as second arguments, typically direct objects. An ambiguous pronoun in the subsequent sentence may access either referent of the first sentence. Various factors have been shown to influence pronoun resolution, including informativity (how informative is the phrase in which the referent is introduced? E.g., the waiter vs. the waiter at the entrance) and information status (is the referent given or new in the context?). While both factors have been independently shown to increase referent accessibility, our visual-world eye-tracking experiment shows an original and quite unexpected effect: informativity and information status interact when it comes to the accessibility of indefinite noun phrases: a higher degree of informativity increases accessibility when a referent is brand-new, but surprisingly decreases accessibility when a referent is inferred. We discuss a potential explanation for this surprising pattern in terms of a mismatch between the denotational type of the indefinite and the type required by the modification. We conclude that indefinites strongly interact with additional semantic, contextual and communicative parameters in establishing their referents.

Online Assessment of Cross-Linguistic Similarity as a Measure of L2 Perceptual Categorization Accuracy

The effect of cross-linguistic similarity on the development of target-like categories in a second or additional language is widely attested. Research also shows that second-language speakers may access both their native and the second-language lexicons when processing second-language speech. Forty-three Catalan learners of English performed a perceptual assimilation task evaluating the perceived similarity between English and Catalan vowels and also participated in a visual world eye-tracking experiment investigating between-language lexical competition. The focus of the study was the English vowel contrasts /iː/-/ɪ/ and /æ/-/ʌ/. The perceptual task confirmed that English /iː/ and /æ/were perceptually closer to native Catalan categories than English /ɪ/ and /ʌ/. The results of the spoken word recognition task indicated that learners experienced greater competition from native words when the target words contained English /iː/ and /æ/, illustrating a close link between the two types of tasks. However, differences in the magnitude of cross-language lexical competition were found to be only weakly related to learners’ degree of perceived similarity to native categories at an individual level. We conclude that online tasks provide a potentially effective method of assessing cross-linguistic similarity without the concerns inherent to more traditional offline approaches.

The Importance of Linguistic Factors: He Likes Subject Referents.

We report the results of one visual-world eye-tracking experiment and two referent selection tasks in which we investigated the effects of information structure in the form of prosody and word order manipulation on the processing of subject pronouns er and der in German. Factors such as subjecthood, focus, and topicality, as well as order of mention have been linked to an increased probability of certain referents being selected as the pronoun's antecedent and described as increasing this referent's prominence, salience, or accessibility. The goal of this study was to find out whether pronoun processing is primarily guided by linguistic factors (e.g., grammatical role) or nonlinguistic factors (e.g., first-mention), and whether pronoun interpretation can be described in terms of referents' "prominence"/"accessibility"/"salience." The results showed an overall subject preference for er, whereas der was affected by the object role and focus marking. While focus increases the attentional load and enhances memory representation for the focused referent making the focused referent more available, ultimately it did not affect the final interpretation of er, suggesting that "prominence" or the related concepts do not explain referent selection preferences. Overall, the results suggest a primacy of linguistic factors in determining pronounresolution.

Limits of short-term perceptual training for enhancing Seoul-Korean listeners’ use of English lexical stress in spoken word recognition

This study investigates whether short-term perceptual training can enhance Seoul-Korean listeners’ use of English lexical stress in spoken word recognition. Unlike English, Seoul Korean does not have lexical stress (or lexical pitch accents/tones). Seoul-Korean speakers at a high-intermediate English proficiency completed a visual-world eye-tracking experiment adapted from Connell et al. (2018) (pre-/post-test). The experiment tested whether pitch in the target stimulus (accented versus unaccented first syllable) and vowel quality in the lexical competitor (reduced versus full first vowel) modulated fixations to the target word (e.g., PARrot; ARson) over the competitor word (e.g., paRADE or PARish; arCHIVE or ARcade). In the training (eight 30-min sessions over eight days), participants heard English lexical-stress minimal pairs uttered by four talkers (high variability) or one talker (low variability), categorized them as noun (first-syllable stress) or verb (second-syllable stress), and received accuracy feedback. The results showed that neither training increased target-over-competitor fixation proportions. Crucially, the same training had been found to improve Seoul- Korean listeners’ recall of English words differing in lexical stress (Tremblay et al., 2022) and their weighting of acoustic cues to English lexical stress (Tremblay et al., 2023). These results suggest that short-term perceptual training has a limited effect on target-over-competitor word activation.

Processing to learn noncanonical word orders: Exploring linguistic and cognitive predictors of reanalysis in early L2 sentence comprehension

Abstract To test the contributions of processing to L2 syntax learning, this study explores (cross-) linguistic and cognitive predictors of sentence reanalysis in the L2 comprehension of relative clauses among low-intermediate L1 German adolescent learners of L2 English. Specifically, we test the degree to which L2 comprehension is affected by L2 proficiency, reanalysis ability in a related, earlier-acquired L2 structure (questions), reanalysis ability of relative clauses in the L1, cognitive control, and cognitive capacity. In visual-world eye-tracking experiments, 141 adolescent German-speaking L2 learners of English selected target pictures for auditorily presented questions and relative clauses in the L1 and in the L2. The results showed a strong subject preference for L2 relative clauses. Learners’ L2 proficiency and their processing of object questions in the L2 predicted reanalysis for object relatives in eye movements, reaction times, and comprehension accuracy. In contrast, there was no evidence that cognitive control or working memory systematically affected the processing of object relatives. These findings suggest that linguistic processing outweighs cognitive processing in accounting for individual differences in low-intermediate L2 acquisition of complex grammar. Specifically, learners recruit shared processing mechanisms and routines across grammatical structures to pave a way in the acquisition of syntax.

The time course of moral decision making in bilinguals' native and foreign language

Previous work has shown that moral decision making depends on whether moral dilemmas are presented in the native or a foreign language (Foreign Language effect). In that work, the focus was on bilinguals' final decision, but the tasks used, failed to capture the processes involved over time. The aim of this study was to examine the time course of moral decision making in bilinguals' native and foreign language prior to and after their moral decision. In a visual-world eye-tracking experiment, 82 Dutch-English bilinguals listened to 20 moral dilemmas (e.g., would you kill one to save five?) in their native or foreign language, while looking at two pictures containing key people involved in the dilemmas. These pictures illustrated (1) the person/people that is/are sacrificed, and (2) the person/people that is/are not sacrificed, depending on the participants' decision which was measured with yes/no-questions. The Foreign Language effect was replicated for the decisions on the personal dilemmas. Importantly, the eye gaze data showed that in the native language, listeners looked at the person who they did not sacrifice, whereas in the foreign language, they looked at the people who they sacrificed. A speculative explanation is that bilinguals might have experienced guilt in the native language, and therefore focused attention on the person they did not sacrifice, while in the foreign language they might have experienced less guilt because they focused attention on the people they sacrificed. More research is needed to understand the influence of factors such as emotion reduction and cognitive load on moral decision making.

Similarity-Based Interference in Sentence Comprehension in Aphasia: a Computational Evaluation of Two Models of Cue-Based Retrieval

Sentence comprehension requires the listener to link incoming words with short-term memory representations in order to build linguistic dependencies. The cue-based retrieval theory of sentence processing predicts that the retrieval of these memory representations is affected by similarity-based interference. We present the first large-scale computational evaluation of interference effects in two models of sentence processing — the activation-based model and a modification of the direct-access model — in individuals with aphasia (IWA) and control participants in German. The parameters of the models are linked to prominent theories of processing deficits in aphasia, and the models are tested against two linguistic constructions in German: pronoun resolution and relative clauses. The data come from a visual-world eye-tracking experiment combined with a sentence-picture matching task. The results show that both control participants and IWA are susceptible to retrieval interference, and that a combination of theoretical explanations (intermittent deficiencies, slow syntax, and resource reduction) can explain IWA’s deficits in sentence processing. Model comparisons reveal that both models have a similar predictive performance in pronoun resolution, but the activation-based model outperforms the direct-access model in relative clauses.

Integration of visual context in early and late bilingual language processing: evidence from eye-tracking.

Previous research on the processing of language embedded in a rich visual context has revealed the strong effect that a recently viewed action event has on language comprehension. It has been shown that listeners are more likely to view the target object of a recently performed event than look at the target object of a plausible future event during sentence utterance, regardless of the tense cue. In the current visual-world eye-tracking experiments, we tested the strength of the recently observed visual context with a group of English monolingual and two groups of English-French early and late bilingual speakers. By comparing these different groups, we examined whether bilingual speakers, as a consequence of greater cognitive flexibility when integrating visual context and language information, show early anticipatory eye-movements toward the target object. We further asked whether early and late bilinguals show differences in their processing. The findings of the three eye-tracking experiments revealed an overall preference for the recently seen event. However, as a result of the early provision of tense cue, this preference was quickly diminished in all three groups. Moreover, the bilingual groups showed an earlier decrease in reliance on the recently seen event compared to monolingual speakers and the early bilinguals showed anticipatory eye-movements toward the plausible future event target. Furthermore, a post-experimental memory test revealed that the bilingual groups recalled the future events marginally better than the recent events, whereas the reverse was found in the monolingual groups.

Evidence for two stages of prediction in non-native speakers: A visual-world eye-tracking study

AbstractComprehenders predict what a speaker is likely to say when listening to non-native (L2) and native (L1) utterances. But what are the characteristics of L2 prediction, and how does it relate to L1 prediction? We addressed this question in a visual-world eye-tracking experiment, which tested when L2 English comprehenders integrated perspective into their predictions. Male and female participants listened to male and female speakers producing sentences (e.g.,I would like to wear the nice…) about stereotypically masculine (target: tie; distractor: drill) and feminine (target: dress; distractor: hairdryer) objects. Participants predicted associatively, fixating objects semantically associated with critical verbs (here, the tie and the dress). They also predicted stereotypically consistent objects (e.g., the tie rather than the dress, given the male speaker). Consistent predictions were made later than associative predictions, and were delayed for L2 speakers relative to L1 speakers. These findings suggest prediction involves both automatic and non-automatic stages.

The role of verbal and working memory skills in Turkish-speaking children’s morphosyntactic prediction

AbstractThe current study investigated the contribution of multiple verbal and working memory (WM) skills to morphosyntactic prediction in Turkish-speaking 4- to 8-year-old children. In a visual world eye-tracking experiment, 76 children were presented with verb-final sentences with nominative and accusative case markers on the initial noun (e.g., the fast rabbitnominative … the carrotaccusativeeatfuture vs. the fast rabbitaccusative … the foxnominativeeatfuture) while they were looking at a visual display with three objects (e.g., rabbit, carrot, and fox). Importantly, the case markers on the initial noun could be used to predict the second noun in these sentences. The results revealed that when children’s early productive vocabulary and language production skills were higher, the better and faster they were in predicting the upcoming noun. The episodic buffer, a component of WM, was also positively associated with children’s morphosyntactic prediction abilities. The implications of these results for the mechanisms of linguistic prediction are discussed.

Correction: Using verb morphology to predict subject number in L1 and L2 sentence processing: A visual-world eye-tracking experiment

This article details a correction to: Koch, E. M., Bulté, B., Housen, A., &amp; Godfroid, A. (2021). Using verb morphology to predict subject number in L1 and L2 sentence processing: A visual-world eye-tracking experiment. <em>Journal of the European Second Language Association, 5</em>(1), 115–132. DOI: <a href="http://doi.org/10.22599/jesla.79" target="_blank">http://doi.org/10.22599/jesla.79</a>.

Do Children Use Multi-Word Information in Real-Time Sentence Comprehension?

Meaning in language emerges from multiple words, and children are sensitive to multi-word frequency from infancy. While children successfully use cues from single words to generate linguistic predictions, it is less clear whether and how they use multi-word sequences to guide real-time language processing and whether they form predictions on the basis of multi-word information or pairwise associations. We address these questions in two visual-world eye-tracking experiments with 5- to 8-year-old children. In Experiment 1, we asked whether children generate more robust predictions for the sentence-final object of highly frequent sequences (e.g., "Throw the ball"), compared to less frequent sequences (e.g., "Throw the book"). We further examined if gaze patterns reflect event knowledge or phrasal frequency by comparing the processing of phrases that have the same event structure but differ in multi-word content (e.g., "Brush your teeth" vs. "Brush her teeth"). In the second study, we employed a training paradigm to ask if children are capable of generating predictio.ns from novel multi-word associations while controlling for the overall frequency of the sequences. While the results of Experiment 1 suggested that children primarily relied on event associations to generate real-time predictions, those of Experiment 2 showed that the same children were able to use recurring novel multi-word sequences to generate real-time linguistic predictions. Together, these findings suggest that children can draw on multi-word information to generate linguistic predictions, in a context-dependent fashion, and highlight the need to account for the influence of multi-word sequences in models of language processing.

Using verb morphology to predict subject number in L1 and L2 sentence processing: A visual-world eye-tracking experiment

We investigated whether adult German native speakers and Dutch-speaking second-language learners of German exploit German regular verb morphology for predictive sentence processing and whether such predictive processing is moderated by working memory capacity and awareness of the predictive cue. In a picture-matching task with visual-world eye-tracking, the participants (first-language group: <em>n</em> = 31; second-language group: <em>n</em> = 30) saw two pictures of action scenes, varying in the number of referents depicted (singular versus plural). Simultaneously, they heard sentences in German in which the verb suffix represented the first reliable cue for grammatical number of the upcoming subject noun phrase. Successful exploitation of this number cue was measured as anticipatory button-press reaction times and eye-movements toward the correct picture, before subject onset. We found significant prediction effects in both the first-language and the second-language group, with anticipatory processing being somewhat faster in the native speakers than in the learners. Faster reaction times on prediction trials were associated with higher working memory scores. Debriefing interviews revealed that all participants had become aware that they could use the verb form predictively, adding further evidence that prediction can be a conscious process.

Prediction involves two stages: Evidence from visual-world eye-tracking

Comprehenders often predict what they are going to hear. But do they make the best predictions possible? We addressed this question in three visual-world eye-tracking experiments by asking when comprehenders consider perspective. Male and female participants listened to male and female speakers producing sentences (e.g., I would like to wear the nice…) about stereotypically masculine (target: tie; distractor: drill) and feminine (target: dress, distractor: hairdryer) objects. In all three experiments, participants rapidly predicted semantic associates of the verb. But participants also predicted consistently – that is, consistent with their beliefs about what the speaker would ultimately say. They predicted consistently from the speaker’s perspective in Experiment 1, their own perspective in Experiment 2, and the character’s perspective in Experiment 3. This consistent effect occurred later than the associative effect. We conclude that comprehenders consider perspective when predicting, but not from the earliest moments of prediction, consistent with a two-stage account.

Influence of Actor's Congruent and Incongruent Gaze on Language Processing.

In interpreting spoken sentences in event contexts, comprehenders both integrate their current interpretation of language with the recent past (e.g., events they have witnessed) and develop expectations about future event possibilities. Tense cues can disambiguate this linking but temporary ambiguity in their interpretation may lead comprehenders to also rely on further, situation-specific cues (e.g., an actor's gaze as a cue to his future actions). How comprehenders reconcile these different cues in real time is an open issue that we must address to accommodate comprehension. It has been suggested that relating a referential expression (e.g., a verb) to a referent (e.g., a recent event) is preferred over relying on other cues that refer to the future and are not yet referentially grounded (“recent-event preference”). Two visual-world eye-tracking experiments compared this recent-event preference with effects of an actor's gaze and of tense/temporal adverbs as cues to a future action event. The results revealed that people overall preferred to focus on the recent (vs. future) event target in their interpretation, suggesting that while a congruent and incongruent actor gaze can jointly with futuric linguistic cues neutralize the recent-event preference late in the sentence, the latter still plays a key role in shaping participants' initial verb-based event interpretation. Additional post-experimental memory tests provided insight into the longevity of the gaze effects.

Integration of Social Context vs. Linguistic Reference During Situated Language Processing.

Research findings on language comprehension suggest that many kinds of non-linguistic cues can rapidly affect language processing. Extant processing accounts of situated language comprehension model these rapid effects and are only beginning to accommodate the role of non-linguistic emotional, cues. To begin with a detailed characterization of distinct cues and their relative effects, three visual-world eye-tracking experiments assessed the relative importance of two cue types (action depictions vs. emotional facial expressions) as well as the effects of the degree of naturalness of social (facial) cues (smileys vs. natural faces). We predicted to replicate previously reported rapid effects of referentially mediated actions. In addition, we assessed distinct world-language relations. If how a cue is conveyed matters for its effect, then a verb referencing an action depiction should elicit a stronger immediate effect on visual attention and language comprehension than a speaker's emotional facial expression. The latter is mediated non-referentially via the emotional connotations of an adverb. The results replicated a pronounced facilitatory effect of action depiction (relative to no action depiction). By contrast, the facilitatory effect of a preceding speaker's emotional face was less pronounced. How the facial emotion was rendered mattered in that the emotional face effect was present with natural faces (Experiment 2) but not with smileys (Experiment 1). Experiment 3 suggests that contrast, i.e., strongly opposing emotional valence information vs. non-opposing valence information, might matter for the directionality of this effect. These results are the first step toward a more principled account of how distinct visual (social) cues modulate language processing, whereby the visual cues that are referenced by language (the depicted action), copresent (the depicted action), and more natural (the natural emotional prime face) tend to exert more pronounced effects.

The Contribution of Form Repetition to Listeners’ Expectation of Givenness in Online Reference Resolution

ABSTRACT Although it is clear that unaccented referring expressions are associated with given information in a discourse, it is less clear what aspects of givenness are relevant. We examine whether listeners’ expectation of givenness depends on repetition of a referring expression or on contextual evocation of a referent. The results from two visual world eye-tracking experiments suggest that for interpretation, listeners associated reduced prominence with a repeated referring expression. Listeners expect previously evoked referents to be candidates for reduced referring expressions only when they are referred to with the exact same referential form. The data also suggest that when referents are referred to with different referential forms across utterances, accenting facilitates linking those forms for co-reference.

Syntactic and Semantic Influences on the Time Course of Relative Clause Processing: The Role of Language Dominance

We conducted a visual world eye-tracking experiment with highly proficient Spanish-English bilingual adults to investigate the effects of relative language dominance, operationalized as a continuous, multidimensional variable, on the time course of relative clause processing in the first-learned language, Spanish. We found that participants exhibited two distinct processing preferences: a semantically driven preference to assign agency to referents of lexically animate noun phrases and a syntactically driven preference to interpret relative clauses as subject-extracted. Spanish dominance was found to exert a distinct influence on each of these preferences, gradiently attenuating the semantic preference while gradiently exaggerating the syntactic preference. While these results might be attributable to particular properties of Spanish and English, they also suggest a possible generalization that greater dominance in a language increases reliance on language-specific syntactic processing strategies while correspondingly decreasing reliance on more domain-general semantic processing strategies.

Developmental effects in the online use of morphosyntactic cues in sentence processing: Evidence from Tagalog

Children must necessarily process their input in order to learn it, yet the architecture of the developing parsing system and how it interfaces with acquisition is unclear. In the current paper we report experimental and corpus data investigating adult and children's use of morphosyntactic cues for making incremental online predictions of thematic roles in Tagalog, a verb-initial symmetrical voice language of the Philippines. In Study 1, Tagalog-speaking adults completed a visual world eye-tracking experiment in which they viewed pictures of causative actions that were described by transitive sentences manipulated for voice and word order. The pattern of results showed that adults process agent and patient voice differently, predicting the upcoming noun in the patient voice but not in the agent voice, consistent with the observation of a patient voice preference in adult sentence production. In Study 2, our analysis of a corpus of child-directed speech showed that children heard more patient voice- than agent voice-marked verbs. In Study 3, 5-, 7-, and 9-year-old children completed a similar eye-tracking task as used in Study 1. The overall pattern of results suggested that, like the adults in Study 1, children process agent and patient voice differently in a manner that reflects the input distributions, with children developing towards the adult state across early childhood. The results are most consistent with theoretical accounts that identify a key role for input distributions in acquisition and language processing.

The interaction of grammatically distinct agreement dependencies in predictive processing

ABSTRACT Previous research has found that comprehenders sometimes predict information that is grammatically unlicensed by sentence constraints. An open question is why such grammatically unlicensed predictions occur. We examined the possibility that unlicensed predictions arise in situations of information conflict, for instance when comprehenders try to predict upcoming words while simultaneously building dependencies with previously encountered elements in memory. German possessive pronouns are a good testing ground for this hypothesis because they encode two grammatically distinct agreement dependencies: a retrospective one between the possessive and its previously mentioned referent, and a prospective one between the possessive and its following nominal head. In two visual world eye-tracking experiments, we estimated the onset of predictive effects in participants’ fixations. The results showed that the retrospective dependency affected resolution of the prospective dependency by shifting the onset of predictive effects. We attribute this effect to an interaction between predictive and memory retrieval processes.