Visual-world Eye-tracking Experiment Research Articles

Online Assessment of Cross-Linguistic Similarity as a Measure of L2 Perceptual Categorization Accuracy

The effect of cross-linguistic similarity on the development of target-like categories in a second or additional language is widely attested. Research also shows that second-language speakers may access both their native and the second-language lexicons when processing second-language speech. Forty-three Catalan learners of English performed a perceptual assimilation task evaluating the perceived similarity between English and Catalan vowels and also participated in a visual world eye-tracking experiment investigating between-language lexical competition. The focus of the study was the English vowel contrasts /iː/-/ɪ/ and /æ/-/ʌ/. The perceptual task confirmed that English /iː/ and /æ/were perceptually closer to native Catalan categories than English /ɪ/ and /ʌ/. The results of the spoken word recognition task indicated that learners experienced greater competition from native words when the target words contained English /iː/ and /æ/, illustrating a close link between the two types of tasks. However, differences in the magnitude of cross-language lexical competition were found to be only weakly related to learners’ degree of perceived similarity to native categories at an individual level. We conclude that online tasks provide a potentially effective method of assessing cross-linguistic similarity without the concerns inherent to more traditional offline approaches.

The Importance of Linguistic Factors: He Likes Subject Referents.

We report the results of one visual-world eye-tracking experiment and two referent selection tasks in which we investigated the effects of information structure in the form of prosody and word order manipulation on the processing of subject pronouns er and der in German. Factors such as subjecthood, focus, and topicality, as well as order of mention have been linked to an increased probability of certain referents being selected as the pronoun's antecedent and described as increasing this referent's prominence, salience, or accessibility. The goal of this study was to find out whether pronoun processing is primarily guided by linguistic factors (e.g., grammatical role) or nonlinguistic factors (e.g., first-mention), and whether pronoun interpretation can be described in terms of referents' "prominence"/"accessibility"/"salience." The results showed an overall subject preference for er, whereas der was affected by the object role and focus marking. While focus increases the attentional load and enhances memory representation for the focused referent making the focused referent more available, ultimately it did not affect the final interpretation of er, suggesting that "prominence" or the related concepts do not explain referent selection preferences. Overall, the results suggest a primacy of linguistic factors in determining pronounresolution.

Processing to learn noncanonical word orders: Exploring linguistic and cognitive predictors of reanalysis in early L2 sentence comprehension

Abstract To test the contributions of processing to L2 syntax learning, this study explores (cross-) linguistic and cognitive predictors of sentence reanalysis in the L2 comprehension of relative clauses among low-intermediate L1 German adolescent learners of L2 English. Specifically, we test the degree to which L2 comprehension is affected by L2 proficiency, reanalysis ability in a related, earlier-acquired L2 structure (questions), reanalysis ability of relative clauses in the L1, cognitive control, and cognitive capacity. In visual-world eye-tracking experiments, 141 adolescent German-speaking L2 learners of English selected target pictures for auditorily presented questions and relative clauses in the L1 and in the L2. The results showed a strong subject preference for L2 relative clauses. Learners’ L2 proficiency and their processing of object questions in the L2 predicted reanalysis for object relatives in eye movements, reaction times, and comprehension accuracy. In contrast, there was no evidence that cognitive control or working memory systematically affected the processing of object relatives. These findings suggest that linguistic processing outweighs cognitive processing in accounting for individual differences in low-intermediate L2 acquisition of complex grammar. Specifically, learners recruit shared processing mechanisms and routines across grammatical structures to pave a way in the acquisition of syntax.

The time course of moral decision making in bilinguals' native and foreign language

Previous work has shown that moral decision making depends on whether moral dilemmas are presented in the native or a foreign language (Foreign Language effect). In that work, the focus was on bilinguals' final decision, but the tasks used, failed to capture the processes involved over time. The aim of this study was to examine the time course of moral decision making in bilinguals' native and foreign language prior to and after their moral decision. In a visual-world eye-tracking experiment, 82 Dutch-English bilinguals listened to 20 moral dilemmas (e.g., would you kill one to save five?) in their native or foreign language, while looking at two pictures containing key people involved in the dilemmas. These pictures illustrated (1) the person/people that is/are sacrificed, and (2) the person/people that is/are not sacrificed, depending on the participants' decision which was measured with yes/no-questions. The Foreign Language effect was replicated for the decisions on the personal dilemmas. Importantly, the eye gaze data showed that in the native language, listeners looked at the person who they did not sacrifice, whereas in the foreign language, they looked at the people who they sacrificed. A speculative explanation is that bilinguals might have experienced guilt in the native language, and therefore focused attention on the person they did not sacrifice, while in the foreign language they might have experienced less guilt because they focused attention on the people they sacrificed. More research is needed to understand the influence of factors such as emotion reduction and cognitive load on moral decision making.

Similarity-Based Interference in Sentence Comprehension in Aphasia: a Computational Evaluation of Two Models of Cue-Based Retrieval

Sentence comprehension requires the listener to link incoming words with short-term memory representations in order to build linguistic dependencies. The cue-based retrieval theory of sentence processing predicts that the retrieval of these memory representations is affected by similarity-based interference. We present the first large-scale computational evaluation of interference effects in two models of sentence processing — the activation-based model and a modification of the direct-access model — in individuals with aphasia (IWA) and control participants in German. The parameters of the models are linked to prominent theories of processing deficits in aphasia, and the models are tested against two linguistic constructions in German: pronoun resolution and relative clauses. The data come from a visual-world eye-tracking experiment combined with a sentence-picture matching task. The results show that both control participants and IWA are susceptible to retrieval interference, and that a combination of theoretical explanations (intermittent deficiencies, slow syntax, and resource reduction) can explain IWA’s deficits in sentence processing. Model comparisons reveal that both models have a similar predictive performance in pronoun resolution, but the activation-based model outperforms the direct-access model in relative clauses.

Integration of visual context in early and late bilingual language processing: evidence from eye-tracking.

Previous research on the processing of language embedded in a rich visual context has revealed the strong effect that a recently viewed action event has on language comprehension. It has been shown that listeners are more likely to view the target object of a recently performed event than look at the target object of a plausible future event during sentence utterance, regardless of the tense cue. In the current visual-world eye-tracking experiments, we tested the strength of the recently observed visual context with a group of English monolingual and two groups of English-French early and late bilingual speakers. By comparing these different groups, we examined whether bilingual speakers, as a consequence of greater cognitive flexibility when integrating visual context and language information, show early anticipatory eye-movements toward the target object. We further asked whether early and late bilinguals show differences in their processing. The findings of the three eye-tracking experiments revealed an overall preference for the recently seen event. However, as a result of the early provision of tense cue, this preference was quickly diminished in all three groups. Moreover, the bilingual groups showed an earlier decrease in reliance on the recently seen event compared to monolingual speakers and the early bilinguals showed anticipatory eye-movements toward the plausible future event target. Furthermore, a post-experimental memory test revealed that the bilingual groups recalled the future events marginally better than the recent events, whereas the reverse was found in the monolingual groups.

Evidence for two stages of prediction in non-native speakers: A visual-world eye-tracking study

AbstractComprehenders predict what a speaker is likely to say when listening to non-native (L2) and native (L1) utterances. But what are the characteristics of L2 prediction, and how does it relate to L1 prediction? We addressed this question in a visual-world eye-tracking experiment, which tested when L2 English comprehenders integrated perspective into their predictions. Male and female participants listened to male and female speakers producing sentences (e.g.,I would like to wear the nice…) about stereotypically masculine (target: tie; distractor: drill) and feminine (target: dress; distractor: hairdryer) objects. Participants predicted associatively, fixating objects semantically associated with critical verbs (here, the tie and the dress). They also predicted stereotypically consistent objects (e.g., the tie rather than the dress, given the male speaker). Consistent predictions were made later than associative predictions, and were delayed for L2 speakers relative to L1 speakers. These findings suggest prediction involves both automatic and non-automatic stages.

The role of verbal and working memory skills in Turkish-speaking children’s morphosyntactic prediction

AbstractThe current study investigated the contribution of multiple verbal and working memory (WM) skills to morphosyntactic prediction in Turkish-speaking 4- to 8-year-old children. In a visual world eye-tracking experiment, 76 children were presented with verb-final sentences with nominative and accusative case markers on the initial noun (e.g., the fast rabbitnominative … the carrotaccusativeeatfuture vs. the fast rabbitaccusative … the foxnominativeeatfuture) while they were looking at a visual display with three objects (e.g., rabbit, carrot, and fox). Importantly, the case markers on the initial noun could be used to predict the second noun in these sentences. The results revealed that when children’s early productive vocabulary and language production skills were higher, the better and faster they were in predicting the upcoming noun. The episodic buffer, a component of WM, was also positively associated with children’s morphosyntactic prediction abilities. The implications of these results for the mechanisms of linguistic prediction are discussed.

Correction: Using verb morphology to predict subject number in L1 and L2 sentence processing: A visual-world eye-tracking experiment

This article details a correction to: Koch, E. M., Bulté, B., Housen, A., &amp; Godfroid, A. (2021). Using verb morphology to predict subject number in L1 and L2 sentence processing: A visual-world eye-tracking experiment. <em>Journal of the European Second Language Association, 5</em>(1), 115–132. DOI: <a href="http://doi.org/10.22599/jesla.79" target="_blank">http://doi.org/10.22599/jesla.79</a>.

Do Children Use Multi-Word Information in Real-Time Sentence Comprehension?

Meaning in language emerges from multiple words, and children are sensitive to multi-word frequency from infancy. While children successfully use cues from single words to generate linguistic predictions, it is less clear whether and how they use multi-word sequences to guide real-time language processing and whether they form predictions on the basis of multi-word information or pairwise associations. We address these questions in two visual-world eye-tracking experiments with 5- to 8-year-old children. In Experiment 1, we asked whether children generate more robust predictions for the sentence-final object of highly frequent sequences (e.g., "Throw the ball"), compared to less frequent sequences (e.g., "Throw the book"). We further examined if gaze patterns reflect event knowledge or phrasal frequency by comparing the processing of phrases that have the same event structure but differ in multi-word content (e.g., "Brush your teeth" vs. "Brush her teeth"). In the second study, we employed a training paradigm to ask if children are capable of generating predictio.ns from novel multi-word associations while controlling for the overall frequency of the sequences. While the results of Experiment 1 suggested that children primarily relied on event associations to generate real-time predictions, those of Experiment 2 showed that the same children were able to use recurring novel multi-word sequences to generate real-time linguistic predictions. Together, these findings suggest that children can draw on multi-word information to generate linguistic predictions, in a context-dependent fashion, and highlight the need to account for the influence of multi-word sequences in models of language processing.

Using verb morphology to predict subject number in L1 and L2 sentence processing: A visual-world eye-tracking experiment

We investigated whether adult German native speakers and Dutch-speaking second-language learners of German exploit German regular verb morphology for predictive sentence processing and whether such predictive processing is moderated by working memory capacity and awareness of the predictive cue. In a picture-matching task with visual-world eye-tracking, the participants (first-language group: <em>n</em> = 31; second-language group: <em>n</em> = 30) saw two pictures of action scenes, varying in the number of referents depicted (singular versus plural). Simultaneously, they heard sentences in German in which the verb suffix represented the first reliable cue for grammatical number of the upcoming subject noun phrase. Successful exploitation of this number cue was measured as anticipatory button-press reaction times and eye-movements toward the correct picture, before subject onset. We found significant prediction effects in both the first-language and the second-language group, with anticipatory processing being somewhat faster in the native speakers than in the learners. Faster reaction times on prediction trials were associated with higher working memory scores. Debriefing interviews revealed that all participants had become aware that they could use the verb form predictively, adding further evidence that prediction can be a conscious process.

Prediction involves two stages: Evidence from visual-world eye-tracking

Comprehenders often predict what they are going to hear. But do they make the best predictions possible? We addressed this question in three visual-world eye-tracking experiments by asking when comprehenders consider perspective. Male and female participants listened to male and female speakers producing sentences (e.g., I would like to wear the nice…) about stereotypically masculine (target: tie; distractor: drill) and feminine (target: dress, distractor: hairdryer) objects. In all three experiments, participants rapidly predicted semantic associates of the verb. But participants also predicted consistently – that is, consistent with their beliefs about what the speaker would ultimately say. They predicted consistently from the speaker’s perspective in Experiment 1, their own perspective in Experiment 2, and the character’s perspective in Experiment 3. This consistent effect occurred later than the associative effect. We conclude that comprehenders consider perspective when predicting, but not from the earliest moments of prediction, consistent with a two-stage account.

Influence of Actor's Congruent and Incongruent Gaze on Language Processing.

In interpreting spoken sentences in event contexts, comprehenders both integrate their current interpretation of language with the recent past (e.g., events they have witnessed) and develop expectations about future event possibilities. Tense cues can disambiguate this linking but temporary ambiguity in their interpretation may lead comprehenders to also rely on further, situation-specific cues (e.g., an actor's gaze as a cue to his future actions). How comprehenders reconcile these different cues in real time is an open issue that we must address to accommodate comprehension. It has been suggested that relating a referential expression (e.g., a verb) to a referent (e.g., a recent event) is preferred over relying on other cues that refer to the future and are not yet referentially grounded (“recent-event preference”). Two visual-world eye-tracking experiments compared this recent-event preference with effects of an actor's gaze and of tense/temporal adverbs as cues to a future action event. The results revealed that people overall preferred to focus on the recent (vs. future) event target in their interpretation, suggesting that while a congruent and incongruent actor gaze can jointly with futuric linguistic cues neutralize the recent-event preference late in the sentence, the latter still plays a key role in shaping participants' initial verb-based event interpretation. Additional post-experimental memory tests provided insight into the longevity of the gaze effects.

Integration of Social Context vs. Linguistic Reference During Situated Language Processing.

Research findings on language comprehension suggest that many kinds of non-linguistic cues can rapidly affect language processing. Extant processing accounts of situated language comprehension model these rapid effects and are only beginning to accommodate the role of non-linguistic emotional, cues. To begin with a detailed characterization of distinct cues and their relative effects, three visual-world eye-tracking experiments assessed the relative importance of two cue types (action depictions vs. emotional facial expressions) as well as the effects of the degree of naturalness of social (facial) cues (smileys vs. natural faces). We predicted to replicate previously reported rapid effects of referentially mediated actions. In addition, we assessed distinct world-language relations. If how a cue is conveyed matters for its effect, then a verb referencing an action depiction should elicit a stronger immediate effect on visual attention and language comprehension than a speaker's emotional facial expression. The latter is mediated non-referentially via the emotional connotations of an adverb. The results replicated a pronounced facilitatory effect of action depiction (relative to no action depiction). By contrast, the facilitatory effect of a preceding speaker's emotional face was less pronounced. How the facial emotion was rendered mattered in that the emotional face effect was present with natural faces (Experiment 2) but not with smileys (Experiment 1). Experiment 3 suggests that contrast, i.e., strongly opposing emotional valence information vs. non-opposing valence information, might matter for the directionality of this effect. These results are the first step toward a more principled account of how distinct visual (social) cues modulate language processing, whereby the visual cues that are referenced by language (the depicted action), copresent (the depicted action), and more natural (the natural emotional prime face) tend to exert more pronounced effects.

The Contribution of Form Repetition to Listeners’ Expectation of Givenness in Online Reference Resolution

ABSTRACT Although it is clear that unaccented referring expressions are associated with given information in a discourse, it is less clear what aspects of givenness are relevant. We examine whether listeners’ expectation of givenness depends on repetition of a referring expression or on contextual evocation of a referent. The results from two visual world eye-tracking experiments suggest that for interpretation, listeners associated reduced prominence with a repeated referring expression. Listeners expect previously evoked referents to be candidates for reduced referring expressions only when they are referred to with the exact same referential form. The data also suggest that when referents are referred to with different referential forms across utterances, accenting facilitates linking those forms for co-reference.

Syntactic and Semantic Influences on the Time Course of Relative Clause Processing: The Role of Language Dominance

We conducted a visual world eye-tracking experiment with highly proficient Spanish-English bilingual adults to investigate the effects of relative language dominance, operationalized as a continuous, multidimensional variable, on the time course of relative clause processing in the first-learned language, Spanish. We found that participants exhibited two distinct processing preferences: a semantically driven preference to assign agency to referents of lexically animate noun phrases and a syntactically driven preference to interpret relative clauses as subject-extracted. Spanish dominance was found to exert a distinct influence on each of these preferences, gradiently attenuating the semantic preference while gradiently exaggerating the syntactic preference. While these results might be attributable to particular properties of Spanish and English, they also suggest a possible generalization that greater dominance in a language increases reliance on language-specific syntactic processing strategies while correspondingly decreasing reliance on more domain-general semantic processing strategies.

Developmental effects in the online use of morphosyntactic cues in sentence processing: Evidence from Tagalog

Children must necessarily process their input in order to learn it, yet the architecture of the developing parsing system and how it interfaces with acquisition is unclear. In the current paper we report experimental and corpus data investigating adult and children's use of morphosyntactic cues for making incremental online predictions of thematic roles in Tagalog, a verb-initial symmetrical voice language of the Philippines. In Study 1, Tagalog-speaking adults completed a visual world eye-tracking experiment in which they viewed pictures of causative actions that were described by transitive sentences manipulated for voice and word order. The pattern of results showed that adults process agent and patient voice differently, predicting the upcoming noun in the patient voice but not in the agent voice, consistent with the observation of a patient voice preference in adult sentence production. In Study 2, our analysis of a corpus of child-directed speech showed that children heard more patient voice- than agent voice-marked verbs. In Study 3, 5-, 7-, and 9-year-old children completed a similar eye-tracking task as used in Study 1. The overall pattern of results suggested that, like the adults in Study 1, children process agent and patient voice differently in a manner that reflects the input distributions, with children developing towards the adult state across early childhood. The results are most consistent with theoretical accounts that identify a key role for input distributions in acquisition and language processing.

The interaction of grammatically distinct agreement dependencies in predictive processing

ABSTRACT Previous research has found that comprehenders sometimes predict information that is grammatically unlicensed by sentence constraints. An open question is why such grammatically unlicensed predictions occur. We examined the possibility that unlicensed predictions arise in situations of information conflict, for instance when comprehenders try to predict upcoming words while simultaneously building dependencies with previously encountered elements in memory. German possessive pronouns are a good testing ground for this hypothesis because they encode two grammatically distinct agreement dependencies: a retrospective one between the possessive and its previously mentioned referent, and a prospective one between the possessive and its following nominal head. In two visual world eye-tracking experiments, we estimated the onset of predictive effects in participants’ fixations. The results showed that the retrospective dependency affected resolution of the prospective dependency by shifting the onset of predictive effects. We attribute this effect to an interaction between predictive and memory retrieval processes.

The time-course of competition from the L1 grammar in L2 sentence processing: Evidence from cross-linguistic structural priming

In two visual-world eye-tracking experiments, we explore the extent to which conflicting first-language (L1) based grammatical parses influence second-language (L2) learners’ on-line and off-line interpretation of sentences in the L2. We used cross-linguistic structural priming to potentially boost competition from the L1 grammar during the processing of wh-questions in English. For L1-German learners (Experiment 1), sentence-final interpretations showed effects of conflicting L1 parses in the greater number of misinterpretations of English subject as object wh-questions than vice versa. In a follow-up experiment with a comparison group of L1-Japanese learners (Experiment 2), we found the reverse pattern in sentence-final interpretations, with lower accuracy on English object than subject wh-questions. The asymmetry in comprehension accuracy between Experiments 1 and 2 demonstrated that the effect observed among L1-German learners reflected grammar competition from the L1 rather than general L2 effects. In both experiments, we observed cross-linguistic priming from L1 object questions to L2 object questions in on-line processing. Yet, cross-linguistic priming did not enhance competition from L1 object questions on L2 subject questions in Experiment 1 as reflected in comprehension accuracy or eye-gaze during processing. Together, these findings present evidence of cross-linguistic structural priming for equivalent interpretations (object–object), but indicate that priming does not extend to competing L1 interpretations of word-order equivalent structures cross-linguistically (object–subject). We suggest that interference from conflicting L1-based grammatical parses is narrowly delimited in intermediate to advanced L2 sentence processing.

The limited role of hippocampal declarative memory in transient semantic activation during online language processing

Recent findings point to a role for hippocampus in the moment-by-moment processing of language, including the use and generation of semantic features in certain contexts. What role the hippocampus might play in the processing of semantic relations in spoken language comprehension, however, is unknown. Here we test patients with bilateral hippocampal damage and dense amnesia in order to examine the necessity of hippocampus for lexico-semantic mapping processes in spoken language understanding. In two visual-world eye-tracking experiments, we monitor eye movements to images that are semantically related to spoken words and sentences. We find no impairment in amnesia, relative to matched healthy comparison participants. These findings suggest, at least for close semantic links and simple language comprehension tasks, a lack of necessity for hippocampus in lexico-semantic mapping between spoken words and simple pictures.