Sentence-picture Verification Task Research Articles

Embodied cognition and L2 sentence comprehension: an eye-tracking study of motor representations.

Evidence from neuroscience and behavioral research has indicated that language meaning is grounded in our motor-perceptual experiences of the world. However, the question of whether motor embodiment occurs at the sentence level in L2 (second language) comprehension has been raised. Furthermore, existing studies on motor embodiment in L2 have primarily focused on the lexical and phrasal levels, often providing conflicting and indeterminate results. Therefore, to address this gap, the present eye-tracking study aimed to explore the embodied mental representations formed during the reading comprehension of L2 action sentences. Specifically, it sought to identify the types of motor representations formed during L2 action sentence comprehension and the extent to which these representations are motor embodied. A total of 56 advanced L2 learners participated in a Sentence-Picture Verification Task, during which their response times (RTs) and eye movements were recorded. Each sentence-picture pair depicted an action that either matched or mismatched the action implied by the sentence. Data analysis focused on areas of interest around the body effectors. RTs in the mismatch condition indicated an impeding effect. Furthermore, fixations on the body effector executing an action were longer in the mismatch condition, especially in late eye-movement measures.

Negative Pragmatic Transfer in Bilinguals: Cross-Linguistic Influence in the Acquisition of Quantifiers

Building on the cross-linguistic variability in the meaning of vague quantifiers, this study explores the potential for negative transfer in Italian-Slovenian bilinguals concerning the use of quantificational determiners, specifically the translational equivalents of the English “many”, that is the Slovenian "precej" and "veliko". The aim is to identify relevant aspects of pragmatic knowledge for cross-linguistic influence. The study presents the results of a sentence-picture verification task in which Slovenian native speakers and Italian-Slovenian bilinguals evaluated sentences of the form "Quantifier X are Y" in relation to visual contexts. The results suggest that Italian learners of Slovenian, unlike Slovenian native speakers, fail to distinguish between "precej" and "veliko". This finding aligns with the negative transfer hypothesis. The study highlights the potential role of pragmatic knowledge in cross-linguistic transfer, particularly in the context of vague quantifiers.

Testing two-step models of negative quantification using a novel machine learning analysis of EEG

ABSTRACT The sentences “More than half of the students passed the exam” and “Fewer than half of the students failed the exam” describe the same set of situations, and yet the former results in shorter reaction times in verification tasks. The two-step model explains this result by postulating that negative quantifiers contain hidden negation, which involves an extra processing stage. To test this theory, we applied a novel EEG analysis technique focused on detecting cognitive stages (HsMM-MVPA) to data from a picture-sentence verification task. We estimated the number of processing stages during reading and verification of quantified sentences (e.g. “Fewer than half of the dots are blue”) that followed the presentation of pictures containing coloured geometric shapes. We did not find evidence for an extra step during the verification of sentences with fewer than half. We provide an alternative interpretation of our results in line with an expectation-based pragmatic account.

The influence of sentence focus on mental simulation: A possible cause of ACE instability.

Recent studies have revealed the instability of the action-sentence compatibility effect (ACE). The current study was designed to demonstrate the hypothesis that the instability of the ACE may be attributed to the instability of focused information in a sentence. A pilot study indicated that the focused information of sentences was relatively stable in the sentence-picture verification task but exhibited significant interindividual variability in the action-sentence compatibility paradigm in previous studies. Experiments 1 and 2 examined the effect of sentence focus on the shape match effect and the ACE by manipulating the focused information of sentences using the focus marker word "" (is). Experiment 1 found that the shape match effect occurred in the original sentence, while it disappeared when the word "" (is) was used to make an object noun no longer the focus of a sentence. Experiment 2 failed to observe the ACE regardless of whether the sentence focus was on the action information. Experiment 3 modified the focus manipulation to observe its impact on the ACE using different fonts and underlines to highlight the focused information. The results indicated that the ACE only occurred when the action information was the sentence focus. These findings suggest that sentence focus influences mental simulation, and the instability of the ACE is likely to be associated with the instability of sentence focus in previous studies. This outcome highlights the crucial role of identifying specific information as the critical element expressed in the current linguistic context for successful simulation.

The comprehension of double negation in Chinese children with reading difficulties

ABSTRACT This study investigated the processing of double negation in Chinese children with reading difficulties. The comprehension of Mandarin affirmative, single negative and double negative sentences was tested with Chinese young poor readers and typical readers, using a sentence-picture verification task. Results showed that double negative sentences were most difficult to process for both groups; the poor readers performed significantly worse than the typical readers in comprehending double negative sentences, while no difference between the two groups was observed in comprehending affirmative and single negative sentences. Besides, morphological awareness correlated with the comprehension of double negative and single negative sentences in poor readers, while this correlation did not emerge with typical readers. Overall, our results suggest that children with reading difficulties experienced great processing difficulty in double negation, confirming that reading disorders are also characterised by oral language difficulties, in particular in the comprehension of sentences requiring high processing costs.

Automatic mental simulation in native and non-native speakers

Pictures of objects are verified faster when they match the implied orientation, shape, and color in a sentence-picture verification task, suggesting that people mentally simulate these features during language comprehension. Previous studies had an unintended correlation between match status and the required response, which may have influenced participants’ responses by eliciting strategic use of this correlation. We removed this correlation by including color-matching filler trials and investigated if the color-match effect was still obtained. In both a native sample (Experiment 1) and a non-native sample (Experiment 2), we found strong evidence for a color-match advantage on median reaction time and error rates. Our results are consistent with the view that color is automatically simulated during language comprehension as predicted by the grounded cognition framework.

Two “many”-Words in Italian? On Molto-Tanto and Cross-linguistic Differences in Quantification

This article investigates the variability in the meaning of vague quantifiers across different languages, focusing on Italian’s m(any)-words molto and tanto. The aim was to replicate a previous analysis conducted on Slovenian m-words – precej and veliko – examining whether the Italian ones exhibit a similar pragmatic strengthening effect. Using a sentence-picture verification task, we tested 88 Italian monolingual participants on their evaluation of sentences of the form “Quantifier X are Y”. Our results showed that, unlike Slovenian speakers, Italian speakers do not exhibit a difference in the evaluation of molto and tanto, suggesting that the two words have the same numerical bound and are interchangeable as amount modifiers. Our analysis suggests that there are underlying semantic distinctions between molto and tanto that require further investigation. These findings contribute to our understanding of the variability in the use of quantifiers across languages and highlight the importance of examining subtle differences in meaning when studying vague quantifiers.

The interplay of computational complexity and memory load during quantifier verification

ABSTRACTFormal analysis of the minimal computational complexity of verification algorithms for natural language quantifiers implies that different classes of quantifiers demand the engagement of different cognitive resources for their verification. In particular, sentences containing proportional quantifiers, e.g. “most”, provably require a memory component, whereas non-proportional quantifiers, e.g. “all”, “three”, do not. In an ERP study, we tested whether previously observed differences between these classes were modulated by memory load. Participants performed a picture-sentence verification task while they had to remember a string of 2 or 4 digits to be compared to a second string at the end of a trial. Relative to non-proportional quantifiers, proportional quantifiers elicited a sentence-internal sustained negativity. Additionally, an interaction between Digit-Load and Quantifier-Class was observed at the sentence-final word. Our results suggest that constraints on cognitive resources deployed during human sentence processing and verification are of the same nature as formal constraints on abstract machines.

The interaction between motor simulation and spatial perspective-taking in action language: a cross-cultural study

Growing evidence has revealed the crucial role of motor simulation and spatial perspective-taking in action language. However, there is still a lack of understanding of how motor and spatial processes interact when there are multiple actors involved, and if embodied processes are consistent across different cultures. To address this gap, we examined the interaction between motor simulation and spatial perspective-taking in action-sentences comprehension, along with the consistency of embodied processes across cultures. We collected data from Italian and US English speakers using an online sentence-picture verification task. The participants completed four conditions: two congruent (i.e., the participant is the agent in the sentence and the photo; the agent is someone else interacting with the participant in both the sentence and the picture) and two incongruent (i.e., the agents of the sentence and the picture do not match). The results show that when the perspective of the picture matched that described in the sentence-processing reaction times (RTs) were faster than in the incongruent conditions. In the congruent conditions where the agent is someone else, RTs were slower compared to the condition where the participant is the agent. This has been interpreted as claiming that motor simulation and perspective-taking are independent processes interacting during sentence comprehension (e.g., motor simulation is always run in the role of the agent, but we can adopt multiple perspectives depending on the pronouns and the contextual cues). Furthermore, Bayesian analysis provided evidence that embodied processing of action language entwines a common mechanism, suggesting cross-cultural consistency of embodied processes.

Pragmatic responses to under-informative some-statements are not scalar implicatures

A highly emblematic paradigm in experimental pragmatics consists in presenting participants with an existentially quantified sentence of the form Some X are Y in a context in which all X are obviously Y. Participants who reject such sentences as false or infelicitous are said to adopt a ‘pragmatic’ instead of a ‘logical’ reading of some, and to derive the scalar implicature Some, but not all X are Y. Although there are several competing accounts of scalar implicatures, virtually all of them assume that a participant who responds pragmatically to an under-informative some-sentence mentally entertains a linguistic representation of the negation of a stronger alternative (All X are Y). Yet, there is no evidence that judging an under-informative some-sentence false or infelicitous actually involves the derivation of the some, but not all scalar implicature. We report three experiments consisting of a sentence-picture verification task followed by a forced choice between two paraphrases of the sentence initially assessed. These experiments robustly show that hearers who reject an under-informative some-sentence do so without explicitly entertaining a some, but not all implicature. Our results represent a strong challenge for grammatical accounts of scalar implicature, which all presuppose a mechanism of negation of stronger alternatives, and force a drastic reinterpretation of processing data on scalar implicatures. More generally, our findings show that one should not conflate psychological models of pragmatic processing with a reconstructed link between sentences and their potential meanings.

Task Demands and Sentence Reading Comprehension among Healthy Older Adults: The Complementary Roles of Cognitive Reserve and Working Memory

Ageing entails different functional brain changes. Education, reading experience, and leisure activities, among others, might contribute to the maintenance of cognitive performance among older adults and are conceptualised as proxies for cognitive reserve. However, ageing also conveys a depletion of working memory capacity, which adversely impacts language comprehension. This study investigated how cognitive reserve proxies and working memory jointly predict the performance of healthy older adults in a sentence reading comprehension task, and how their predictive value changes depending on sentence structure and task demands. Cognitively healthy older adults (n = 120) completed a sentence-picture verification task under two conditions: concurrent viewing of the sentence and picture or their sequential presentation, thereby imposing greater demands on working memory. They also completed a questionnaire on cognitive reserve proxies as well as a verbal working memory test. The sentence structure was manipulated by altering the canonical word order and modifying the amount of propositional information. While the cognitive reserve was the main predictor in the concurrent condition, the predictive role of working memory increased under the sequential presentation, particularly for complex sentences. These findings highlight the complementary roles played by cognitive reserve and working memory in the reading comprehension of older adults.

Processing pragmatic inferences in L2 French speakers

A large amount of literature exists on how native speakers derive and process pragmatic inferences, yet few studies have examined the issue in second language learners, despite a controversial debate of second language (L2) ultimate attainment of phenomena situated at external interfaces. This study contributes to the debate on the integration of external interfaces in highly proficient end-state adult L2 grammars. In an effort to bridge the empirical gap in the past literature, this article reports on two sentence-picture verification tasks designed to test the processing cost of the exhaustive inference associated with cleft sentences in the L2 French of English learners with different proficiency levels. Truth-value judgments and reaction times were recorded in contexts that violated or supported the exhaustive inference. Overall, results show that L2 learners diverge from the natives in their online processing of the exhaustive inference only and that proficiency plays an important role in predicting their behavior. Nevertheless, what post-hoc observations of the data reveal is that L2 length of exposure to native input might be an even better predictor, since only those L2 speakers who have had more exposure to native French input compute the exhaustive inference in clefts as fast as French natives.

Simulating background settings during spoken and written sentence comprehension.

Previous findings from the sentence-picture verification task demonstrated that comprehenders simulate visual information about intrinsic attributes of described objects. Of interest is whether comprehenders may also simulate the setting in which an event takes place, such as, for example, the light information. To address this question, four experiments were conducted in which participants (total N = 412) either listened to (Experiment 1) or read (Experiment 3) sentences like “The sun is shining onto a bench” followed by a picture with the matching object (bench) and either the matching lighting condition of the scene (sunlit bench against the sunlit background) or the mismatching one (moonlit bench against the moonlit background). In both experiments, response times (RTs) were shorter when the lighting condition of the pictured scene matched the one implied in the sentence. However, no difference in RTs was observed when the processing of spoken sentences was interfered with visual noise (Experiment 2). Specifically, the results showed that visual interference disrupted incongruent visual content activated by listening to the sentences, as evidenced by faster responses on mismatching trials. Similarly, no difference in RTs was observed when the lighting condition of the pictured scene matched sentence context, but the target object presented for verification mismatched sentence context (Experiment 4). Thus, the locus of simulation effect is on the lighting representation of the target object rather than the lighting representation of the background. These findings support embodied and situated accounts of cognition, suggesting that comprehenders do not simulate objects independently of background settings.

Perceptual simulation in language comprehension and Chinese character reading among third-grade Hong Kong children

The current study investigated perceptual simulation and its relationship with literacy ability in Chinese children. Ninety-three third-grade Hong Kong Chinese children completed a sentence-picture verification task for perceptual simulation. In this task, a sentence mentioning an object was presented first, followed by a picture involving the object. The picture was either a perceptual match with the preceding sentence or a mismatch. The participants were asked to judge whether the object in the picture was mentioned in the preceding sentence. Literacy ability was measured by a Chinese character reading task. Model analysis revealed a significant mismatch effect: the children gave faster correct responses to the perceptually matched pictures than to the mismatched ones. Furthermore, the children with lower literacy abilities displayed larger mismatch effects than those with higher literacy abilities. The results suggest that children with lower literacy abilities may rely more heavily on reactivated perceptual representations in language comprehension.

Semantic integration of multidimensional perceptual information in L1 sentence comprehension

AbstractMany studies have substantiated the perceptual symbol system, which assumes a routine generation of perceptual information during language comprehension, but little is known about the processing format in which the perceptual information of different dimensions is conveyed simultaneously during sentence comprehension. The current study provides the first experimental evidence of how multidimensional perceptual information (color and shape) was processed during online sentence comprehension in Mandarin. We designed three consecutive sentence–picture verification tasks that only differed in the delay of the display of pictures preceded by declarative sentences. The processing was analyzed in three stages based on time intervals (i.e., 0ms, +750ms, +1500ms). The response accuracy and response time data were reported. The initial stage (i.e., ISI=0ms) attested the match effect of color and shape, but the simulated representation of color and shape did not interact. In the intermediate stage (i.e., ISI=750ms), the routinely simulated color and shape interacted, but the match facilitation was found only in cases where one perceptual information was in mismatch while the other was not. In the final stage (i.e., ISI=1500ms), the match facilitation of one particular perceptual property was influenced by a mismatch with the other perceptual property. These results suggested that multiple perceptual information presented simultaneously was processed in an additive manner to a large extent before entering into the final stage, where the simulated perceptual information was integrated in a multiplicative manner. The results also suggested that color and shape were comparable to object recognition when conjointly conveyed. In relation to other evidence from behavioral and event-related potential studies on sentence reading in the discussion, we subscribed to the idea that the full semantic integration became available over time.

Simulating semantics: Are individual differences in motor imagery related to sensorimotor effects in language processing?

In embodied theories of semantic representation, the processes and mechanisms of modal simulations that are engaged during semantic processing have tended to be underspecified. We investigated the possibility that motor imagery may be a mechanism of simulation, using an individual differences approach. In this preregistered study, we assessed motor imagery abilities (n = 161) with implicit and explicit measures and identified two latent factors. We then examined whether those factors account for significant variations in sensorimotor effects observed in three different language tasks: a lexical-decision task, syntactic classification task, and sentence-picture verification task. In the language tasks, when all participants were considered together, we replicated some previously reported sensorimotor effects (e.g., body-object interaction [BOI], effects in semantic processing, wherein words associated with more sensorimotor information were processed more quickly than words associated with less sensorimotor information) and did not replicate others (e.g., BOI effects in the lexical decision task, congruency effects in sentence picture verification task). There were no significant relationships between imagery factor scores and sensorimotor effects. A follow-up analysis using scores from each motor imagery measure revealed a significant interaction between hand movement imagery and BOI effects in the syntactic classification task, with those higher in this imagery ability showing a larger BOI effect. This latter result may suggest that specific types of motor imagery are related to sensorimotor effects in semantic processing; however, further investigation is needed. In general, our findings provide little support for the possibility that motor imagery is an underlying mechanism of sensorimotor simulation during language processing. (PsycInfo Database Record (c) 2022 APA, all rights reserved).

The comprehension of grammaticalized implicit meanings in SPCD and ASD children: A comparative study.

Autism spectrum disorder (ASD) and social pragmatic communication disorder (SPCD) are two neurodevelopmental disorders with many similarities in affected individuals' impairments in social-communicative and pragmatic development. A central question pertaining to their differentiation concerns whether the distinction is truly qualitative or, instead, quantitative in nature, and indeed, defining the boundary between SPCD and ASD with IQ in the normal range often presents differential-diagnostic difficulties. While deficits in the comprehension of certain linguistically systematic implicit verbal meanings have been targeted by experimental research in ASD, to date they have not been investigated in controlled experiments in SPCD. The empirical objectives of our study are twofold. First, it is explored whether the comprehension of a set of highly systematic, grammaticalized implicit meanings is impaired in ASD and SPCD children compared to their typically developing (TD) peers, and whether ASD and SPCD children differ from each other in accessing these verbal meanings. Second, it is investigated whether receptive grammatical competence and first-order ToM abilities are associated with children's performance in any way and whether there is a difference in this regard between the ASD and the SPCD group. Our main experiment, using a sentence-picture verification task, tested the comprehension of highly systematic implicit verbal meanings, including grammaticalized implicatures, presuppositions, and entailments. The experiment was complemented with a false-belief (ToM) task and a test of receptive grammar, among other measures. Seventy-one 4-to-9-year-old children participated in the study (ASD: n=19, SPCD: n=13, TD controls: n=39). While both children with SPCD and children with ASD performed significantly more poorly than the TD group, only the comprehension profile of the SPCD group differed significantly from that of the TD group. Importantly, while ASDs' performance exhibited an association with their ToM results, the performance of SPCDs showed a correlation with their receptive grammar skills. By contrast, the performance of TDs correlated with neither. These findings reveal potential divergences in the cognitive developmental mechanisms that underlie the semantic-pragmatic difficulties in the two clinical groups, suggesting that the communicative impairments in ASD and in SPCD differ qualitatively, rather than quantitatively. Specific implications for theories of pragmatic impairments in ASD and in SPCD are discussed. What is already known on the subject Linguistically systematic implicit meanings are understudied both in ASD and in SPCD. Within this domain of verbal meaning, the majority of relevant experimental work on ASD, concentrated on generalized (mostly: scalar) implicatures, has yielded somewhat divergent results, while the comprehension difficulties in SPCD have remained barely charted territory. Linguistically more highly conventionalized implicit verbal meanings have not been experimentally investigated in either neurodevelopmental disorder. What this study adds A primary finding of our study is that although both the SPCD and the ASD group show significant deficit in the comprehension of highly conventional, grammaticalized implicit meanings, SPCD children may diverge more in their comprehension profile from their TD peers than ASD children. Another key result is that the comprehension of grammaticalized implicit meanings is linked with different cognitive functions in ASD and in SPCD. While comprehension performance is associated with ToM in ASD but not in SPCD or in TD, it is correlated with receptive grammar skills in SPCD but not in ASD or in TD. Clinical implications of this study These findings provide potential support for the hypothesis that the difference between ASD and SPCD is qualitative rather than quantitative in nature, thereby casting doubt on the conception that pragmatic limitations in SPCD are to be approached as a less severe form of similar deficits in ASD. Uncovering differences in the underlying cognitive sources and in the comprehension deficits of children with ASD and SPCD are critical for the improvement of the accuracy of SPCD children's early diagnosis and timely therapeutic intervention.

Verifying Negative Sentences

In the long history of psycholinguistic research on verifying negative sentences, an often-reported finding is that participants take longer to correctly judge negative sentences true than false, while being faster to judge their positive counterparts true (e.g. Clark & Chase, Cogn Psychol 3(3):472−517, 1972; Carpenter & Just, Psychol Rev 82(1):45–73, 1975). While many linguists and psycholinguists have strongly advocated the idea that the costs and complexity of negation can be explained by appeal to context, context-based approaches have not been able to provide a satisfying account of this polarity*truth-value interaction. By contrast, the alternative theory of negation processing, which says that negation is processed by separately representing the positive, does provide a plausible account. Our proposals provide a means for reconciliation between the two views since we argue that negation is a strong cue to a positive context. Here we present our account of why and when negation is often apparently processed via the positive. We review many of the factors that are seen to be at play in sentence verification involving negation. We present evidence that participants’ adoption of the positive-first procedure in sentence-picture verification tasks is conditioned by context.

Adjectival polarity and the processing of scalar inferences

In a seminal study, Bott & Noveck (2004) found that the computation of the scalar inference of ‘some’ implying ‘not all’ was associated with increased sentence verification times, suggesting a processing cost. Recently, van Tiel and colleagues (2019b) hypothesised that the presence of this processing cost critically depends on the polarity of the scalar word. We comprehensively evaluated this polarity hypothesis on the basis of a sentence-picture verification task in which we tested the processing of 16 types of adjectival scalar inferences. We develop a quantitative measure of adjectival polarity which combines insights from linguistics and psychology. In line with the polarity hypothesis, our measure of polarity reliably predicted the presence or absence of a processing cost (i.e., an increase in sentence verification times). We conclude that the alleged processing cost for scalar inferencing in verification tasks is not due to the process of drawing a scalar inference, but rather to the cognitive difficulty of verifying negative information.

Electrophysiological correlates of the action vs. role relations congruencies in visually situated auditory sentence processing in Korean

ABSTRACT This study investigated brain responses to verbal action mismatches (e.g. The brother catches/*bites the sister) and role relation mismatches (e.g. The sister catches the brother ) in a picture-sentence verification task in Korean using event–related potentials (ERPs). EEG data collected from 30 native Korean speakers revealed a large N400 at the verb for the action mismatch, but an early negativity at the mismatched case-marker of the first NP as well as sustained negativity at the verb for the role mismatch. These results conform to the English-based data (i.e. Knoeferle et al., 2014. Different mechanisms for role relations versus verb–action congruence effects: Evidence from ERPs in picture–sentence verification. Acta Psychologica, 152, 133–148. https://doi.org/10.1016/j.actpsy.2014.08.004) in that the meaning of verb action and the thematic role relations involve different mechanisms in language comprehension. However, cross-lingustic differences were also found in terms of comprehenders’ relative sensitivity to different linguistic features in individual languages (e.g. word order in English and case-marking in Korean). Implications of the findings are discussed regrading predictive processing in language comprehension.