Judgment Experiment Research Articles

The grammaticality asymmetry in agreement attraction reflects response bias: Experimental and modeling evidence

Memory access mechanisms such as cue-based retrieval have come to dominate theories of the processing of linguistic dependencies such as subject-verb agreement. One phenomenon that has been regarded as demonstrating the role of such mechanisms is the grammaticality asymmetry in agreement attraction, which is the observation that nouns other than the grammatical controller of agreement can influence the computation of subject-verb agreement in ungrammatical, but not grammatical, sentences. This asymmetry is most often accounted for via the dynamics of retrieval interference. We challenge this interpretation, arguing that the asymmetry largely reflects response bias. Three forced-choice judgment experiments show that neutralizing response bias results in a decrease in the size of the grammaticality asymmetry, or its elimination altogether. Together with the response time patterns in these experiments, this result favors an account that attributes attraction effects to a continuous and equivocal representation of number, rather than to the dynamics of retrieval interference. We implement a model of grammaticality judgments that links a continuous representation of number to the rate of evidence accumulation in a diffusion process. This model accounts for the presence or absence of the grammaticality asymmetry through shifts in the decisional starting point (i.e. response bias), and highlights the importance of monitoring for response bias effects in judgment tasks.

On the Island Sensitivity of Topicalization in Norwegian: An Experimental Investigation

Mainland Scandinavian languages have been reported to allow movement from embedded questions, relative clauses, and complex NPs—domains commonly considered to be islands crosslinguistically. Yet in formal acceptability studies Scandinavian participants often show ‘island effects’: they reject island-violating movement similarly to native speakers of ‘island-sensitive’ languages. To investigate this apparent mismatch between informal and formal judgments, we conducted two acceptability judgment experiments testing the acceptability of topicalization from various island domains in Norwegian. We were interested in determining whether we could (i) find evidence for island insensitivity and (ii) pin down the source of qualitatively different island effects. We asked whether such effects are best explained as reflecting violations of a uniform syntactic constraint or extrasyntactic factors. Our results suggest that embedded questions and relative clauses are not uniform syntactic islands for topicalization, but complex NPs are. Unexpectedly, we also found evidence suggesting that conditional adjunct clauses may not be islands.

The Impact of Task Abandonment in Crowdsourcing

Crowdsourcing has become a standard methodology to collect manually annotated data such as relevance judgments at scale. On crowdsourcing platforms like Amazon MTurk or FigureEight, crowd workers select tasks to work on based on different dimensions such as task reward and requester reputation. Requesters then receive the judgments of workers who self-selected into the tasks and completed them successfully. Several crowd workers, however, preview tasks, begin working on them, reaching varying stages of task completion without finally submitting their work. Such behavior results in unrewarded effort which remains invisible to requesters. In this paper, we conduct an investigation of the phenomenon of task abandonment , the act of workers previewing or beginning a task and deciding not to complete it. We follow a three-fold methodology which includes 1) investigating the prevalence and causes of task abandonment by means of a survey over different crowdsourcing platforms, 2) data-driven analysis of logs collected during a large-scale relevance judgment experiment, and 3) controlled experiments measuring the effect of different dimensions on abandonment. Our results show that task abandonment is a widely spread phenomenon. Apart from accounting for a considerable amount of wasted human effort, this bears important implications on the hourly wages of workers as they are not rewarded for tasks that they do not complete. We also show how task abandonment may have strong implications on the use of collected data (for example, on the evaluation of Information Retrieval systems).

Perceptual Similarity between English and Korean: Evidence from Online Adaptation

Korean loanwords borrowed from English that end in voiceless stops frequently undergo vowel insertion. This vowel insertion is interesting since voiceless stops are permissible in final position in Korean native words. The perceptual similarity approach claims that this seemingly unnecessary vowel insertion is motivated by perceptual similarity between English and Korean forms. The current study reports on a perception experiment designed to examine whether Korean listeners actually judge English forms to be similar to Korean forms. In an AXB similarity judgment experiment, Korean learners of English listened to a triplet, consisting of an English stop-final form (CVC) and two Korean forms, one ending in a stop (CVC) and the other ending in stop-vowel (CVCV), and indicated which of the two Korean forms the English form sounded more similar to. The experimental result showed that Korean listeners were more likely to judge Korean CVCV as more similar to English CVC than Korean CVC when the English form ended in a released stop, and that final stress did not have a significant effect. This finding turns out to be consistent with the perceptual similarity approach, indicating that the English form could be acoustically similar to the Korean pronunciation.

Interpreting the ‘quality of law’ at the European Court of Human Rights: Metaphorical Framing and Evaluative Judgment

This paper looks at the use of metaphor and its effect on the interpretation of the ‘quality of law’ of contracting parties in Art. 8 cases of the European Court of Human Rights. This paper demonstrates the Court’s reliance on and reproduction of specific metaphorical frames - a finding consistent with the use of metaphor in judgment experiments in cognitive linguistics, but a first in studies of legal language. The Court employs metaphors conceptually coherent with those used in their cited precedent, in their representation of the successful pleadings within their judgments, and insists (implicitly) on different metaphors in dissent. This paper argues that rather than being simple judicial rhetoric, the use of congruent metaphors may be indicative of metaphor as a modulating factor in how judges reason. In the least, it is a significantly understudied phenomenon and this paper provides evidence for the saliency of its approach for understanding judicial reasoning. It approaches case law as data to understand the effects metaphorical framing has within them.

Units of Language Mixing: A Cross-Linguistic Perspective.

Language mixing is a ubiquitous phenomenon characterizing bilingual speakers. A frequent context where two languages are mixed is the word-internal level, demonstrating how tightly integrated the two grammars are in the mind of a speaker and how they adapt to each other. This raises the question of what the minimal unit of language mixing is, and whether or not this unit differs depending on what the languages are. Some scholars have argued that an uncategorized root serves as a unit, others argue that the unit needs to have been categorized prior to mixing. We will discuss the question of what the relevant unit for language mixing is by studying word-internal mixing in Cypriot Greek-English, English-Norwegian, Greek-English, Greek-German, and Spanish-German varieties that have been reported in the literature based on data from judgment experiments and spoken corpora. By understanding and modeling the units of language mixing across languages, we will gain insight into how languages adapt to each other word-internally, and what some possible outcomes of language contact are in the minds of speakers.

Articulation and Dynamics Influence the Perceptual Attack Time of Saxophone Sounds.

Perceptual attack time (PAT) is defined as the moment when the most salient rhythmical feature of a sound is perceived. This paper focuses on the PAT of saxophone sounds, investigating how the location of this point in time changes when a note is played with different characteristics. Nine saxophone sounds that differ in articulation and dynamics were examined. Ground truth for PAT was determined in a synchronization judgment experiment with 40 participants. Articulation (p < 0.001, η2 = 0.316), dynamics (p < 0.001, η2 = 0.098), and their interaction (p < 0.001, η2 = 0.094) affected the placement of the PAT. The onset rise time, which has been used as a predictor for PAT in earlier studies, was only weakly correlated with PAT (r = 0.143, p = 0.006).

Exhaustivity in single bare &lt;i&gt;wh&lt;/i&gt;-questions: A differential-analysis of exhaustivity

Despite a large body of research, the linguistic nature of exhaustivity in single wh-questions is unresolved. Moreover, little empirical evidence exists as to which related structures pattern with bare wh-questions regarding exhaustivity. This paper explores the felicity of various exhaustivity violations in unembedded single bare wh-questions in German and compares them to related structures. In two novel felicity judgment experiments, a total of 441 participants rated exhaustive as well as non-exhaustive plural and non-exhaustive singleton answers to wh-questions or statements in a questionnaire. Answers were based on picture stimuli depicting individuals performing various actions. The felicity of non-exhaustive answers was compared across four main test conditions: bare wh-questions (wer ‘who’), wh-questions with a lexical exhaustivity marker (wer alles ‘who all’), plural definite descriptions contained in a restrictive relative clause (e.g., “the people who are fishing in the garden”), and the scalar quantifier “some” (e.g., “some people who are fishing in the garden”).We employ a novel methodological approach to improve the interpretability of statistical differences between experimental conditions by using the statistical measure of Minimal Important Difference (MID). Our results from estimated MIDs reveal that adults’ felicity judgments of non-exhaustive plural answers to bare wh-questions pattern with those to wer alles-questions and to plural definite descriptions: exhaustivity violations in the bare wh, the wer alles and the plural definite conditions were rated as less felicitous than exhaustivity violations in the some-condition.

Topic situations: Coherence by inclusion

Topic situations have been studied in the linguistic literature but for the most part have not been studied psycholinguistically. Five experiments tested predictions of the hypothesis that a sentence-initial prepositional phrase (PP) in English introduces a Topic Situation, which by default restricts the interpretation of the following discourse. Participants in judgment experiments interpreted later discourse events as being more likely to take place in the location specified by a PP when that PP had appeared sentence-initially compared to other sentence positions, and they rated such sentences as less natural when the discourse event was implausible in the specified location. Participants in two additional experiments made naturalness judgments of sentences containing an initial PP that introduced a situation that has a usual range of durations. Sentences with a final temporal phrase that fell outside this range were judged to be unnatural, suggesting that this temporal phrase was (implausibly) interpreted as being included in the Topic Situation introduced by the PP. We suggest that these findings can advance understanding of discourse phenomena such as presupposition and domain restriction.

Two switches in the theory of counterfactuals

Based on a crowdsourced truth value judgment experiment, we provide empirical evidence challenging two classical views in semantics, and we develop a novel account of counterfactuals that combines ideas from inquisitive semantics and causal reasoning. First, we show that two truth-conditionally equivalent clauses can make different semantic contributions when embedded in a counterfactual antecedent. Assuming compositionality, this means that the meaning of these clauses is not fully determined by their truth conditions. This finding has a clear explanation in inquisitive semantics: truth-conditionally equivalent clauses may be associated with different propositional alternatives, each of which counts as a separate counterfactual assumption. Second, we show that our results contradict the common idea that the interpretation of a counterfactual involves minimizing change with respect to the actual state of affairs. We propose to replace the idea of minimal change by a distinction between foreground and background for a given counterfactual assumption: the background is held fixed in the counterfactual situation, while the foreground can be varied without any minimality constraint.

A processing-based account of subliminal wh-island effects

Wh-island effects are notorious for their cross-linguistic variation. However, experimental syntax studies observed super-additive wh-island effects in some languages which were previously argued to be immune to them. In four acceptability judgment experiments, we investigate the origin of super-additivity in acceptable wh-islands, so-called “subliminal island effects,” focusing on Hebrew. Our first experiment reveals a super-additive wh-island effect in Hebrew. However, we suggest that the super-additivity measure for wh-islands is contaminated by processing factors. Specifically, we propose that in this case, the decrease in acceptability reflects interference caused by the simultaneous maintenance of two dependencies, rather than grammatical islandhood. The following experiments demonstrate that super-additivity can be observed in binding structures, where it cannot be attributed to a violation of a grammatical constraint, but rather to processing costs related to interference for maintenance processes. We also show that when minimizing these costs, super-additivity does not emerge in Hebrew wh-islands. We conclude that processing costs underlie the apparent wh-island effect in Hebrew, and perhaps in additional languages, and that the super-additivity paradigm should be fine-tuned in order to avoid these confounds.

The nominative/accusative alternation in Japanese and information structure

In Japanese, desiderative (and potential) predicates derived from transitive verbs allow their direct object to be marked with the nominative marker ga, instead of the expected accusative marker o. This article argues that the nominative/accusative alternation in a desiderative construction has an information-structural implication. Nominative-marking on the object indicates its focushood, i.e., that it is either the focus of the utterance or part thereof, whereas accusative-marking has no such information-structural bearing. This claim is motivated by the observation that the direct object of a desiderative predicate resists nominative-marking when it is not adjacent to the predicate (“adjacency effect”). Under our account, the adjacency effect can be regarded as a variety of the garden-path effect, stemming from the discrepancy between the default (expected) locus of the focus and the pragmatic information conveyed by nominative-marking. With three sets of experimental data (from two acceptability judgment experiments with written stimuli and one rating experiment with auditory stimuli), we demonstrate that (i) the adjacency effect is real, and (ii) it can be mitigated by prosodic or contextual cues signaling the focushood of the object. The second finding conforms well to our hypothesis that the adjacency effect is a processing-based phenomenon, rather than a reflection of a purely syntactic constraint.

The Reference of Proper Names: Testing Usage and Intuitions.

Experiments on theories of reference have mostly tested referential intuitions. We think that experiments should rather be testing linguistic usage. Substantive Aim (I): to test classical description theories of proper names against usage by "elicited production." Our results count decisively against those theories. Methodological Aim (I): Machery, Olivola, and de Blanc () claim that truth-value judgment experiments test usage. Martí () disagrees. We argue that Machery etal. are right and offer some results that are consistent with that conclusion. Substantive Aim (II): Machery etal. provide evidence that the usage of a name varies, being sometimes descriptive, sometimes not. In seven out of eight tests of usage, we did not replicate this variation. Methodological Aim (II): to test the reliability of referential intuitions by comparing them with linguistic usage. Earlier studies led us to predict that we would find those intuitions unreliable, but we did not. Our results add to evidence that tests of referential intuition are susceptible to unpredictable wording effects.

Frisian strong and weak verbs in the face of Dutch influence: a synchronic and experimental approach

Like other Germanic languages, Frisian has both strong and weak verbal inflection. Despite a strong diachronic tendency for change towards weak inflection, strong inflection patterns are available synchronically to speakers to form the past tense and past participle of new or nonce verbs. Using a measure for ‘potential productivity’ developed by Knooihuizen and Strik (Folia Linguist Hist 35:173–200, 2014) for Dutch, we investigate the relative strength of available patterns in Frisian in an elicitation and an acceptability judgment experiment. Despite the multitude of different patterns in the strong verbal inflection system, strong inflection makes up 35% of the elicited nonce forms; these forms cannot all be explained by analogy. Analogically formed strong inflections of nonce verbs receive relatively high acceptability ratings at 4.2 on a 7-point scale. The elicitation experiment also produced many weak forms (12% of participles) that are not normatively possible with the -e infinitives in the elicitation prompt. These alternative weak forms were not included in the acceptability judgment experiment. We discuss the experimental results in the context of diachronically attested language change in Frisian and of intensive language contact with Dutch.

Object symmetry effects in Germanic

This paper focuses on passive symmetry effects in Germanic. We describe two large-sample judgment experiments with native speakers of Norwegian and Swedish, two partially symmetric passive languages. The results fail to support predictions of Anagnostopoulou’s (2003) seminal locality approach to passive symmetry in these languages. We propose that constraints on object ordering in these varieties are better modeled on a revised version of classic case-based theories. On this approach, patterns of object ordering are governed by variation in the way that case is assigned to objects. In addition, the Norwegian results suggest a shape conservation effect in object shift contexts not previously reported in the literature. Theme-recipient orders in Norwegian object shift contexts are available for just those speakers who also accept theme-recipient orders in active non-object shift contexts. This object ordering constraint applies in the same environment that another, much better described ordering constraint applies, namely Holmberg’s Generalization effects. We show that these results are explained by Fox and Pesetsky’s (2005) cyclic linearization algorithm together with the assumption that theme-recipient orders vP-internally reflect short theme-movement above the recipient.

Morphosyntactic form of Korean fragments is relevant to their resolution

We offer evidence for a structural identity constraint between a fragment and the structurally parallel position in the antecedent (which we term correspondent here). We ask if there is a preference for morphosyntactic match (generally in terms of syntactic category, but in terms of case marking in the Korean data discussed here) between a fragment and its correspondent. This question follows from the idea that in order to interpret fragments, the parser directly accesses content-addressable representations stored in memory, using as retrieval cues the linguistic information that fragments provide. We explore this preference using experimental data from Korean. In three acceptability judgment experiments, we demonstrate that (1) morphosyntactic match between fragments and correspondents is favored over mismatch, (2) the acceptability of mismatch is directional, favoring fragments that are morphosyntactically less complex than correspondents over the reverse, and (3) caseless fragments are degraded when paired with implicit correspondents compared to explicit ones.

Focus in heritage Hungarian

ABSTRACTBilinguals have been shown to differ from monolinguals especially in the realization and interpretation of phenomena that operate at the syntax/discourse interface. Hungarian has a well-known interface structure—identificational focus—which has been widely studied in the theoretical literature but never with bilinguals. The present article fills that gap, using three acceptability judgment experiments to test the realization and interpretation of identificational focus by heritage speakers and L1-dominant native speakers of Hungarian. The results reveal that both groups pattern similarly, showing some vulnerability at the syntax/discourse interface with regard to focus interpretation but less so with its realization. Further, the findings bear on the description of Hungarian focus in the theoretical literature, providing experimental evidence that focus is realized preverbally and that its interpretation is the result of a pragmatic implicature rather than part of the sentence’s semantics.

한국어 화자의 섬 구조 언어처리에서 나타나는 담화연결 효과에 관한 연구

Sungsoo Ok and Soo-Yeon Kim. 2017. Sensitivity to D-linking Effects in Island Construction Processing by Korean Speakers. Studies in Modern Grammar 96, 109-127. Questioning across-the-board effects of islands and D-linking in wh-interrogative construction, this study presents the results of two acceptability judgement experiments where judgements of Korean native speakers for Korean wh-interrogative constructions and for English wh-interrogative constructions are provided and compared. With respect to Korean wh-questions, native Korean speakers show different acceptability patterns depending on whether wh-phrases move overtly or not. When they are scrambled, presence of islands and the D-linking feature of wh-phrase affect acceptability in Korean as if they do in English: the former degrades the acceptability while there is an increase in acceptability status in the latter. As for in-situ cases, however, D-linking rather downgrades the acceptability of the sentences. In the second experiment on English wh-questions with Korean EFL learners, there are statistically significant, but comparatively marginal, effects of island constructions and D-linking on whinterrogative judgements. These results indicate that island effects and D-linking effects are not ubiquitous with the same degree but that they do vary depending on the language type, overtness of movement, and whether the tested language is L1 or L2.

Effects of Aging on Interference During Pronoun Resolution.

The purpose of this study is to investigate the effects of healthy aging on the ability to suppress grammatically illicit antecedents during pronoun resolution. In 2 reading-based acceptability-judgment experiments, younger and older speakers of German read sentences containing an object pronoun and 2 potential antecedent noun phrases, only 1 of which was a grammatically licit antecedent. Using a gender-mismatch paradigm, we compared to what extent younger and older speakers were sensitive to feature (mis)matches between the pronoun and either of the 2 antecedents. All participants were fluent readers of German and had finished at least secondary education. Experiment 1 used a self-paced reading paradigm. Older speakers showed greater sensitivity than younger ones to mismatching licit antecedents, but no group showed any evidence of interference from an intervening competitor antecedent. In Experiment 2, we increased the processing demand by using paced word-by-word stimulus presentation and longer sentences. Here, older participants showed reduced sensitivity, in comparison with younger people, to mismatching licit antecedents. Unlike our younger participants, they showed signs of distraction by the presence of a linearly closer but grammatically inappropriate antecedent when no appropriate antecedent was available. Together, our results show that older (but not younger) speakers' ability to compute intrasentential referential dependencies is vulnerable to increased task demands. We briefly discuss a potential role for executive functions, such as interference control.

Split intransitivity in Japanese is syntactic: Evidence for the Unaccusative Hypothesis from sentence acceptability and truth value judgment experiments

Split intransitivity (SI) is the generalization that intransitive verbs form two subclasses: the subjects of some intransitives behave like direct objects of transitive verbs, whereas the subjects of other intransitives behave like transitive subjects. The Unaccusative Hypothesis (UH) accounts for SI syntactically, arguing that the subjects of some intransitives (unaccusatives) are initially internal arguments, just like transitive objects, while the subjects of other intransitives (unergatives) are true subjects, just like transitive subjects. The UH has been enormously influential and inspired a rich and diverse literature on SI, including many recent experimental studies. Yet, no conclusive evidence for or against the UH has been put forward, largely because SI diagnostics (SIDs) – phenomena that motivate SI – are often amenable to both syntactic and non-syntactic analyses. In order to present unequivocal evidence for the UH, therefore, SIDs that make crucial reference to the UH must be carefully identified and studied. This study argues that two Japanese SIDs, quantifier scope and floating numeral quantifier licensing (FNQ-licensing), are such SIDs, as (i) they make crucial reference to the UH and the putative syntactic difference between unaccusative and unergative subjects, and (ii) they reliably classify Japanese intransitive verbs into two subclasses because of a syntactic difference. The evidence for this claim comes from the results of truth value judgment and sentence acceptability judgment experiments. The study argues that the results of these experiments show that Japanese intransitive verbs form two subclasses because of a syntactic difference, supporting the UH.