Lexical Probabilities Research Articles

A robust temporal map of speech monitoring from planning to articulation

Speakers continuously monitor their own speech to optimize fluent production, but the precise timing and underlying variables influencing speech monitoring remain insufficiently understood. Through two EEG experiments, this study aimed to provide a comprehensive temporal map of monitoring processes ranging from speech planning to articulation.In both experiments, participants were primed to switch the consonant onsets of target word pairs read aloud, eliciting speech errors of either lexical or articulatory-phonetic (AP) origin. Experiment I used pairs of the same stimuli words, creating lexical or non-lexical errors when switching initial consonants, with the degree of shared AP features not fully balanced but considered in the analysis. Experiment II followed a similar methodology but used different words in pairs for the lexical and non-lexical conditions, fully orthogonalizing the number of shared AP features.As error probability is higher in trials primed to result in lexical versus non-lexical errors and AP-close compared to AP-distant errors, more monitoring is required for these conditions. Similarly, error trials require more monitoring compared to correct trials. We used high versus low error probability on correct trials and errors versus correct trials as indices of monitoring.Across both experiments, we observed that on correct trials, lexical error probability effects were present during initial stages of speech planning, while AP error probability effects emerged during speech motor preparation. In contrast, error trials showed differences from correct utterances in both early and late speech motor preparation and during articulation. These findings suggest that (a) response conflict on ultimately correct trials does not persist during articulation; (b) the timecourse of response conflict is restricted to the time window during which a given linguistic level is task-relevant (early on for response appropriateness-related variables and later for articulation-relevant variables); and (c) monitoring during the response is primarily triggered by pre-response monitoring failure. These results support that monitoring in language production is temporally distributed and rely on multiple mechanisms.

Lexical Surprisal Shapes the Time Course of Syntactic Structure Building.

When we understand language, we recognize words and combine them into sentences. In this article, we explore the hypothesis that listeners use probabilistic information about words to build syntactic structure. Recent work has shown that lexical probability and syntactic structure both modulate the delta-band (<4 Hz) neural signal. Here, we investigated whether the neural encoding of syntactic structure changes as a function of the distributional properties of a word. To this end, we analyzed MEG data of 24 native speakers of Dutch who listened to three fairytales with a total duration of 49 min. Using temporal response functions and a cumulative model-comparison approach, we evaluated the contributions of syntactic and distributional features to the variance in the delta-band neural signal. This revealed that lexical surprisal values (a distributional feature), as well as bottom-up node counts (a syntactic feature) positively contributed to the model of the delta-band neural signal. Subsequently, we compared responses to the syntactic feature between words with high- and low-surprisal values. This revealed a delay in the response to the syntactic feature as a consequence of the surprisal value of the word: high-surprisal values were associated with a delayed response to the syntactic feature by 150-190 ms. The delay was not affected by word duration, and did not have a lexical origin. These findings suggest that the brain uses probabilistic information to infer syntactic structure, and highlight an importance for the role of time in this process.

The role of probability and duration in perception of speech sounds

When listeners are hearing sounds in the speech signal and have not yet heard enough acoustic cues to know what a sound is, they might predict the upcoming sound based on likelihood in the language or likelihood of occurrence after the preceding sound (overall and conditional probability of sounds). A related factor may be how long the sound goes on: listeners might be more able to identify sounds that go on longer because they have more time to hear the acoustic cues. However, additional duration may only be useful to listeners if it contains new acoustic cues (acoustic change). This paper investigates whether these factors have a role in how listeners perceive or predict sounds. This study presents further analysis of the perception data collected by the English Diphones project (Warner et al., 2014). The results show that listeners predict upcoming phonemes based on a guessing strategy, not based on lexical probability, with a small influence of syllable structure. Longer duration in itself does not produce better identification, but rather plays a role when important acoustic changes occur within the additional duration. The present study suggests that listeners use acoustic information rather than probability, and that they are sensitive to dynamic changes in spectral structure.

Phonotactically probable word shapes represent attractors in the cultural evolution of sound patterns

Abstract Words are processed more easily when they have canonical phonotactic shapes, i.e., shapes that are frequent both in the lexicon and in usage. We explore whether this cognitively grounded constraint or preference implies testable predictions about the implementation of sound change. Specifically, we hypothesise that words with canonical shapes favour, or ‘select for’, sound changes that (re-)produce words with the same shapes. To test this, we investigate a Middle English sound change known as Open Syllable Lengthening (OSL). OSL lengthened vowels in disyllables such as ME /ma.kə/ make, but more or less only when they became monosyllabic and when their vowels were non-high. We predict that word shapes produced by this implementation pattern should correspond to the shapes that were most common among morphologically simple monosyllables and disyllables at the time when OSL occurred. We test this prediction against Early Middle English corpus data. Our results largely confirm our prediction: monosyllables produced by OSL indeed conformed to the shapes that were most frequent among already existing monosyllables. At the same time, the failure of OSL to affect disyllables (such as body) prevented them from assuming shapes that were far more typical of morphologically complex word forms than of simple ones. This suggests that the actuation and implementation of sound changes may be even more sensitive to lexical probabilities than hitherto suspected. Also, it demonstrates how diachronic data can be used to test hypotheses about constraints on word recognition and processing.

Predictive Neural Computations Support Spoken Word Recognition: Evidence from MEG and Competitor Priming.

Human listeners achieve quick and effortless speech comprehension through computations of conditional probability using Bayes rule. However, the neural implementation of Bayesian perceptual inference remains unclear. Competitive-selection accounts (e.g., TRACE) propose that word recognition is achieved through direct inhibitory connections between units representing candidate words that share segments (e.g., hygiene and hijack share /haidʒ/). Manipulations that increase lexical uncertainty should increase neural responses associated with word recognition when words cannot be uniquely identified. In contrast, predictive-selection accounts (e.g., Predictive-Coding) propose that spoken word recognition involves comparing heard and predicted speech sounds and using prediction error to update lexical representations. Increased lexical uncertainty in words, such as hygiene and hijack, will increase prediction error and hence neural activity only at later time points when different segments are predicted. We collected MEG data from male and female listeners to test these two Bayesian mechanisms and used a competitor priming manipulation to change the prior probability of specific words. Lexical decision responses showed delayed recognition of target words (hygiene) following presentation of a neighboring prime word (hijack) several minutes earlier. However, this effect was not observed with pseudoword primes (higent) or targets (hijure). Crucially, MEG responses in the STG showed greater neural responses for word-primed words after the point at which they were uniquely identified (after /haidʒ/ in hygiene) but not before while similar changes were again absent for pseudowords. These findings are consistent with accounts of spoken word recognition in which neural computations of prediction error play a central role.SIGNIFICANCE STATEMENT Effective speech perception is critical to daily life and involves computations that combine speech signals with prior knowledge of spoken words (i.e., Bayesian perceptual inference). This study specifies the neural mechanisms that support spoken word recognition by testing two distinct implementations of Bayes perceptual inference. Most established theories propose direct competition between lexical units such that inhibition of irrelevant candidates leads to selection of critical words. Our results instead support predictive-selection theories (e.g., Predictive-Coding): by comparing heard and predicted speech sounds, neural computations of prediction error can help listeners continuously update lexical probabilities, allowing for more rapid word identification.

Morphosyntactic but not lexical corpus-based probabilities can substitute for cloze probabilities in reading experiments.

During reading or listening, people can generate predictions about the lexical and morphosyntactic properties of upcoming input based on available context. Psycholinguistic experiments that study predictability or control for it conventionally rely on a human-based approach and estimate predictability via the cloze task. Our study investigated an alternative corpus-based approach for estimating predictability via language predictability models. We obtained cloze and corpus-based probabilities for all words in 144 Russian sentences, correlated the two measures, and found a strong correlation between them. Importantly, we estimated how much variance in eye movements registered while reading the same sentences was explained by each of the two probabilities and whether the two probabilities explain the same variance. Along with lexical predictability (the activation of a particular word form), we analyzed morphosyntactic predictability (the activation of morphological features of words) and its effect on reading times over and above lexical predictability. We found that for predicting reading times, cloze and corpus-based measures of both lexical and morphosyntactic predictability explained the same amount of variance. However, cloze and corpus-based lexical probabilities both independently contributed to a better model fit, whereas for morphosyntactic probabilities, the contributions of cloze and corpus-based measures were interchangeable. Therefore, morphosyntactic but not lexical corpus-based probabilities can substitute for cloze probabilities in reading experiments. Our results also indicate that in languages with rich inflectional morphology, such as Russian, when people engage in prediction, they are much more successful in predicting isolated morphosyntactic features than predicting the particular lexeme and its full morphosyntactic markup.

A Lexical Dependency Probability Model for Mongolian Based On Integration of Morphological and Syntactic Features

Syntactic parsing is an important topic in the field of Mongolian language information processing. Compared with English and Chinese dependency parsing, Mongolian dependency parsing is still at the beginning stage. Mongolian syntactic parsing lack of Treebank resources seriously, under such conditions, a high quality syntactic parser cannot be developed by statistical methods simply. Aiming at the characteristics that Mongolian language has rich morphological features, this paper presented a rule and statistics-based dependency parsing model using Mongolian Dependency Treebank as training and evaluation data. The morphological and syntactic rules are represented using complex features and unification operations. The statistical model is represented using lexical dependent probability. This model has now achieved accuracies of 77.18%, 69.42% and 95.44% for the unlabelled annotation score, the labeled annotation score and the head word annotation score respectively.

Towards Machine Translation in Semantic Vector Space

Measuring the quality of the translation rules and their composition is an essential issue in the conventional statistical machine translation (SMT) framework. To express the translation quality, the previous lexical and phrasal probabilities are calculated only according to the co-occurrence statistics in the bilingual corpus and may be not reliable due to the data sparseness problem. To address this issue, we propose measuring the quality of the translation rules and their composition in the semantic vector embedding space (VES). We present a recursive neural network (RNN)-based translation framework, which includes two submodels. One is the bilingually-constrained recursive auto-encoder, which is proposed to convert the lexical translation rules into compact real-valued vectors in the semantic VES. The other is a type-dependent recursive neural network, which is proposed to perform the decoding process by minimizing the semantic gap (meaning distance) between the source language string and its translation candidates at each state in a bottom-up structure. The RNN-based translation model is trained using a max-margin objective function that maximizes the margin between the reference translation and the n-best translations in forced decoding. In the experiments, we first show that the proposed vector representations for the translation rules are very reliable for application in translation modeling. We further show that the proposed type-dependent, RNN-based model can significantly improve the translation quality in the large-scale, end-to-end Chinese-to-English translation evaluation.

Information Density and Syntactic Repetition.

In noun phrase (NP) coordinate constructions (e.g., NP and NP), there is a strong tendency for the syntactic structure of the second conjunct to match that of the first; the second conjunct in such constructions is therefore low in syntactic information. The theory of uniform information density predicts that low-information syntactic constructions will be counterbalanced by high information in other aspects of that part of the sentence, and high-information constructions will be counterbalanced by other low-information components. Three predictions follow: (a) lexical probabilities (measured by N-gram probabilities and head-dependent probabilities) will be lower in second conjuncts than first conjuncts; (b) lexical probabilities will be lower in matching second conjuncts (those whose syntactic expansions match the first conjunct) than nonmatching ones; and (c) syntactic repetition should be especially common for low-frequency NP expansions. Corpus analysis provides support for all three of these predictions.

Locally Non-Linear Learning for Statistical Machine Translation via Discretization and Structured Regularization

Linear models, which support efficient learning and inference, are the workhorses of statistical machine translation; however, linear decision rules are less attractive from a modeling perspective. In this work, we introduce a technique for learning arbitrary, rule-local, non-linear feature transforms that improve model expressivity, but do not sacrifice the efficient inference and learning associated with linear models. To demonstrate the value of our technique, we discard the customary log transform of lexical probabilities and drop the phrasal translation probability in favor of raw counts. We observe that our algorithm learns a variation of a log transform that leads to better translation quality compared to the explicit log transform. We conclude that non-linear responses play an important role in SMT, an observation that we hope will inform the efforts of feature engineers.

Lexicon expansion for latent variable grammars

This study investigates the use of unlabeled data, i.e., raw texts, to strengthen latent variable probabilistic context-free grammars, in particular lexical models. A graph-based lexicon expansion approach is proposed to achieve this goal. It aims to discover additional lexical knowledge from a large amount of unlabeled data to help the syntax parsing. The proposed approach is based on a transductive graph-based label propagation technique. The approach builds k-nearest-neighbor (k-NN) similarity graphs over the words of labeled and unlabeled data, for propagating lexical emission probabilities. The intuition is that different word under similar syntactic environment should have approximate lexical emission distributions. The derived words, together with lexical emission probabilities, are incorporated into the parsing. This approach is very effective in parsing out-of-vocabulary (OOV) words. Empirical results for English, Chinese, and Portuguese revealed its effectiveness.

Chinese unknown word recognition for PCFG-LA parsing.

This paper investigates the recognition of unknown words in Chinese parsing. Two methods are proposed to handle this problem. One is the modification of a character-based model. We model the emission probability of an unknown word using the first and last characters in the word. It aims to reduce the POS tag ambiguities of unknown words to improve the parsing performance. In addition, a novel method, using graph-based semisupervised learning (SSL), is proposed to improve the syntax parsing of unknown words. Its goal is to discover additional lexical knowledge from a large amount of unlabeled data to help the syntax parsing. The method is mainly to propagate lexical emission probabilities to unknown words by building the similarity graphs over the words of labeled and unlabeled data. The derived distributions are incorporated into the parsing process. The proposed methods are effective in dealing with the unknown words to improve the parsing. Empirical results for Penn Chinese Treebank and TCT Treebank revealed its effectiveness.

A Context-Theoretic Framework for Compositionality in Distributional Semantics

Formalizing “meaning as context” mathematically leads to a new, algebraic theory of meaning, in which composition is bilinear and associative. These properties are shared by other methods that have been proposed in the literature, including the tensor product, vector addition, point-wise multiplication, and matrix multiplication. Entailment can be represented by a vector lattice ordering, inspired by a strengthened form of the distributional hypothesis, and a degree of entailment is defined in the form of a conditional probability. Approaches to the task of recognizing textual entailment, including the use of subsequence matching, lexical entailment probability, and latent Dirichlet allocation, can be described within our framework.

Parts of Speech Tagging for Afaan Oromo

The main aim of this study is to develop part-of-speech tagger for Afaan Oromo language. After reviewing literatures on Afaan Oromo grammars and identifying tagset and word categories, the study adopted Hidden Markov Model (HMM) approach and has implemented unigram and bigram models of Viterbi algorithm. Unigram model is used to understand word ambiguity in the language, while bigram model is used to undertake contextual analysis of words. For training and testing purpose 159 sentences (with a total of 1621 words) that are manually annotated sample corpus are used. The corpus is collected from different public Afaan Oromo newspapers and bulletins to make the sample corpus balanced. A database of lexical probabilities and transitional probabilities are developed from the annotated corpus. These two probabilities are from which the tagger learn and tag sequence of words in sentences. The performance of the prototype, Afaan Oromo tagger is tested using tenfold cross validation mechanism. The result shows that in both unigram and bigram models 87.58% and 91.97% accuracy is obtained, respectively.

Sura Length and Lexical Probability Estimation in Cluster Analysis of the Qur’an

Thabet [2005] applied cluster analysis to the Qur’an in the hope of generating a classification of the (suras) that is useful for understanding of its thematic structure. The result was positive, but variation in (sura) length was a problem because clustering of the shorter was found to be unreliable. The present discussion addresses this problem in four parts. The first part summarizes Thabet’s work. The second part argues that unreliable clustering of the shorter is a consequence of poor estimation of lexical population probabilities in those. The third part proposes a solution to the problem based on calculation of a minimum length threshold using concepts from statistical sampling theory followed by selection of and lexical variables based on that threshold. The fourth part applies the proposed solution to a reanalysis of the Qur’an.

Can CV intelligibility predict speech intelligibility

This paper begins by reviewing the speech perception literature to predict cues that would and would not be expected to survive energetic masking of various types. The focus is especially (but not exclusively) on spectrotemporal cues to stops in the vicinity of the segment boundary in CV syllables. The second part of the paper discusses influences that can restrict the generality of research findings from isolated CV syllables. This includes ways in which CV syllables change in different phonetic contexts and styles of speech, contributions of the visual modality, and other uses of top‐down information, such as phonotactic, lexical, semantic and syntactic probability. The paper concludes by asking whether‐and how‐simple measures such as CV intelligibility can be used to reflect intelligibility of speech in real‐life communicative situations.

Part-of-speech tagging of Modern Hebrew text

AbstractWords in Semitic texts often consist of a concatenation ofword segments, each corresponding to a part-of-speech (POS) category. Semitic words may be ambiguous with regard to their segmentation as well as to the POS tags assigned to each segment. When designing POS taggers for Semitic languages, a major architectural decision concerns the choice of the atomic input tokens (terminal symbols). If the tokenization is at the word level, the output tags must be complex, and represent both the segmentation of the word and the POS tag assigned to each word segment. If the tokenization is at the segment level, the input itself must encode the different alternative segmentations of the words, while the output consists of standard POS tags. Comparing these two alternatives is not trivial, as the choice between them may have global effects on the grammatical model. Moreover, intermediate levels of tokenization between these two extremes are conceivable, and, as we aim to show, beneficial. To the best of our knowledge, the problem of tokenization for POS tagging of Semitic languages has not been addressed before in full generality. In this paper, we study this problem for the purpose of POS tagging of Modern Hebrew texts. After extensive error analysis of the two simple tokenization models, we propose a novel, linguistically motivated, intermediate tokenization model that gives better performance for Hebrew over the two initial architectures. Our study is based on the well-known hidden Markov models (HMMs). We start out from a manually devised morphological analyzer and a very small annotated corpus, and describe how to adapt an HMM-based POS tagger for both tokenization architectures. We present an effective technique for smoothing the lexical probabilities using an untagged corpus, and a novel transformation for casting the segment-level tagger in terms of a standard, word-level HMM implementation. The results obtained using our model are on par with the best published results on Modern Standard Arabic, despite the much smaller annotated corpus available for Modern Hebrew.

Second mention reduction in Indian, English, and Korean

Phonetic reduction of the second relative to the first mention of a word in a discourse (Second Mention Reduction, or SMR) is a well-documented feature of spoken American English (AE). This phenomenon could be mediated by discourse-level prosodic structure: second mentions are reduced because they are less likely than first mentions to be accented in AE; or it could occur independently of prosodic structure. Impressionistic studies indicate that Indian English (IE) speakers do not deaccent second mentions like AE speakers [J. Gumperz, Discourse Strategies (1982)]. Korean marks information structure through phrasing and word order rather than pitch accents. Therefore, SMR in these languages would support a direct link between lexical probability and duration. Six IE speakers read five paragraphs containing 59 repeated words, and five Korean speakers read a Korean translation of one of these paragraphs. IE speakers were significantly less likely to deaccent second mentions than AE speakers (sign-test, p&amp;lt;.05), and four IE speakers had identical accenting rates for both mentions, providing empirical evidence that IE speakers do not deaccent second mentions. Moreover, significant SMR was found in both Korean (W=43, p&amp;lt;.05) and IE (W=1340, p&amp;lt;.001), suggesting that probabilistic effects on word duration are not mediated by prosodic prominence.

Modeling the word recognition data of Vitevitch and Luce (1998): Is it ARTful?

Vitevitch and Luce (1998) showed that the probability with which phonemes co-occur in the language (phonotactic probability) affects the speed with which words and nonwords are named. Words with high phonotactic probabilities between phonemes were named more slowly than words with low probabilities, whereas with nonwords, just the opposite was found. To reproduce this reversal in performance, a model would seem to require not merely sublexical representations, but sublexical representations that are relatively independent of lexical representations. ARTphone (Grossberg, Boardman, & Cohen, 1997) is designed to meet these requirements. In this study, we used a technique called parameter space partitioning to analyze ARTphone's behavior and to learn if it can mimic human behavior and, if so, to understand how. To perform best, differences in sublexical node probabilities must be amplified relative to lexical node probabilities to offset the additional source of inhibition (from top-down masking) that is found at the sublexical level.

Early semantic context integration and lexical access as revealed by event-related brain potentials

A late neurophysiological response, the N400, is well known to reflect word processing and semantic context integration. Nevertheless, behavioural and neurophysiological data have demonstrated that word features and semantic sentence context influence linguistic processes already within the first 200 ms following the onset of the critical word stimulus. The lack of early word-related ERP effects in some studies might be due to large stimulus variance on relevant linguistic parameters, for example the length of written words or their frequency of usage. Here we investigated the effect of the length of words, their standardized lexical frequency and probability to occur in a given semantic context, by orthogonally varying these factors in a sentence-reading task. We found word frequency and probability to be reflected neurophysiologically already at 120 and 180 ms after written word onset. However, at these early stages frequency and probability effects were modulated by word length, as documented by significant interactions, whereas at later stages, around 300–500 ms, additive effects of these variables were found. These results indicate that semantic context integration may take place at a surprisingly early stage and near-simultaneously with the processing of information about the form of a word and its lexical properties, therefore challenging serial models of psycholinguistic information access. A methodological implication of this study is that, in order to obtain early psycholinguistic ERP effects, stimulus variance must be kept to a minimum.