Word Prediction Research Articles

Word predictability blurs the lines between production and comprehension: Evidence from the production effect in memory

Predictions about likely upcoming input may promote rapid language processing, but the mechanisms by which such predictions are generated remain unclear. One hypothesis is that comprehenders use their production system to covertly produce what they would say if they were the speaker. If reading predictable words involves covert production, this act might have consequences for memory. The present study capitalized on the production effect, which is the observation that words read aloud are remembered better than words read silently. Participants read sentence-final predictable and unpredictable words aloud or silently, followed by a surprise recognition memory task. If reading predictable words involves covert production, the memory improvement from actually producing the words should be smaller for predictable words than for unpredictable words. This was confirmed in Experiment 1, which tested item memory using old/new judgments. Experiment 2 followed the same procedure, except that participants now made aloud/silent judgments probing their memory for prior acts of production. Here the hypothesis was that, relative to unpredictable words, it should be more difficult to remember whether predictable words had been read aloud or silently. Indeed, word predictability tended to make it harder to tell the difference, suggesting that predictability blurred the lines between production and comprehension. Taken together, the findings support the idea that reading predictable words can involve covert production and show that this act has consequences for what readers retain.

Anterior temporal lobe is necessary for efficient lateralised processing of spoken word identity

In the healthy human brain, the processing of language is strongly lateralised, usually to the left hemisphere, while the processing of complex non-linguistic sounds recruits brain regions bilaterally. Here we asked whether the anterior temporal lobes, strongly implicated in semantic processing, are critical to this special treatment of spoken words. Nine patients with semantic dementia (SD) and fourteen age-matched controls underwent magnetoencephalography and structural MRI. Voxel based morphometry demonstrated the stereotypical pattern of SD: severe grey matter loss restricted to the anterior temporal lobes, with the left side more affected. During magnetoencephalography, participants listened to word sets in which identity and meaning were ambiguous until word completion, for example PLAYED versus PLATE. Whereas left-hemispheric responses were similar across groups, patients demonstrated increased right hemisphere activity 174–294 msec after stimulus disambiguation. Source reconstructions confirmed recruitment of right-sided analogues of language regions in SD: atrophy of anterior temporal lobes was associated with increased activity in right temporal pole, middle temporal gyrus, inferior frontal gyrus and supramarginal gyrus. Overall, the results indicate that anterior temporal lobes are necessary for normal and efficient lateralised processing of word identity by the language network.

GateCap: Gated spatial and semantic attention model for image captioning

Visual attention has been widely used in deep image captioning models for its capacity of selectively aligning visual features to the corresponding words, i.e., the word-to-region alignment. In many cases, existing attention modules may not highlight task-related image regions for lack of high-level semantics. To advance captioning model, it is non-trivial for image captioning to effectively leverage high-level semantics. To defeat such issues, we propose a gated spatial and semantic attention captioning model (GateCap) which adaptively fuses spatial attention features with semantic attention features to achieve this goal. In particular, GateCap brings into two novel aspects: 1) spatial and semantic attention features are further enhanced via triple LSTMs in a divide-and-fuse learning manner, and 2) a context gate module is explored to reweigh spatial and semantic attention features in a fair manner. Benefitting from them, GateCap could reduce the side effect of the word-to-region misalignment at a time step over subsequent word prediction, thereby possibly alleviating emergence of incorrect words during testing. Experiments on MSCOCO dataset verify the efficacy of the proposed GateCap model in terms of quantitative and qualitative results.

Word predictability depends on parafoveal preview validity in Chinese reading

ABSTRACTResearch with alphabetic scripts shows that providing an invalid parafoveal preview eliminates or diminishes effects of contextual predictability on word identification, revealing that such effects depend on the interplay between top-down contextual expectations and bottom-up perceptual information. Whether similar effects are observed in character-based scripts like Chinese is unknown. However, such knowledge would extend our understanding of contextual prediction in different writing systems. Accordingly, we conducted an eye movement experiment using the boundary paradigm to assess contextual predictability effects on the processing of target words with valid and invalid parafoveal previews. Interactions between predictability and preview validity were observed in early reading times but not word-skipping for targets. This suggests an interplay between top-down and bottom-up processes drives contextual processing in Chinese reading, but that word-skipping is not strongly mediated by contextual expectations in this script. We consider these findings in relation to differences between alphabetic and non-alphabetic writing systems.

Natural Language Word Prediction Model Based on Multi-Window Convolution and Residual Network

In this paper, we propose MCNN-ReMGU model based on multi-window convolution and residual-connected minimal gated unit (MGU) network for the natural language word prediction. First, the convolution kernels with different sizes are used to extract the local feature information of different graininess between the word sequences. Then, the extracted features are fed to the residual-connected MGU network. Finally, the prediction results are output by the SoftMax layer. Through the residual-connection processing of MGU network in the model, not only the problems of vanishing gradient and network degradation are effectively solved, but also the long-term dependence between word sequences is effectively extracted to predict the next word accurately. Meanwhile, the introduction of the convolution kernel in a convolutional neural network (CNN) enables the feature information between word sequences to be extracted more fully. The experimental results on the Penn Treebank and WikiText-2 datasets show that the proposed method has certain advantages in the word prediction task.

Learning context-dependent word embeddings based on dependency parsing

Word embeddings constitute the basic methods of text representation. Whether they are the inputs to a machine learning algorithm or the features used in a natural language processing application, embeddings have proven helpful in solving various text processing tasks. In natural language texts, contextual information exerts a crucial influence on the semantics of word representations. In current research, most training models are based on shallow textual information and do not fully exploit deep relationships in sentences. To overcome this problem, this paper proposes the dependency-based continuous bag-of-words model which integrates the dependency relationships between words and sentences into the context with weights, thereby increasing the influence of specific contextual information on the prediction of target words. This method increases the abundancy of word context information and enhances the semantics of word embeddings. The experimental results show that the proposed method highlights semantic relations and improves the performance of word representations.

Learning context-dependent word embeddings based on dependency parsing

Word embeddings constitute the basic methods of text representation. Whether they are the inputs to a machine learning algorithm or the features used in a natural language processing application, embeddings have proven helpful in solving various text processing tasks. In natural language texts, contextual information exerts a crucial influence on the semantics of word representations. In current research, most training models are based on shallow textual information and do not fully exploit deep relationships in sentences. To overcome this problem, this paper proposes the dependency-based continuous bag-of-words model which integrates the dependency relationships between words and sentences into the context with weights, thereby increasing the influence of specific contextual information on the prediction of target words. This method increases the abundancy of word context information and enhances the semantics of word embeddings. The experimental results show that the proposed method highlights semantic relations and improves the performance of word representations.

Make It Directly: Event Extraction Based on Tree-LSTM and Bi-GRU

Event extraction is an important research direction in the field of natural language processing (NLP) applications including information retrieval (IR). Traditional event extraction is realized with two methods: the pipeline and the joint extraction methods. The pipeline method determines the event by triggering word recognition to further implement event extraction and is prone to error cascading. The joint extraction method applies deep learning to implement the completion of the trigger word and the argument role classification task. Most studies with the joint extraction method adopt the CNN or RNN network structure. However, in the case of event extraction, deeper understanding of complex contexts is required. Existing studies do not make full use of syntactic relations. This paper proposes a novel event extraction model, which is built upon a Tree-LSTM network and a Bi-GRU network and carries syntactically related information. It is illustrated that this method simultaneously uses Tree-LSTM and Bi-GRU to obtain a representation of the candidate event sentence and identify the event type, which results in a better performance compared to the ones that use chain structured LSTM, CNN or only Tree-LSTM. Finally, the hidden state of each node is used in Tree-LSTM to predict a label for candidate arguments and identify/classify all arguments of an event. Lab results show that the proposed event extraction model achieves competitive results compared to previous works.

Are Some Words Worth More than Others?

Current evaluation metrics for language modeling and generation rely heavily on the accuracy of predicted (or generated) words as compared to a reference ground truth. While important, token-level accuracy only captures one aspect of a language model's behavior, and ignores linguistic properties of words that may allow some mis-predicted tokens to be useful in practice. Furthermore, statistics directly tied to prediction accuracy (including perplexity) may be confounded by the Zipfian nature of written language, as the majority of the prediction attempts will occur with frequently-occurring types. A model's performance may vary greatly between high- and low-frequency words, which in practice could lead to failure modes such as repetitive and dull generated text being produced by a downstream consumer of a language model. To address this, we propose two new intrinsic evaluation measures within the framework of a simple word prediction task that are designed to give a more holistic picture of a language model's performance. We evaluate several commonly-used large English language models using our proposed metrics, and demonstrate that our approach reveals functional differences in performance between the models that are obscured by more traditional metrics.

The First Azeri (Azerbaijani) Language Next Word Predictor

Azeri (Azerbaijani) language is one of the more than 50 Turkic languages which it is a little studied language in terms of using the modern signal processing algorithms. This paper tackles the problem of Hidden Markov Models (HMMs) based next word prediction for this language based on Natural Language Processing (NLP) principles using Python high-level programming language. The software is included a small Azeri vocabulary database, the various Python libraries, a HMM model and a Web based interface. In this research, the database was constructed by a predictor parser which it was implemented for the first time for Azeri language. The database was concluded by the most general Azeri language words to introduce HMMs based generated word pairs. The Model was trained by 90% of the database, hence, predicting the next 5 words on the test data resulted 54% accuracy.

Spelling Mistake Tolerant Word Prediction for Working, Writing and Communicating

Spelling Mistake Tolerant Word Prediction for Working, Writing and Communicating

Dissociable effects of prediction and integration during language comprehension: evidence from a large-scale study using brain potentials.

Composing sentence meaning is easier for predictable words than for unpredictable words. Are predictable words genuinely predicted, or simply more plausible and therefore easier to integrate with sentence context? We addressed this persistent and fundamental question using data from a recent, large-scale (n = 334) replication study, by investigating the effects of word predictability and sentence plausibility on the N400, the brain's electrophysiological index of semantic processing. A spatio-temporally fine-grained mixed-effect multiple regression analysis revealed overlapping effects of predictability and plausibility on the N400, albeit with distinct spatio-temporal profiles. Our results challenge the view that the predictability-dependent N400 reflects the effects of either prediction or integration, and suggest that semantic facilitation of predictable words arises from a cascade of processes that activate and integrate word meaning with context into a sentence-level meaning. This article is part of the theme issue 'Towards mechanistic models of meaning composition'.

Word frequency, predictability, and return-sweep saccades: Towards the modeling of eye movements during paragraph reading.

Models of eye movement control during reading focus on the reading of single lines of text. Within these models, word frequency and predictability are important input variables which influence fixation probabilities and durations. However, a comprehensive model of eye movement control will have to account for readers' eye movements across multiline texts. Line-initial words are unlike those presented midline; they are routinely unavailable for parafoveal preprocessing. Therefore, it is unclear whether and how word frequency and predictability influence reading times on line-initial words. To address this, we present an analysis of the Provo Corpus (Luke & Christianson, 2018) followed by a novel eye-movement experiment. We conclude that word frequency and predictability impact single-fixation and gaze durations on line-initial words. We also observed that return-sweep error (undersweep-fixations) may, among several other possibilities, allow for parafoveal processing of line-initial words prior to their direct fixation. Implications for models of eye movement control during reading are discussed. (PsycINFO Database Record (c) 2019 APA, all rights reserved).

The Relation between Alpha/Beta Oscillations and the Encoding of Sentence induced Contextual Information

Pre-stimulus alpha (8–12 Hz) and beta (16–20 Hz) oscillations have been frequently linked to the prediction of upcoming sensory input. Do these frequency bands serve as a neural marker of linguistic prediction as well? We hypothesized that if pre-stimulus alpha and beta oscillations index language predictions, their power should monotonically relate to the degree of predictability of incoming words based on past context. We expected that the more predictable the last word of a sentence, the stronger the alpha and beta power modulation. To test this, we measured neural responses with magnetoencephalography of healthy individuals during exposure to a set of linguistically matched sentences featuring three levels of sentence context constraint (high, medium and low constraint). We observed fluctuations in alpha and beta power before last word onset, and modulations in M400 amplitude after last word onset. The M400 amplitude was monotonically related to the degree of context constraint, with a high constraining context resulting in the strongest amplitude decrease. In contrast, pre-stimulus alpha and beta power decreased more strongly for intermediate constraints, followed by high and low constraints. Therefore, unlike the M400, pre-stimulus alpha and beta dynamics were not indexing the degree of word predictability from sentence context.

Future-Aware Knowledge Distillation for Neural Machine Translation

Although future context is widely regarded useful for word prediction in machine translation, it is quite difficult in practice to incorporate it into neural machine translation. In this paper, we propose a future-aware knowledge distillation framework (FKD) to address this issue. In the FKD framework, we learn to distill future knowledge from a backward neural language model (teacher) to future-aware vectors (student) during the training phase. The future-aware vector for each word position is computed in a bridge network and optimized towards the corresponding hidden state in the backward neural language model via a knowledge distillation mechanism. We further propose an algorithm to jointly train the neural machine translation model, neural language model and knowledge distillation module end-to-end. The learned future-aware vectors are incorporated into the attention layer of the decoder to provide full-range context information during the decoding phase. Experiments on the NIST Chinese-English and WMT English-German translation tasks show that the proposed method significantly improves translation quality and word alignment.

Recurrent Neural Network based Models for Word Prediction

Globally, people are spending a cumulative amount of time on their mobile device, laptop, tab, desktop, etc,. for messaging, sending emails, banking, interaction through social media, and all other activities. It is necessary to cut down the time spend on typing through these devices. It can be achieved when the device can provide the user more options for what the next word might be for the current typed word. It also increases the speed of typing. In this paper, we suggest and presented a comparative study on various models like Recurrent Neural Network, Stacked Recurrent Neural Network, Long Short Term Memory network (LSTM) and Bi-directional LSTM that gives solution for the above said problem. Our primary goal is to suggest the best model among the four models to predict the next word for the given current word in English Language. Our survey says that for predicting next word RNN provide accuracy 60% and loss 40%, Stacked RNN provide accuracy 62% and loss 38%, LSTM provide accuracy 64% and loss 36% and Bidirectional LSTM provide accuracy 72% and loss 28%.

Slowly but surely: Interpreting facilitates L2 morphological anticipation based on suprasegmental and segmental information

AbstractNative speakers use suprasegmental information to predict words, but less is known about segmental information. Moreover, anticipatory studies with non-native speakers are scarce and mix proficiency with anticipatory experience. To address these limitations, we investigated whether Spanish monolinguals and advanced English learners of Spanish use suprasegmentals (stress: oxytone, paroxytone) and segmentals (syllabic structure: CVC, CV) to predict word suffixes, and whether increased anticipatory experience acquired via interpreting will facilitate anticipation in non-interpreting L2 situations. Eye-tracking data revealed that: (1) the three groups made use of the linguistic variables, and L2 groups did not anticipate in CV paroxytones; (2) everybody anticipated better with the less frequent conditions (oxytones, CVC) having fewer lexical competitors; (3) monolinguals anticipated earlier than L2 learners; and (4) interpreters anticipated at a faster rate in some conditions. These findings indicate that less frequent suprasegmental and segmental information and anticipatory experience facilitate native and non-native spoken word prediction.

Show, Tell, and Polish: Ruminant Decoding for Image Captioning

The encoder-decoder framework has been the base of popular image captioning models, which typically predicts the target sentence based on the encoded source image one word at a time in sequence. However, such a single-pass decoding framework encounters two problems. First, mistakes in the predicted words cannot be corrected and may propagate to the entire sentence. Second, because the single-pass decoder cannot access the following un-generated words, it can only perform local planning to choose every single word according to the preceding words, while lacks the global planning ability as for maintaining the semantic consistency and fluency of the whole sentence. In order to address the above two problems, in this work, we design a ruminant captioning framework which contains an image encoder, a base decoder, and a ruminant decoder. Specifically, the outputs of the former/base decoder are utilized as the global information to guide the words prediction of the latter/ruminant decoder, in an attempt to mimic human polishing process. We enable jointly training of the whole framework and overcome the non-differential problem of discrete words by designing a novel reinforcement learning based optimization algorithm. Experiments on two datasets (MS COCO and Flickr30 k) demonstrate that our ruminant decoding method can bring significant improvements over traditional single-pass decoding based models and achieves state-of-the-art performance.

Can’t forget: disruption of the right prefrontal cortex impairs voluntary forgetting in a recognition test

ABSTRACTThe causal influence of the dorsolateral prefrontal cortex (DLPFC) in voluntary forgetting remains unclear. Here, we employed repetitive transcranial magnetic stimulation (rTMS) over the right DLPFC to temporarily disrupt function of this brain region and examined its influence on an item-method directed forgetting (DF) task with both neutral and negative emotional memories. Participants were assigned to either an active or a sham rTMS group, in which we administered stimulation for 20 min before the DF task. We then examined the explicit and implicit DF effects with an explicit recognition and an implicit word completion test. We found that while participants in the sham group showed the classic DF effects in both explicit and implicit memory tests, temporally disrupting activity of the right DLPFC selectively reduced the DF effect on explicit recognition, but not on implicit word completion test. Our findings provide novel evidence that the right DLPFC plays a causal role in voluntary forgetting and support the direct inhibition account of voluntary memory control. Intriguingly, preserved implicit DF effects in the active stimulation group suggest that unintentional expressions of unwanted memories may be more sensitive to DF and less dependent on the right DLPFC.

Robust effects of predictability across experimental contexts: Evidence from event-related potentials

Many experiments have shown that comprehenders can generate predictions about upcoming inputs on the fly, but relatively little is known about whether and how comprehenders' sensitivity to predictability may be modulated by the experimental context. The present study used event-related potentials (ERPs) in two experiments to ask whether changing the overall predictive validity of the stimuli will affect comprehenders' brain responses to predictable as well as unpredictable words by manipulating the filler sentences, which made up 50% of the stimuli in each experiment. Contrary to the prediction that predictable words should be processed more easily and elicit a smaller N400 response in a more prediction-encouraging experimental context, we found that participants' N400 response to predictable as well as unpredictable words was smaller when the overall predictive validity of the stimuli was low (that is, when the filler items were incongruous compared to when they were predictable). Further, even though the use of different filler sentences did modulate comprehenders' ERP and behavioural responses, it did not modulate the effect of target word predictability on participants' ERP responses at all. We take the present findings to suggest that healthy young adults’ brain responses are inherently sensitive to the predictability of the incoming linguistic stimuli, and that this robust sensitivity can be observed regardless of the make-up of the experimental stimuli.