Word Prediction Research Articles

The Entropy of Morphological Systems in Natural Languages Is Modulated by Functional and Semantic Properties

ABSTRACT In most natural languages, grammatical gender and number features encode semantic attributes concerning animacy, sex, and numerosity. Despite the likely advantage of promptly communicating about such salient attributes, inflectional systems rarely display consistently bijective correspondences between the semantic attributes and the grammatical feature values. In a study on Italian, we explored how this apparently noisy encoding depends on a trade-off between the semantic and the functional aspects of grammatical features. Using entropy metrics, we assessed the primarily functional purpose of gender and number features in the lexicon, observing a distribution of nouns that can optimally serve agreement-based parsing and prediction of words in sentences. A novel context entropy measure, introduced in this study to assess meaning specificity, revealed a semantic underspecification in masculine and singular nouns denoting animate referents. We argue that underspecification is the hallmark of the particular type of information compression occurring in inflectional systems. In binary inflectional systems, one value specifically encodes a semantic attribute, while the other value does not encode any semantic information, and surfaces as a default for functional purposes. By providing an information-theoretical account of the role of grammatical features, we set the basis for a scientifically informed pursue of language inclusiveness.

Odor-based context-dependent memory: influence of olfactory cues on declarative and nondeclarative memory indices.

Reinstating the olfactory learning context can increase access to memory information, but it is not fully clear which memory functions are subject to an enhancing odor context reinstatement effect. Here, we tested whether congruent odor context during encoding and recall positively affected declarative and nondeclarative memory scores using a novel method for manipulation of an odorous environment; namely, intranasal Nosa plugs. Recall of a text and a complex figure as well as performance in a priming task were assessed immediately and 1 wk after encoding. We found that congruent odor exposure at encoding and recall aided free retrieval of a story at delayed testing but had no significant effect on a complex figure recall or a word completion task. Differences between the assessed memory indices suggest that olfactory environmental cues may be primarily efficient in free verbal recall tasks.

Semantic Relation Extraction from Cultural Heritage Archives

Digital preservation technologies are now being increasingly adopted by cultural heritage organizations. This cultural heritage data is often disseminated in the form of digital text through a variety of channels such as Wikipedia, cultural heritage archives, etc. To acquire knowledge from digital data, the extraction technique becomes an important part. However, in the case of digital text, which has characteristics such as ambiguity, complex grammar structures such as the Thai language, and others, it makes it more challenging to extract information with a high level of accuracy. We thus propose a method for improving the performance of data extraction techniques based on word features, multiple instance learning, and unseen word mapping. Word features are used to improve the quality of word definition by concatenating parts of speech (POS) and word position is used to establish the accurate definition of a word and convert all of this into a vector. In addition, we use multiple instance learning to solve issues where words do not fully express the meaning of the triple. We also cluster the particular word to find the predicate word by removing words that are irrelevant between the subject and the object. The difficulty of having a new set of words that have never been trained before can be overcome by using unseen word mapping with sub-word and nearest neighbor word mapping. We conducted several experiments on a cultural heritage knowledge graph to show the efficacy of the proposed method. The results demonstrated that our proposed technique outperforms existing models currently utilized in relation to extraction systems. It can achieve excellent accuracy since its precision, recall, and F1 score are 0.89, 0.88, and 0.89, respectively. Furthermore, it also performed well in terms of unseen word prediction, precision, recall, and F1 score, which were 0.81, 0.87, and 0.84, respectively.

BERT-BU12 Hate Speech Detection using Bidirectional Encoder-Decoder

In the recent times transfer learning models have known to exhibited good results in the area of text classification for question-answering, summarization, next word prediction but these learning models have not been extensively used for the problem of hate speech detection yet. We anticipate that these networks may give better results in another task of text classification i.e. hate speech detection. This paper introduces a novel method of hate speech detection based on the concept of attention networks using the BERT attention model. We have conducted exhaustive experiments and evaluation over publicly available datasets using various evaluation metrics (precision, recall and F1 score). We show that our model outperforms all the state-of-the-art methods by almost 4%. We have also discussed in detail the technical challenges faced during the implementation of the proposed model.

Image Visual Attention Mechanism-based Global and Local Semantic Information Fusion for Multi-modal English Machine Translation

<p>Machine translation is a hot research topic at present. Traditional machine translation methods are not effective because they require a large number of training samples. Image visual semantic information can improve the effect of the text machine translation model. Most of the existing works fuse the whole image visual semantic information into the translation model, but the image may contain different semantic objects. These different local semantic objects have different effects on the words prediction of the decoder. Therefore, this paper proposes a multi-modal machine translation model based on the image visual attention mechanism via global and local semantic information fusion. The global semantic information in the image and the local semantic information are fused into the text attention weight as the image attention. Thus, the alignment information between the hidden state of the decoder and the text of the source language is further enhanced. Experimental results on the English-German translation pair and the Indonesian-Chinese translation pair on the Multi30K dataset show that the proposed model has a better performance than the state-of-the-art multi-modal machine translation models, the BLEU values of English-German translation results and Indonesian-Chinese translation results exceed 43% and 29%, which proves the effectiveness of the proposed model.</p> <p> </p>

A Study on Chinese-English Machine Translation Based on Transfer Learning and Neural Networks

The existing Chinese-English machine translation has problems such as inaccurate word translation and difficult translation of long sentences. To this end, this paper proposes a new machine translation model based on bidirectional Chinese-English translation incorporating translation knowledge and transfer learning, and the components of this model include a recurrent neural network-based translation quality assessment model and a self-focused network-based model. The experimental results demonstrate that our method works better on the dataset of machine translation quality assessment task for Chinese-English translation with more information, and the Pearson correlation coefficient of its quality assessment feature vector (such as word prediction vector representation) is higher.

Are older adults more risky readers? Evidence from meta-analysis.

According to an influential account of aging effects on reading, older adults (65+ years) employ a more “risky” reading strategy compared to young adults (18–30 years), in which they attempt to compensate for slower processing by using lexical and contextual knowledge to guess upcoming (i.e., parafoveal) words more often. Consequently, while older adults may read more slowly, they might also skip words more often (by moving their gaze past words without fixating them), especially when these are of higher lexical frequency or more predictable from context. However, this characterization of aging effects on reading has been challenged recently following several failures to replicate key aspects of the risky reading hypothesis, as well as evidence that key effects predicted by the hypothesis are not observed in Chinese reading. To resolve this controversy, we conducted a meta-analysis of 102 eye movement experiments comparing the reading performance of young and older adults. We focused on the reading of sentences displayed normally (i.e., without unusual formatting or structures, or use of gaze-contingent display-change techniques), conducted using an alphabetic script or Chinese, and including experiments manipulating the frequency or predictability of a specific target word. Meta-analysis confirmed that slower reading by older compared to younger adults is accompanied by increased word-skipping, although only for alphabetic scripts. Meta-analysis additionally showed that word-skipping probabilities are unaffected by age differences in word frequency or predictability effects, casting doubt on a central component of the risky reading hypothesis. We consider implications for future research on aging effects on reading.

Multimodal Lip-Reading for Tracheostomy Patients in the Greek Language

Voice loss constitutes a crucial disorder which is highly associated with social isolation. The use of multimodal information sources, such as, audiovisual information, is crucial since it can lead to the development of straightforward personalized word prediction models which can reproduce the patient’s original voice. In this work we designed a multimodal approach based on audiovisual information from patients before loss-of-voice to develop a system for automated lip-reading in the Greek language. Data pre-processing methods, such as, lip-segmentation and frame-level sampling techniques were used to enhance the quality of the imaging data. Audio information was incorporated in the model to automatically annotate sets of frames as words. Recurrent neural networks were trained on four different video recordings to develop a robust word prediction model. The model was able to correctly identify test words in different time frames with 95% accuracy. To our knowledge, this is the first word prediction model that is trained to recognize words from video recordings in the Greek language.

With or against others? Pay-for-Performance activates aggressive aspects of competitiveness

ABSTRACT While paying employees for performance (PfP) has been shown to elicit increased motivation by way of competitive processes, the present paper investigates whether the same competitive processes inherent in PfP can also encourage aggressiveness. We tested our hypothesis in three studies that conceptually build on each other: First, in a word completion experiment (N = 104), we find that PfP triggers the implicit activation of the fighting and defeating facets of competitiveness. Second, in a multi-source field study (N = 94), co-workers reported more interpersonal deviance from colleagues when the latter received a performance bonus than when they did not. In our final field study (N = 286), we tested the full model, assessing the effect of PfP and interpersonal deviance mediated by competitiveness: Employees with a bonus self-reported higher interpersonal deviance towards their co-workers, which was mediated by individual competitiveness. These findings underscore that PfP can entail powerful yet widely unstudied collateral effects.

A Rapid Combined Model for Automatic Generating Web UI Codes

Encoder-Decoder network is usually applied to image caption to automatically generate descriptive text for a picture. Web user interface (Web UI) is a special type of image and is usually described by HTML (hypertext marked language). Consequently, it becomes possible to use the encoder-decoder network to generate the corresponding code from a screenshot of Web UI. The basic structure of the decoder is RNN, LSTM, GRU, or other recurrent neural networks. However, this kind of decoder needs a long training time, so it increases the time complexity of training and prediction. The HTML language is a typically structured language to describe the Web UI, but it is hard to express the timing characteristics of the word sequence and the complex context. To resolve these problems efficiently, a rapid combined model RCM (rapid combined model) is designed in this paper. The basic structure of the RCM is an encoder-decoder network. The word embedding matrix and visual model are included in the encoder. The word embedding matrix uses fully connected units. Compared with LSTM, the accuracy of the word embedding matrix is basically unchanged, but the training and prediction speed have been significantly improved. In the visual model, the pretrained InceptionV3 network is used to generate the image vector, which not only improves the quality of the recognition of the Web UI interface image but also reduces the training time of the RCM significantly. In the decoder, the word embedding vector and the image vector are integrated together and input into the prediction model for word prediction.

Language Models Explain Word Reading Times Better Than Empirical Predictability.

Though there is a strong consensus that word length and frequency are the most important single-word features determining visual-orthographic access to the mental lexicon, there is less agreement as how to best capture syntactic and semantic factors. The traditional approach in cognitive reading research assumes that word predictability from sentence context is best captured by cloze completion probability (CCP) derived from human performance data. We review recent research suggesting that probabilistic language models provide deeper explanations for syntactic and semantic effects than CCP. Then we compare CCP with three probabilistic language models for predicting word viewing times in an English and a German eye tracking sample: (1) Symbolic n-gram models consolidate syntactic and semantic short-range relations by computing the probability of a word to occur, given two preceding words. (2) Topic models rely on subsymbolic representations to capture long-range semantic similarity by word co-occurrence counts in documents. (3) In recurrent neural networks (RNNs), the subsymbolic units are trained to predict the next word, given all preceding words in the sentences. To examine lexical retrieval, these models were used to predict single fixation durations and gaze durations to capture rapidly successful and standard lexical access, and total viewing time to capture late semantic integration. The linear item-level analyses showed greater correlations of all language models with all eye-movement measures than CCP. Then we examined non-linear relations between the different types of predictability and the reading times using generalized additive models. N-gram and RNN probabilities of the present word more consistently predicted reading performance compared with topic models or CCP. For the effects of last-word probability on current-word viewing times, we obtained the best results with n-gram models. Such count-based models seem to best capture short-range access that is still underway when the eyes move on to the subsequent word. The prediction-trained RNN models, in contrast, better predicted early preprocessing of the next word. In sum, our results demonstrate that the different language models account for differential cognitive processes during reading. We discuss these algorithmically concrete blueprints of lexical consolidation as theoretically deep explanations for human reading.

The learnability consequences of Zipfian distributions in language

While the languages of the world differ in many respects, they share certain commonalties, which can provide insight on our shared cognition. Here, we explore the learnability consequences of one of the striking commonalities between languages. Across languages, word frequencies follow a Zipfian distribution, showing a power law relation between a word's frequency and its rank. While their source in language has been studied extensively, less work has explored the learnability consequences of such distributions for language learners. We propose that the greater predictability of words in this distribution (relative to less skewed distributions) can facilitate word segmentation, a crucial aspect of early language acquisition. To explore this, we quantify word predictability using unigram entropy, assess it across languages using naturalistic corpora of child-directed speech and then ask whether similar unigram predictability facilitates word segmentation in the lab. We find similar unigram entropy in child-directed speech across 15 languages. We then use an auditory word segmentation task to show that the unigram predictability levels found in natural language are uniquely facilitative for word segmentation for both children and adults. These findings illustrate the facilitative impact of skewed input distributions on learning and raise questions about the possible role of cognitive pressures in the prevalence of Zipfian distributions in language.

A comprehensive review and evaluation on text predictive and entertainment systems

One of the most important ways to experience communication and interact with the systems is by handling the prediction of the most likely words to happen after typing letters or words. It is helpful for people with disabilities due to disabling people who could type or enter texts at a limited slow speed. Also, it is beneficial for people with dyslexia and those people who are not well with spells of words. Though, an input technology, for instance, the next word suggestion facilitates the typing process in smartphones as an example. This means that when a user types a word, then the system suggests the next words to be chosen in which the necessary word by the user. Besides, it can be used in entertainment as a gam, for example, to determine a target word and reach it or tackle it within 10 attempts of prediction. Generally, the systems depend on a text corpus, which was provided in the system to conduct the prediction. Writing every single word is time-consuming, therefore, it is vitally important to decrease time consumption by reducing efforts to input texts in the systems by offering most probable words for the user to select, this could be done via next word prediction systems. There are several techniques can be found in literature, which is utilized to conduct a variety of next word prediction systems by using different approaches. In this paper, a survey of miscellaneous techniques towards the next word prediction systems will be addressed. Besides, the evaluation of the prediction systems will be discussed. Then, a modal technique will be determined to be utilized for the next word prediction system from the perspective of easiness of implementation and obtaining a good result.

English and Romanian Brain-to-Text Brain-Computer Interface Word Prediction System

Brain-Computer Interface (BCI) can recognise the thoughts of a human through various electrophysiological signals. Electrodes (sensors) placed on the scalp are used to detect these signals, or by using electrodes implanted inside the brain. Usually, BCI can detect brain activity through different neuroimage methods, but the most preferred is Electroencephalography (EEG) because it is a non-invasive and non-critical method. BCI systems applications are very helpful in restoring functionalities to people suffering from disabilities due to different reasons. In this study, a novel brain-to-text BCI system is presented to predict the word that the subject is thinking. This brain-to-text can assist mute people or those who cannot communicate with others due to different diseases to restore some of their abilities to interact with the surrounding environment and express themselves. In addition, brain-to-text may be used in different control or entertainment applications. EMOTIV™ Insight headset has been used to collect EEG signals from the subject’s brain. Feature extraction of EEG signals for BCI systems is very important to classification performance. Statistical-based feature extraction has been used in this system to extract valuable features to be used for classification. The datasets are sentences involving some commonly used words in English and Romanian languages. The results of the English language elucidated that K-Nearest neighbour (KNN) has a prediction accuracy of 86.7%, 86.1% for Support Vector Machine (SVM), and 79.2% for Linear discriminant analysis (LDA), while the Romanian language has a prediction accuracy of 96.1%, 97.1%, and 94.8% for SVM, LDA, and KNN respectively. This system is a step forward in developing advanced brain-to-text BCI prediction systems.

Context-Based Text Completion System for Amharic Language

This paper discusses the practical aspects of easiness in communication using Short Message Service (SMS), E-mailing, correcting misspelt words and checking the grammatical mistakes. There are different data entry mechanisms to insert a text on the computer machine as well as a mobile device, such as a keyboard, soft keys, speech etc. The paper proposed to develop a contextbased auto text completion system for the Amharic language specifically to correct misspelling on Short Message Service (SMS), E-mailing and helps to correct the grammar mistakes as well. Data entry technique can be inserted with the support of text completion (predictive) or non-predictive. Therefore, we are using a statistical model, Predictive Partial Match (PPM) and Support Vector Machine (SVM) approaches for implementing the Amharic contextbased text completion system. Since the system is developed by using the context-based and statistical model, we adopted the Amharic Part of Speech (POS) tagger system. For training and testing the system, we are using 395,464 unique words with frequency and 750,000 sentences that has been prepared by the Walta Information Centre (WIA) and Ethiopia News Agency (ENA). All those data have been used to build the Amharic dictionary, the corpus of the system and to calculate the frequency occurrences of each word as well. Finally, the results show a 14% improvement from traditional frequency-based Amharic word prediction system.

Study of consumer perception about low‐sodium foods and characteristics related to perception and purchase of low‐sodium spreadable processed cheese

AbstractThis study interviewed 1,511 consumers using an online questionnaire to understand consumer behavior on reduced‐sodium products and the buying interest for processed cheese with low sodium content. Likert scale techniques were applied, which was assessed by analyzing factors (49.38%), Cronbach's alpha (0.694) and KMO (0.703), which conferred reliability to the applied test, and word association and completion task, which were evaluated chi‐square and chi‐square per cell, respectively showing X2 = 799,635 and X2 = 6,428.59 (positive and negative stimuli), and X2 = 652.55 (purchase condition stimulus) for p < .0001. The consumer understands the problem of high sodium consumption, understands the use of products reduced in sodium, and recognizes the need to reduce the sodium content in cheeses, associates the purchase condition with the high cost and potential health benefits. However, the lack of technical knowledge about the product negatively influences the decision.Practical applicationsSensory analysis techniques helps to elucidate the behavior of consumers towards a specific niche of products, clearly understand positive and negative points that directly influence the buying attitude, and clarify the limits of consumers' knowledge about products reduced in sodium, specifically evaluating the perception, behavior and purchase an interest in the face of a new product developed. The study takes on great relevance when it diagnoses the lack of information about salt, sodium, cheese technology processed by the consumer and demonstrates important points that interfere with safety and how it affects the purchase decision. It also highlights the need for greater transparency of the dairy industries with the consuming public when it comes to better information about the product, both related to the components and the applied technology. The work also provides information that aggregates both in research with consumers and offers a way of approximation between industry and consumer.

Context-based facilitation of semantic access follows both logarithmic and linear functions of stimulus probability

Stimuli are easier to process when context makes them predictable, but does context-based facilitation arise from preactivation of a limited set of relatively probable upcoming stimuli (with facilitation then linearly related to probability) or, instead, because the system maintains and updates a probability distribution across all items (with facilitation logarithmically related to probability)? We measured the N400, an index of semantic access, to words of varying probability, including unpredictable words. Word predictability was measured using both cloze probabilities and a state-of-the-art machine learning language model (GPT-2). We reanalyzed five datasets (n = 138) to demonstrate and then replicate that context-based facilitation on the N400 is graded, even among unpredictable words. Furthermore, we established that the relationship between word predictability and context-based facilitation combines linear and logarithmic functions. We argue that this composite function reveals properties of the mapping between words and semantic features and how feature- and word-related information is activated on-line.

The Rediscovery Hypothesis: Language Models Need to Meet Linguistics

There is an ongoing debate in the NLP community whether modern language models contain linguistic knowledge, recovered through so-called probes. In this paper, we study whether linguistic knowledge is a necessary condition for the good performance of modern language models, which we call the rediscovery hypothesis.
 In the first place, we show that language models that are significantly compressed but perform well on their pretraining objectives retain good scores when probed for linguistic structures. This result supports the rediscovery hypothesis and leads to the second contribution of our paper: an information-theoretic framework that relates language modeling objectives with linguistic information. This framework also provides a metric to measure the impact of linguistic information on the word prediction task. We reinforce our analytical results with various experiments, both on synthetic and on real NLP tasks in English.

Dynamic is optimal: Effect of three alternative auto-complete on the usability of in-vehicle dialing displays and driver distraction

Objective Auto-complete (AC) has become ubiquitous on domain-specific systems and is mainly divided into two types (static-AC and dynamic-AC). Specifically, static-AC only presents the possible completions not changing with user input in the suggestion list for users to select. Dynamic-AC constantly filters out inconsistent content with user input and shows the possible completions at the top of the suggestion list. However, the details of the driver’s interactions with AC in the vehicle are poorly understood. Therefore, we investigated the effect of three alternative AC (non-AC, static-AC, and dynamic-AC) on the usability of in-vehicle dialing displays and driver distraction. As a reference, the baseline task (only driving) was also surveyed in each AC condition. Methods A simulated driving experiment consisting of 24 participants was conducted. The primary task was a lane-keeping task with speed ranging between 60 and 120 km/h over the stretch. The secondary task was dialing an 11-digit phone number. Usability metrics (task completion time and number of errors) and driver distraction metrics (NASA-reduced task load index (NASA-RTLX), mean speed, lateral position variation, total glance time, number of glances, mean glance time, and number of glances over 1.6 s) in each condition were measured. A series of one-way repeated measure analyses of variance was used to examine whether and which type of AC can maximize the usability of in-vehicle dialing displays and minimize driver distraction. Results Generally, the AC-based in-vehicle dialing display gains a more positive effect. Specifically, we observed that among the three alternative AC conditions, dynamic-AC performed optimally on usability metrics similar to previous studies and various driver distraction metrics, notwithstanding it is still not up to the level of the baseline condition. However, static-AC did not exhibit the advantages described in previous studies except for fewer errors and NASA-RTLX owing to the possibility of position bias and boundary effect. Conclusions This study provides valuable insights into drivers’ interactions with AC-based in-vehicle dialing displays and broadened its applications in safety-critical situations. More importantly, it informs the design of a more effective in-vehicle system, which positively contributes to mitigating driver distraction and preventing traffic accidents.

Predictive pre-activation of orthographic and lexical-semantic representations facilitates visual word recognition.

To a crucial extent, the efficiency of reading results from the fact that visual word recognition is faster in predictive contexts. Predictive coding models suggest that this facilitation results from pre-activation of predictable stimulus features across multiple representational levels before stimulus onset. Still, it is not sufficiently understood which aspects of the rich set of linguistic representations that are activated during reading-visual, orthographic, phonological, and/or lexical-semantic-contribute to context-dependent facilitation. To investigate in detail which linguistic representations are pre-activated in a predictive context and how they affect subsequent stimulus processing, we combined a well-controlled repetition priming paradigm, including words and pseudowords (i.e., pronounceable nonwords), with behavioral and magnetoencephalography measurements. For statistical analysis, we used linear mixed modeling, which we found had a higher statistical power compared to conventional multivariate pattern decoding analysis. Behavioral data from 49 participants indicate that word predictability (i.e., context present vs. absent) facilitated orthographic and lexical-semantic, but not visual or phonological processes. Magnetoencephalography data from 38 participants show sustained activation of orthographic and lexical-semantic representations in the interval before processing the predicted stimulus, suggesting selective pre-activation at multiple levels of linguistic representation as proposed by predictive coding. However, we found more robust lexical-semantic representations when processing predictable in contrast to unpredictable letter strings, and pre-activation effects mainly resembled brain responses elicited when processing the expected letter string. This finding suggests that pre-activation did not result in "explaining away" predictable stimulus features, but rather in a "sharpening" of brain responses involved in word processing.