Word Prediction Research Articles

Comprehensive Review of Deep learning Techniques in Electronic Medical Records

A digital collection of patient’s health care data like diagnosis history of patient, treatment details, medical prescriptions are stored electronically. This electronic patient health records (EPHR) model provides huge volume of real time data and used for clinical research. Natural Language processing (NLP) automatically retrieve the patient’s information based on decision support system. NLP performs traditional techniques of machine learning, deep learning algorithms and focussing on word embeddings, classification and prediction, extraction, knowledge graphs, phenotyping, etc. By using NLP technique, extract the information from clinical data and analysis it provides valuable patient medical information. NLP based on clinical systems are evaluated on document level annotations which contains document of patient report, health status of patient, document section types contain past medical history of patient, summary of discharge statement, etc. similarly the semantic properties contain severity of disease in the aspects of positivity, negativity. These documents are developed and implemented on word level or sentence level. In this survey article, we summarize the recent NLP techniques which are used in EPHR applications. This survey paper focuses on prediction, classification, extraction, embedding, phenotyping, multilingually etc techniques.

A Potential Solution to Chinese Students’ Unintelligible English Accent: A Study of the Effectiveness of a Stress Placement Intervention to Improve Chinese English Speakers’ Lexical Pronunciation

Due to a lack of authentic learning resources, Chinese students often do not receive native pronunciation models in their classes. This study provided Chinese students with an approach to improve their English pronunciation, specifically to sound more like British Received Pronunciation (RP) or General American English Pronunciation (GAEP). The study proposes word stress prediction rules to help non-native Chinese learners predict the primary word stress of English words, improving their vocabulary pronunciation. Repeated-measures tests confirmed the effectiveness of this model, and statistical results showed that these rules were both applicable and teachable. These rules successfully enabled Chinese English learners to stress the correct vowel in a new English word without consulting a dictionary. Chinese students can rely on these rules for predicting the word stress of dictionary words, enabling them to pronounce new words as accurately as possible to native speakers.

Word Data Research and Prediction Based on Wordle Game

Word Data Research and Prediction Based on Wordle Game

Multimodal prediction of profanity based on speech analysis

With increasing multimedia content and social activities, moderation problems increase. There are different approaches to moderation and automation. However, they have limitations in terms of usage in real-time. The analysis of scientific papers revealed that most of the more common approaches solve the task of detection instead of prediction by considering the final utterance. For this reason, calls are unprotected in toxic languages, and online broadcasts can be unpredictable.In this work, a new way for automatic speech moderation in terms of dynamic word prediction was suggested. The considered task involves the analysis of the auditory and textual channels of speech. Words can have different meanings depending on the context, so in solving the problem it is planned to consider profanity, which is socially unacceptable regardless of the context.In this paper approaches for working with speech stream in the task of profanity prediction were proposed. It can be possible to have smaller latency with usage of audio features. We also suggest the pipeline for real-time (with the ability to predict the sequence with a higher duration than the latency of the processing) prediction for multimodal prediction, which compensates the latency of ASR systems. As a result, in this paper, we compared different solutions for the next color prediction task for English speech and reached the F1 score of 86.6 for 3 class prediction.

Curriculum Compositional Continual Learning for Neural Machine Translation

Current trends in language modelling leverage large language models pre-trained on a huge corpus of data to achieve state of the art results on several NLP tasks. On the other hand, humans acquire language from small amount of data using cognitive principles. Recently, a continual learning approach using compositionality to disentangle the syntax and semantics of an input sentence for downstream sequence to sequence tasks was proposed. In this work, we show how curriculum learning can be incorporated with this framework to improve performance. More specifically, first, we show that using the model of interest with reduced hidden size as the auxiliary model to generate curriculum is not necessarily optimal and second, we propose a novel variant of the one best score approach for curriculum learning where, a sequence to sequence model is used as the auxiliary model to generate the conditional probabilities of word predictions (proxy for difficulty) and consequently used this to generate a curriculum. Results on a variety of translation tasks, demonstrate the superiority of the proposed approach compared to several baselines, enabling the improvement of sentence accuracy with respect to knowledge transfer and catastrophic-forgetting both by at least a significant margin of 35% with respect to the best performing baseline on the English-French translation task.

A Systematic Review of Mobile Learning Trends in Supporting the Mastery of Spelling

Teaching and learning to spell forms the basis for mastering English Language. The emerging mobile learning trends in the current 21st century education is useful in teaching and learning spelling. However, there were limited reviews about them. As such, a systematic review was conducted to provide an up-to-date synthesis on the mobile learning trends in supporting the mastery of spelling from year 2012 to year 2022. This review adhered to Preferred Reporting Items for Systematic Reviews and Meta-Analyses which is also known as the PRISMA guideline. Three databases were involved: Web of Science, Scopus and Educational Resources Information Centre. After a rigorous screening process, a total of 19 articles fitted its objectives. Firstly, significant findings from the studies portrayed the use of four platforms in mobile learning trends in supporting the mastery of spelling namely mobile apps, spell-checker apps, word prediction apps and text messaging apps. Secondly, the primary level of education was more dominant in the related studies than the secondary and tertiary levels. Thirdly, it emphasised more on the students with a lower prevalence on teachers and both teachers and students as the respondents in the associated studies. This paper contributes to systematically reviewing 19 eligible related studies through a step-by-step protocol of identification, screening, eligibility and included as in the PRISMA guideline to establish quality review. It benefits both the English Language teachers and students in utilising various mobile learning platforms to support the mastery of spelling at all levels of education. It aspires to foreground the paucity in this topic in which more substantial studies are required to address the research gaps. The ultimate goal is to impose effective and practical strategies to advance mobile learning trends in supporting the mastery of spelling.

The subjective feeling of a gap between conceptual and perceptual fluency is interpreted as a metacognitive signal of pastness.

The present study aimed to address the following question: does the discrepancy between an expected word and its readability enhances or impair its memorability? We used an adaptation of the sentence stem paradigm (Whittlesea in J Exp Psycol 19:1235-1253, 1993) and manipulated the perceptual clarity of the words by introducing some Gaussian noise (Reber in Psycol Sci 9:45-48, 1998). The target words were semantically predictable or otherwise (conceptual fluency) or were easy or difficult to read (perceptual fluency). The first experiment was conducted to ensure that the two manipulated factors had an impact on the readability of the words. In particular, results showed that when the words were written against a noisy background their predictability enhanced the judgement of readability. The second experiment aimed to test the hypothesis that recognition would be influenced by the discrepancy between conceptual and perceptual fluency. The results showed that with a noisy background, the predictability of the target words had an impact on recognition judgement; with a clear background, the effect on the recognition judgement was caused by the non-predictability of the target words. Conversely, confidence in judgement increased when the two factors went in the same direction, that is, predictability with clarity and non-predictability with low clarity. The results showed that (a) depending on the task, the effects of conceptual and perceptual fluency did not go in the same direction; (b) the kinds of fluency (conceptual and perceptual) were not independent; and (c) recognition judgements were affected by the gap between conceptual and perceptual fluency.

A recent survey of augmentative and alternative communication use and service delivery experiences of people with amyotrophic lateral sclerosis in the United States

Purpose The objective of this study was to explore and describe current trends in the augmentative and alternative communication (AAC) use and service delivery experiences of people with amyotrophic lateral sclerosis (PALS) in the U.S. Methods Cross-sectional data were collected from 216 PALS via an anonymous online questionnaire in 2021. Results Over 70% of participants reported at least some detectable speech disturbance, and approximately half used aided communication during face-to-face interactions. Among respondents with severe speech impairment, over 90% reported using speech-generating devices, and just over half reported using low-tech AAC. Most participants had met with an SLP to discuss speech and communication, but varied in both timing of the initial intervention and frequency of ongoing intervention. Fewer than half reported that their family members or other important people had received education or support related to communication for PALS. Participants also shared their use of and experiences with telephone and video calls, access methods, mounting systems, word prediction and stored phrases, and message and voice banking. Conclusions Results highlight the importance of early referral for AAC intervention, ongoing re-evaluation and treatment, involvement of communication partners and support for multimodal communication and adaptation to changing needs.

Separate Syntax and Semantics: Part-of-Speech-Guided Transformer for Image Captioning

Transformer-based image captioning models have recently achieved remarkable performance by using new fully attentive paradigms. However, existing models generally follow the conventional language model of predicting the next word conditioned on the visual features and partially generated words. They treat the predictions of visual and nonvisual words equally and usually tend to produce generic captions. To address these issues, we propose a novel part-of-speech-guided transformer (PoS-Transformer) framework for image captioning. Specifically, a self-attention part-of-speech prediction network is first presented to model the part-of-speech tag sequences for the corresponding image captions. Then, different attention mechanisms are constructed for the decoder to guide the caption generation by using the part-of-speech information. Benefiting from the part-of-speech guiding mechanisms, the proposed framework not only adaptively adjusts the weights between visual features and language signals for the word prediction, but also facilitates the generation of more fine-grained and grounded captions. Finally, a multitask learning is introduced to train the whole PoS-Transformer network in an end-to-end manner. Our model was trained and tested on the MSCOCO and Flickr30k datasets with the experimental evaluation standard CIDEr scores of 1.299 and 0.612, respectively. The qualitative experimental results indicated that the captions generated by our method conformed to the grammatical rules better.

Medical Text Prediction and Suggestion Using Generative Pretrained Transformer Models with Dental Medical Notes.

Generative pretrained transformer (GPT) models are one of the latest large pretrained natural language processing models that enables model training with limited datasets and reduces dependency on large datasets, which are scarce and costly to establish and maintain. There is a rising interest to explore the use of GPT models in health care. We investigate the performance of GPT-2 and GPT-Neo models for medical text prediction using 374,787 free-text dental notes. We fine-tune pretrained GPT-2 and GPT-Neo models for next word prediction on a dataset of over 374,000 manually written sections of dental clinical notes. Each model was trained on 80% of the dataset, validated on 10%, and tested on the remaining 10%. We report model performance in terms of next word prediction accuracy and loss. Additionally, we analyze the performance of the models on different types of prediction tokens for categories. For comparison, we also fine-tuned a non-GPT pretrained neural network model, XLNet (large), for next word prediction. We annotate each token in 100 randomly sampled notes by category (e.g., names, abbreviations, clinical terms, punctuation, etc.) and compare the performance of each model by token category. Models present acceptable accuracy scores (GPT-2: 76%; GPT-Neo: 53%), and the GPT-2 model also performs better in manual evaluations, especially for names, abbreviations, and punctuation. Both GPT models outperformed XLNet in terms of accuracy. The results suggest that pretrained models have the potential to assist medical charting in the future. We share the lessons learned, insights, and suggestions for future implementations. The results suggest that pretrained models have the potential to assist medical charting in the future. Our study presented one of the first implementations of the GPT model used with medical notes.

Concept formation through multimodal integration using multimodal BERT and VQ-VAE

Humans form concepts from multimodal information obtained from multiple sensory organs and understand the environment and language by predicting unobserved information through these concepts. Robots can also make better predictions by forming concepts from multimodal information in a bottom-up manner. In contrast, BERT, a language model, acquires semantic representations of sentences applicable to various language comprehension tasks by learning relationships between words through mask prediction learning. We believe that BERT can be extended to multimodality and learn multimodal representations for world understanding by learning relationships among data across modalities. Therefore, we propose a multimodal learning model using BERT and VQ-VAEs with multimodal information as input and evaluate the model in terms of model predictability and concept formation. To verify the usefulness of the proposed model, experiments were conducted using a multimodal dataset of objects acquired by a robot. The experimental results showed that the proposed model has higher category classification accuracy and word prediction performance than multimodal latent Dirichlet allocation (MLDA), which is used as a concept formation model. An analysis of the representation of the model showed that clusters of object categories were formed within the model.

Peer-to-peer deep learning with non-IID data

Collaborative training of deep neural networks using edge devices has attracted substantial research interest recently. The two main architecture approaches for the training process are centrally orchestrated Federated Learning and fully decentralized peer-to-peer learning. In decentralized systems, edge devices, known as agents, collaborate in a peer-to-peer architecture, avoiding the need for a central system to orchestrate the process. Decentralized peer-to-peer (P2P) learning techniques are well researched under the assumption of independent and identically distributed (IID) data across the agents. IID data is seldom observed in real-world distributed systems, and the training performance varies significantly with non-IID data. This paper proposes a decentralized learning variant of the P2P gossip averaging method with Batch Normalization (BN) adaptation for P2P architectures. It is well-known that BN layers accelerate the convergence of the non-distributed deep learning models. Recent research confirms that Federated Learning methods benefit from using the BN method with some aggregation alterations. Our work demonstrated BN effectiveness in P2P architectures by mitigating the non-IID data characteristics across decentralized agents. We also introduce a variant of the early stopping technique that, combined with BN layers, acts as a fine-tuning technique for agent models. We validated our approach by conducting numerous simulations of different model-topology-communication combinations and comparing them to other decentralized baseline approaches. The evaluations were conducted on the next word prediction task using user comments from the Reddit and StackOverflow datasets representing comments from two different domains. Simulations showed that our approach, on average, achieves a mean relative top accuracy increase of 16.9% in ring (19.9% for Reddit, 13.9% for StackOverflow) and 29.8% in sparse (32.9% for Reddit, 26.6% for StackOverflow) communication topologies compared to the best baseline approach. Our code is available at https://github.com/fipu-lab/p2p_bn.

Explaining neural activity in human listeners with deep learning via natural language processing of narrative text

Deep learning (DL) approaches may also inform the analysis of human brain activity. Here, a state-of-art DL tool for natural language processing, the Generative Pre-trained Transformer version 2 (GPT-2), is shown to generate meaningful neural encodings in functional MRI during narrative listening. Linguistic features of word unpredictability (surprisal) and contextual importance (saliency) were derived from the GPT-2 applied to the text of a 12-min narrative. Segments of variable duration (from 15 to 90 s) defined the context for the next word, resulting in different sets of neural predictors for functional MRI signals recorded in 27 healthy listeners of the narrative. GPT-2 surprisal, estimating word prediction errors from the artificial network, significantly explained the neural data in superior and middle temporal gyri (bilaterally), in anterior and posterior cingulate cortices, and in the left prefrontal cortex. GPT-2 saliency, weighing the importance of context words, significantly explained the neural data for longer segments in left superior and middle temporal gyri. These results add novel support to the use of DL tools in the search for neural encodings in functional MRI. A DL language model like the GPT-2 may feature useful data about neural processes subserving language comprehension in humans, including next-word context-related prediction.

A resource-rational model of human processing of recursive linguistic structure

A major goal of psycholinguistic theory is to account for the cognitive constraints limiting the speed and ease of language comprehension and production. Wide-ranging evidence demonstrates a key role for linguistic expectations: A word's predictability, as measured by the information-theoretic quantity of surprisal, is a major determinant of processing difficulty. But surprisal, under standard theories, fails to predict the difficulty profile of an important class of linguistic patterns: the nested hierarchical structures made possible by recursion in human language. These nested structures are better accounted for by psycholinguistic theories of constrained working memory capacity. However, progress on theory unifying expectation-based and memory-based accounts has been limited. Here we present a unified theory of a rational trade-off between precision of memory representations with ease of prediction, a scaled-up computational implementation using contemporary machine learning methods, and experimental evidence in support of the theory's distinctive predictions. We show that the theory makes nuanced and distinctive predictions for difficulty patterns in nested recursive structures predicted by neither expectation-based nor memory-based theories alone. These predictions are confirmed 1) in two language comprehension experiments in English, and 2) in sentence completions in English, Spanish, and German. More generally, our framework offers computationally explicit theory and methods for understanding how memory constraints and prediction interact in human language comprehension and production.

Word prediction using closely and moderately related verbs in Down syndrome.

People with Down syndrome (DS) have several difficulties in language learning, and one of the areas most affected is language production. Theoretical frameworks argue that prediction depends on the production system. Yet, people with DS can predict upcoming nouns using semantically related verbs. Possibly, prediction skills in people with DS are driven by their associative mechanism rather than by the prediction mechanism based on the production system. This study explores prediction mechanisms in people with DS and their relationship with production skills. Three groups were evaluated in a preferential-looking task: young adults, children with DS, and a typically developing control group paired by sex and mental age. Participants saw two images, a target and a distractor. They also heard a sentence in one of the three conditions: with a verb that was closely related to the object (e.g., “The woman read the book”), with a verb that was moderately related to the object (e.g., “My uncle waited for the bus”), or with a verb that was unrelated to the object (e.g., “My sister threw a broom”). Their productive vocabulary was then measured. In the young adult and typically developing groups, the results showed prediction in sentences with highly and moderately related verbs. Participants with DS, however, showed prediction skills only in the highly related context. There was no influence of chronological age, mental age, or production on prediction skills. These results indicate that people with DS base prediction mainly on associative mechanisms and they have difficulty in generating top-down predictions.

Southern dialects of United States English and automatic speech recognition

Automatic Speech Recognition (ASR) technology is designed to provide more human to machine communication options. Emergent research has observed variant performance of ASR for African American English (AAE) and other minority dialects. Word prediction was observed to be better for AAE compared to majority dialects. The opposite was true for word identification accuracy. The researchers hypothesize the higher word error rate for AAE is related to phonetic factors of vowel duration, consonant production, rhythm, pitch, and syllable accent. This work evaluates that hypothesis. Recordings of two AAE and two White AE speaking women from North Carolina reading Comma Gets a Cure were submitted for transcription to the Microsoft ASR program. Several common transcription errors were noted across speaker groups with a greater variety of errors noted for the AAE women. Acoustic analyses of vowel duration, and the spectral acoustics of vowel and consonant production by talker and group will be completed. The results of this analysis will be useful to describing the specific aspects of AAE speech that may perturb the tested ASR. These data provide insight into the speech tasks and the sub and supra-segmental acoustic data that should be included in future ASR programming iterations.

The electrophysiological correlates of word pre-activation during associative word learning

Human beings continuously make use of learned associations to generate predictions about future occurrences in the environment. Such memory-related predictive processes provide a scaffold for learning in that mental representations of foreseeable events can be adjusted or strengthened based on a specific outcome. Learning the meaning of novel words through picture-word associations constitutes a prime example of associative learning because pictures preceding words can trigger word prediction through the pre-activation of the related mnemonic representations. In the present electroencephalography (EEG) study, we used event-related potentials (ERPs) to compare neural indices of word pre-activation between a word learning condition with maximal prediction likelihood and a non-learning control condition with low prediction. Results revealed that prediction-related N400 amplitudes in response to pictures decreased over time at central electrodes as a function of word learning, whereas late positive component (LPC) amplitudes increased. Notably, N400 but not LPC changes were also predictive of word learning performance, suggesting that the N400 component constitutes a sensitive marker of word pre-activation during associative word learning.

Automatic Grammar Error Correction Model Based on Encoder-decoder Structure for English Texts

The role of information transmission in social life is irreplaceable, and language is a very important information carrier. Among all kinds of languages, English always occupies an important position. In the process of English learning, grammar error has become a difficult problem for most learners. In this paper, we propose an automatic grammar error correction model based on encoder-decoder structure. Different from traditional encoders, we design a dual-encoder structure to capture the information of source sentence and context sentence separately. The decoder is designed with a gated structure, it can effectively integrate output information of encoders. At the same time, the self-attention mechanism is combined to better solve the problem of long-distance information extraction. In addition, we propose a dynamic beam search algorithm to improve the accuracy of the word prediction process, and achieve dynamic extraction of the decoder output by combining kernel sampling techniques. We add a penalty factor to reduce the probability of generating repeated words, while suppressing the model's preference for generating shorter sentences. Finally, the proposed method is validated on the official English grammar error correction dataset. Experiments show that the dual encoder model in this paper has a good performance.

Frequency and predictability effects in first and second language of different script bilinguals.

Reading efficiently in a second language (L2) is a crucial skill, but it is not universally achieved. Here we ask whether L2 reading efficiency is better captured as a language specific skill or whether it is mostly shared across L1 and L2, relying on general language abilities. To this end, we examined word frequency and predictability effects in sentence reading, and tested the same readers in L1 and L2, recording participants' eye-movements. Participants were 57 undergraduate bilingual speakers of Hebrew and English, languages that use different scripts, allowing for a clearer distinction between L1 and L2 processing. Both word frequency and word predictability effects were more pronounced in participants' L2 than in the L1, suggesting that both lower level and higher-order processes in reading are sensitive to language proficiency. Further, frequency effects in the L2 were linked with L2 proficiency but not general language abilities, and L2 predictability effects were not associated with either variable. Finally, readers' frequency and predictability effects in L1 and L2 were not associated with each other. Taken together, these results suggest that for different-script bilinguals, efficient reading in the L2 is a highly specific skill, dependent upon proficiency in that language, and drawing less on L1 and general language ability. (PsycInfo Database Record (c) 2022 APA, all rights reserved).

A Deep OCR for Degraded Bangla Documents

Despite the significant success of document image analysis techniques, efficient Optical Character Recognition (OCR) of degraded document images still remains an open problem. Although a body of work has been reported on degraded document recognition for English language, only little attention has been paid to Indic scripts. In this work, we focus on developing a degraded OCR for Bangla, a major Indian language. In general, an OCR system includes segmentation of the foreground text part from the background followed by recognition of the extracted text. The text segmentation module aims to assign the foreground or background label to each pixel of the document image. In this paper, we present a new OCR system which is particularly suitable for degraded quality Bangla document images. The contribution is two fold. In the first phase, we use a semi-supervised Markov Random Field (MRF)- based Generative Adversarial Network (GAN) model (which we call MRF-GAN ) for foreground segmentation of texts from degraded text. In the proposed MRF-GAN , we extend the concept of GAN to a multitask learning mechanism where discriminator-classifier networks differentiate between real/fake images and also assign a foreground or background label to each pixel. In the second phase, we propose to use a new encoder-decoder based recognizer that incorporates an attention-based character to a word prediction model, which has the capability of minimizing Word Error Rate (WER) . We optimize this network using a Multitask based Transfer Learning scheme (MTTL) . We perform experiments on a publicly available degraded Bangla document image dataset as well as on a new degraded printed Hindi document image dataset, which has been created as a part of the present study. Results of the experimentations demonstrate the efficacy of the proposed OCR.