Sentence Prediction Research Articles

사전학습 언어모델의 기술 분석 연구

The pre-training language model, BERT has achieved a great success in natural language processing by transferring knowledge from rich-resource pre-training task to the low-resource downstream tasks. The model was recognized as a breakthrough or an innovative technology that changed the paradigm of natural language processing. In this paper, a number of studies have been analyzed to classify and compare the research directions. We examined a technical challenges after BERT. In the pre-training process, self-supervised learning is performed, which relies entirely on training data. If we introduce the linguistic knowledge in the course of the training, it would be possible to get better result more effectively. Therefore, it is necessary to develop a method to insert external knowledge such as linguistic information in the training process. The mask language model and the next sentence prediction are being used in BERT's pre-training tasks. Though, to get much deeper understanding of the natural language, some other effective methods are to be studied and developed. Lastly, we should aim to develop eXplainable Artificial Intelligence (XAI) technology in natural language processing, helping us look into the transparent processing. The pre-trained language model focuses on the development of skills that can be used for all the tasks of natural language understanding. A lot of researches are focused on how to adapt language model effectively to downstream tasks based on common language models, even with the case with little data. It is also hoped that the technical analysis reviewed in this study will provide linguists and computer researchers with an opportunity to understand recent technological achievements in the field of natural language processing and to seek joint research.

Automated Lip Reading using Word Level Sentence Prediction based on LSTMs and Attention

Automated Lip Reading using Word Level Sentence Prediction based on LSTMs and Attention

Pupillary responses signify more than just effort: Windows into processing, prediction, reflection, and uncertainty

Pupil dilation has long been used to index effort or general cognitive arousal in a variety of tasks. Traditionally, the main outcome measure is peak pupil dilation or the mean pupil dilation within some target window. In this talk, we explore how the time course of the growth and decay of pupil dilation can lend further insight into auditory processing. A series of pupillometry experiments revealed elevated listening effort resulting from various influences such as poor spectral resolution, unpredictability of sentence content, selective attention, or the need to reflect on what was heard to produce a creative response. Peak and mean pupillary responses generally confirmed original hypotheses, but more interesting findings emerge upon inspection of unforeseen trends in the data. Specifically, the timing of the pupil decay response can arguably be interpreted as a signature of the completion of sentence prediction, or the completion of processing that relies on memory trace (rather than perception) of the signal. The ultimate goal of this discussion is to unpack a new approach to pupillometry experiments that is aimed at separating processing that stems from perception versus processing that stems from cognitive processing, or reflecting upon a memory trace of the stimulus.

Celebrity defendant: The effect of the negative image of celebrity on public sentence prediction

Celebrity defendant: The effect of the negative image of celebrity on public sentence prediction

The Effects of AAC Sentence Prediction Function on the Speed and Accuracy of Sentence Generation

The Effects of AAC Sentence Prediction Function on the Speed and Accuracy of Sentence Generation

Measuring the Performance of a Location-Aware Text Prediction System

In recent years, some works have discussed the conception of location-aware Augmentative and Alternative Communication (AAC) systems with very positive feedback from participants. However, in most cases, complementary quantitative evaluations have not been carried out to confirm those results. To contribute to clarifying the validity of these approaches, our study quantitatively evaluated the effect of using language models with location knowledge on the efficiency of a word and sentence prediction system. Using corpora collected for three different locations (classroom, school cafeteria, home), location-specific language models were trained with sentences from each location and compared with a traditional all-purpose language model, trained on all corpora. User tests showed a modest mean improvement of 2.4% and 1.3% for Words Per Minute (WPM) and Keystroke Saving Rate (KSR), respectively, but the differences were not statistically significant. Since our text prediction system relies on the concept of sentence reuse, we ran a set of simulations with language models having different sentence knowledge levels (0%, 25%, 50%, 75%, 100%). We also introduced in the comparison a second location-aware strategy that combines the location-specific approach with the all-purpose approach (mixed approach). The mixed language models performed better under low sentence-reuse conditions (0%, 25%, 50%) with 1.0%, 1.3%, and 1.2% KSR improvements, respectively. The location-specific language models performed better under high sentence-reuse conditions (75%, 100%) with 1.7% and 1.5% KSR improvements, respectively.

A Stochastic Prediction Interface for Urdu

This work lays down a foundation for text prediction of an inflected and under-resourced language Urdu. The interface developed is not limited to a T9 (Text on 9 keys) application used in embedded devices, which can only predict a word after typing initial characters. It is capable of predicting a word like T9 and also a sequence of word after a word in a continuous manner for fast document typing. It is based on Ngram language model. This stochastic interface deals with three N-gram levels from unary to ternary independently. The unigram mode is being in use for applications like T9, while the bigram and tri-gram modes are being in use for sentence prediction. The measures include a percentage of keystrokes saved, keystrokes until completion and a percentage of time saved during the typing. Two different corpora are merged to build a sufficient amount of data. The test data is divided into a test and a held out data equally for an experimental purpose. This whole exercise enables the QASKU system outperforms the FastType with almost 15% more saved keystrokes.

Word and Sentence Prediction: Using the best of the two worlds to assist AAC users

Word and Sentence Prediction: Using the best of the two worlds to assist AAC users

Automatic evaluation of syntactic learners in typologically-different languages

Human syntax acquisition involves a system that can learn constraints on possible word sequences in typologically-different human languages. Evaluation of computational syntax acquisition systems typically involves theory-specific or language-specific assumptions that make it difficult to compare results in multiple languages. To address this problem, a bag-of-words incremental generation (BIG) task with an automatic sentence prediction accuracy (SPA) evaluation measure was developed. The BIG–SPA task was used to test several learners that incorporated n-gram statistics which are commonly found in statistical approaches to syntax acquisition. In addition, a novel Adjacency–Prominence learner, that was based on psycholinguistic work in sentence production and syntax acquisition, was also tested and it was found that this learner yielded the best results in this task on these languages. In general, the BIG–SPA task is argued to be a useful platform for comparing explicit theories of syntax acquisition in multiple languages.

On the Nature of Connectionist Conceptualizations and Connectionist Explanations

argued that connectionist explanations are too vague to qualify as theories of cognitive functions. Much of the argument hinges on the claim that hidden unit activation patterns of connectionist networks are currently too difficult to analyze, and that such opacity renders connectionist accounts virtually ineffective. However, recent attempts at analyzing connectionist hidden units using statistical techniques such as hierarchical clustering and principal component analysis reveal that connectionist networks easily yield categories which we traditionally associate with constituent structures (Elman, 1990). In this paper, we will focus on Elman's statistical analyses of the hidden unit activation patterns in his simple recurrent network on sentence prediction, first to highlight the feasibility of such analyses, and then to show how connectionist explanations contribute to the development of effective explanatory theories. In addition, in response to Fantuzzi's (1993) argument that connectionist conceptualizations do not qualify as connectionist explanations, we argue for the evolutionary nature of explanations as they relate to theory development, and for the role of conceptualizations as constructive intermediate explanations. Finally, to address Fantuzzi's (1993) criticism that pre-simulation connectionist conceptualizations are unproductive exercises akin to putting the cart before the horse, we provide examples to show that the relationship between