Neural Network Language Model Research Articles

Educational Resource Material Translation System

The proposed "Educational Resource Material Translation System" represents a significant leap in bridging language barriers, particularly for individuals who lack proficiency in English. By leveraging advanced technologies such as Python, Flask, Recurrent Neural Network (RNN) and Large Language Models (LLM), the system offers a robust solution for translating educational content from English to prominent Indian regional languages like Kannada, Telugu and Hindi. Its user- friendly web interface, developed with Flask, ensures a seamless and intuitive translation experience for users of varying technical backgrounds. Beyond mere language translation, this project facilitates cross- cultural communication and enhances accessibility to educational resources, thereby empowering individuals and businesses alike. The utilization of Machine Learning further underscores the system's adaptability and potential for delivering precise and efficient translations. In essence, the "Language Translator" not only serves as a valuable tool for linguistic inclusivity but also fosters knowledge dissemination and cultural exchange on a broader scale.

Lexical-Semantic Content, Not Syntactic Structure, Is the Main Contributor to ANN-Brain Similarity of fMRI Responses in the Language Network.

Representations from artificial neural network (ANN) language models have been shown to predict human brain activity in the language network. To understand what aspects of linguistic stimuli contribute to ANN-to-brain similarity, we used an fMRI data set of responses to n = 627 naturalistic English sentences (Pereira et al., 2018) and systematically manipulated the stimuli for which ANN representations were extracted. In particular, we (i) perturbed sentences' word order, (ii) removed different subsets of words, or (iii) replaced sentences with other sentences of varying semantic similarity. We found that the lexical-semantic content of the sentence (largely carried by content words) rather than the sentence's syntactic form (conveyed via word order or function words) is primarily responsible for the ANN-to-brain similarity. In follow-up analyses, we found that perturbation manipulations that adversely affect brain predictivity also lead to more divergent representations in the ANN's embedding space and decrease the ANN's ability to predict upcoming tokens in those stimuli. Further, results are robust as to whether the mapping model is trained on intact or perturbed stimuli and whether the ANN sentence representations are conditioned on the same linguistic context that humans saw. The critical result-that lexical-semantic content is the main contributor to the similarity between ANN representations and neural ones-aligns with the idea that the goal of the human language system is to extract meaning from linguistic strings. Finally, this work highlights the strength of systematic experimental manipulations for evaluating how close we are to accurate and generalizable models of the human language network.

Artificial Neural Network Language Models Predict Human Brain Responses to Language Even After a Developmentally Realistic Amount of Training.

Artificial neural networks have emerged as computationally plausible models of human language processing. A major criticism of these models is that the amount of training data they receive far exceeds that of humans during language learning. Here, we use two complementary approaches to ask how the models' ability to capture human fMRI responses to sentences is affected by the amount of training data. First, we evaluate GPT-2 models trained on 1 million, 10 million, 100 million, or 1 billion words against an fMRI benchmark. We consider the 100-million-word model to be developmentally plausible in terms of the amount of training data given that this amount is similar to what children are estimated to be exposed to during the first 10 years of life. Second, we test the performance of a GPT-2 model trained on a 9-billion-token dataset to reach state-of-the-art next-word prediction performance on the human benchmark at different stages during training. Across both approaches, we find that (i) the models trained on a developmentally plausible amount of data already achieve near-maximal performance in capturing fMRI responses to sentences. Further, (ii) lower perplexity-a measure of next-word prediction performance-is associated with stronger alignment with human data, suggesting that models that have received enough training to achieve sufficiently high next-word prediction performance also acquire representations of sentences that are predictive of human fMRI responses. In tandem, these findings establish that although some training is necessary for the models' predictive ability, a developmentally realistic amount of training (∼100 million words) may suffice.

Alice's Tales: the transformation of the composition, fairy-tale formulas and contexts of the voice assistant in the skill "Let's come up with"

The subject of the study is a spontaneously generated text by the voice assistant Alice when it is creating a fairy tale together with the user, and the purpose of the study is the transformation of the structure, fairy-tale formulas and context in terms of the selection of linguistic elements and meanings using artificial intelligence technology. Particular attention has focused on the skill "Let's make it up", which became available to users in the spring of 2023. The collision and interaction of folklore canons with the realities of the 21st century give rise to an ambiguous reaction to the interactive opportunity to play the role of a storyteller together with a voice assistant. The main research method was a continuous sample, which was used to distribute the steps, stages and actions when it is creating a fairy-tale plot together with a voice assistant. In addition, methods such as comparative and contextual analyses were used to identify similarities and differences between traditional Russian fairy tales and a spontaneously generated fairy tale plot. To obtain the data and subsequent analysis of the components, a linguistic experiment with the voice assistant Alice from Yandex was conducted and described. The rapid development of neural network language models allows us to talk about the scientific novelty of the material under study, since this area is unexplored and is being modified too quickly. It is important to emphasize that to date, the texts of spontaneously generated fairy-tale, their structural division and the correspondence of fairy-tale formulas in them to folklore canons have not been studied. The main conclusion of the study is that the user's share in creating a fairy tale with the voice assistant Alice is greatly exaggerated.

Machine learning drug discovery based on graph neural network and large language model

The COVID-19 pandemic has presented an urgent need to understand the long-term health implications faced by survivors. Post-COVID-19 complications, such as acute kidney injury, arrhythmia, and stroke, pose significant challenges to public health. Despite extensive research on COVID-19 complications, a comprehensive understanding of the risk factors remains elusive due to the potential confounding variables present in the data. Traditional statistical models, while insightful, may not fully capture the causal relationships between these risk factors and post-COVID-19 complications. Motivated by this gap in the literature, we propose a novel approach using causal inference models to predict the likelihood of post-COVID-19 complications based on patient demographics and pre-existing conditions. Our model, trained on a dataset of COVID-19 inpatients in Wuhan Province, China, estimates the causal effect of these factors on the likelihood of patients experiencing post-COVID-19 complications. This approach allows us to isolate the causal impact of each factor while accounting for potential confounders, providing a more accurate understanding of the underlying mechanisms driving these relationships. Unlike traditional models that predict the probability of certain outcomes, our model provides insights into the causal relationships between risk factors and complications, offering a more reliable and comprehensive understanding of the underlying mechanisms. This approach can help identify at-risk patients, inform targeted interventions, and contribute to the development of effective prevention and treatment strategies. Our work aims to contribute to the current understanding of the virus and inform public health policies and interventions.

The Construction of “Interactive” English Translation Teaching Mode in the Background of Internet

Abstract This paper discusses the construction of “interactive” English translation teaching mode in the background of the Internet, and analyzes the application of neural machine translation model in improving the effect of translation teaching. The research adopts Word2vec algorithm to train word vectors, combines neural network language model and encoder-decoder structure, and constructs Transformer model and CNN+Transformer model to improve translation efficiency and quality. The translation experiments on sentences of different lengths show that the models perform better in short sentence translation. The experimental results show that the proposed model outperforms the traditional model in terms of BLEU value, and the best translation effect is found in test set 1, where the BLEU value is improved by 10.6 and 11.7 compared with that of the baseline model and the CNN model, respectively. The “interactive” teaching mode designed with POA theory significantly outperforms the traditional teaching mode regarding students’ lexical, grammatical and semantic translation quality. The “interactive” English translation teaching with neural translation model can effectively improve the quality of translation teaching and students’ learning effect.

Формирование лексического модуля прикладной онтологии для её обучения

The complexity of developing ontologies is associated with their learning, which involves the automation of solving problems related to the extraction of ontological concepts and relationships from natural language texts. For a created ontology used in an information system, there will most likely be a need to expand its conceptual system due to the complication or change in data processing. Subsequent application of text analysis to discover new concepts and in-corporate them into the ontology, that is, linking them with existing concepts, will require identifying the latter in sen-tences of natural language texts. This paper considers the problem of automating the formation of the lexical module of applied ontology, which includes formalized representations of ontological concepts in natural language texts. A brief review of existing works devoted to the use of lexical information about ontology components in solving problems related to the analysis of textual data is presented. The ways of using the OntoLex-Lemon model for determining the structure of lexical representations of ontology concepts are considered. A procedure is proposed for the formation of lexical representations based on the analysis of texts in the subject area, taking into account the case when concepts have names consisting of several words. The results of applying the obtained module for the automatic formation of a training set of a neural network language model used in the ontology learning task for discovering new concepts in the corpus of subject texts are presented.

Transformer-Based Language Models as Psycholinguistic Subjects: Focusing on Understanding Metaphor

Metaphor is a fundamental aspect of human language and cognition, playing a crucial role in communication, comprehension, and creative expression. In light of the recent advancements demonstrated by prominent language models, a pivotal question arises: Can these expansive language models effectively discern metaphorical knowledge? The primary objective involves comparing the surprisal values estimated from neural network language models like autoregressive and bidirectional language models to the reaction times of human when exposed to both metaphorical and literal sentences. Our secondary objective involves assessing the AI's comprehension of metaphors by utilizing the sensicality ratings generated by sophisticated ChatGPT. To achieve this, we used psycholinguistic methods, and adopted the experimental materials from Lai, Currana, and Menna (2009). We found the surprisal values estimated from the autoregressive language model demonstrate metaphor processing that closely resembles that of native speakers. Furthermore, ChatGPT's processing of conventional metaphorical sentences closely resembles its approach to literal sentences, mirroring the convergence observed in native speakers' ERP response to conventional metaphorical sentences and their alignment with that of literal sentences.

Language Model Method for Collocation Rules of Parts of Speech in Machine Translation System

With the development of the times, modern society has now entered the Internet of Things (IoT) information age and Machine Translation (MT) plays an important role in increasingly frequent cross-language communication. In recent years, China's artificial intelligence industry has been in a stage of rapid construction, and the scale of its core industries has grown explosively, and a large number of artificial intelligence companies, including issuers, have emerged. Part of speech has always been a major problem in MT. One of the reasons is that there are a large number of multi-category words in Chinese and a large number of polysemy words in English, so part of speech collocation problems account for a large proportion of MT errors, which to some extent affects the credibility and accuracy of the translation. To reduce the error problem in MT of part of speech collocation, this paper used Machine Learning (ML) methods to study the Language Model (LM) of part of speech collocation based on recurrent neural network (NN) and compared it with the traditional statistical LM. In terms of the accuracy rate of the two LMS in the automatic evaluation index of machine translation, the experimental results show that the recursive NN LM established by the ML method had an accuracy rate of 80.42% and 83.57% respectively for the part-of-speech matching rules of the IoT machine translation system in the dialogue between Chinese and English and the translation of articles. The accuracy of traditional statistical LM evaluation was 71.29% and 69.52%, respectively. Compared to traditional statistical LM, the accuracy of translation was higher. This showed that the recurrent NN LM reduced the number of errors in the collocation of parts of speech in MT and improved the accuracy and credibility of MT.

Expectations modulate retrieval interference during ellipsis resolution

Memory operations during language comprehension are subject to interference: retrieval is harder when items are linguistically similar to each other. We test how such interference effects might be modulated by linguistic expectations. Theories differ in how these factors might interact; we consider three possibilities: (i) predictability determines the need for retrieval, (ii) predictability affects cue-preference during retrieval, or (iii) word predictability moderates the effect of noise in memory during retrieval. We first demonstrate that expectations for a target word modulate retrieval interference in Mandarin noun-phrase ellipsis in an electroencephalography (EEG) experiment. This result obtains in globally ungrammatical sentences – termed “facilitatory interference.” Such a pattern is inconsistent with theories that focus only on the need for retrieval. To tease apart cue-preferences from noisy-memory representations, we operationalize the latter using a Transformer neural network language model. Confronting the model with our stimuli reveals an interference effect, consistent with prior work, but that effect does not interact with predictability in contrast to human EEG results. Together, these data are most consistent with the hypothesis that the predictability of target items affects cue-preferences during retrieval.

Toward enriched decoding of mandarin spontaneous speech

A deep neural network (DNN)-based automatic speech recognition (ASR) method for enriched decoding of Mandarin spontaneous speech is proposed. It adopts an enhanced approach over the baseline model built with factored time delay neural networks (TDNN-f) and rescored with RNNLM to first building a baseline system composed of a TDNN-f acoustic model (AM), a trigram language model (LM), and a recurrent neural network language model (RNNLM) to generate a word lattice. It then sequentially incorporates a multi-task Part-of-Speech-RNNLM (POS-RNNLM), a hierarchical prosodic model (HPM), and a reduplication-word LM (RLM) into the decoding process by expanding the word lattice and performing rescoring to improve recognition performance and enrich the decoding output with syntactic parameters of POS and punctuation (PM), prosodic tags of word-juncture break types and syllable prosodic states, and an edited recognition text with reduplication words being eliminated. Experimental results on the Mandarin conversational dialogue corpus (MCDC) showed that SER, CER, and WER of 13.2 %, 13.9 %, and 19.1 % were achieved when incorporating the POS-RNNLM and HPM into the baseline system. They represented relative SER, CER, and WER reductions of 7.7 %, 7.9 % and 5.0 % as comparing with those of the baseline system. Futhermore, the use of the RLM resulted in additional 3 %, 4.6 %, and 4.5 % relative SER, CER, and WER reductions through eliminating reduplication words.

An intelligent method for generating a list of job profile requirements based on neural network language models using ESCO taxonomy and online job corpus

Online recruitment systems have accumulated a huge amount of data on the real labor market in recent years. Of particular interest to the study are the data on the real requirements of the labor market contained in the texts of online vacancies, as well as the process of extracting and structuring them for further analysis and use. The stage of compiling an up-to-date list of requirements for a position profile in the recruitment process is very time-consuming and requires a large amount of effort from an HR specialist related to monitoring changes in entire industries and professions, as well as analyzing relevance of existing requirements on the market. In this article, the author proposes a conceptual model of a recommendation system that allows one to reduce the burden on an HR specialist at the stage of forming an up-to-date list of requirements for a position profile in the recruitment process. The model is based on a combination of the following components: a graph model of labor market requirements based on the ESCO taxonomy adapted for the Russian language; and an intelligent method of forming recommendations for compiling an up-to-date list of requirements in the recruitment process based on neural network models of the language on the architecture of transformers, ESCO skills taxonomy and corpus online vacancies of the Russian labor market. The article also provides a conceptual algorithm for the work of the recommendation system and possible options for recommendations on updating the list of requirements of the position profile in the recruitment process based on an analysis of the needs of the real labor market.

The impact of African swine fever news sentiment on the Korean meat market.

Our study analyzed the impact of African swine fever (ASF) news on the Korean meat market using sentiment analysis. We applied a neural network language model (NNLM) to generate a sentiment index indicating whether the news had a positive or negative impact on consumer expectations. We analyzed 24,143 news articles to estimate the impulse responses of meat price variables to sentiment shocks. Our study contributes significantly to agricultural economics as it applies NNLM to generate a sentiment index. The empirical results indicated that ASF news sentiment has a substantial impact on meat prices in Korea, and there is evidence of substitution effects among different types of meat. ASF news has a positive impact on the price of pork, negative effects on beef and chicken prices, and a greater impact on the price of chicken than beef. The findings imply that the effect of ASF news on demand outweighs its impact on supply in the pork market, whereas the effect on supply surpasses the effect on demand in the beef and chicken market. We believe our methods and results will inspire discussions among applied economists studying consumer behavior in this specific market and could encourage the application of big data analysis to the agricultural economy.

The Misinformation Susceptibility Test (MIST): A psychometrically validated measure of news veracity discernment.

Interest in the psychology of misinformation has exploded in recent years. Despite ample research, to date there is no validated framework to measure misinformation susceptibility. Therefore, we introduce Verification done, a nuanced interpretation schema and assessment tool that simultaneously considers Veracity discernment, and its distinct, measurable abilities (real/fake news detection), and biases (distrust/naïvité-negative/positive judgment bias). We then conduct three studies with seven independent samples (Ntotal = 8504) to show how to develop, validate, and apply the Misinformation Susceptibility Test (MIST). In Study 1 (N = 409) we use a neural network language model to generate items, and use three psychometric methods-factor analysis, item response theory, and exploratory graph analysis-to create the MIST-20 (20 items; completion time < 2 minutes), the MIST-16 (16 items; < 2 minutes), and the MIST-8 (8 items; < 1 minute). In Study 2 (N = 7674) we confirm the internal and predictive validity of the MIST in five national quota samples (US, UK), across 2 years, from three different sampling platforms-Respondi, CloudResearch, and Prolific. We also explore the MIST's nomological net and generate age-, region-, and country-specific norm tables. In Study 3 (N = 421) we demonstrate how the MIST-in conjunction with Verification done-can provide novel insights on existing psychological interventions, thereby advancing theory development. Finally, we outline the versatile implementations of the MIST as a screening tool, covariate, and intervention evaluation framework. As all methods are transparently reported and detailed, this work will allow other researchers to create similar scales or adapt them for any population of interest.

What should be encoded by position embedding for neural network language models?

AbstractWord order is one of the most important grammatical devices and the basis for language understanding. However, as one of the most popular NLP architectures, Transformer does not explicitly encode word order. A solution to this problem is to incorporate position information by means of position encoding/embedding (PE). Although a variety of methods of incorporating position information have been proposed, the NLP community is still in want of detailed statistical researches on position information in real-life language. In order to understand the influence of position information on the correlation between words in more detail, we investigated the factors that affect the frequency of words and word sequences in large corpora. Our results show that absolute position, relative position, being at one of the two ends of a sentence and sentence length all significantly affect the frequency of words and word sequences. Besides, we observed that the frequency distribution of word sequences over relative position carries valuable grammatical information. Our study suggests that in order to accurately capture word–word correlations, it is not enough to focus merely on absolute and relative position. Transformers should have access to more types of position-related information which may require improvements to the current architecture.

Increased top-down semantic processing in natural speech linked to better reading in dyslexia

Early research proposed that individuals with developmental dyslexia use contextual information to facilitate lexical access and compensate for phonological deficits. Yet at present there is no corroborating neuro-cognitive evidence. We explored this with a novel combination of magnetoencephalography (MEG), neural encoding and grey matter volume analyses. We analysed MEG data from 41 adult native Spanish speakers (14 with dyslexic symptoms) who passively listened to naturalistic sentences. We used multivariate Temporal Response Function analysis to capture online cortical tracking of both auditory (speech envelope) and contextual information. To compute contextual information tracking we used word-level Semantic Surprisal derived using a Transformer Neural Network language model. We related online information tracking to participants’ reading scores and grey matter volumes within the reading-linked cortical network. We found that right hemisphere envelope tracking was related to better phonological decoding (pseudoword reading) for both groups, with dyslexic readers performing worse overall at this task. Consistently, grey matter volume in the superior temporal and bilateral inferior frontal areas increased with better envelope tracking abilities. Critically, for dyslexic readers only, stronger Semantic Surprisal tracking in the right hemisphere was related to better word reading. These findings further support the notion of a speech envelope tracking deficit in dyslexia and provide novel evidence for top-down semantic compensatory mechanisms.

Examining Homophily, Language Coordination, and Analytical Thinking in Web-Based Conversations About Vaccines on Reddit: Study Using Deep Neural Network Language Models and Computer-Assisted Conversational Analyses

Vaccine hesitancy has been deemed one of the top 10 threats to global health. Antivaccine information on social media is a major barrier to addressing vaccine hesitancy. Understanding how vaccine proponents and opponents interact with each other on social media may help address vaccine hesitancy. We aimed to examine conversations between vaccine proponents and opponents on Reddit to understand whether homophily in web-based conversations impedes opinion exchange, whether people are able to accommodate their languages to each other in web-based conversations, and whether engaging with opposing viewpoints stimulates higher levels of analytical thinking. We analyzed large-scale conversational text data about human vaccines on Reddit from 2016 to 2018. Using deep neural network language models and computer-assisted conversational analyses, we obtained each Redditor's stance on vaccines, each post's stance on vaccines, each Redditor's language coordination score, and each post or comment's analytical thinking score. We then performed chi-square tests, 2-tailed t tests, and multilevel modeling to test 3 questions of interest. The results show that both provaccine and antivaccine Redditors are more likely to selectively respond to Redditors who indicate similar views on vaccines (P<.001). When Redditors interact with others who hold opposing views on vaccines, both provaccine and antivaccine Redditors accommodate their language to out-group members (provaccine Redditors: P=.044; antivaccine Redditors: P=.047) and show no difference in analytical thinking compared with interacting with congruent views (P=.63), suggesting that Redditors do not engage in motivated reasoning. Antivaccine Redditors, on average, showed higher analytical thinking in their posts and comments than provaccine Redditors (P<.001). This study shows that although vaccine proponents and opponents selectively communicate with their in-group members on Reddit, they accommodate their language and do not engage in motivated reasoning when communicating with out-group members. These findings may have implications for the design of provaccine campaigns on social media.

A deep hierarchy of predictions enables online meaning extraction in a computational model of human speech comprehension

Understanding speech requires mapping fleeting and often ambiguous soundwaves to meaning. While humans are known to exploit their capacity to contextualize to facilitate this process, how internal knowledge is deployed online remains an open question. Here, we present a model that extracts multiple levels of information from continuous speech online. The model applies linguistic and nonlinguistic knowledge to speech processing, by periodically generating top-down predictions and incorporating bottom-up incoming evidence in a nested temporal hierarchy. We show that a nonlinguistic context level provides semantic predictions informed by sensory inputs, which are crucial for disambiguating among multiple meanings of the same word. The explicit knowledge hierarchy of the model enables a more holistic account of the neurophysiological responses to speech compared to using lexical predictions generated by a neural network language model (GPT-2). We also show that hierarchical predictions reduce peripheral processing via minimizing uncertainty and prediction error. With this proof-of-concept model, we demonstrate that the deployment of hierarchical predictions is a possible strategy for the brain to dynamically utilize structured knowledge and make sense of the speech input.

Learned Text Representation for Amharic Information Retrieval and Natural Language Processing

Over the past few years, word embeddings and bidirectional encoder representations from transformers (BERT) models have brought better solutions to learning text representations for natural language processing (NLP) and other tasks. Many NLP applications rely on pre-trained text representations, leading to the development of a number of neural network language models for various languages. However, this is not the case for Amharic, which is known to be a morphologically complex and under-resourced language. Usable pre-trained models for automatic Amharic text processing are not available. This paper presents an investigation on the essence of learned text representation for information retrieval and NLP tasks using word embeddings and BERT language models. We explored the most commonly used methods for word embeddings, including word2vec, GloVe, and fastText, as well as the BERT model. We investigated the performance of query expansion using word embeddings. We also analyzed the use of a pre-trained Amharic BERT model for masked language modeling, next sentence prediction, and text classification tasks. Amharic ad hoc information retrieval test collections that contain word-based, stem-based, and root-based text representations were used for evaluation. We conducted a detailed empirical analysis on the usability of word embeddings and BERT models on word-based, stem-based, and root-based corpora. Experimental results show that word-based query expansion and language modeling perform better than stem-based and root-based text representations, and fastText outperforms other word embeddings on word-based corpus.

Towards POI-based large-scale land use modeling: spatial scale, semantic granularity, and geographic context

ABSTRACT The combination of spatial distribution, semantic characteristics, and sometimes temporal dynamics of POIs inside a geographic region can capture its unique land use characteristics. Most previous studies on POI-based land use modeling research focused on one geographic region and select one spatial scale and semantic granularity for land use characterization. There is a lack of understanding on the impact of spatial scale, semantic granularity, and geographic context on POI-based land use modeling, particularly large-scale land use modeling. In this study, we developed a scalable POI-based land use modeling framework and examined the impact of these three factors on POI-based land use characterization using data from three geographic regions. We developed a unified semantic representation framework for POI semantics that can help fuse heterogeneous POI data sources. Then, by combining POIs with a neural network language model, we developed a spatially explicit approach to learn the embedding representation of POIs and AOIs. We trained multiple supervised classifiers using AOI embeddings as input features to predict AOI land use at different semantic granularities. The classification performance of different land use classes was analyzed and compared across three geographic regions to identify the semantic representativeness of POI-based AOI embedding and the impact of geographic context.