Neural Network Language Model Research Articles

Neural Network Language Models, Mutual Sense Affinity, and Short Text Ranking According to Closeness to a Semantic Standard

Neural Network Language Models, Mutual Sense Affinity, and Short Text Ranking According to Closeness to a Semantic Standard

An abstract linguistic space for transmitting information from one mind to another

In this issue of Neuron, Zada etal.1 examine how linguistic information flows from a speaker's brain to a listener's brain during face-to-face spontaneous conversation. The authors use intracranial recordings from five pairs of epilepsy patients and neural network language models to establish the existence of an abstract, linguistic space that is shared during conversation.

Research on Leisure Tourism Destination Image Innovation Driven by Artificial Intelligence

As the public’s demand for tourism experience continues to improve, the demand for tourism products gradually develops in the direction of personalization, customization and differentiation. In order to mine the image of leisure tourism destinations, this paper conducts theme mining on the review texts of leisure tourism destinations by means of LDA model. An improved neural network language model CBOW and a Glove model are used to train the review word vectors of leisure tourism destination texts. Based on the deep fully connected neural network model, the perceptual features of tourist destinations are studied and analyzed. Taking Sanya tourist attractions as a research case, by analyzing the perception of tourist attractions, the cognitive image vocabulary of Sanya tourist attractions includes three major categories. Among them, tourism attractions, tourism infrastructure and services, tourism service atmosphere, and tourists’ travel behavior accounted for 53.1%, 22.3%, 5.1%, and 19.5% respectively. Then the emotional vocabulary in the perception of tourism destination image in Sanya City was categorized in terms of positive, neutral and negative, and from the neutral vocabulary, the word frequency number of words such as waiting (101), understanding (185), fun (61), and relaxation (56) was high. Regression analysis was used to explore the factors influencing the effect of tourists’ perception of destination image. Among them, tangibility (0.43), reliability (0.463), and responsiveness (0.434) were positively and significantly related to tourist satisfaction. Improving the image of leisure tourism destination can focus on these three aspects.

Language Model Method for Collocation Rules of Parts of Speech in Machine Translation System

With the development of the times, modern society has now entered the Internet of Things (IoT) information age and Machine Translation (MT) plays an important role in increasingly frequent cross-language communication. In recent years, China's artificial intelligence industry has been in a stage of rapid construction, and the scale of its core industries has grown explosively, and a large number of artificial intelligence companies, including issuers, have emerged. Part of speech has always been a major problem in MT. One of the reasons is that there are a large number of multi-category words in Chinese and a large number of polysemy words in English, so part of speech collocation problems account for a large proportion of MT errors, which to some extent affects the credibility and accuracy of the translation. To reduce the error problem in MT of part of speech collocation, this paper used Machine Learning (ML) methods to study the Language Model (LM) of part of speech collocation based on recurrent neural network (NN) and compared it with the traditional statistical LM. In terms of the accuracy rate of the two LMS in the automatic evaluation index of machine translation, the experimental results show that the recursive NN LM established by the ML method had an accuracy rate of 80.42% and 83.57%, respectively, for the part-of-speech matching rules of the IoT machine translation system in the dialogue between Chinese and English and the translation of articles. The accuracy of traditional statistical LM evaluation was 71.29% and 69.52%, respectively. Compared to traditional statistical LM, the accuracy of translation was higher. This showed that the recurrent NN LM reduced the number of errors in the collocation of parts of speech in MT and improved the accuracy and credibility of MT.

A Dual-Stream Cross AGFormer-GPT Network for Traffic Flow Prediction Based on Large-Scale Road Sensor Data.

Traffic flow prediction can provide important reference data for managers to maintain traffic order, and can also be based on personal travel plans for optimal route selection. On account of the development of sensors and data collection technology, large-scale road network historical data can be effectively used, but their high non-linearity makes it meaningful to establish effective prediction models. In this regard, this paper proposes a dual-stream cross AGFormer-GPT network with prompt engineering for traffic flow prediction, which integrates traffic occupancy and speed as two prompts into traffic flow in the form of cross-attention, and uniquely mines spatial correlation and temporal correlation information through the dual-stream cross structure, effectively combining the advantages of the adaptive graph neural network and large language model to improve prediction accuracy. The experimental results on two PeMS road network data sets have verified that the model has improved by about 1.2% in traffic prediction accuracy under different road networks.

Fuzzing JavaScript engines with a syntax-aware neural program model

Neural network language modeling has become a remarkable approach in the generation of test cases for fuzzing JavaScript engines. Fuzzers built upon neural language models offer several advantages. They obviate the need for manually developing code generation rules, enable the extraction of patterns from high-quality seed sets, and exhibit commendable portability. Nevertheless, existing works confront challenges in three key aspects: diminished language modeling performance attributable to extensive vocabularies, potential semantic errors within generated test cases, and the limitation of black-box fuzzing, which fails to leverage the internal feedback from the target engine.This paper proposes an innovative neural model-based grey-box fuzzing approach for JavaScript engines. We incorporate the context-free grammar of JavaScript into the neural language model to mitigate the challenges associated with extensive vocabularies, thereby enhancing the model’s performance. Furthermore, to enhance the semantic validity of the generated test cases, we introduce semantic constraints into the mutation process. Notably, this work pioneers the integration of grey-box testing into a fuzzer built upon a neural language model, thereby enhancing the exploration of deep paths. Our prototype, PMFuzz, surpasses NNLM-based counterparts in both language modeling performance and test case generation capabilities. PMFuzz demonstrates a high level of competitiveness in exploring the software state space when compared to traditional coverage-guided grey-box fuzzers. In our evaluation, PMFuzz successfully identified 20 new defects within mainstream JS engines. Eight of them have been confirmed and fixed. Moreover, upon applying our method to C compilers, PMFuzz has revealed 11 new defects.

Educational Resource Material Translation System

The proposed "Educational Resource Material Translation System" represents a significant leap in bridging language barriers, particularly for individuals who lack proficiency in English. By leveraging advanced technologies such as Python, Flask, Recurrent Neural Network (RNN) and Large Language Models (LLM), the system offers a robust solution for translating educational content from English to prominent Indian regional languages like Kannada, Telugu and Hindi. Its user- friendly web interface, developed with Flask, ensures a seamless and intuitive translation experience for users of varying technical backgrounds. Beyond mere language translation, this project facilitates cross- cultural communication and enhances accessibility to educational resources, thereby empowering individuals and businesses alike. The utilization of Machine Learning further underscores the system's adaptability and potential for delivering precise and efficient translations. In essence, the "Language Translator" not only serves as a valuable tool for linguistic inclusivity but also fosters knowledge dissemination and cultural exchange on a broader scale.

Lexical-Semantic Content, Not Syntactic Structure, Is the Main Contributor to ANN-Brain Similarity of fMRI Responses in the Language Network.

Representations from artificial neural network (ANN) language models have been shown to predict human brain activity in the language network. To understand what aspects of linguistic stimuli contribute to ANN-to-brain similarity, we used an fMRI data set of responses to n = 627 naturalistic English sentences (Pereira et al., 2018) and systematically manipulated the stimuli for which ANN representations were extracted. In particular, we (i) perturbed sentences' word order, (ii) removed different subsets of words, or (iii) replaced sentences with other sentences of varying semantic similarity. We found that the lexical-semantic content of the sentence (largely carried by content words) rather than the sentence's syntactic form (conveyed via word order or function words) is primarily responsible for the ANN-to-brain similarity. In follow-up analyses, we found that perturbation manipulations that adversely affect brain predictivity also lead to more divergent representations in the ANN's embedding space and decrease the ANN's ability to predict upcoming tokens in those stimuli. Further, results are robust as to whether the mapping model is trained on intact or perturbed stimuli and whether the ANN sentence representations are conditioned on the same linguistic context that humans saw. The critical result-that lexical-semantic content is the main contributor to the similarity between ANN representations and neural ones-aligns with the idea that the goal of the human language system is to extract meaning from linguistic strings. Finally, this work highlights the strength of systematic experimental manipulations for evaluating how close we are to accurate and generalizable models of the human language network.

Artificial Neural Network Language Models Predict Human Brain Responses to Language Even After a Developmentally Realistic Amount of Training.

Artificial neural networks have emerged as computationally plausible models of human language processing. A major criticism of these models is that the amount of training data they receive far exceeds that of humans during language learning. Here, we use two complementary approaches to ask how the models' ability to capture human fMRI responses to sentences is affected by the amount of training data. First, we evaluate GPT-2 models trained on 1 million, 10 million, 100 million, or 1 billion words against an fMRI benchmark. We consider the 100-million-word model to be developmentally plausible in terms of the amount of training data given that this amount is similar to what children are estimated to be exposed to during the first 10 years of life. Second, we test the performance of a GPT-2 model trained on a 9-billion-token dataset to reach state-of-the-art next-word prediction performance on the human benchmark at different stages during training. Across both approaches, we find that (i) the models trained on a developmentally plausible amount of data already achieve near-maximal performance in capturing fMRI responses to sentences. Further, (ii) lower perplexity-a measure of next-word prediction performance-is associated with stronger alignment with human data, suggesting that models that have received enough training to achieve sufficiently high next-word prediction performance also acquire representations of sentences that are predictive of human fMRI responses. In tandem, these findings establish that although some training is necessary for the models' predictive ability, a developmentally realistic amount of training (∼100 million words) may suffice.

Alice's Tales: the transformation of the composition, fairy-tale formulas and contexts of the voice assistant in the skill "Let's come up with"

The subject of the study is a spontaneously generated text by the voice assistant Alice when it is creating a fairy tale together with the user, and the purpose of the study is the transformation of the structure, fairy-tale formulas and context in terms of the selection of linguistic elements and meanings using artificial intelligence technology. Particular attention has focused on the skill "Let's make it up", which became available to users in the spring of 2023. The collision and interaction of folklore canons with the realities of the 21st century give rise to an ambiguous reaction to the interactive opportunity to play the role of a storyteller together with a voice assistant. The main research method was a continuous sample, which was used to distribute the steps, stages and actions when it is creating a fairy-tale plot together with a voice assistant. In addition, methods such as comparative and contextual analyses were used to identify similarities and differences between traditional Russian fairy tales and a spontaneously generated fairy tale plot. To obtain the data and subsequent analysis of the components, a linguistic experiment with the voice assistant Alice from Yandex was conducted and described. The rapid development of neural network language models allows us to talk about the scientific novelty of the material under study, since this area is unexplored and is being modified too quickly. It is important to emphasize that to date, the texts of spontaneously generated fairy-tale, their structural division and the correspondence of fairy-tale formulas in them to folklore canons have not been studied. The main conclusion of the study is that the user's share in creating a fairy tale with the voice assistant Alice is greatly exaggerated.

Machine learning drug discovery based on graph neural network and large language model

The COVID-19 pandemic has presented an urgent need to understand the long-term health implications faced by survivors. Post-COVID-19 complications, such as acute kidney injury, arrhythmia, and stroke, pose significant challenges to public health. Despite extensive research on COVID-19 complications, a comprehensive understanding of the risk factors remains elusive due to the potential confounding variables present in the data. Traditional statistical models, while insightful, may not fully capture the causal relationships between these risk factors and post-COVID-19 complications. Motivated by this gap in the literature, we propose a novel approach using causal inference models to predict the likelihood of post-COVID-19 complications based on patient demographics and pre-existing conditions. Our model, trained on a dataset of COVID-19 inpatients in Wuhan Province, China, estimates the causal effect of these factors on the likelihood of patients experiencing post-COVID-19 complications. This approach allows us to isolate the causal impact of each factor while accounting for potential confounders, providing a more accurate understanding of the underlying mechanisms driving these relationships. Unlike traditional models that predict the probability of certain outcomes, our model provides insights into the causal relationships between risk factors and complications, offering a more reliable and comprehensive understanding of the underlying mechanisms. This approach can help identify at-risk patients, inform targeted interventions, and contribute to the development of effective prevention and treatment strategies. Our work aims to contribute to the current understanding of the virus and inform public health policies and interventions.

The Construction of “Interactive” English Translation Teaching Mode in the Background of Internet

Abstract This paper discusses the construction of “interactive” English translation teaching mode in the background of the Internet, and analyzes the application of neural machine translation model in improving the effect of translation teaching. The research adopts Word2vec algorithm to train word vectors, combines neural network language model and encoder-decoder structure, and constructs Transformer model and CNN+Transformer model to improve translation efficiency and quality. The translation experiments on sentences of different lengths show that the models perform better in short sentence translation. The experimental results show that the proposed model outperforms the traditional model in terms of BLEU value, and the best translation effect is found in test set 1, where the BLEU value is improved by 10.6 and 11.7 compared with that of the baseline model and the CNN model, respectively. The “interactive” teaching mode designed with POA theory significantly outperforms the traditional teaching mode regarding students’ lexical, grammatical and semantic translation quality. The “interactive” English translation teaching with neural translation model can effectively improve the quality of translation teaching and students’ learning effect.

Формирование лексического модуля прикладной онтологии для её обучения

The complexity of developing ontologies is associated with their learning, which involves the automation of solving problems related to the extraction of ontological concepts and relationships from natural language texts. For a created ontology used in an information system, there will most likely be a need to expand its conceptual system due to the complication or change in data processing. Subsequent application of text analysis to discover new concepts and in-corporate them into the ontology, that is, linking them with existing concepts, will require identifying the latter in sen-tences of natural language texts. This paper considers the problem of automating the formation of the lexical module of applied ontology, which includes formalized representations of ontological concepts in natural language texts. A brief review of existing works devoted to the use of lexical information about ontology components in solving problems related to the analysis of textual data is presented. The ways of using the OntoLex-Lemon model for determining the structure of lexical representations of ontology concepts are considered. A procedure is proposed for the formation of lexical representations based on the analysis of texts in the subject area, taking into account the case when concepts have names consisting of several words. The results of applying the obtained module for the automatic formation of a training set of a neural network language model used in the ontology learning task for discovering new concepts in the corpus of subject texts are presented.

Transformer-Based Language Models as Psycholinguistic Subjects: Focusing on Understanding Metaphor

Metaphor is a fundamental aspect of human language and cognition, playing a crucial role in communication, comprehension, and creative expression. In light of the recent advancements demonstrated by prominent language models, a pivotal question arises: Can these expansive language models effectively discern metaphorical knowledge? The primary objective involves comparing the surprisal values estimated from neural network language models like autoregressive and bidirectional language models to the reaction times of human when exposed to both metaphorical and literal sentences. Our secondary objective involves assessing the AI's comprehension of metaphors by utilizing the sensicality ratings generated by sophisticated ChatGPT. To achieve this, we used psycholinguistic methods, and adopted the experimental materials from Lai, Currana, and Menna (2009). We found the surprisal values estimated from the autoregressive language model demonstrate metaphor processing that closely resembles that of native speakers. Furthermore, ChatGPT's processing of conventional metaphorical sentences closely resembles its approach to literal sentences, mirroring the convergence observed in native speakers' ERP response to conventional metaphorical sentences and their alignment with that of literal sentences.

Expectations modulate retrieval interference during ellipsis resolution

Memory operations during language comprehension are subject to interference: retrieval is harder when items are linguistically similar to each other. We test how such interference effects might be modulated by linguistic expectations. Theories differ in how these factors might interact; we consider three possibilities: (i) predictability determines the need for retrieval, (ii) predictability affects cue-preference during retrieval, or (iii) word predictability moderates the effect of noise in memory during retrieval. We first demonstrate that expectations for a target word modulate retrieval interference in Mandarin noun-phrase ellipsis in an electroencephalography (EEG) experiment. This result obtains in globally ungrammatical sentences – termed “facilitatory interference.” Such a pattern is inconsistent with theories that focus only on the need for retrieval. To tease apart cue-preferences from noisy-memory representations, we operationalize the latter using a Transformer neural network language model. Confronting the model with our stimuli reveals an interference effect, consistent with prior work, but that effect does not interact with predictability in contrast to human EEG results. Together, these data are most consistent with the hypothesis that the predictability of target items affects cue-preferences during retrieval.

Toward enriched decoding of mandarin spontaneous speech

A deep neural network (DNN)-based automatic speech recognition (ASR) method for enriched decoding of Mandarin spontaneous speech is proposed. It adopts an enhanced approach over the baseline model built with factored time delay neural networks (TDNN-f) and rescored with RNNLM to first building a baseline system composed of a TDNN-f acoustic model (AM), a trigram language model (LM), and a recurrent neural network language model (RNNLM) to generate a word lattice. It then sequentially incorporates a multi-task Part-of-Speech-RNNLM (POS-RNNLM), a hierarchical prosodic model (HPM), and a reduplication-word LM (RLM) into the decoding process by expanding the word lattice and performing rescoring to improve recognition performance and enrich the decoding output with syntactic parameters of POS and punctuation (PM), prosodic tags of word-juncture break types and syllable prosodic states, and an edited recognition text with reduplication words being eliminated. Experimental results on the Mandarin conversational dialogue corpus (MCDC) showed that SER, CER, and WER of 13.2 %, 13.9 %, and 19.1 % were achieved when incorporating the POS-RNNLM and HPM into the baseline system. They represented relative SER, CER, and WER reductions of 7.7 %, 7.9 % and 5.0 % as comparing with those of the baseline system. Futhermore, the use of the RLM resulted in additional 3 %, 4.6 %, and 4.5 % relative SER, CER, and WER reductions through eliminating reduplication words.

An intelligent method for generating a list of job profile requirements based on neural network language models using ESCO taxonomy and online job corpus

Online recruitment systems have accumulated a huge amount of data on the real labor market in recent years. Of particular interest to the study are the data on the real requirements of the labor market contained in the texts of online vacancies, as well as the process of extracting and structuring them for further analysis and use. The stage of compiling an up-to-date list of requirements for a position profile in the recruitment process is very time-consuming and requires a large amount of effort from an HR specialist related to monitoring changes in entire industries and professions, as well as analyzing relevance of existing requirements on the market. In this article, the author proposes a conceptual model of a recommendation system that allows one to reduce the burden on an HR specialist at the stage of forming an up-to-date list of requirements for a position profile in the recruitment process. The model is based on a combination of the following components: a graph model of labor market requirements based on the ESCO taxonomy adapted for the Russian language; and an intelligent method of forming recommendations for compiling an up-to-date list of requirements in the recruitment process based on neural network models of the language on the architecture of transformers, ESCO skills taxonomy and corpus online vacancies of the Russian labor market. The article also provides a conceptual algorithm for the work of the recommendation system and possible options for recommendations on updating the list of requirements of the position profile in the recruitment process based on an analysis of the needs of the real labor market.

The impact of African swine fever news sentiment on the Korean meat market.

Our study analyzed the impact of African swine fever (ASF) news on the Korean meat market using sentiment analysis. We applied a neural network language model (NNLM) to generate a sentiment index indicating whether the news had a positive or negative impact on consumer expectations. We analyzed 24,143 news articles to estimate the impulse responses of meat price variables to sentiment shocks. Our study contributes significantly to agricultural economics as it applies NNLM to generate a sentiment index. The empirical results indicated that ASF news sentiment has a substantial impact on meat prices in Korea, and there is evidence of substitution effects among different types of meat. ASF news has a positive impact on the price of pork, negative effects on beef and chicken prices, and a greater impact on the price of chicken than beef. The findings imply that the effect of ASF news on demand outweighs its impact on supply in the pork market, whereas the effect on supply surpasses the effect on demand in the beef and chicken market. We believe our methods and results will inspire discussions among applied economists studying consumer behavior in this specific market and could encourage the application of big data analysis to the agricultural economy.

The Misinformation Susceptibility Test (MIST): A psychometrically validated measure of news veracity discernment.

Interest in the psychology of misinformation has exploded in recent years. Despite ample research, to date there is no validated framework to measure misinformation susceptibility. Therefore, we introduce Verification done, a nuanced interpretation schema and assessment tool that simultaneously considers Veracity discernment, and its distinct, measurable abilities (real/fake news detection), and biases (distrust/naïvité-negative/positive judgment bias). We then conduct three studies with seven independent samples (Ntotal = 8504) to show how to develop, validate, and apply the Misinformation Susceptibility Test (MIST). In Study 1 (N = 409) we use a neural network language model to generate items, and use three psychometric methods-factor analysis, item response theory, and exploratory graph analysis-to create the MIST-20 (20 items; completion time < 2 minutes), the MIST-16 (16 items; < 2 minutes), and the MIST-8 (8 items; < 1 minute). In Study 2 (N = 7674) we confirm the internal and predictive validity of the MIST in five national quota samples (US, UK), across 2 years, from three different sampling platforms-Respondi, CloudResearch, and Prolific. We also explore the MIST's nomological net and generate age-, region-, and country-specific norm tables. In Study 3 (N = 421) we demonstrate how the MIST-in conjunction with Verification done-can provide novel insights on existing psychological interventions, thereby advancing theory development. Finally, we outline the versatile implementations of the MIST as a screening tool, covariate, and intervention evaluation framework. As all methods are transparently reported and detailed, this work will allow other researchers to create similar scales or adapt them for any population of interest.

What should be encoded by position embedding for neural network language models?

AbstractWord order is one of the most important grammatical devices and the basis for language understanding. However, as one of the most popular NLP architectures, Transformer does not explicitly encode word order. A solution to this problem is to incorporate position information by means of position encoding/embedding (PE). Although a variety of methods of incorporating position information have been proposed, the NLP community is still in want of detailed statistical researches on position information in real-life language. In order to understand the influence of position information on the correlation between words in more detail, we investigated the factors that affect the frequency of words and word sequences in large corpora. Our results show that absolute position, relative position, being at one of the two ends of a sentence and sentence length all significantly affect the frequency of words and word sequences. Besides, we observed that the frequency distribution of word sequences over relative position carries valuable grammatical information. Our study suggests that in order to accurately capture word–word correlations, it is not enough to focus merely on absolute and relative position. Transformers should have access to more types of position-related information which may require improvements to the current architecture.