Neural Network Language Model Research Articles

Increased top-down semantic processing in natural speech linked to better reading in dyslexia

Early research proposed that individuals with developmental dyslexia use contextual information to facilitate lexical access and compensate for phonological deficits. Yet at present there is no corroborating neuro-cognitive evidence. We explored this with a novel combination of magnetoencephalography (MEG), neural encoding and grey matter volume analyses. We analysed MEG data from 41 adult native Spanish speakers (14 with dyslexic symptoms) who passively listened to naturalistic sentences. We used multivariate Temporal Response Function analysis to capture online cortical tracking of both auditory (speech envelope) and contextual information. To compute contextual information tracking we used word-level Semantic Surprisal derived using a Transformer Neural Network language model. We related online information tracking to participants’ reading scores and grey matter volumes within the reading-linked cortical network. We found that right hemisphere envelope tracking was related to better phonological decoding (pseudoword reading) for both groups, with dyslexic readers performing worse overall at this task. Consistently, grey matter volume in the superior temporal and bilateral inferior frontal areas increased with better envelope tracking abilities. Critically, for dyslexic readers only, stronger Semantic Surprisal tracking in the right hemisphere was related to better word reading. These findings further support the notion of a speech envelope tracking deficit in dyslexia and provide novel evidence for top-down semantic compensatory mechanisms.

Examining Homophily, Language Coordination, and Analytical Thinking in Web-Based Conversations About Vaccines on Reddit: Study Using Deep Neural Network Language Models and Computer-Assisted Conversational Analyses

Vaccine hesitancy has been deemed one of the top 10 threats to global health. Antivaccine information on social media is a major barrier to addressing vaccine hesitancy. Understanding how vaccine proponents and opponents interact with each other on social media may help address vaccine hesitancy. We aimed to examine conversations between vaccine proponents and opponents on Reddit to understand whether homophily in web-based conversations impedes opinion exchange, whether people are able to accommodate their languages to each other in web-based conversations, and whether engaging with opposing viewpoints stimulates higher levels of analytical thinking. We analyzed large-scale conversational text data about human vaccines on Reddit from 2016 to 2018. Using deep neural network language models and computer-assisted conversational analyses, we obtained each Redditor's stance on vaccines, each post's stance on vaccines, each Redditor's language coordination score, and each post or comment's analytical thinking score. We then performed chi-square tests, 2-tailed t tests, and multilevel modeling to test 3 questions of interest. The results show that both provaccine and antivaccine Redditors are more likely to selectively respond to Redditors who indicate similar views on vaccines (P<.001). When Redditors interact with others who hold opposing views on vaccines, both provaccine and antivaccine Redditors accommodate their language to out-group members (provaccine Redditors: P=.044; antivaccine Redditors: P=.047) and show no difference in analytical thinking compared with interacting with congruent views (P=.63), suggesting that Redditors do not engage in motivated reasoning. Antivaccine Redditors, on average, showed higher analytical thinking in their posts and comments than provaccine Redditors (P<.001). This study shows that although vaccine proponents and opponents selectively communicate with their in-group members on Reddit, they accommodate their language and do not engage in motivated reasoning when communicating with out-group members. These findings may have implications for the design of provaccine campaigns on social media.

A deep hierarchy of predictions enables online meaning extraction in a computational model of human speech comprehension.

Understanding speech requires mapping fleeting and often ambiguous soundwaves to meaning. While humans are known to exploit their capacity to contextualize to facilitate this process, how internal knowledge is deployed online remains an open question. Here, we present a model that extracts multiple levels of information from continuous speech online. The model applies linguistic and nonlinguistic knowledge to speech processing, by periodically generating top-down predictions and incorporating bottom-up incoming evidence in a nested temporal hierarchy. We show that a nonlinguistic context level provides semantic predictions informed by sensory inputs, which are crucial for disambiguating among multiple meanings of the same word. The explicit knowledge hierarchy of the model enables a more holistic account of the neurophysiological responses to speech compared to using lexical predictions generated by a neural network language model (GPT-2). We also show that hierarchical predictions reduce peripheral processing via minimizing uncertainty and prediction error. With this proof-of-concept model, we demonstrate that the deployment of hierarchical predictions is a possible strategy for the brain to dynamically utilize structured knowledge and make sense of the speech input.

Learned Text Representation for Amharic Information Retrieval and Natural Language Processing

Over the past few years, word embeddings and bidirectional encoder representations from transformers (BERT) models have brought better solutions to learning text representations for natural language processing (NLP) and other tasks. Many NLP applications rely on pre-trained text representations, leading to the development of a number of neural network language models for various languages. However, this is not the case for Amharic, which is known to be a morphologically complex and under-resourced language. Usable pre-trained models for automatic Amharic text processing are not available. This paper presents an investigation on the essence of learned text representation for information retrieval and NLP tasks using word embeddings and BERT language models. We explored the most commonly used methods for word embeddings, including word2vec, GloVe, and fastText, as well as the BERT model. We investigated the performance of query expansion using word embeddings. We also analyzed the use of a pre-trained Amharic BERT model for masked language modeling, next sentence prediction, and text classification tasks. Amharic ad hoc information retrieval test collections that contain word-based, stem-based, and root-based text representations were used for evaluation. We conducted a detailed empirical analysis on the usability of word embeddings and BERT models on word-based, stem-based, and root-based corpora. Experimental results show that word-based query expansion and language modeling perform better than stem-based and root-based text representations, and fastText outperforms other word embeddings on word-based corpus.

Towards POI-based large-scale land use modeling: spatial scale, semantic granularity, and geographic context

ABSTRACT The combination of spatial distribution, semantic characteristics, and sometimes temporal dynamics of POIs inside a geographic region can capture its unique land use characteristics. Most previous studies on POI-based land use modeling research focused on one geographic region and select one spatial scale and semantic granularity for land use characterization. There is a lack of understanding on the impact of spatial scale, semantic granularity, and geographic context on POI-based land use modeling, particularly large-scale land use modeling. In this study, we developed a scalable POI-based land use modeling framework and examined the impact of these three factors on POI-based land use characterization using data from three geographic regions. We developed a unified semantic representation framework for POI semantics that can help fuse heterogeneous POI data sources. Then, by combining POIs with a neural network language model, we developed a spatially explicit approach to learn the embedding representation of POIs and AOIs. We trained multiple supervised classifiers using AOI embeddings as input features to predict AOI land use at different semantic granularities. The classification performance of different land use classes was analyzed and compared across three geographic regions to identify the semantic representativeness of POI-based AOI embedding and the impact of geographic context.

Predictive Coding or Just Feature Discovery? An Alternative Account of Why Language Models Fit Brain Data.

Many recent studies have shown that representations drawn from neural network language models are extremely effective at predicting brain responses to natural language. But why do these models work so well? One proposed explanation is that language models and brains are similar because they have the same objective: to predict upcoming words before they are perceived. This explanation is attractive because it lends support to the popular theory of predictive coding. We provide several analyses that cast doubt on this claim. First, we show that the ability to predict future words does not uniquely (or even best) explain why some representations are a better match to the brain than others. Second, we show that within a language model, representations that are best at predicting future words are strictly worse brain models than other representations. Finally, we argue in favor of an alternative explanation for the success of language models in neuroscience: These models are effective at predicting brain responses because they generally capture a wide variety of linguistic phenomena.

Analysis of neural network language models for solving problems of text data processing

Analysis of neural network language models for solving problems of text data processing

Accuracy Analysis of the End-to-End Extraction of Related Named Entities from Russian Drug Review Texts by Modern Approaches Validated on English Biomedical Corpora

An extraction of significant information from Internet sources is an important task of pharmacovigilance due to the need for post-clinical drugs monitoring. This research considers the task of end-to-end recognition of pharmaceutically significant named entities and their relations in texts in natural language. The meaning of “end-to-end” is that both of the tasks are performed within a single process on the “raw” text without annotation. The study is based on the current version of the Russian Drug Review Corpus—a dataset of 3800 review texts from the Russian segment of the Internet. Currently, this is the only corpus in the Russian language appropriate for research of the mentioned type. We estimated the accuracy of the recognition of the pharmaceutically significant entities and their relations in two approaches based on neural-network language models. The first core approach is to sequentially solve tasks of named-entities recognition and relation extraction (the sequential approach). The second one solves both tasks simultaneously with a single neural network (the joint approach). The study includes a comparison of both approaches, along with the hyperparameters selection to maximize resulting accuracy. It is shown that both approaches solve the target task at the same level of accuracy: 52–53% macro-averaged F1-score, which is the current level of accuracy for “end-to-end” tasks on the Russian language. Additionally, the paper presents the results for English open datasets ADE and DDI based on the joint approach, and hyperparameter selection for the modern domain-specific language models. The result is that the achieved accuracies of 84.2% (ADE) and 73.3% (DDI) are comparable or better than other published results for the datasets.

ТЕМАТИЧЕСКОЕ МОДЕЛИРОВАНИЕ И СУММАРИЗАЦИЯ ТЕКСТОВ В ОБЛАСТИ КИБЕРБЕЗОПАСНОСТИ

Purpose: improving the quality of text document analysis through the use of machine learning and intellectual analysis models in summarizing and topic modeling tasks, which will reduce the load on an expert who analyzes and generalizes significant volumes of semi-structured text data on information security topics from various sources. Methods: machine learning methods were used for the operational processing and analysis of large volumes of heterogeneous ill-structured information in natural language (NL). Methods of thematic modeling and summarization of texts based on deep neural networks, including neural network language models based on the architecture of transformers, are applied. Practical relevance: the main stages of the machine procedure of thematic modeling and summarization of professional texts in the field of information security are highlighted. The results of a comparative evaluation of the effectiveness of using clustering models, latent semantic analysis, Fast Text, Text Rank language models and BERT transformers for these purposes are presented. Recommendations are given regarding the prospects for the practical application of these models as a means of intellectual support for the professional activities of cybersecurity specialists. Scientific novelty: a complex of machine learning models for thematic modeling and summarization of professional texts is proposed, based on neural network models of attachments and transformer models, characterized by an algorithm for preparing a corpus of texts for training models and the use of a learning transfer algorithm, which will increase the efficiency of analysis and generalization of domain-specific corpora of texts.

Machine-Learning-Assisted Discovery of High-Efficient Oxygen Evolution Electrocatalysts

Iridium oxide (IrO2) is the predominant electrocatalyst for the oxygen evolution reaction (OER), but its low efficiency and high cost limit its applications. In this work, we have developed a strategy by combination of high-throughput density functional theory (DFT) and machine learning (ML) techniques for material discovery on IrO2-based electrocatalysts with enhanced OER activity. A total of 36 kinds of metal dopants are considered to substitute for Ir to form binary and ternary metal oxides, and the most stable surface structures are selected from a total of 4648 structures for OER activity evaluation. Utilizing the neural network language model (NNLM), we associate the atomic environment with the formation energies of crystals and free energies of OER intermediates, and finally a series of potential candidates have been screened as the superior OER catalysts. Our strategy could efficiently explore promising electrocatalysts, especially for evaluating complex multi-metallic compounds.

Mining of risk perception dimensions of Chinese tourists' outbound tourism based on word vector method.

Safety is the lifeline of tourism development. The article's goal is to examin how Chinese tourists perceive risk when travelling aboard. In order to create the initial corpus, this study first uses "outbound tourism"as the key word to crawl the question and answer (hereinafter referred to as "Q & A") data from 4 Chinese online travel operator platforms, then preprocesses the "Q & A" data in Python. Secondly, after being extracted, the feature words are converted into the word vector model using the word vector method based on neural network language model. Finally, the word vectors are clustered and classified. It is found that there are six dimensions of risk perception of Chinese tourists' outbound tourism, namely traffic risk, planning risk, service risk, communication risk, financial risk and functional risk. Important and practical information for government and tourism enterprises is provided to accurately grasp the risk perception of Chinese tourists' outbound tourism and continuously improve the supply of tourism risk information.

Morphology aware data augmentation with neural language models for online hybrid ASR

AbstractRecognition of Hungarian conversational telephone speech is challenging due to the informal style and morphological richness of the language. Neural Network Language Models (NNLMs) can provide remedy for the high perplexity of the task; however, their high complexity makes them very difficult to apply in the first (single) pass of an online system. Recent studies showed that a considerable part of the knowledge of NNLMs can be transferred to traditional n-grams by using neural text generation based data augmentation. Data augmentation with NNLMs works well for isolating languages; however, we show that it causes a vocabulary explosion in a morphologically rich language. Therefore, we propose a new, morphology aware neural text augmentation method, where we retokenize the generated text into statistically derived subwords. We compare the performance of word-based and subword-based data augmentation techniques with recurrent and Transformer language models and show that subword-based methods can significantly improve the Word Error Rate (WER) while greatly reducing vocabulary size and memory requirements. Combining subword-based modeling and neural language model-based data augmentation, we were able to achieve 11% relative WER reduction and preserve real-time operation of our conversational telephone speech recognition system. Finally, we also demonstrate that subword-based neural text augmentation outperforms the word-based approach not only in terms of overall WER but also in recognition of Out-of-Vocabulary (OOV) words.

Supply chain disruptions and labor shortages: COVID in perspective

We use a neural network language model and dependency parsing to extract textual information about shortages from US corporate filings. This allows us to study the COVID pandemic in the broader context of other macroeconomic environments over the past 25 years. Shortages are much more prevalent during the pandemic than they are during other times, especially in the goods sector and for intermediate inputs. Firms also report significantly more increases in costs. The largest number of shortages occurs at the very end of our sample, in the first half of 2022. Our data are consistent with several known economic facts that are unrelated to the pandemic. • We use a neural network language model and dependency parsing to extract textual information about shortages from US corporate filings. • Shortages are much more prevalent during the pandemic than they are during other times. • This is most pronounced in the goods sector and for intermediate inputs. • Shortages are most prevalent at the very end of our sample, in the first half of 2022. • Our data are consistent with several known economic facts that are unrelated to the pandemic.

Differentiable [formula omitted]-gram objective on abstractive summarization

ROUGE is a standard automatic evaluation metric based on N-gram for sequence-to-sequence tasks like abstractive summarization, while cross-entropy loss is an essential objective that optimizes at unigram level for neural network language models. In this paper we present differentiable N-gram objectives, attempting to alleviate the discrepancy between training and evaluating criteria. The novelty of our work is the objective does not ceil the number of matched sub-sequences by the ground truth count of N-gram in reference sequence and weights the matched sub-sequences equally. Therefore, our proposed objective can maximize the probabilistic weight of matched sub-sequences. We jointly optimize cross-entropy loss and the objective, providing decent evaluation scores enhancement including ROUGE over abstractive summarization datasets CNN/DM and XSum, outperforming competitive N-gram objectives.

Do fake online comments pose a threat to regulatory policymaking? Evidence from Internet regulation in the United States

AbstractBroad stakeholder participation in regulatory policymaking via online commenting platforms has become the norm in many advanced democracies around the world. In recent years, a policy debate has emerged over the dangers posed to the process by fake comments that impersonate ordinary citizens. This paper helps to clarify the terms of this debate by evaluating a contentious and prominent case in the United States, the nearly 24 million comments from the Federal Communications Commission's 2017 Restoring Internet Freedom proceeding. Using a two‐step methodology that combines a computationally efficient search algorithm and a neural network language model, I show that regulators were able to cite much of the relevant information submitted in public comments by relying on longstanding methods of information gathering through interest groups, despite the fact that fake comments outnumbered others 3 to 1. The results suggest that fake comments did not impede regulators' ability to extract specialized information from the public consultation process, but may have distorted signals from mass comment campaigns about constituent mobilization.

Toward the nature of automatic associations: item-level computational semantic similarity and IAT-based alcohol-valence associations

Automatic associations involving alcohol have been proposed to play a role in drinking behavior. Such associations are often assessed using implicit measures such as the Implicit Association Test (IAT). Neural network language models provide computational measures of semantic relationships between words. These model-based measures could be related to behavioral alcohol-related associations as observed using the IAT. If so, this could provide a step toward better understanding of the nature of automatic associations and their relationship to behavior. The current study therefore aimed to test whether there is a systematic covariation over items between model-based and behavior-based associations. Analyses were performed for two single-target IATs from a previously published study. One task involved alcohol versus nonalcohol drinks and positive associates, and the other alcohol versus nonalcohol drinks and negative associates. The GenSim library and a pretrained word2vec model were used to calculate a relative computational association between specific items from the positive and negative categories, respectively, and the alcohol versus nonalcohol word sets. In both tasks, a significant covariance between items’ computational and behavioral measures of association was found over participants. The results thus add to the information on the relationship between neural network language models and psychological associations. They may provide methodological strategies for task design and data analysis. Models of semantic associations connect computational linguistics and social-cognitive psychology and may provide a theoretical link between measures of alcohol-related associations using verbal stimuli and alcohol-related cognition and behaviors.

Serum Raman spectroscopy combined with convolutional neural network for rapid diagnosis of HER2-positive and triple-negative breast cancer

Breast cancer is common in women, and its number of patients ranks first among female malignant tumors. Breast cancer is highly heterogeneous, and different types of breast cancer have different biological behaviors and prognoses. Therefore, identifying the different types of breast cancer is of great help in formulating individualized treatment plans. Based on serum Raman spectroscopy and deep learning algorithms, we propose a fast and low-cost diagnosis method for screening triple-negative breast cancer, human epidermal growth factor receptor 2 (HER2)-positive breast cancer, and healthy controls. We collected 75 serum samples in this study, including 23 triple-negative breast cancers, 22 HER2-positive breast cancers, and 30 healthy controls. Using the preprocessed Raman spectra as the input of deep learning, three deep learning models, neural network language model (NNLM), bidirectional long-short-term memory network (BiLSTM), and convolutional neural network (CNN), were established, and the accuracy rates of the three models were 87.78%, 90.37%, and 91.11%, respectively. The experimental results demonstrate the feasibility of serum Raman spectroscopy combined with deep learning algorithms to diagnose breast cancer, which can be used as an effective auxiliary diagnosis method for breast cancer.

Non-Life Insurance Risk Classification Using Categorical Embedding

This article presents several actuarial applications of categorical embedding in the context of non-life insurance risk classification. In non-life insurance, many rating factors are naturally categorical, and often the categorical variables have a large number of levels. The high cardinality of categorical rating variables presents challenges in the implementation of traditional actuarial methods. Categorical embedding that is proposed in the machine learning literature for handling categorical variables has recently received attention in actuarial studies. The method is inspired by the neural network language models for learning text data and maps a categorical variable into a real-valued representation in the Euclidean space. Using a property insurance claims we demonstrate the use of categorical embedding in three applications. The first shows how embeddings are used to construct rating classes and calculate rating relativities for a single insurance risk. The second concerns predictive modeling for multivariate insurance risks and emphasizes the effects of dependence on tail risks. The third focuses on pricing new products where transfer learning is used to gather knowledge from existing products.

A transformer-Based neural language model that synthesizes brain activation maps from free-form text queries.

Neuroimaging studies are often limited by the number of subjects and cognitive processes that can be feasibly interrogated. However, a rapidly growing number of neuroscientific studies have collectively accumulated an extensive wealth of results. Digesting this growing literature and obtaining novel insights remains to be a major challenge, since existing meta-analytic tools are constrained to keyword queries. In this paper, we present Text2Brain, an easy to use tool for synthesizing brain activation maps from open-ended text queries. Text2Brain was built on a transformer-based neural network language model and a coordinate-based meta-analysis of neuroimaging studies. Text2Brain combines a transformer-based text encoder and a 3D image generator, and was trained on variable-length text snippets and their corresponding activation maps sampled from 13,000 published studies. In our experiments, we demonstrate that Text2Brain can synthesize meaningful neural activation patterns from various free-form textual descriptions. Text2Brain is available at https://braininterpreter.com as a web-based tool for efficiently searching through the vast neuroimaging literature and generating new hypotheses.

A Data-Driven Model for Automated Chinese Word Segmentation and POS Tagging.

Chinese natural language processing tasks often require the solution of Chinese word segmentation and POS tagging problems. Traditional Chinese word segmentation and POS tagging methods mainly use simple matching algorithms based on lexicons and rules. The simple matching or statistical analysis requires manual word segmentation followed by POS tagging, which leads to the inability to meet the practical requirements for label prediction accuracy. With the continuous development of deep learning technology, data-driven machine learning models provide new opportunities for automated Chinese word segmentation and POS tagging. Therefore, a data-driven automated Chinese word segmentation and POS tagging model is proposed in order to address the above problems. Firstly, the main idea and overall framework of the proposed automated model are outlined, and the tagging strategy and neural network language model used are described. Secondly, two main optimisations are made on the input side of the model: (1) the use of word2Vec for the representation of text features, thus representing the text as a distributed word vector; and (2) the use of an improved AlexNet for efficient encoding of long-range word, and the addition of an attention mechanism to the model. Finally, on the output side, an additional auxiliary loss function was designed to optimise the Chinese text based on its frequency. The experimental results show that the proposed model can significantly improve the accuracy and operational efficiency of Chinese word segmentation and POS tagging compared with other existing models, thus verifying its effectiveness and advancement.