Human Language Processing Research Articles

Understanding public opinion and discussion dynamics of digital humans on social media: an analysis of sentiment, themes and user characteristics

PurposeThe emergence of artificial intelligence-generated content (AIGC) technology has markedly enhanced the capabilities of digital human content generation and natural language processing, thus further advancing the development of digital humans. To enable enterprises and governments to effectively address the challenges and opportunities arising from the rapid development of digital humans, it is imperative to understand the public opinion and discussion dynamics of digital humans.Design/methodology/approachThis study initially analyzed the trends and distribution patterns of public attention to digital humans. By utilizing word cloud technology, we explored the primary focal points of public interest and conducted a topic analysis using latent Dirichlet allocation (LDA) techniques. Subsequently, content analysis was conducted on the popular application domains of digital humans. Finally, this study examined the influence of user characteristics on emotional scores toward digital humans and the presence of differences in focus across user groups.FindingsThe results indicate a sustained increase in public attention toward digital humans, accompanied by notable geographic disparities in the distribution of discussions. Discussions on Weibo are primarily focused on four domains, whereas areas within the digital human application domain that provoke widespread discussion include live streaming, service, cultural entertainment and digital avatars. Significant impacts of user characteristics on sentiment scores were observed, revealing divergent focal points of interest among different user groups toward digital humans.Originality/valueThrough the deep analysis of Weibo data, this study offers new insights into the digital human industry, enabling governments and businesses to understand industry trends and develop targeted digital human customization strategies based on customer characteristics.

On the influence of discourse connectives on the predictions of humans and language models.

Psycholinguistic literature has consistently shown that humans rely on a rich and organized understanding of event knowledge to predict the forthcoming linguistic input during online sentence comprehension. We, the authors, expect sentences to maintain coherence with the preceding context, making congruent sentence sequences easier to process than incongruent ones. It is widely known that discourse relations between sentences (e.g., temporal, contingency, comparison) are generally made explicit through specific particles, known as discourse connectives, (e.g., and, but, because, after). However, some relations that are easily accessible to the speakers, given their event knowledge, can also be left implicit. The goal of this paper is to investigate the importance of discourse connectives in the prediction of events in human language processing and pretrained language models, with a specific focus on concessives and contrastives, which signal to comprehenders that their event-related predictions have to be reversed. Inspired by previous work, we built a comprehensive set of story stimuli in Italian and Mandarin Chinese that differ in the plausibility and coherence of the situation being described and the presence or absence of a discourse connective. We collected plausibility judgments and reading times from native speakers for the stimuli. Moreover, we correlated the results of the experiments with the predictions given by computational modeling, using Surprisal scores obtained via Transformer-based language models. The human judgements were collected using a seven-point Likert scale and analyzed using cumulative link mixed modeling (CLMM), while the human reading times and language model surprisal scores were analyzed using linear mixed effects regression (LMER). We found that Chinese NLMs are sensitive to plausibility and connectives, although they struggle to reproduce expectation reversal effects due to a connective changing the plausibility of a given scenario; Italian results are even less aligned with human data, with no effects of either plausibility and connectives on Surprisal.

Language Learning, Representation, and Processing in Humans and Machines: Introduction to the Special Issue

Abstract Large Language Models (LLMs) and humans acquire knowledge about language without direct supervision. LLMs do so by means of specific training objectives, while humans rely on sensory experience and social interaction. This parallelism has created a feeling in NLP and cognitive science that a systematic understanding of how LLMs acquire and use the encoded knowledge, could provide useful insights for studying human cognition. Conversely, methods and findings from the field of cognitive science have occasionally inspired language model development. Yet, the differences in the way that language is processed by machines and humans—in terms of learning mechanisms, amounts of data used, grounding and access to different modalities—make a direct translation of insights challenging. The aim of this edited volume has been to create a forum of exchange and debate along this line of research, inviting contributions that further elucidate similarities and differences between humans and LLMs.

Leveraging AI automated emergency response with natural language processing: Enhancing real-time decision making and communication

Artificial Intelligence (AI) and Natural Language Processing (NLP) technologies play an essential role in the ongoing development and advancements of emergency response systems. This paper explores the new opportunities and possibilities offered by the incorporation of AI-driven automated emergency response systems, and the benefits of using NLP to increase the efficiency of human language processing and communication in real time. Using case studies and applications, we analyse how integrating AI and NLP technologies can reduce response times, improve situational awareness and coordination between emergency services. We then explore how AI-driven models can be used to predict natural disasters, optimise emergency vehicle routes and identify areas of need in real time through the analysis of social media and emergency communications via NLP. However, we also discuss the issues that come with implementing these technologies, such as data-privacy challenges, technical limitations and the need for sustained retraining of AI systems. The conclusions of this research indicate that continued collaboration between stakeholders is necessary to overcome these barriers, and that substantial innovation is needed to maximise the potential of AI and NLP technologies in emergency management.

Exploring temporal sensitivity in the brain using multi-timescale language models: an EEG decoding study

Abstract The brain’s ability to perform complex computations at varying timescales is crucial, ranging from understanding single words to grasping the overarching narrative of a story. Recently, multi-timescale long short-term memory (MT-LSTM) models (Mahto et al. 2020; Jain et al. 2020) have been introduced, which use temporally-tuned parameters to induce sensitivity to different timescales of language processing (i.e. related to near/distant words). However, there has not been an exploration of the relation between such temporally-tuned information processing in MT-LSTMs and the brain’s language processing using high temporal resolution recording modalities, such as electroencephalography (EEG). To bridge this gap, we used an EEG dataset recorded while participants listened to Chapter 1 of “Alice in Wonderland” and trained ridge regression models to predict the temporally-tuned MT-LSTM embeddings from EEG responses. Our analysis reveals that EEG signals can be used to predict MT-LSTM embeddings across various timescales. For longer timescales, our models produced accurate predictions within an extended time window of ±2 s around word onset, while for shorter timescales, significant predictions are confined to a narrow window ranging from −180 ms to 790 ms. Intriguingly, we observed that short timescale information is not only processed in the vicinity of word onset but also at distant time points. These observations underscore the parallels and discrepancies between computational models and the neural mechanisms of the brain. As word embeddings are used more as in silico models of semantic representation in the brain, a more explicit consideration of timescale-dependent processing enables more targeted explorations of language processing in humans and machines.

Connectivity Reveals the Relationships between Human Brain Areas Associated with High-Level Linguistic Processing and Macaque Brain Areas.

Cross-species research has advanced human understanding of brain regions, with cross-species comparisons using magnetic resonance imaging technology becoming increasingly common. Currently, cross-species research on human language regions has primarily focused on traditional brain areas such as the Broca region. While some studies have indicated that human language function also involves other language regions, the corresponding relationships between these brain regions in humans and macaques remain unclear. This study calculated the strength of the connections between the high-level language processing regions in human and macaque brains, identified homologous target areas based on the structural connections of white-matter fiber bundles, and compared the connectivity profiles of both species. The results of the experiment demonstrated that macaques possess brain regions which exhibit connectivity patterns resembling those found in human high-level language processing regions. This discovery suggests that while the function of a human brain region is specialized, it still maintains a structural connectivity similar to that seen in macaques.

Rethinking tokenization

Abstract Tokenization significantly influences language models (LMs)’ performance. This paper traces the evolution of tokenizers from word-level to subword-level, analyzing how they balance tokens and types to enhance model adaptability while controlling complexity. Despite subword tokenizers like Byte Pair Encoding (BPE) overcoming many word tokenizer limitations, they encounter difficulties in handling non-Latin languages and depend heavily on extensive training data and computational resources to grasp the nuances of multiword expressions (MWEs). This article argues that tokenizers, more than mere technical tools, should drawing inspiration from the cognitive science about human language processing. This study then introduces the “Principle of Least Effort” from cognitive science, that humans naturally seek to reduce cognitive effort, and discusses the benefits of this principle for tokenizer development. Based on this principle, the paper proposes that the Less-is-Better (LiB) model could be a new approach for LLM tokenizer. The LiB model can autonomously learn an integrated vocabulary consisting of subwords, words, and MWEs, which effectively reduces both the numbers of tokens and types. Comparative evaluations show that the LiB tokenizer outperforms existing word and BPE tokenizers, presenting an innovative method for tokenizer development, and hinting at the possibility of future cognitive science-based tokenizers being more efficient.

Artificial Neural Network Language Models Predict Human Brain Responses to Language Even After a Developmentally Realistic Amount of Training.

Artificial neural networks have emerged as computationally plausible models of human language processing. A major criticism of these models is that the amount of training data they receive far exceeds that of humans during language learning. Here, we use two complementary approaches to ask how the models' ability to capture human fMRI responses to sentences is affected by the amount of training data. First, we evaluate GPT-2 models trained on 1 million, 10 million, 100 million, or 1 billion words against an fMRI benchmark. We consider the 100-million-word model to be developmentally plausible in terms of the amount of training data given that this amount is similar to what children are estimated to be exposed to during the first 10 years of life. Second, we test the performance of a GPT-2 model trained on a 9-billion-token dataset to reach state-of-the-art next-word prediction performance on the human benchmark at different stages during training. Across both approaches, we find that (i) the models trained on a developmentally plausible amount of data already achieve near-maximal performance in capturing fMRI responses to sentences. Further, (ii) lower perplexity-a measure of next-word prediction performance-is associated with stronger alignment with human data, suggesting that models that have received enough training to achieve sufficiently high next-word prediction performance also acquire representations of sentences that are predictive of human fMRI responses. In tandem, these findings establish that although some training is necessary for the models' predictive ability, a developmentally realistic amount of training (∼100 million words) may suffice.

Word Frequency and Predictability Dissociate in Naturalistic Reading.

Many studies of human language processing have shown that readers slow down at less frequent or less predictable words, but there is debate about whether frequency and predictability effects reflect separable cognitive phenomena: are cognitive operations that retrieve words from the mental lexicon based on sensory cues distinct from those that predict upcoming words based on context? Previous evidence for a frequency-predictability dissociation is mostly based on small samples (both for estimating predictability and frequency and for testing their effects on human behavior), artificial materials (e.g., isolated constructed sentences), and implausible modeling assumptions (discrete-time dynamics, linearity, additivity, constant variance, and invariance over time), which raises the question: do frequency and predictability dissociate in ordinary language comprehension, such as story reading? This study leverages recent progress in open data and computational modeling to address this question at scale. A large collection of naturalistic reading data (six datasets, >2.2 M datapoints) is analyzed using nonlinear continuous-time regression, and frequency and predictability are estimated using statistical language models trained on more data than is currently typical in psycholinguistics. Despite the use of naturalistic data, strong predictability estimates, and flexible regression models, results converge with earlier experimental studies in supporting dissociable and additive frequency and predictability effects.

Large-scale evidence for logarithmic effects of word predictability on reading time

During real-time language comprehension, our minds rapidly decode complex meanings from sequences of words. The difficulty of doing so is known to be related to words' contextual predictability, but what cognitive processes do these predictability effects reflect? In one view, predictability effects reflect facilitation due to anticipatory processing of words that are predictable from context. This view predicts a linear effect of predictability on processing demand. In another view, predictability effects reflect the costs of probabilistic inference over sentence interpretations. This view predicts either a logarithmic or a superlogarithmic effect of predictability on processing demand, depending on whether it assumes pressures toward a uniform distribution of information over time. The empirical record is currently mixed. Here, we revisit this question at scale: We analyze six reading datasets, estimate next-word probabilities with diverse statistical language models, and model reading times using recent advances in nonlinear regression. Results support a logarithmic effect of word predictability on processing difficulty, which favors probabilistic inference as a key component of human language processing.

Metaphor forces argument overtness

Abstract This paper uncovers how metaphor forces argument overtness – across languages and parts of speech. It addresses the relationship between semantically unsaturated terms, functors, and the argument terms that complete them. When the component terms’ default senses clash semantically, a metaphor arises. In such cases, the argument must be overt, in contrast to literal uses. It is possible to say Everyone was waiting at the hotel. Finally, Kim arrived. By contrast, people do not use arrived metaphorically without a goal argument: Everything had been pointing to that conclusion all along. *Finally, Kim arrived. What they say is Finally, Kim arrived at it. We illustrate the phenomenon with powerful and diverse evidence: three corpus studies (Indo-Aryan languages, British English, Vera’a) and a sentence-completion experiment with around 250 native speakers of English. Both the corpus studies and the experiment show no or almost no exceptions to metaphor-driven argument overtness. The strength of the effect contrasts with a complete lack of speaker awareness. We propose that metaphor-driven argument overtness – as well as the lack of speaker consciousness – is a universal phenomenon that can be accounted for in terms of human language processing.

Evolutionary continuity and divergence of auditory dorsal and ventral pathways in primates revealed by ultra-high field diffusion MRI

Auditory dorsal and ventral pathways in the human brain play important roles in supporting speech and language processing. However, the evolutionary root of the dual auditory pathways in the primate brain is unclear. By parcellating the auditory cortex of marmosets (a New World monkey species), macaques (an Old World monkey species), and humans using the same individual-based analysis method and tracking the pathways from the auditory cortex based on multi-shell diffusion-weighted MRI (dMRI), homologous auditory dorsal and ventral fiber tracks were identified in these primate species. The ventral pathway was found to be well conserved in all three primate species analyzed but extend to more anterior temporal regions in humans. In contrast, the dorsal pathway showed a divergence between monkey and human brains. First, frontal regions in the human brain have stronger connections to the higher-level auditory regions than to the lower-level auditory regions along the dorsal pathway, while frontal regions in the monkey brain show opposite connection patterns along the dorsal pathway. Second, the left lateralization of the dorsal pathway is only found in humans. Moreover, the connectivity strength of the dorsal pathway in marmosets is more similar to that of humans than macaques. These results demonstrate the continuity and divergence of the dual auditory pathways in the primate brains along the evolutionary path, suggesting that the putative neural networks supporting human speech and language processing might have emerged early in primate evolution.

<b>Systematic Review of Semantic Analysis Methods</b>

NLP (natural language processing) is a very broad subcategory of computer science that focuses on human language processing in computers. Each different NLP processing technique focuses on different parts of linguistics, with semantics being the main focus of this manuscript. This manuscript aims to utilize a systematic review of several published papers ranging from 1998 to the current day to review and summarize critical analysis regarding the three main models that this paper focuses on: Latent Semantic Analysis (LSA), Explicit Semantic Analysis (ESA), and Neural Network based models. This manuscript compares these models against each other, weighs their advantages and disadvantages, and provides their uses. The selected studies in this review were analyzed and examined to ensure that they meet the quality and standards of the proposed research methodology. The results show that Neural network-based solutions are the most popular Semantic Analysis model in Academia (doubling the number of results of ESA and LSA combined), and they are usually the best in most tasks. However, there are specific scenarios and circumstances in which relying on the older LSA and ESA models could be beneficial.

Abstract 15459: Statin Twitter: Human and "Bot" Contributions, 2010-2022

Background: Half of patients with ASCVD do not take statin medications despite proven efficacy and safety. Misinformation about statins drives skepticism about their use. Social media “bots” are known to proliferate misinformation, but involvement in statin-related discourse is unknown. Objectives: 1) to examine temporal trends in volume and sentiment of statin-related Twitter posts (tweets); 2) to compare statin-related content generated by humans and bots Methods: We queried Twitter for all tweets with statin-related terms from 2010-2022. We used a machine learning-derived classifier to determine tweet authorship as bot, human, or intermediate, and natural language processing to assign a sentiment score to each tweet. Manual qualitative analysis was performed on a randomly selected sample of all tweets and highly influential tweets to quantify statin skepticism. Results: We identified 1,155,735 original tweets (excluding retweets) with statin-related terms, the most common being “statin” (n=832,627, 72%) and “Lipitor” (n=197,015, 17%). Bots produced 333,689 (29%), humans produced 699,876 (61%), and intermediate accounts produced 104,966 (9.1%). Over the study period, the proportion of bot tweets decreased from 48% to 11%, human tweets increased from 44% to 80%, and intermediate tweets decreased from 7.7% to 6.7%. The proportion of negative sentiment tweets increased from 28% to 43% for bots, 31% to 38% for humans, and 29% to 40% for intermediate accounts. Including retweets, negative sentiment increased from 30% to 45% across all tweets. In a qualitative analysis of highly influential tweets, statin skepticism increased from 7% to 29% for bots and 25% to 36% for humans. Conclusions: Since 2010, humans have overtaken bots as generators of statin-related content on Twitter. Negative sentiment and statin skepticism have increased, especially among highly influential tweets. Misinformation interventions should focus on human users and influential bots.

A fusion scheme for eliminating input interference induced by spelling errors

Spelling errors are a prevalent form of interference that may result from human handwriting or natural language processing systems, such as optical character recognition. This type of interference presents a challenge for natural language processing systems that rely on context-based embeddings to comprehend the exact semantics of a sentence. The proposed fusion scheme is the main focus of this paper, which aims to correct spelling errors in sentences. The scheme employs both the original input and the masked input. The masked input is acquired by utilizing a detection module to mask any spelling errors present in the original input. By utilizing the masked input, the scheme is capable of learning the real sentence semantic information that is not affected by spelling errors. Utilizing the original input can prevent the loss of input information resulting from masking operations. Finally, the utilization of the Kullback-Leibler divergence is employed to establish concordance between the two output distributions within the framework, consequently facilitating the acquisition of comprehensive input information, all while mitigating the influence of spelling errors. Experiments were conducted on two widely used benchmarks. Compared with previous methods, the approach presented herein exhibited superior performance as it achieved improvements on two benchmarks.

Development of an automatic grammar checker for Yorùbá word processing using Government and Binding Theory

Grammar checkers have been developed for well-resourced languages such as French, German and English. However, there is a dearth of automatic grammar checkers for African languages including Standard Yorùbá (SY) and consequently no standard software application. This study developed a software system for Yorùbá grammar checking. Requirements for a Yorùbá grammar checking system were investigated by gathering information on Yorùbá syntax from Textbooks and Journals. Based on the information gathered, a computational model was formulated using the theoretical framework of Government and Binding (GB). The formulated system was designed using the Context Free Grammar Tool in Automata Theory. The system was implemented using the Python programming language version 3.5 and Natural Language Tool kit (NLTK). The system was tested for accuracy using a total of three hundred sentences. In detecting grammatically correct and incorrect sentences, an accuracy of 86.67%, 88.89% and 88.89% were achieved for simple, complex and compound sentences respectively. The grammar checker has been able to function as an automatic grammatical error checker of Yorùbá digital text. Its implementation in software applications has the potential to increase the quality of Yorùbá word processing. Also, the dictionary of words created can be used in other areas of human language processing.

Algoritma Stemming Teks Bahasa Batak Angkola Berbasis Aturan Tata Bahasa

The Angkola Batak language is a variety of Batak languages, to be precise in the southern Tapanuli area, which is still used and maintained as an everyday language. Until now, the resources of the Angkola Batak language are not yet available in digital form that can be used by researchers in the analytical stages of human natural language processing. Natural language processing (NLP Taks) for the Angkola Batak language must follow the stages of text processing starting from tokenization, lexical analysis, syntax, semantics, and phragmatics. This study conducted natural language processing in the first stage, namely lexical analysis. At the lexical analysis stage, one of the most important NLP tasks is stemming. Stemming is the process of determining root words from affixed words. In this research, an analysis and design of the Angkola Batak stemming algorithm have been carried out based on grammar rules. The stages in this research are starting from collecting the grammar rules of the Angkola Batak language, collecting basic words in the Angkola Batak language as a database dictionary, and removing affixes from root words. The output of this research is the stemmer of the Angkola Batak language in the form of PHP. Based on tests conducted on 450 words originating from the Batak Angkola folklore, 448 test words were correct (99.56%) and 2 test words were wrong (0.44%). The wrong test word is obtained because the root word is not found in the dictionary.

A case study of using natural language processing to extract consumer insights from tweets in American cities for public health crises

BackgroundThe COVID-19 pandemic was a “wake up” call for public health agencies. Often, these agencies are ill-prepared to communicate with target audiences clearly and effectively for community-level activations and safety operations. The obstacle is a lack of data-driven approaches to obtaining insights from local community stakeholders. Thus, this study suggests a focus on listening at local levels given the abundance of geo-marked data and presents a methodological solution to extracting consumer insights from unstructured text data for health communication.MethodsThis study demonstrates how to combine human and Natural Language Processing (NLP) machine analyses to reliably extract meaningful consumer insights from tweets about COVID and the vaccine. This case study employed Latent Dirichlet Allocation (LDA) topic modeling, Bidirectional Encoder Representations from Transformers (BERT) emotion analysis, and human textual analysis and examined 180,128 tweets scraped by Twitter Application Programming Interface’s (API) keyword function from January 2020 to June 2021. The samples came from four medium-sized American cities with larger populations of people of color.ResultsThe NLP method discovered four topic trends: “COVID Vaccines,” “Politics,” “Mitigation Measures,” and “Community/Local Issues,” and emotion changes over time. The human textual analysis profiled the discussions in the selected four markets to add some depth to our understanding of the uniqueness of the different challenges experienced.ConclusionsThis study ultimately demonstrates that our method used here could efficiently reduce a large amount of community feedback (e.g., tweets, social media data) by NLP and ensure contextualization and richness with human interpretation. Recommendations on communicating vaccination are offered based on the findings: (1) the strategic objective should be empowering the public; (2) the message should have local relevance; and, (3) communication needs to be timely.

POSSIBILITIES OF APPLICATION OF ARTIFICIAL INTELLIGENCE TECHNOLOGY IN TEACHING FOREIGN LANGUAGES IN UNIVERSITIES

This article deals with the features of using AI technologies in foreign language learning at universities, as well as its advantages for both students and teachers. Special attention is paid to neurolinguistic programming (an area that combines artificial intelligence and linguistics, and is associated with automated processing of human language), machine and deep learning (a method that uses artificial neural networks to learn from large datasets).

Why Does Surprisal From Larger Transformer-Based Language Models Provide a Poorer Fit to Human Reading Times?

AbstractThis work presents a linguistic analysis into why larger Transformer-based pre-trained language models with more parameters and lower perplexity nonetheless yield surprisal estimates that are less predictive of human reading times. First, regression analyses show a strictly monotonic, positive log-linear relationship between perplexity and fit to reading times for the more recently released five GPT-Neo variants and eight OPT variants on two separate datasets, replicating earlier results limited to just GPT-2 (Oh et al., 2022). Subsequently, analysis of residual errors reveals a systematic deviation of the larger variants, such as underpredicting reading times of named entities and making compensatory overpredictions for reading times of function words such as modals and conjunctions. These results suggest that the propensity of larger Transformer-based models to ‘memorize’ sequences during training makes their surprisal estimates diverge from humanlike expectations, which warrants caution in using pre-trained language models to study human language processing.