Language Understanding Research Articles

Halliday, Critical Discourse Analysis and ideology

Abstract While Halliday’s work has been a source of inspiration to the fields of Critical Linguistics and Critical Discourse Analysis, Halliday has never been embraced by either school. This paper reviews the engagement of CL and CDA with Halliday, examining what ideas were borrowed, and what was either rejected or modified beyond recognition. It then examines how and why the key working assumptions in Halliday’s framework — in particular, metafunction, realisation and instantiation — are essential to understanding the empiric, ideological power of language. As humanity heads deeper into its greatest crisis and challenge, a clear-sighted understanding of the power of language is more important than ever. Halliday’s attention to understanding language as a sociological phenomenon makes his theory ideal for linguistic projects trying to deconstruct and understand the force of the most consequential ideologies in human societies.

A simple yet effective self-debiasing framework for transformer models

Current Transformer-based natural language understanding (NLU) models heavily rely on dataset biases, while failing to handle real-world out-of-distribution (OOD) instances. Many methods have been proposed to deal with this issue, but they ignore the fact that the features learned in different layers of Transformer-based NLU models are different. In this paper, we first conduct preliminary studies to obtain two conclusions: 1) both low- and high-layer sentence representations encode common biased features during training; 2) the low-layer sentence representations encode fewer unbiased features than the high-layer ones. Based on these conclusions, we propose a simple yet effective self-debiasing framework for Transformer-based NLU models. Concretely, we first stack a classifier on a selected low layer. Then, we introduce a residual connection that feeds the low-layer sentence representation to the top-layer classifier. In this way, the top-layer sentence representation will be trained to ignore the common biased features encoded by the low-layer sentence representation and focus on task-relevant unbiased features. During inference, we remove the residual connection and directly use the top-layer sentence representation to make predictions. Extensive experiments and in-depth analyses on NLU tasks demonstrate the superiority of our framework, achieving a new state-of-the-art (SOTA) on three OOD test sets.

Using deep learning and word embeddings for predicting human agreeableness behavior

The latest advancements of deep learning have resulted in a new era of natural language processing. The machines now possess an unparallel ability to interpret and engage with various tasks such as text classification, content generation and natural language understanding. This development extended to the analysis of human behavior, where deep learning models are used to decode human personality. Due to the rise of social media, generating huge amounts of textual data that reshaped communication patterns. Understanding personality traits is a challenging topic which helps us to explore the patterns of thoughts, feelings and behaviors which are helpful for recruitment, career counselling and consumers’ behavior for marketing, etc. In this research study, the main aim is to predict the human personality trait of agreeableness showing whether a person is emotional who feels a lot or thinker who is logical and has rational thinking. This behavior leads to analyzing them as cooperative, friendly and respecting difference of views. For comprehensive empirical analysis, shallow machine learning models, ensemble models, and deep learning technique including state of the art transformer-based models are applied on real-world dataset of MBTI. For feature engineering, textual features of TF-IDF and POS tagging and word embeddings such as word2vec, glove and sentence embeddings are explored. The results analysis shows the highest performance 91.57% with sentence embeddings utilizing Bi-LSTM algorithm that highlights the power of this study as compared to existing studies in the relevant literature.

A Survey on Deep Neural Network Pruning: Taxonomy, Comparison, Analysis, and Recommendations.

Modern deep neural networks, particularly recent large language models, come with massive model sizes that require significant computational and storage resources. To enable the deployment of modern models on resource-constrained environments and to accelerate inference time, researchers have increasingly explored pruning techniques as a popular research direction in neural network compression. More than three thousand pruning papers have been published from 2020 to 2024. However, there is a dearth of up-to-date comprehensive review papers on pruning. To address this issue, in this survey, we provide a comprehensive review of existing research works on deep neural network pruning in a taxonomy of 1) universal/specific speedup, 2) when to prune, 3) how to prune, and 4) fusion of pruning and other compression techniques. We then provide a thorough comparative analysis of eight pairs of contrast settings for pruning (e.g., unstructured/structured, one-shot/iterative, data-free/data-driven, initialized/pre-trained weights, etc.) and explore several emerging topics, including pruning for large language models, vision transformers, diffusion models, and large multimodal models, post-training pruning, and different levels of supervision for pruning to shed light on the commonalities and differences of existing methods and lay the foundation for further method development. Finally, we provide some valuable recommendations on selecting pruning methods and prospect several promising research directions for neural network pruning. To facilitate future research on deep neural network pruning, we summarize broad pruning applications (e.g., adversarial robustness, natural language understanding, etc.) and build a curated collection of datasets, networks, and evaluations on different applications. We maintain a repository on https://github.com/hrcheng1066/awesome-pruning that serves as a comprehensive resource for neural network pruning papers and corresponding open-source codes. We will keep updating this repository to include the latest advancements in the field.

Aspect-based sentiment analysis: natural language understanding for implicit review

Aspect-based sentiment analysis: natural language understanding for implicit review

Design and implementation of EventsKG for situational monitoring and security intelligence in India: An open-source intelligence gathering approach

This paper presents a method to construct and implement an Events Knowledge Graph (EventsKG) for security-related open-source intelligence gathering, focusing on event exploration for situation monitoring in India. The EventsKG is designed to process news articles, extract events of national security significance, and represent them in a consistent and intuitive manner. This method utilizes state-of-the-art natural language understanding techniques and the capabilities of graph databases to extract and organize events. A domain-specific ontology is created for effective storage and retrieval. In addition, we provide a user-friendly dashboard for querying and a complete visualization of events across India. The effectiveness of the EventsKG is assessed through a human evaluation of the information retrieval quality. Our approach contributes to rapid data availability and decision-making through a comprehensive understanding of events, including local events, from every part of India in just a few clicks. The system is evaluated against a manually annotated dataset and by involving human evaluators through a feedback survey, and it has shown good retrieval accuracy. The EventsKG can also be used for other applications such as threat intelligence, incident response, and situational awareness.

HC 2L: Hybrid and Cooperative Contrastive Learning for Cross-Lingual Spoken Language Understanding.

State-of-the-art model for zero-shot cross-lingual spoken language understanding performs cross-lingual unsupervised contrastive learning to achieve the label-agnostic semantic alignment between each utterance and its code-switched data. However, it ignores the precious intent/slot labels, whose label information is promising to help capture the label-aware semantics structure and then leverage supervised contrastive learning to improve both source and target languages' semantics. In this paper, we propose Hybrid and Cooperative Contrastive Learning to address this problem. Apart from cross-lingual unsupervised contrastive learning, we design a holistic approach that exploits source language supervised contrastive learning, cross-lingual supervised contrastive learning and multilingual supervised contrastive learning to perform label-aware semantics alignments in a comprehensive manner. Each kind of supervised contrastive learning mechanism includes both single-task and joint-task scenarios. In our model, one contrastive learning mechanism's input is enhanced by others. Thus the total four contrastive learning mechanisms are cooperative to learn more consistent and discriminative representations in the virtuous cycle during the training process. Experiments show that our model obtains consistent improvements over 9 languages, achieving new state-of-the-art performance.

BhashaBlend: Enabling Multilingual understanding of videos through NLP for Deaf and Hearing Impaired users

BhashaBlend: Enabling Multilingual understanding of videos through NLP for Deaf and Hearing Impaired users

LIPT: Improving Prompt Tuning with Late Inception Reparameterization

Prompt tuning is a mainstream technique for fine-tuning large language models (LLMs), offering minimal parameter adjustments by learning task-specific prompt vectors. However, it suffers from training costs due to network-wide backpropagation and weaker performance compared to methods like adapters and LoRA, likely due to the limited capacity of soft prompts to encode task-specific information. This study introduces Late Inception Prompt Tuning (LIPT), a novel approach to soft prompt learning that enhances performance and efficiency by shortening backpropagation paths and employing a multidimensional bottleneck network with greater capacity. LIPT surpasses existing prompt tuning techniques on various benchmark tasks, delivering a 1.3% gain over LPT and a 5% improvement compared to standard prompt tuning when applied to RoBERTa-large, while converging more rapidly. It achieves an average accuracy of 90% across ten benchmark datasets. Notably, in certain scenarios, LIPT’s performance approaches that of full-parameter fine-tuning methods. To evaluate parameter-efficient fine-tuning (PEFT) comprehensively, we propose an Efficiency Indicator (EI) that balances accuracy and cost. LIPT is well suited for natural language understanding tasks, like sentiment analysis and text classification, with potential extensions to larger-scale models and tasks like text generation. This framework advances the scalability and practicality of fine-tuning methods for diverse applications.

GKA-GPT: Graphical knowledge aggregation for multiturn dialog generation

In human interaction, effective communication relies on shared cognitive processes that facilitate the ability of individuals to comprehend the intended message of their interlocutors. Recent research in multiturn dialog generation seeks to emulate human-like responses by incorporating external knowledge into generative models to enhance language understanding. These models often utilize graphical representations of knowledge and employ graph neural networks (GNNs) to capture dialog semantics. However, sole reliance on external knowledge can fall short as human cognition integrates universal commonsense and personal knowledge, with the latter being derived from individual experiences and frequently disregarded. To remedy this, we propose GKA-GPT, a novel GNN-based approach that merges commonsense and personal knowledge into a comprehensive cognition graph to enhance the relevance and diversity of responses in multiturn dialog scenarios. Furthermore, GKA-GPT introduces a multigrained graphical knowledge aggregation mechanism for effective semantic information processing across various levels. Our experiments demonstrate that GKA-GPT outperforms existing baselines by generating more relevant and informative responses.

HealthWise: AI-Powered Medical Chatbot with Llama 2

This research paper presents the development of a medical chatbot utilizing cutting- edge technologies, specifically the Langchain framework and the Llama 2 language model, in conjunction with the PineCone database for efficient data storage and retrieval. The primary objective of this project is to create an intelligent conversational agent capable of providing reliable medical information, assisting users in symptom assessment, and guiding them towards appropriate healthcare resources. The integration of Langchain facilitates seamless interaction between the user interface and the underlying LLM, enabling natural language understanding and generation. Leveraging the advanced capabilities of the Llama 2 model ensures that the chatbot can engage in meaningful dialogues while maintaining context and relevance. Additionally, PineCone serves as a robust solution for managing and indexing medical data, enhancing the chatbot's ability to deliver accurate and context-aware responses. This study explores the design, implementation, and potential implications of our medical chatbot, contributing to the emerging field of AI-driven healthcare solutions and underscoring its role in improving patient engagement and access to medical advice. Keywords – Medical Chatbot, Langchain, Llama-2, Pine Cone, Natural Language Processing, Artificial Intelligence, Machine Learning, Generative Artificial Intelligence.

Design and validation of a Primary Reading Proficiency Test (PCL-P)

Reading proficiency is one of the most important skills to be developed by students at school. However, the data on reading proficiency performance in Spanish schoolchildren show a downward trend. Thus, to measure this performance, a reading proficiency assessment test has been developed for Primary School students (PCL-P). Based on a multilevel model (executive, functional, instrumental, and epistemic), the test content is evaluated by 5 pedagogical and linguistic experts (inter-judge agreement: Fleiss' Kappa was 0.88). After that, it is applied to a sample of 216 students (6–12 years old) of the autonomous community of the Basque Country. Cronbach's alpha was 0.83. On the other hand, the exploratory factor analysis (principal components method and Varimax rotation) results in 5 factors explaining 59% of the variance: (1) word and text recognition and structural understanding; (2) evaluation and critical reflection of information for the creation of new ideas; (3) advanced/deep reading strategies; (4) phonological awareness; and (5) structural understanding of language and detection of key elements. In conclusion, the PCL-P is a valid instrument for the assessment of reading proficiency in Primary.

A Review of the Limitations of Language Models in NLP

Abstract. In the field of Natural Language Processing (NLP), large neural language models have successfully been applied to a variety of tasks, including machine translation and reading comprehension. These models often learn the structure of language (such as grammar and semantics) by predicting the next word or character. However, there's an overly optimistic assessment of the current state of natural language understanding, which assumes that models can handle text at a semantic level. This article primarily explores the successes of large neural language models (such as BERT and GPT-2) in numerous tasks that require the understanding of linguistic meaning in the domain of NLP. Language models trained purely on form cannot learn meaning, as understanding meaning involves the relationship between linguistic form and non-linguistic communicative intent. The article aims to guide scientific research in the field of Natural Language Understanding (NLU) by clearly distinguishing the concepts of form and meaning. The learning, generalization, and other capabilities of different models under various attributes show significant differences, suggesting that a combination of different models and properties can achieve unexpected results. The simple learning combinations of some models may offer new insights into the understanding of linguistic meaning by models.

ChatGPT: Technology Frontiers and Cybersecurity Challenges

This paper examines the evolution and application of OpenAI's advanced conversational AI, ChatGPT, particularly within the domain of cybersecurity. With an architecture built on the Transformer model, ChatGPT demonstrates significant capabilities in language understanding and generation. It leverages vast datasets, ranging from social media posts to technical documents, ensuring the model adapts to diverse fields and maintains compliance with privacy and security regulations. The paper explores ChatGPT's role in network security, highlighting its proficiency in threat detection, vulnerability assessment, and incident response, essential as regulations like GDPR and CCPA become more stringent. Furthermore, the study addresses potential security risks associated with AI, such as phishing and misinformation, and discusses mitigation strategies through advanced training techniques like adversarial training and multi-task learning. A novel variational autoencoder (VAE)-based method, T-VAE, is introduced, offering enhanced generalization capabilities across different tasks and scenarios. The findings suggest that while ChatGPT has made significant strides in cybersecurity applications, continuous improvements in model robustness and adaptability are necessary to mitigate emerging threats and adapt to evolving digital landscapes.

Организация занятий художественного восприятия на историко-мифологических портретов в изобразительном искусстве

This article is devoted to the problems of depicting historical and mythological portrait painting and the analysis of portraits of the same name by artists, which are topical issues of teaching painting. From the emergence of the idea of a portrait with a mythological appearance in the article to its full formation, it sometimes goes a long and complex way. Thus, there are no clear boundaries between the birth of the idea of a portrait and the finished version, it is impossible to separate the creative process as a whole, moreover, the finished work does not always differ from the original sketch. At the initial stage of portrait work, it is necessary to perform large and diverse work. The essence and components of the future work are analyzed, the options are compared, everything is considered, the most suitable for clarifying the idea is selected and evaluated. The article will consider classes on the topic "portrait of great figures of Fine Art", in which a meaningful conversation of a teacher explaining the illustrative material used in a simple and understandable language can arouse interest among students.

Advancements and Applications of Large Language Models in Natural Language Processing: A Comprehensive Review

Abstract. Large language models (LLMs) have revolutionized the field of natural language processing (NLP), demonstrating remarkable capabilities in understanding, generating, and manipulating human language. This comprehensive review explores the development, applications, optimizations, and challenges of LLMs. This paper begin by tracing the evolution of these models and their foundational architectures, such as the Transformer, GPT, and BERT. We then delve into the applications of LLMs in natural language understanding tasks, including sentiment analysis, named entity recognition, question answering, and text summarization, highlighting real-world use cases. Next, we examine the role of LLMs in natural language generation, covering areas such as content creation, language translation, personalized recommendations, and automated responses. We further discuss LLM applications in other NLP tasks like text style transfer, text correction, and language model pre-training. Subsequently, we explore techniques for optimizing and improving LLMs, including model compression, explainability, robustness, and security. Finally, we address the challenges posed by the significant computational requirements, sample inefficiency, and ethical considerations surrounding LLMs. We conclude by discussing potential future research directions, such as efficient architectures, few-shot learning, bias mitigation, and privacy-preserving techniques, which will shape the ongoing development and responsible deployment of LLMs in NLP.

A cross-language short text classification model based on BERT and multilayer collaborative convolutional neural network (MCNN)

This study focuses on cross-lingual short text classification tasks and aims to combine the advantages of BERT and Multi-layer Collaborative Convolutional Neural Network (MCNN) to build an efficient classification model. BERT model provides rich semantic information for text classification with its powerful language understanding and bidirectional context modeling ability, while MCNN effectively extracts local and global features in text through multi-layer convolution structure and collaborative working mechanism. In this study, the output of BERT is used as the input of MCNN, and MCNN is used to further mine the deep features in the text, so as to realize the high-precision classification of cross-lingual short text. The experimental results show that the model has achieved significant performance improvement on the dataset, which provides a new effective solution for cross-lingual short text classification tasks.

Assessing the Reliability and Validity of a Questionnaire Evaluating Medical Students’ Attitudes, Knowledge, and Perceptions of Antibiotic Education and Antimicrobial Resistance in University Training

The misuse and overuse of antibiotics represent a critical global issue and one of the most pressing public health challenges of the 21st century. Training future healthcare professionals effectively is essential for ensuring responsible antibiotic use. This study aimed to validate a questionnaire designed to evaluate the knowledge, attitudes, and perceptions of medical students regarding the education they receive on infectious diseases, antimicrobial resistance, and antibiotic stewardship during their university studies. Methods: A self-administered questionnaire was developed and distributed to medical students at the University of Santiago de Compostela. Comprising 44 items, the questionnaire assessed eight key dimensions: “infection diagnosis”, “criteria for not prescribing antibiotics”, “initial antibiotic therapy”, “re-evaluation of therapy”, “quality of care”, “communication skills”, “antibiotic resistance”, and “teaching methodology”. Validation was carried out in two stages: Phase 1 involved content and face validity, while Phase 2 focused on reliability analysis. Results: A total of 295 students completed the questionnaire, with a mean age of 23.15 ± 1.78 years. The sample included 86 male (29.2%) and 209 female (70.8%) respondents. Content and face validity were established by a nominal group of five experts and a focus group of medicine and pharmacy students to ensure consensus on item understanding in the Spanish language. The questionnaire demonstrated high internal consistency with a Cronbach’s alpha of 0.92 and satisfactory item discrimination. Construct validity was confirmed through principal component analysis, which supported the presence of the eight predefined dimensions. Conclusions: The validated questionnaire exhibited strong reliability and validity, making it a valuable tool for assessing medical students’ training in antibiotic-related topics. Its application will enable the identification of areas for improvement in university curricula, ultimately contributing to the promotion of appropriate antibiotic use and the reduction of antimicrobial resistance.

Practical Design and Implementation of Virtual Chatbot Assistants for Bioinformatics Based on a NLU Open Framework

In this work, we describe the implementation of an infrastructure of conversational chatbots by using natural language processing and training within the Rasa framework. We use this infrastructure to create a chatbot assistant for the users of a bioinformatics suite. This suite provides a customized interface solution for omic pipelines and workflows, and it is named GPRO. The infrastructure has also been used to build another chatbot for a Laboratory Information Management System (LIMS). The two chatbots (namely, Genie and Abu) have been built on an open framework that uses natural language understanding (NLU) and machine learning techniques to understand user queries and respond to them. Users can seamlessly interact with the chatbot to receive support on navigating the GPRO pipelines and workflows. The chatbot provides a bridge between users and the wealth of bioinformatics knowledge available online.

Teachers’ beliefs and experiences regarding collaboration across language subjects

ABSTRACT In schools worldwide, there are increasing expectations for teacher collaboration across subjects. However, despite the potential benefits for students and teachers of viewing language subjects as interrelated, limited research has explored how meaningful collaboration across different language subjects can be undertaken. Addressing this gap, this study investigates language teachers’ experiences with and perspectives on collaboration across language subjects. Data were generated from focus group discussions with teachers of Norwegian, English, French, German, and Spanish in four Norwegian secondary schools and analysed through reflexive thematic analysis. The analysis revealed that teachers had limited prior experience in this area but recognised several potential benefits, such as increased language awareness and a more holistic understanding of languages. However, teachers’ suggestions for collaboration mainly focused on sharing ideas and coordinating tasks, whereas deeper forms of joint work are likely necessary to achieve lasting learning effects. Moreover, teachers lacked knowledge of each other’s subject curricula, whereas a crucial starting point for effective collaboration would be for teachers to become familiar with the contents of all language subject curricula in school. Finally, the analysis suggests that effective collaboration requires active leadership support to address practical challenges and develop a shared understanding of what collaboration may entail.