Natural Language Processing Tasks Research Articles

Towards a semi-automatic classifier of malware through tweets for early warning threat detection

This paper presents a method for developing a malware ontology structure by detecting malware instances on Twitter. The ontology represents a semi-automatic classifier fed by the data extracted from tweets. In particular, the automatic part of the presented methodology relies on a pattern-based approach to detect trigger expressions leading to new information about malware, whilst the manual one covers the evaluation of the results by domain-experts, who also validate the reliability of the semantic relationships within the ontology framework. We present preliminary results on the application of our methodology to tweets extracted from MalwareBazaar database showing how the documents’ collection analysis, through Natural Language Processing (NLP) tasks, can support the knowledge retrieval and documents’ classification procedures for building early warning system of detected malware. Results obtained from this research paper within the time framework of 2023 are referred to the previous version of the current social network X.

An in-depth evaluation of federated learning on biomedical natural language processing for information extraction

Language models (LMs) such as BERT and GPT have revolutionized natural language processing (NLP). However, the medical field faces challenges in training LMs due to limited data access and privacy constraints imposed by regulations like the Health Insurance Portability and Accountability Act (HIPPA) and the General Data Protection Regulation (GDPR). Federated learning (FL) offers a decentralized solution that enables collaborative learning while ensuring data privacy. In this study, we evaluated FL on 2 biomedical NLP tasks encompassing 8 corpora using 6 LMs. Our results show that: (1) FL models consistently outperformed models trained on individual clients’ data and sometimes performed comparably with models trained with polled data; (2) with the fixed number of total data, FL models training with more clients produced inferior performance but pre-trained transformer-based models exhibited great resilience. (3) FL models significantly outperformed pre-trained LLMs with few-shot prompting.

A Bibliometric Review of Large Language Models Research from 2017 to 2023

Large language models (LLMs), such as OpenAI’s Generative Pre-trained Transformer (GPT), are a class of language models that have demonstrated outstanding performance across a range of natural language processing (NLP) tasks. LLMs have become a highly sought-after research area because of their ability to generate human-like language and their potential to revolutionize science and technology. In this study, we conduct bibliometric and discourse analyses of scholarly literature on LLMs. Synthesizing over 5,000 publications, this paper serves as a roadmap for researchers, practitioners, and policymakers to navigate the current landscape of LLMs research. We present the research trends from 2017 to early 2023, identifying patterns in research paradigms and collaborations. We start with analyzing the core algorithm developments and NLP tasks that are fundamental in LLMs research. We then investigate the applications of LLMs in various fields and domains, including medicine, engineering, social science, and humanities. Our review also reveals the dynamic, fast-paced evolution of LLMs research. Overall, this paper offers valuable insights into the current state, impact, and potential of LLMs research and its applications.

Potential of Large Language Models in Health Care: Delphi Study.

A large language model (LLM) is a machine learning model inferred from text data that captures subtle patterns of language use in context. Modern LLMs are based on neural network architectures that incorporate transformer methods. They allow the model to relate words together through attention to multiple words in a text sequence. LLMs have been shown to be highly effective for a range of tasks in natural language processing (NLP), including classification and information extraction tasks and generative applications. The aim of this adapted Delphi study was to collect researchers' opinions on how LLMs might influence health care and on the strengths, weaknesses, opportunities, and threats of LLM use in health care. We invited researchers in the fields of health informatics, nursing informatics, and medical NLP to share their opinions on LLM use in health care. We started the first round with open questions based on our strengths, weaknesses, opportunities, and threats framework. In the second and third round, the participants scored these items. The first, second, and third rounds had 28, 23, and 21 participants, respectively. Almost all participants (26/28, 93% in round 1 and 20/21, 95% in round 3) were affiliated with academic institutions. Agreement was reached on 103 items related to use cases, benefits, risks, reliability, adoption aspects, and the future of LLMs in health care. Participants offered several use cases, including supporting clinical tasks, documentation tasks, and medical research and education, and agreed that LLM-based systems will act as health assistants for patient education. The agreed-upon benefits included increased efficiency in data handling and extraction, improved automation of processes, improved quality of health care services and overall health outcomes, provision of personalized care, accelerated diagnosis and treatment processes, and improved interaction between patients and health care professionals. In total, 5 risks to health care in general were identified: cybersecurity breaches, the potential for patient misinformation, ethical concerns, the likelihood of biased decision-making, and the risk associated with inaccurate communication. Overconfidence in LLM-based systems was recognized as a risk to the medical profession. The 6 agreed-upon privacy risks included the use of unregulated cloud services that compromise data security, exposure of sensitive patient data, breaches of confidentiality, fraudulent use of information, vulnerabilities in data storage and communication, and inappropriate access or use of patient data. Future research related to LLMs should not only focus on testing their possibilities for NLP-related tasks but also consider the workflows the models could contribute to and the requirements regarding quality, integration, and regulations needed for successful implementation in practice.

Multi-Task Learning in Natural Language Processing: An Overview

Deep learning approaches have achieved great success in the field of Natural Language Processing (NLP). However, directly training deep neural models often suffer from overfitting and data scarcity problems that are pervasive in NLP tasks. In recent years, Multi-Task Learning (MTL), which can leverage useful information of related tasks to achieve simultaneous performance improvement on these tasks, has been used to handle these problems. In this paper, we give an overview of the use of MTL in NLP tasks. We first review MTL architectures used in NLP tasks and categorize them into four classes, including parallel architecture, hierarchical architecture, modular architecture, and generative adversarial architecture. Then we present optimization techniques on loss construction, gradient regularization, data sampling, and task scheduling to properly train a multi-task model. After presenting applications of MTL in a variety of NLP tasks, we introduce some benchmark datasets. Finally, we make a conclusion and discuss several possible research directions in this field.

Recurrent event query decoder for document-level event extraction

Document-level event extraction is a challenging task in natural language processing, as it involves multiple events within a document and scattered event arguments across sentences. To tackle these challenges, we propose a recurrent event query decoder, i.e., a recurrent module that dynamically updates event queries to capture cross-event dependencies. Our approach then generates arguments by extracting role-argument relations using bilinear mapping, which helps address the issue of scattered arguments. Experimental results demonstrate that our proposed approach outperforms state-of-the-art models on a large-scale public dataset and actual application data, achieving significant improvements in F1-score. In domain-specific event extraction applications, our method achieves higher accuracy with fewer resources compared to general-purpose large language models.

Multi Task Learning Based Shallow Parsing for Indian Languages

Shallow Parsing is an important step for many Natural Language Processing tasks. Although shallow parsing has a rich history for resource rich languages, it is not the case for most Indian languages. Shallow Parsing consists of POS Tagging and Chunking. Our study focuses on developing shallow parsers for Indian languages. As part of shallow parsing we included morph analysis as well. For the study, we first consolidated available shallow parsing corpora for 7 Indian Languages (Hindi, Kannada, Bangla, Malayalam, Marathi, Urdu, Telugu) for which treebanks are publicly available. We then trained models to achieve state of the art performance for shallow parsing in these languages for multiple domains. Since analyzing the performance of model predictions at sentence level is more realistic, we report the performance of these shallow parsers not only at the token level, but also at the sentence level. We also present machine learning techniques for multitask shallow parsing. Our experiments show that fine-tuned contextual embedding with multi-task learning improves the performance of multiple as well as individual shallow parsing tasks across different domains. We show the transfer learning capability of these models by creating shallow parsers (only with POS and Chunk) for Gujarati, Odia, and Punjabi for which no treebanks are available. As a part of this work, we will be releasing the Indian Languages Shallow Linguistic (ILSL) benchmarks for 10 Indian languages including both the major language families Indo-Aryan and Dravidian as common building blocks that can be used to evaluate and understand various linguistic phenomena found in Indian languages and how well newer approaches can tackle them.

A multi-view representation learning framework for commonsense knowledge bases

Commonsense knowledge bases play an essential role in a wide range of natural language processing tasks. This paper studies the problem of representation learning for commonsense knowledge bases to effectively incorporate their knowledge into numerical models. Most existing knowledge base representation learning methods are difficult to apply to commonsense knowledge bases since they are much sparser than general knowledge bases. Hence, in this paper, we propose a novel method for commonsense knowledge base representation learning. Specifically, we first model the nodes from multiple views, including word/phase information, context information, and graph information. Then, we design a scoring function to measure whether the commonsense triplets are established through relation representation learning. We conduct extensive experiments on two tasks and the results show that our proposed model outperforms other knowledge base representation learning methods.

FAULT-TRANSFORMER: AN AUTOMATIC FAULT DETECTION ALGORITHM ON SEISMIC IMAGES USING A TRANSFORMER ENHANCED NEURAL NETWORK

Seismic fault detection is a key step in seismic interpretation and reservoir characterization that often requires a large amount of human labor and interpretation time. Therefore, automatic seismic fault detection is critical for improving the efficiency of seismic data processing and interpretation. Existing AI methods are mostly based on convolutional neural networks (CNNs) with a U-shaped encoder-decoder structure, known as the U-net. However, the convolution is limited in modeling long-range correlative features. Instead, Transformers, utilizing self-attention mechanisms, avoid the local nature of the convolution, which has the potential to extract long-distance correlations. Transformers are proven to perform well in natural language processing, image classification, and segmentation tasks in precision and recall. Here, we develop a new deep neural network with transformers and a U-net-like structure: Fault-Transformer, to perform the fault detection task. The new network outperforms traditional U-net in the application with synthetic data sets.

Exploring noise-induced techniques to strengthen deep learning model resilience

AbstractIn artificial intelligence, combating overfitting and enhancing model generalization is crucial. This research explores innovative noise-induced regularization techniques, focusing on natural language processing tasks. Inspired by gradient noise and Dropout, this study investigates the interplay between controlled noise, model complexity, and overfitting prevention. Utilizing long short-term memory and bidirectional long short term memory architectures, this study examines the impact of noise-induced regularization on robustness to noisy input data. Through extensive experimentation, this study shows that introducing controlled noise improves model generalization, especially in language understanding. This contributes to the theoretical understanding of noise-induced regularization, advancing reliable and adaptable artificial intelligence systems for natural language processing.

RT: a Retrieving and Chain-of-Thought framework for few-shot medical named entity recognition.

This article aims to enhance the performance of larger language models (LLMs) on the few-shot biomedical named entity recognition (NER) task by developing a simple and effective method called Retrieving and Chain-of-Thought (RT) framework and to evaluate the improvement after applying RT framework. Given the remarkable advancements in retrieval-based language model and Chain-of-Thought across various natural language processing tasks, we propose a pioneering RT framework designed to amalgamate both approaches. The RT approach encompasses dedicated modules for information retrieval and Chain-of-Thought processes. In the retrieval module, RT discerns pertinent examples from demonstrations during instructional tuning for each input sentence. Subsequently, the Chain-of-Thought module employs a systematic reasoning process to identify entities. We conducted a comprehensive comparative analysis of our RT framework against 16 other models for few-shot NER tasks on BC5CDR and NCBI corpora. Additionally, we explored the impacts of negative samples, output formats, and missing data on performance. Our proposed RT framework outperforms other LMs for few-shot NER tasks with micro-F1 scores of 93.50 and 91.76 on BC5CDR and NCBI corpora, respectively. We found that using both positive and negative samples, Chain-of-Thought (vs Tree-of-Thought) performed better. Additionally, utilization of a partially annotated dataset has a marginal effect of the model performance. This is the first investigation to combine a retrieval-based LLM and Chain-of-Thought methodology to enhance the performance in biomedical few-shot NER. The retrieval-based LLM aids in retrieving the most relevant examples of the input sentence, offering crucial knowledge to predict the entity in the sentence. We also conducted a meticulous examination of our methodology, incorporating an ablation study. The RT framework with LLM has demonstrated state-of-the-art performance on few-shot NER tasks.

Houses classification using vision transformer with shifted patch tokenization

Deep learning has recently made great progress leading to revolutionizing image recognition, speech recognition, and natural language processing tasks that were previously challenging to make using traditional techniques. Image classification offers a lot of potential for architectural design, even though it is rarely used to uncover new techniques. It can be used to determine the client’s preferences and design a building that satisfies those preferences. The different architectural styles based on culture, region, and time are one of the main challenges for image classification in architecture. Hence, it can be challenging for untrained clients to recognize an architectural style, and sometimes some buildings are made up of various types that are difficult to classify as a single style. This paper investigates the potential of employing state-of-art cutting-edge image classification algorithms in houses classification. In addition, the paper proposes the uses of Shifted Patch Tokenization (SPT) and Locality Self-Attention (LSA) in order to enhance the performance of Vision transformer (ViT) when trained to classify house images with a small dataset, opposed to the regular ViT which requires huge dataset in order to converge. Experimentally, these techniques proved to have a positive impact on the performance of the ViT, which reached 96.85% accuracy when SPT and LSA are employed.

USE OF NATURAL LANGUAGE PROCESSING TECHNIQUES TO PREDICT PATIENT SELECTION FOR TOTAL HIP ARTHROPLASTY: RESULTS FROM THE AI TO REVOLUTIONISE THE PATIENT CARE PATHWAY IN HIP AND KNEE ARTHROPLASTY (ARCHERY) PROJECT

To examine whether Natural Language Processing (NLP) using a state-of-the-art clinically based Large Language Model (LLM) could predict patient selection for Total Hip Arthroplasty (THA), across a range of routinely available clinical text sources.Data pre-processing and analyses were conducted according to the Ai to Revolutionise the patient Care pathway in Hip and Knee arthroplasty (ARCHERY) project protocol (https://www.researchprotocols.org/2022/5/e37092/). Three types of deidentified Scottish regional clinical free text data were assessed: Referral letters, radiology reports and clinic letters. NLP algorithms were based on the GatorTron model, a Bidirectional Encoder Representations from Transformers (BERT) based LLM trained on 82 billion words of de-identified clinical text. Three specific inference tasks were performed: assessment of the base GatorTron model, assessment after model-fine tuning, and external validation.There were 3911, 1621 and 1503 patient text documents included from the sources of referral letters, radiology reports and clinic letters respectively. All letter sources displayed significant class imbalance, with only 15.8%, 24.9%, and 5.9% of patients linked to the respective text source documentation having undergone surgery. Untrained model performance was poor, with F1 scores (harmonic mean of precision and recall) of 0.02, 0.38 and 0.09 respectively. This did however improve with model training, with mean scores (range) of 0.39 (0.31–0.47), 0.57 (0.48–0.63) and 0.32 (0.28–0.39) across the 5 folds of cross-validation. Performance deteriorated on external validation across all three groups but remained highest for the radiology report cohort.Even with further training on a large cohort of routinely collected free-text data a clinical LLM fails to adequately perform clinical inference in NLP tasks regarding identification of those selected to undergo THA. This likely relates to the complexity and heterogeneity of free-text information and the way that patients are determined to be surgical candidates.

Harnessing large language models for coding, teaching and inclusion to empower research in ecology and evolution

Abstract Large language models (LLMs) are a type of artificial intelligence (AI) that can perform various natural language processing tasks. The adoption of LLMs has become increasingly prominent in scientific writing and analyses because of the availability of free applications such as ChatGPT. This increased use of LLMs not only raises concerns about academic integrity but also presents opportunities for the research community. Here we focus on the opportunities for using LLMs for coding in ecology and evolution. We discuss how LLMs can be used to generate, explain, comment, translate, debug, optimise and test code. We also highlight the importance of writing effective prompts and carefully evaluating the outputs of LLMs. In addition, we draft a possible road map for using such models inclusively and with integrity. LLMs can accelerate the coding process, especially for unfamiliar tasks, and free up time for higher level tasks and creative thinking while increasing efficiency and creative output. LLMs also enhance inclusion by accommodating individuals without coding skills, with limited access to education in coding, or for whom English is not their primary written or spoken language. However, code generated by LLMs is of variable quality and has issues related to mathematics, logic, non‐reproducibility and intellectual property; it can also include mistakes and approximations, especially in novel methods. We highlight the benefits of using LLMs to teach and learn coding, and advocate for guiding students in the appropriate use of AI tools for coding. Despite the ability to assign many coding tasks to LLMs, we also reaffirm the continued importance of teaching coding skills for interpreting LLM‐generated code and to develop critical thinking skills. As editors of MEE, we support—to a limited extent—the transparent, accountable and acknowledged use of LLMs and other AI tools in publications. If LLMs or comparable AI tools (excluding commonly used aids like spell‐checkers, Grammarly and Writefull) are used to produce the work described in a manuscript, there must be a clear statement to that effect in its Methods section, and the corresponding or senior author must take responsibility for any code (or text) generated by the AI platform.

A head-to-head attention with prompt text augmentation for text classification

Short-text classification is an important task in natural language processing (NLP). The classification results are unsatisfactory due to the sparsity of Chinese short texts, insufficient annotation data, single tasks, and classification imbalance problems. Therefore, we propose a method based on a self-attention mechanism to focus on the relationships between attention heads, which can improve the performance of models represented by self-attention for short-text classification tasks. In addition, we designed a text augmentation template based on prompt learning with embedded labels. This allows single-task classification to be transformed into multitask classification while allowing the model to focus on the semantic consistency of the labels with the text. We conducted experiments on the CHNSenticorp, COLD, and SST-2 datasets to achieve better results than several popular text classification methods.

Improving Robustness of Intent Detection Under Adversarial Attacks: A Geometric Constraint Perspective.

Deep neural networks (DNNs)-based natural language processing (NLP) systems are vulnerable to being fooled by adversarial examples presented in recent studies. Intent detection tasks in dialog systems are no exception, however, relatively few works have been attempted on the defense side. The combination of linear classifier and softmax is widely used in most defense methods for other NLP tasks. Unfortunately, it does not encourage the model to learn well-separated feature representations. Thus, it is easy to induce adversarial examples. In this article, we propose a simple, yet efficient defense method from the geometric constraint perspective. Specifically, we first propose an M-similarity metric to shrink variances of intraclass features. Intuitively, better geometric conditions of feature space can bring lower misclassification probability (MP). Therefore, we derive the optimal geometric constraints of anchors within each category from the overall MP (OMP) with theoretical guarantees. Due to the nonconvex characteristic of the optimal geometric condition, it is hard to satisfy the traditional optimization process. To this end, we regard such geometric constraints as manifold optimization processes in the Stiefel manifold, thus naturally avoiding the above challenges. Experimental results demonstrate that our method can significantly improve robustness compared with baselines, while retaining the excellent performance on normal examples.

Using ChatGPT to Annotate a Dataset: A Case Study in Intelligent Tutoring Systems

Large language models like ChatGPT can learn in-context (ICL) from examples. Studies showed that, due to ICL, ChatGPT achieves impressive performance in various natural language processing tasks. However, to the best of our knowledge, this is the first study that assesses ChatGPT's effectiveness in annotating a dataset for training instructor models in intelligent tutoring systems (ITSs). The task of an ITS instructor model is to automatically provide effective tutoring instruction given a student's state, mimicking human instructors. These models are typically implemented as hardcoded rules, requiring expertise, and limiting their ability to generalize and personalize instructions. These problems could be mitigated by utilizing machine learning (ML). However, developing ML models requires a large dataset of student states annotated by corresponding tutoring instructions. Using human experts to annotate such a dataset is expensive, time-consuming, and requires pedagogical expertise. Thus, this study explores ChatGPT's potential to act as a pedagogy expert annotator. Using prompt engineering, we created a list of instructions a tutor could recommend to a student. We manually filtered this list and instructed ChatGPT to select the appropriate instruction from the list for the given student's state. We manually analyzed ChatGPT's responses that could be considered incorrectly annotated. Our results indicate that using ChatGPT as an annotator is an effective alternative to human experts. The contributions of our work are (1) a novel dataset annotation methodology for the ITS, (2) a publicly available dataset of student states annotated with tutoring instructions, and (3) a list of possible tutoring instructions.

SAMT-generator: A second-attention for image captioning based on multi-stage transformer network

In recent years, Transformer has been widely used in the crossing task of computer vision (CV) and natural language processing (NLP), e.g., image captioning. The prior works of image captioning based on Transformer have achieved remarkable progress. In order to further improve the model’s ability to describe detail features and generate high-quality sentences, in the paper, we first construct a Multi-stage Transformer Feature Enhancement Network (MT-FEN), which obtains more semantics by fusing features at different scales. Furthermore, we crucially propose a novel second-attention (SA) that can focus more on valuable features and filter out noises in MT-FEN. Besides that, to generate more sensible sentences, we use multiple decoding layers to choose the best-described word via the Max-out module. The overall model is dubbed as SAMT-Generator. After conducting extensive experiments on two well-known datasets: MS COCO and Flickr8k, the results demonstrate the effectiveness of our proposed model, and it is comparable with SOTA methods.

Fast, Simple, and Accurate Time Series Analysis with Large Language Models: An Example of Mean-motion Resonances Identification

Classical machine learning has been actively utilized in astronomy to address various challenges, including predicting orbital stability, classifying asteroids, galaxies, and other objects, and analyzing images. However, the emerging trend in artificial intelligence involves the use of large language models such as GPT-4 and ChatGPT. These models are trained on a large corpus of text and can perform a wide range of natural language processing tasks, including text generation, translation, summarization, and classification. Surprisingly, these capabilities present significant potential for application in astronomy. This paper demonstrates how the new model gpt-4-vision-preview can analyze visual patterns and accurately classify asteroids as resonant or nonresonant with high accuracy. This process requires no training, fine-tuning, or coding beyond writing the appropriate prompt in natural language. Moreover, this approach can be extended to other common problems within astronomy.

A Developed Graphical User Interface-Based on Different Generative Pre-trained Transformers Models

Objective: The article investigates the integration of advanced Generative Pretrained Transformers (GPT) models into a user-friendly Graphical User Interface (GUI). The primary objective of this work is to simplify access to complex Natural Language Processing (NLP) tasks for a diverse range of users, including those with limited technical background. Method: The development process of the GUI was comprehensive and systematic: Needs Assessment: This stage involved understanding the requirements and expectations of potential users to ensure the GUI effectively addresses their needs. Preliminary Design and Development: The initial designs were created and developed into a functional GUI, emphasizing the integration of features supporting various NLP tasks like text summarization, translation, and question-answering. Iterative Refinement: Continuous improvements were made based on user feedback, focusing on enhancing user experience, ease of navigation, and customization capabilities. Results: The developed GUI successfully integrated GPT models, including GPT-4 Turbo and GPT-3.5, resulting in an intuitive and adaptable interface. It demonstrated efficiency in performing various NLP tasks, thereby making these advanced language processing tools accessible to a broader audience. The GUI's design, emphasizing user-friendliness and adaptability, was particularly noted for its ability to cater to both technical and non-technical users. Conclusion: In conclusion, the article illustrates the significant impact of combining advanced GPT models with a Graphical User Interface to democratize the use of NLP tools. This integration not only makes complex language processing more accessible but also marks a pivotal step in the inclusive application of AI technology across various domains. The successful implementation of the GUI highlights the potential of AI in enhancing user interaction and broadening the scope of technology usage in everyday tasks.