Field Of Natural Language Processing Research Articles

Exploring the relationship between team institutional composition and novelty in academic papers based on fine-grained knowledge entities

Purpose The composition of author teams is a significant factor affecting the novelty of academic papers. Existing research lacks studies focusing on institutional types and measures of novelty remained at a general level, making it difficult to analyse the types of novelty in papers and to provide a detailed explanation of novelty. This study aims to take the field of natural language processing (NLP) as an example to analyse the relationship between team institutional composition and the fine-grained novelty of academic papers. Design/methodology/approach Firstly, author teams are categorized into three types: academic institutions, industrial institutions and mixed academic and industrial institutions. Next, the authors extract four types of entities from the full paper: methods, data sets, tools and metric. The novelty of papers is evaluated using entity combination measurement methods. Additionally, pairwise combinations of different types of fine-grained entities are analysed to assess their contributions to novel papers. Findings The results of the study found that in the field of NLP, for industrial institutions, collaboration with academic institutions has a higher probability of producing novel papers. From the contribution rate of different types of fine-grained knowledge entities, the mixed academic and industrial institutions pay more attention to the novelty of the combination of method indicators, and the industrial institutions pay more attention to the novelty of the combination of method tools. Originality/value This paper explores the relationship between the team institutional composition and the novelty of academic papers and reveals the importance of cooperation between industry and academia through fine-grained novelty measurement, which provides key guidance for improving the quality of papers and promoting industry–university–research cooperation.

ChatGPT With GPT-4 Outperforms Emergency Department Physicians in Diagnostic Accuracy: Retrospective Analysis.

OpenAI's ChatGPT is a pioneering artificial intelligence (AI) in the field of natural language processing, and it holds significant potential in medicine for providing treatment advice. Additionally, recent studies have demonstrated promising results using ChatGPT for emergency medicine triage. However, its diagnostic accuracy in the emergency department (ED) has not yet been evaluated. This study compares the diagnostic accuracy of ChatGPT with GPT-3.5 and GPT-4 and primary treating resident physicians in an ED setting. Among 100 adults admitted to our ED in January 2023 with internal medicine issues, the diagnostic accuracy was assessed by comparing the diagnoses made by ED resident physicians and those made by ChatGPT with GPT-3.5 or GPT-4 against the final hospital discharge diagnosis, using a point system for grading accuracy. The study enrolled 100 patients with a median age of 72 (IQR 58.5-82.0) years who were admitted to our internal medicine ED primarily for cardiovascular, endocrine, gastrointestinal, or infectious diseases. GPT-4 outperformed both GPT-3.5 (P<.001) and ED resident physicians (P=.01) in diagnostic accuracy for internal medicine emergencies. Furthermore, across various disease subgroups, GPT-4 consistently outperformed GPT-3.5 and resident physicians. It demonstrated significant superiority in cardiovascular (GPT-4 vs ED physicians: P=.03) and endocrine or gastrointestinal diseases (GPT-4 vs GPT-3.5: P=.01). However, in other categories, the differences were not statistically significant. In this study, which compared the diagnostic accuracy of GPT-3.5, GPT-4, and ED resident physicians against a discharge diagnosis gold standard, GPT-4 outperformed both the resident physicians and its predecessor, GPT-3.5. Despite the retrospective design of the study and its limited sample size, the results underscore the potential of AI as a supportive diagnostic tool in ED settings.

Green AI: exploring carbon footprints, mitigation strategies, and trade offs in large language model training

Prominent works in the space of Natural Language Processing (NLP) have long attempted to create new innovative models by improving upon previous model training approaches, altering model architecture, and developing more in-depth datasets to better their performance. However, with the quickly advancing field of NLP comes increased greenhouse gas emissions, posing concerns over the environmental damage caused by training LLMs. Gaining a comprehensive understanding of the various costs, particularly those pertaining to environmental aspects, that are associated with artificial intelligence serves as the foundational basis for ensuring safe AI models. Currently, investigations into the CO2 emissions of AI models remain an emerging area of research, and as such, we evaluate the CO2 emissions of well-known large language models, which have an especially high carbon footprint due to their significant amount of model parameters. We argue for the training of LLMs in a way that is responsible and sustainable by suggesting measures for reducing carbon emissions. Furthermore, we discuss how the choice of hardware affects CO2 emissions by contrasting the CO2 emissions during model training for two widely used GPUs. Based on our results, we present the benefits and drawbacks of our proposed solutions and make the argument for the possibility of training more environmentally safe AI models without sacrificing their robustness and performance.

Natural Language Processing for Ancient Greek

Abstract Computational methods have produced meaningful and usable results to study word semantics, including semantic change. These methods, belonging to the field of Natural Language Processing, have recently been applied to ancient languages; in particular, language modelling has been applied to Ancient Greek, the language on which we focus. In this contribution we explain how vector representations can be computed from word co-occurrences in a corpus and can be used to locate words in a semantic space, and what kind of semantic information can be extracted from language models. We compare three different kinds of language models that can be used to study Ancient Greek semantics: a count-based model, a word embedding model and a syntactic embedding model; and we show examples of how the quality of their representations can be assessed. We highlight the advantages and potential of these methods, especially for the study of semantic change, together with their limitations.

Research on the Training and Application Methods of a Lightweight Agricultural Domain-Specific Large Language Model Supporting Mandarin Chinese and Uyghur

In the field of Natural Language Processing (NLP), the lack of support for minority languages, especially Uyghur, the scarcity of Uyghur language corpora in the agricultural domain, and the lightweight nature of large language models remain prominent issues. This study proposes a method for constructing a bilingual (Uyghur and Chinese) lightweight specialized large language model for the agricultural domain. By utilizing a mixed training approach of Uyghur and Chinese, we extracted Chinese corpus text from agricultural-themed books in PDF format using OCR (Optical Character Recognition) technology, converted the Chinese text corpus into a Uyghur corpus using a rapid translation API, and constructed a bilingual mixed vocabulary. We applied the parameterized Transformer model algorithm to train the model for the agricultural domain in both Chinese and Uyghur. Furthermore, we introduced a context detection and fail-safe mechanism for the generated text. The constructed model possesses the ability to support bilingual reasoning in Uyghur and Chinese in the agricultural domain, with higher accuracy and a smaller size that requires less hardware. It (our work) addresses issues such as the scarcity of Uyghur corpora in the agricultural domain, mixed word segmentation and word vector modeling in Uyghur for widespread agricultural languages, model lightweighting and deployment, and the fragmentation of non-relevant texts during knowledge extraction from small-scale corpora. The lightweight design of the model reduces hardware requirements, facilitating deployment in resource-constrained environments. This advancement promotes agricultural intelligence, aids in the development of specific applications and minority languages (such as agriculture and Uyghur), and contributes to rural revitalization.

Every Problem, Every Step, All in Focus: Learning to Solve Vision-Language Problems With Integrated Attention.

Integrating information from vision and language modalities has sparked interesting applications in the fields of computer vision and natural language processing. Existing methods, though promising in tasks like image captioning and visual question answering, face challenges in understanding real-life issues and offering step-by-step solutions. In particular, they typically limit their scope to solutions with a sequential structure, thus ignoring complex inter-step dependencies. To bridge this gap, we propose a graph-based approach to vision-language problem solving. It leverages a novel integrated attention mechanism that jointly considers the importance of features within each step as well as across multiple steps. Together with a graph neural network method, this attention mechanism can be progressively learned to predict sequential and non-sequential solution graphs depending on the characterization of the problem-solving process. To tightly couple attention with the problem-solving procedure, we further design new learning objectives with attention metrics that quantify this integrated attention, which better aligns visual and language information within steps, and more accurately captures information flow between steps. Experimental results on VisualHow, a comprehensive dataset of varying solution structures, show significant improvements in predicting steps and dependencies, demonstrating the effectiveness of our approach in tackling various vision-language problems.

Effectiveness Analysis of Entrepreneurial Legal Risk Prevention Based on Multi-Modal Deep Learning Model

With the emergence of the upsurge of entrepreneurship, entrepreneurs are increasingly concerned about legal risks. In the process of entrepreneurship, legal risk is the biggest hidden danger of entrepreneurial enterprises. The prevention and avoidance of legal risks is a long and arduous process, and not all risks can be identified and avoided in time. The deep learning method has brought great changes to the fields of speech recognition, image recognition and natural language processing. The tasks in these fields only involve single-mode input, but more recent applications need to involve multi-mode intelligence. Multimodal deep learning mainly includes three aspects: multimodal learning representation, multimodal signal fusion and multimodal application. Through the research on the legal risks in recent years, this paper believed that there are many legal problems in the development of entrepreneurial enterprises, including contract disputes, trade secret disputes, trademark infringement disputes, copyright infringement disputes, and so on. This paper aimed to study the problem of Legal Risks of Entrepreneurship (LRE) that entrepreneurs are concerned about, and proposed a new solution from the perspective of image recognition technology. 486 entrepreneurs were investigated by questionnaire. In the process of entrepreneurship, 87.04% of entrepreneurs encountered legal risks, and they would turn to lawyers for help, which is a better way. However, in most cases, entrepreneurs are exposed to LRE because of their insufficient understanding of economic law, so they solve it through lawyers. However, if a lawyer is hired, the cost would be very high, which would bring great economic pressure to the enterprise. Only 17.90% of entrepreneurs would safeguard their legitimate rights and interests through their own knowledge and legal weapons without resorting to lawyers. It can be seen that entrepreneurs have relatively low practical ability in the use of LRE, and their legal practical ability is obviously insufficient.

ESMSec: Prediction of Secreted Proteins in Human Body Fluids Using Protein Language Models and Attention.

The secreted proteins of human body fluid have the potential to be used as biomarkers for diseases. These biomarkers can be used for early diagnosis and risk prediction of diseases, so the study of secreted proteins of human body fluid has great application value. In recent years, the deep-learning-based transformer language model has transferred from the field of natural language processing (NLP) to the field of proteomics, leading to the development of protein language models (PLMs) for protein sequence representation. Here, we propose a deep learning framework called ESM Predict Secreted Proteins (ESMSec) to predict three types of proteins secreted in human body fluid. The ESMSec is based on the ESM2 model and attention architecture. Specifically, the protein sequence data are firstly put into the ESM2 model to extract the feature information from the last hidden layer, and all the input proteins are encoded into a fixed 1000 × 480 matrix. Secondly, multi-head attention with a fully connected neural network is employed as the classifier to perform binary classification according to whether they are secreted into each body fluid. Our experiment utilized three human body fluids that are important and ubiquitous markers. Experimental results show that ESMSec achieved average accuracy of 0.8486, 0.8358, and 0.8325 on the testing datasets for plasma, cerebrospinal fluid (CSF), and seminal fluid, which on average outperform the state-of-the-art (SOTA) methods. The outstanding performance results of ESMSec demonstrate that the ESM can improve the prediction performance of the model and has great potential to screen the secretion information of human body fluid proteins.

Exploring motivations for algorithm mention in the domain of natural language processing: A deep learning approach

With the formation of the fourth paradigm of scientific research, algorithms have become increasingly important in scientific research. In academic papers, algorithms may be mentioned by scholars with various motivations, using, comparing, or improving algorithms to solve complex research tasks. Identifying these motivations can help scholars discover the relationships between algorithms and further assess their roles and values. Therefore, taking the field of natural language processing (NLP) as an example, this article proposes a complete method to conduct the identification, distribution, and evolution of motivations for mentioning algorithms at the sentence level. Specifically, using manual annotation and machine learning methods, we identify algorithm entities and sentences in the full text of papers, classify motivations for mentioning algorithms by pre-training models and data augmentation techniques, and finally analyze the distribution and evolution of motivations. The results show that the deep learning models trained with the augmented data outperform the traditional machine learning models in the classification task. In academic papers, more than half of the sentences show the direct use of algorithms, while the lowest percentage of motivations are improving algorithms, and the diversity of motivations has been increasing with time. For specific algorithms, grammatical algorithms are mentioned more by the motivation of “description,” while more motivations of “use” are found in the machine learning algorithms category. As time passed, the “use” motivations gradually replaced the “description” motivations for different algorithms, and the number of motivation types decreased significantly. Our research explores the identification, distribution, and evolution of authors’ motivations for mentioning algorithm entities, which could provide a basis for future algorithm relationship identification and influence evaluation using motivations.

FuzzyTP-BERT: Enhancing extractive text summarization with fuzzy topic modeling and transformer networks

In the rapidly evolving field of natural language processing, the demand for efficient automated text summarization systems that not only distill extensive documents but also capture their nuanced thematic elements has never been greater. This paper introduces the FuzzyTP-BERT framework, a novel approach in extractive text summarization that synergistically combines Fuzzy Topic Modeling (FuzzyTM) with the advanced capabilities of Bidirectional Encoder Representations from Transformers (BERT). Unlike traditional extractive methods, FuzzyTP-BERT integrates fuzzy logic to refine topic modeling, enhancing the semantic sensitivity of summaries by allowing a more nuanced representation of word-topic relationships. This integration results in summaries that are not only coherent but also thematically rich, addressing a significant gap in current summarization technology. Extensive evaluations on benchmark datasets demonstrate that FuzzyTP-BERT significantly outperforms existing models in terms of ROUGE scores, effectively balancing topical relevance with semantic coherence. Our findings suggest that incorporating fuzzy logic into deep learning frameworks can markedly improve the quality of automated text summaries, potentially benefiting a wide range of applications in the information overload age.

FedREAS: A Robust Efficient Aggregation and Selection Framework for Federated Learning

In the field of Natural Language Processing (NLP), Deep Learning (DL) and Neural Network (NN) technologies have been widely applied to machine translation and sentiment analysis and have demonstrated outstanding performance. In recent years, NLP applications have also combined multimodal data, such as visual and audio, continuously improving language processing performance. At the same time, the size of Neural Network models is increasing, and many models cannot be deployed on devices with limited computing resources. Deploying models on cloud platforms has become a trend. However, deploying models in the cloud introduces new privacy risks for endpoint data, despite overcoming computational limitations. Federated Learning (FL) methods protect local data by keeping the data on the client side and only sending local updates to the central server. However, the FL architecture still has problems, such as vulnerability to adversarial attacks and non-IID data distribution. In this work, we propose a Federated Learning aggregation method called FedREAS. The server uses a benchmark dataset to train a global model and obtains benchmark updates in this method. Before aggregating local updates, the server adjusts the local updates using the benchmark updates and then returns the adjusted benchmark updates. Then, based on the similarity between the adjusted local updates and the adjusted benchmark updates, the server aggregates these local updates to obtain a more robust update. This method also improves the client selection process. FedREAS selects suitable clients for training at the beginning of each round based on specific strategies, the similarity of the previous round’s updates, and the submitted data. We conduct experiments on different datasets and compare FedREAS with other Federated Learning methods. The results show that FedREAS outperforms other methods regarding model performance and resistance to attacks.

From text to model: Leveraging natural language processing for system dynamics model development

AbstractTextual data is abundantly available, and natural language processing (NLP) facilitates its analysis. However, system dynamics (SD) modelling relies on the modeller to identify relevant information. We explore the ability of NLP models to support SD modelling by identifying causal sentences in texts. We provide a primer on the notion of causality in SD and on the linguistic properties of causality, followed by an introduction of NLP models suitable for this task. Using three test cases, we evaluate the performance of the NLP models using common evaluation metrics and an SD model completeness metric. We conclude that NLP models can add considerable value to SD modelling, provided that remaining challenges are addressed. One such caveat is the difference we observe between information regarded as causal and information relevant for describing system structure. We discuss how these challenges can be addressed through collaboration between the NLP and SD fields. © 2024 The Author(s). System Dynamics Review published by John Wiley &amp; Sons Ltd on behalf of System Dynamics Society.

Transformative Trends: A Comprehensive Review of Large Language Models (LLMs) in Healthcare

Recent years have witnessed notable progress in Language Model (LM) technology, especially with the introduction of Large Language Models (LLMs). With the use of complex neural networks and enormous volumes of data, these models—like the GPT (Generative Pre-trained Transformer) series—have completely changed the field of natural language processing. This study offers a thorough analysis of LLMs, examining their uses as well as the difficulties and moral issues that come with implementing LLMs in practical settings. Moreover, it investigates the wide range of uses of LLMs in many fields, such as question answering, drug discovery, language translation, and text production. By means of this review, we hope to offer ideas on future directions in this quickly developing field as well as insights into the state-of-the-art in LLM research today. Key Words: Large Language Models, LangChain, Natural language processing, GPT, ChatGPT, AI chatbots, Indirect Prompt Injection

A Survey of Visual Transformers.

Transformer, an attention-based encoder-decoder model, has already revolutionized the field of natural language processing (NLP). Inspired by such significant achievements, some pioneering works have recently been done on employing Transformer-liked architectures in the computer vision (CV) field, which have demonstrated their effectiveness on three fundamental CV tasks (classification, detection, and segmentation) as well as multiple sensory data stream (images, point clouds, and vision-language data). Because of their competitive modeling capabilities, the visual Transformers have achieved impressive performance improvements over multiple benchmarks as compared with modern convolution neural networks (CNNs). In this survey, we have reviewed over 100 of different visual Transformers comprehensively according to three fundamental CV tasks and different data stream types, where taxonomy is proposed to organize the representative methods according to their motivations, structures, and application scenarios. Because of their differences on training settings and dedicated vision tasks, we have also evaluated and compared all these existing visual Transformers under different configurations. Furthermore, we have revealed a series of essential but unexploited aspects that may empower such visual Transformers to stand out from numerous architectures, e.g., slack high-level semantic embeddings to bridge the gap between the visual Transformers and the sequential ones. Finally, two promising research directions are suggested for future investment. We will continue to update the latest articles and their released source codes at https://github.com/liuyang-ict/awesome-visual-transformers.

From RNN techniques to pre-trained models: emphasis on the use in Arabic machine translation

&lt;p&gt;Neural Machine Translation (NMT) has gained increasing attention in recent years due to its promising performance over conventional approaches such as Statistical Machine Translation. Nevertheless, when applied to languages with different structures, such as the pair (English, Arabic) that interests us in this work, its efficiency is degraded.&lt;br /&gt;Alternatively, the use of unsupervised pre-training of large neural models has recently made a significant leap forward in the field of Natural Language Processing (NLP). By warm starting from published checkpoints, NLP practitioners have been able to push the state-of-the-art boundaries on multiple benchmarks while simultaneously saving considerable computational time. The emphasis so far has been mainly on natural language understanding problems. In this paper, we show the efficacy of pre-trained checkpoints for Arabic Machine Translation. We have evolved a transformer-based sequence-to-sequence model which is compatible with pre-trained checkpoints publicly available for Arabic Bidirectional Encoder Representations from Transformers (AraBERT) and Arabic Generative Pre-trained Transformer (AraGPT) and conducted a thorough empirical study on the usefulness of initializing our Arabic MT model, both encoder and decoder, with these checkpoints. Our models yield new state-of-the-art results in Arabic MT.&lt;/p&gt;

Evaluating the machine learning models based on natural language processing tasks

&lt;span lang="EN-US"&gt;In the realm of natural language processing (NLP), a diverse array of language models has emerged, catering to a wide spectrum of tasks, ranging from speaker recognition and auto-correction to sentiment analysis and stock prediction. The significance of language models in enabling the execution of these NLP tasks cannot be overstated. This study proposes an approach to enhance accuracy by leveraging a hybrid language model, combining the strengths of long short-term memory (LSTM) and gated recurrent unit (GRU). LSTM excels in preserving long-term dependencies in data, while GRU's simpler gating mechanism expedites the training process. The research endeavors to evaluate four variations of this hybrid model: LSTM, GRU, &lt;a name="_Hlk158992630"&gt;&lt;/a&gt;bidirectional long short-term memory (Bi-LSTM), and a combination of LSTM with GRU. These models are subjected to rigorous testing on two distinct datasets: one focused on IBM stock price prediction, and the other on Jigsaw toxic comment classification (sentiment analysis). This work represents a significant stride towards democratizing NLP capabilities, ensuring that even in resource-constrained settings, NLP models can exhibit improved performance. The anticipated implications of these findings span a wide spectrum of real-world applications and hold the potential to stimulate further research in the field of NLP. &lt;/span&gt;

Predicting multi-label emojis, emotions, and sentiments in code-mixed texts using an emojifying sentiments framework

In the era of social media, the use of emojis and code-mixed language has become essential in online communication. However, selecting the appropriate emoji that matches a particular sentiment or emotion in the code-mixed text can be difficult. This paper presents a novel task of predicting multiple emojis in English-Hindi code-mixed sentences and proposes a new dataset called SENTIMOJI, which extends the SemEval 2020 Task 9 SentiMix dataset. Our approach is based on exploiting the relationship between emotion, sentiment, and emojis to build an end-to-end framework. We replace the self-attention sublayers in the transformer encoder with simple linear transformations and use the RMS-layer norm instead of the normal layer norm. Moreover, we employ Gated Linear Unit and Fully Connected layers to predict emojis and identify the emotion and sentiment of a tweet. Our experimental results on the SENTIMOJI dataset demonstrate that the proposed multi-task framework outperforms the single-task framework. We also show that emojis are strongly linked to sentiment and emotion and that identifying sentiment and emotion can aid in accurately predicting the most suitable emoji. Our work contributes to the field of natural language processing and can help in the development of more effective tools for sentiment analysis and emotion recognition in code-mixed languages. The codes and data will be available at https://www.iitp.ac.in/~ai-nlp-ml/resources.html#SENTIMOJI to facilitate research.

Sample-Adaptive Classification Inference Network

Existing pre-trained models have yielded promising results in terms of computational time reduction. However, these models only focus on pruning simple sentences or less salient words, while neglecting the treatment of relatively complex sentences. It is frequently these sentences that cause the loss of model accuracy. This shows that the adaptation of the existing models is one-sided. To address this issue, in this paper, we propose a sample-adaptive training and inference model. Specifically, complex samples are extracted from the training datasets and a dedicated data augmentation module is trained to extract global and local semantic information of complex samples. During inference, simple samples can exit the model via the Sample Adaptive Exit Mechanism, Normal samples pass through the whole backbone model before inference, while complex samples are processed by the Characteristic Enhancement Module after passing through the backbone model. In this way, all samples are processed adaptively. Our extensive experiments on classification tasks datasets in the field of Natural Language Processing demonstrate that our method enhances model accuracy and reduces model inference time for multiple datasets. Moreover, our method is transferable and can be applied to multiple pre-trained models.

Identifying widespread and recurrent variants of genetic parts to improve annotation of engineered DNA sequences.

Engineered plasmids have been workhorses of recombinant DNA technology for nearly half a century. Plasmids are used to clone DNA sequences encoding new genetic parts and to reprogram cells by combining these parts in new ways. Historically, many genetic parts on plasmids were copied and reused without routinely checking their DNA sequences. With the widespread use of high-throughput DNA sequencing technologies, we now know that plasmids often contain variants of common genetic parts that differ slightly from their canonical sequences. Because the exact provenance of a genetic part on a particular plasmid is usually unknown, it is difficult to determine whether these differences arose due to mutations during plasmid construction and propagation or due to intentional editing by researchers. In either case, it is important to understand how the sequence changes alter the properties of the genetic part. We analyzed the sequences of over 50,000 engineered plasmids using depositor metadata and a metric inspired by the natural language processing field. We detected 217 uncatalogued genetic part variants that were especially widespread or were likely the result of convergent evolution or engineering. Several of these uncatalogued variants are known mutants of plasmid origins of replication or antibiotic resistance genes that are missing from current annotation databases. However, most are uncharacterized, and 3/5 of the plasmids we analyzed contained at least one of the uncatalogued variants. Our results include a list of genetic parts to prioritize for refining engineered plasmid annotation pipelines, highlight widespread variants of parts that warrant further investigation to see whether they have altered characteristics, and suggest cases where unintentional evolution of plasmid parts may be affecting the reliability and reproducibility of science.

Textual Compression Using Lamini-LM

In the age of digital overload, the ability to efficiently summarize large amount of text has become a critical capability. This project focuses on the development of a document summarization system that uses the power of the LaMini-LM model, a member of the Langchain family of large language models (LLMs). By using the natural language processing (NLP) techniques through the Langchain LLM technology it creates a robust and practical solution for automatically generating concise and informative summaries of text documents. The methodological approach of this project combines the strengths of NLP techniques, including the transformer architecture, the T5 model, and specialized components such as the T5 tokenizer and the Pipeline API. This strategic integration of technologies allows to create a comprehensive and efficient document summarization system that can effectively process and summarize a diverse range of text documents. The successful development and implementation of the document summarization system using the LaMini-LM model represents a significant advancement in the field of natural language processing.