Field Of Natural Language Processing Research Articles

A new computationally efficient method to tune BERT networks – transfer learning

BERT is a pre-trained language model that achieves state-of-the-art performance on natural language processing (NLP) tasks. Once it was published, it quickly became one of the most popular models in the NLP field. The official recommended method for applying BERT to downstream tasks is fine-tuning. However, we argue that transfer learning is also a very practical approach to applying BERT, and this approach has its own advantages compared to fine-tuning. In this paper, we explore the advantages of the transfer learning approach through the method that uses transfer learning and fine-tuning approaches separately to apply BERT to the same eight GLUE benchmark tasks and compare the training time spent and model performance scores, such as accuracy or F1 score, of the two approaches. Finally, this paper finds that among all eight GLUE tasks and on small training sets, transfer learning saves 30% to 50% of the time compared to a fine-tuning approach to train the models on the data of downstream tasks and can achieve very similar performance. Besides, for the same amount of time, transfer learning can obtain higher performance scores than fine-tuning on larger training sets. In conclusion, on small training sets, transfer learning has a huge advantage in time consumption under the premise of approximate performance, and it also performs better on large training sets under the same length of time, compared to fine-tuning.

Semantic micro-dynamics as a reflex of occurrence frequency: a semantic networks approach

Abstract This article correlates fine-grained semantic variability and change with measures of occurrence frequency to investigate whether a word’s degree of semantic change is sensitive to how often it is used. We show that this sensitivity can be detected within a short time span (i.e., 20 years), basing our analysis on a large corpus of German allowing for a high temporal resolution (i.e., per month). We measure semantic variability and change with the help of local semantic networks, combining elements of deep learning methodology and graph theory. Our micro-scale analysis complements previous macro-scale studies from the field of natural language processing, corroborating the finding that high token frequency has a negative effect on the degree of semantic change in a lexical item. We relate this relationship to the role of exemplars for establishing form–function pairings between words and their habitual usage contexts.

TBC-YOLOv7: a refined YOLOv7-based algorithm for tea bud grading detection.

Accurate grading identification of tea buds is a prerequisite for automated tea-picking based on machine vision system. However, current target detection algorithms face challenges in detecting tea bud grades in complex backgrounds. In this paper, an improved YOLOv7 tea bud grading detection algorithm TBC-YOLOv7 is proposed. The TBC-YOLOv7 algorithm incorporates the transformer architecture design in the natural language processing field, integrating the transformer module based on the contextual information in the feature map into the YOLOv7 algorithm, thereby facilitating self-attention learning and enhancing the connection of global feature information. To fuse feature information at different scales, the TBC-YOLOv7 algorithm employs a bidirectional feature pyramid network. In addition, coordinate attention is embedded into the critical positions of the network to suppress useless background details while paying more attention to the prominent features of tea buds. The SIOU loss function is applied as the bounding box loss function to improve the convergence speed of the network. The results of the experiments indicate that the TBC-YOLOv7 is effective in all grades of samples in the test set. Specifically, the model achieves a precision of 88.2% and 86.9%, with corresponding recall of 81% and 75.9%. The mean average precision of the model reaches 87.5%, 3.4% higher than the original YOLOv7, with average precision values of up to 90% for one bud with one leaf. Furthermore, the F1 score reaches 0.83. The model's performance outperforms the YOLOv7 model in terms of the number of parameters. Finally, the results of the model detection exhibit a high degree of correlation with the actual manual annotation results ( =0.89), with the root mean square error of 1.54. The TBC-YOLOv7 model proposed in this paper exhibits superior performance in vision recognition, indicating that the improved YOLOv7 model fused with transformer-style module can achieve higher grading accuracy on densely growing tea buds, thereby enables the grade detection of tea buds in practical scenarios, providing solution and technical support for automated collection of tea buds and the judging of grades.

A Comprehensive Study of ChatGPT: Advancements, Limitations, and Ethical Considerations in Natural Language Processing and Cybersecurity

This paper presents an in-depth study of ChatGPT, a state-of-the-art language model that is revolutionizing generative text. We provide a comprehensive analysis of its architecture, training data, and evaluation metrics and explore its advancements and enhancements over time. Additionally, we examine the capabilities and limitations of ChatGPT in natural language processing (NLP) tasks, including language translation, text summarization, and dialogue generation. Furthermore, we compare ChatGPT to other language generation models and discuss its applicability in various tasks. Our study also addresses the ethical and privacy considerations associated with ChatGPT and provides insights into mitigation strategies. Moreover, we investigate the role of ChatGPT in cyberattacks, highlighting potential security risks. Lastly, we showcase the diverse applications of ChatGPT in different industries and evaluate its performance across languages and domains. This paper offers a comprehensive exploration of ChatGPT’s impact on the NLP field.

Hybrid photonic deep convolutional residual spiking neural networks for text classification.

Spiking neural networks (SNNs) offer powerful computation capability due to its event-driven nature and temporal processing. However, it is still limited to shallow structure and simple tasks due to the training difficulty. In this work, we propose a deep convolutional residual spiking neural network (DCRSNN) for text classification tasks. In the DCRSNN, the feature extraction is achieved via a convolution SNN with residual connection, using the surrogate gradient direct training technique. Classification is performed by a fully-connected network. We also suggest a hybrid photonic DCRSNN, in which photonic SNNs are used for classification with a converted training method. The accuracy of hard and soft reset methods, as well as three different surrogate functions, were evaluated and compared across four different datasets. Results indicated a maximum accuracy of 76.36% for MR, 91.03% for AG News, 88.06% for IMDB and 93.99% for Yelp review polarity. Soft reset methods used in the deep convolutional SNN yielded slightly better accuracy than their hard reset counterparts. We also considered the effects of different pooling methods and observation time windows and found that the convergence accuracy achieved by convolutional SNNs was comparable to that of convolutional neural networks under the same conditions. Moreover, the hybrid photonic DCRSNN also shows comparable testing accuracy. This work provides new insights into extending the SNN applications in the field of text classification and natural language processing, which is interesting for the resources-restrained scenarios.

BamnetTL: Bidirectional Attention Memory Network with Transfer Learning for Question Answering Matching

In KBQA (knowledge base question answering), questions are processed using NLP (natural language processing), and knowledge base technology is used to generate the corresponding answers. KBQA is one of the most challenging tasks in the field of NLP. Q&A (question and answer) matching is an important part of knowledge base QA (question answering), in which the correct answer is selected from candidate answers. At present, Q&A matching task faces the problem of lacking training data in new fields, which leads to poor performance and low efficiency of the question answering system. The paper puts forward a KBQA Q&A matching model for deep feature transfer based on a bidirectional attention memory network, BamnetTL. It uses biattention to collect information from the knowledge base and question sentences in both directions in order to improve the accuracy of Q&A matching and transfers knowledge from different fields through a deep dynamic adaptation network. BamnetTL improves the accuracy of Q&A matching in the target domain by transferring the knowledge in the source domain with more training resources to the target domain with fewer training resources. The experimental results show that the proposed method is effective.

Low-Resource Language Information Processing using Dwarf Mongoose Optimization with Deep Learning Based Sentiment Classification

Asian and low-resource language information processing refers to the field of computational linguistics that aims to develop natural language processing (NLP) technologies for languages that have fewer available language resources or are less commonly spoken. This is an important field of study because many languages in Asia and other parts of the world are underrepresented in the field of NLP, which may limit access to information and technology for speakers of these languages. The growing volume of user-generated content on the web has made sentiment analysis (SA) a significant tool for extracting data regarding human emotional states. Twitter sentiment detectors provide a superior solution for assessing the quality of products and services compared to other conventional technologies. The detection performance and classifier accuracy of SA, which can be highly dependent on classifier methods and the quality of input features have been utilised. Deep learning (DL) methods use distinct techniques to extract data from raw data such as tweets or texts and represent them in different forms of models. Therefore, this article presents a Dwarf Mongoose Optimization with Deep Learning-Based Twitter Sentiment Classification (DMODL-TSC) technique to classify sentiments based on tweets. The presented DMODL-TSC technique leverages the concepts of natural language processing (NLP) and DL. Primarily, the raw tweets are preprocessed to transform them into a useful format. Next, the DMODL-TSC technique uses the advanced FastText word embedding technique. Moreover, the bidirectional recurrent neural network (BiRNN) method is utilized for the recognition of sentiments. Finally, the DMO technique is utilized for the optimal hyperparameter optimization of the BiRNN method, which leads to effective classification performance. The comprehensive result examination of the DMODL-TSC system was tested on three datasets, and the obtained outcomes illustrate the supremacy of the DMODL-TSC approach.

Sturdies Advanced in chatbots based on Deep learning

Chat robots have always been a hot research topic in the field of natural language processing and human-computer interaction, aiming to build models to understand human input and provide logical answers. Early chat robots were mostly based on matching or rules, but due to the lack of manual feature representation capabilities and the complexity of rules, they could not meet practical application requirements. Thanks to the rapid development of deep learning, chat robots based on convolutional neural networks have made breakthroughs in accuracy and speed in recent years. According to the different design ideas of chat robots, the existing work mainly includes two categories: retrieval type chat robots and generation type robots. Focusing on the two main frameworks mentioned above, this article introduces the latest research progress of chat robots in detail, including the basic steps, design ideas, advantages and disadvantages of representative algorithms. We also introduced common chat robot modeling data sets and evaluation indicators. Finally, we discussed the existing problems and future development directions in the field of chat robots.

FAAQA-QAD: A Frequently Asked Arabic Question Answering Dataset for Diseases Detection System

Domain-specific question-answering (QA) systems are a specialized field in Natural Language Processing. Their objective is to generate answers to questions within a specific domain, such as healthcare. The existing Arabic datasets are often characterized by their low quality, resulting from inadequate preprocessing or inaccurate annotation because that translated from another language.

Word Alignment of Chinese Poetry Parallel Corpus based on Word Embedding Technology

The research of word alignment technology provides the basis for many fields in natural language processing, such as speech recognition, bilingual dictionary writing, corpus construction and information retrieval. Based on the existing research foundation of parallel corpus for poetry sentence alignment, this paper studies how to align English-Chinese words. Firstly, the parallel corpus of Chinese poetry is preprocessed and translated into words, and the English poetry word vector is trained based on word embedding method, and the optimal corresponding word is judged by similarity calculation, and finally getting word alignment from parallel corpus sentence alignment.

The State of the Art of Natural Language Processing—A Systematic Automated Review of NLP Literature Using NLP Techniques

ABSTRACT Nowadays, natural language processing (NLP) is one of the most popular areas of, broadly understood, artificial intelligence. Therefore, every day, new research contributions are posted, for instance, to the arXiv repository. Hence, it is rather difficult to capture the current “state of the field” and thus, to enter it. This brought the id-art NLP techniques to analyse the NLP-focused literature. As a result, (1) meta-level knowledge, concerning the current state of NLP has been captured, and (2) a guide to use of basic NLP tools is provided. It should be noted that all the tools and the dataset described in this contribution are publicly available. Furthermore, the originality of this review lies in its full automation. This allows easy reproducibility and continuation and updating of this research in the future as new researches emerge in the field of NLP.

Grammar Correction for Multiple Errors in Chinese Based on Prompt Templates

Grammar error correction (GEC) is a crucial task in the field of Natural Language Processing (NLP). Its objective is to automatically detect and rectify grammatical mistakes in sentences, which possesses immense application research value. Currently, mainstream grammar-correction methods primarily rely on sequence labeling and text generation, which are two kinds of end-to-end methods. These methods have shown exemplary performance in areas with low error density but often fail to deliver satisfactory results in high-error density situations where multiple errors exist in a single sentence. Consequently, these methods tend to overcorrect correct words, leading to a high rate of false positives. To address this issue, we researched the specific characteristics of the Chinese grammar error correction (CGEC) task in high-error density situations. We proposed a grammar-correction method based on prompt templates. Firstly, we proposed a strategy for constructing prompt templates suitable for CGEC. This strategy transforms the CGEC task into a masked fill-in-the-blank task compatible with the masked language model BERT. Secondly, we proposed a method for dynamically updating templates, which incorporates already corrected errors into the template through dynamic updates to improve the template quality. Moreover, we used the phonetic and graphical resemblance knowledge from the confusion set as guiding information. By combining this with BERT’s prediction results, the model can more accurately select the correct characters, significantly enhancing the accuracy of the model’s prediction correction results. Our methods were validated through experiments on a public grammar-correction dataset. The results indicate that our method achieves higher correction performance and lower false correction rates in high-error density scenarios.

Impact of Advances in Artificial Intelligence on Health Tech Industry

Abstract: The healthcare sector is being rapidly transformed by artificial intelligence (AI), which has the potential to enhance care quality, lower costs, and improve patient outcomes. This study examines the newest developments in artificial intelligence (AI) in the fields of robotics, computer vision, and natural language processing. It also looks at the potential and problems brought on by these new technologies, as well as how they affect patient participation, research, and healthcare delivery. A significant benefit that aids in the development of many AI applications is the availability of a huge volume of data supported by limitless cloud storage. The learning algorithms are improved through training with this data. New knowledge in diagnosis and therapy is made possible by the algorithms' interactions with the training data. Hence, it enhances the outcomes for patients. A case study on Predictive Analytics for Early Detection of Sepsis was discussed and a suitable architecture is proposed.

Deciphering “the language of nature”: A transformer-based language model for deleterious mutations in proteins

Various machine-learning models, including deep neural network models, have already been developed to predict deleteriousness of missense (non-synonymous) mutations. Potential improvements to the current state of the art, however, may still benefit from a fresh look at the biological problem using more sophisticated self-adaptive machine-learning approaches. Recent advances in the field of natural language processing show that transformer models-a type of deep neural network-to be particularly powerful at modeling sequence information with context dependence. In this study, we introduce MutFormer, a transformer-based model for the predictionof deleterious missense mutations, which uses reference and mutated protein sequences from the human genome as the primary features. MutFormer takes advantage of a combination of self-attention layers and convolutional layers to learn both long-range and short-range dependencies between amino acid mutations in a protein sequence. We first pre-trained MutFormer on reference protein sequences and mutated protein sequences resulting from common genetic variants observed in human populations. We next examined different fine-tuning methods to successfully apply the model to deleteriousness prediction of missense mutations. Finally, we evaluated MutFormer's performance on multiple testing datasets. We found that MutFormer showed similar or improved performance over a variety of existing tools, including thosethat used conventional machine-learning approaches. In conclusion, MutFormer considers sequence features that are not explored in previous studies and can complement existing computational predictions or empirically generated functional scores to improve our understanding of disease variants.

A New Model for Emotion-Driven Behavior Extraction from Text

Emotion analysis is currently a popular research direction in the field of natural language processing. However, existing research focuses primarily on tasks such as emotion classification, emotion extraction, and emotion cause analysis, while there are few investigations into the relationship between emotions and their impacts. To address these limitations, this paper introduces the emotion-driven behavior extraction (EDBE) task, which addresses these limitations by separately extracting emotions and behaviors to filter emotion-driven behaviors described in text. EDBE comprises three sub-tasks: emotion extraction, behavior extraction, and emotion–behavior pair filtering. To facilitate research in this domain, we have created a new dataset, which is accessible to the research community. To address the EDBE task, we propose a pipeline approach that incorporates the causal relationship between emotions and driven behaviors. Additionally, we adopt the prompt paradigm to improve the model’s representation of cause-and-effect relationships. In comparison to state-of-the-art methods, our approach demonstrates notable improvements, achieving a 1.32% improvement at the clause level and a 1.55% improvement at the span level on our newly curated dataset in terms of the F1 score, which is a commonly used metric to measure the performance of models. These results underscore the effectiveness and superiority of our approach in relation to existing methods.

On solving textual ambiguities and semantic vagueness in MRC based question answering using generative pre-trained transformers.

Machine reading comprehension (MRC) is one of the most challenging tasks and active fields in natural language processing (NLP). MRC systems aim to enable a machine to understand a given context in natural language and to answer a series of questions about it. With the advent of bi-directional deep learning algorithms and large-scale datasets, MRC achieved improved results. However, these models are still suffering from two research issues: textual ambiguities and semantic vagueness to comprehend the long passages and generate answers for abstractive MRC systems. To address these issues, this paper proposes a novel Extended Generative Pretrained Transformers-based Question Answering (ExtGPT-QA) model to generate precise and relevant answers to questions about a given context. The proposed architecture comprises two modified forms of encoder and decoder as compared to GPT. The encoder uses a positional encoder to assign a unique representation with each word in the sentence for reference to address the textual ambiguities. Subsequently, the decoder module involves a multi-head attention mechanism along with affine and aggregation layers to mitigate semantic vagueness with MRC systems. Additionally, we applied syntax and semantic feature engineering techniques to enhance the effectiveness of the proposed model. To validate the proposed model's effectiveness, a comprehensive empirical analysis is carried out using three benchmark datasets including SQuAD, Wiki-QA, and News-QA. The results of the proposed ExtGPT-QA outperformed state of art MRC techniques and obtained 93.25% and 90.52% F1-score and exact match, respectively. The results confirm the effectiveness of the ExtGPT-QA model to address textual ambiguities and semantic vagueness issues in MRC systems.

Chinese medical entity recognition based on the dual-branch TENER model.

Named Entity Recognition (NER) is a long-standing fundamental problem in various research fields of Natural Language Processing (NLP) and has been practiced in many application scenarios. However, the application results of NER methods in Chinese electronic medical records (EMRs) are not satisfactory, mainly due to the following two problems: (1) Existing methods do not take into account the impact of medical terminology on model recognition performance, resulting in poor model performance. (2) Existing methods do not fully utilize the Chinese language features contained in EMR, resulting in poor model robustness. Therefore, it is imminent to solve these two problems regarding the performance of the NER model for EMRs. In this paper, a TENER-based radical feature and entity augmentation model for NER in Chinese EMRs is proposed. The TENER model is first used in the pre-training stage to extract deep semantic information from each layer of the feature extractor. In the decoder part, the recognition of medical entity boundary and entity category are divided into two branch tasks. We compare the overall performance of the proposed model with existing models on different datasets using the computed F1 score evaluation metric. The experimental results show that our model achieves the best F1 score of 82.67%, 74.37%, 70.16% on the CCKS2019, ERTCMM, and CEMR data sets. Meanwhile, in the CMeEE challenge, our model surpassed the top-3 with the F1 score of 68.39%. Our proposed model is the first to divide the NER task into a two-branch tasks, entity boundary and types recognition. Firstly, the medical entity dictionary information is integrated into TENER to obtain the feature information of professional terms in Chinese EMRs. Secondly, the features of Chinese radicals in Chinese EMRs extracted by CNN are added to the entity category recognition task. Finally, the effectiveness of the model is validated on four datasets and competitive results are achieved.

A Short-Text Similarity Model Combining Semantic and Syntactic Information

As one of the prominent research directions in the field of natural language processing (NLP), short-text similarity has been widely used in search recommendation and question-and-answer systems. Most of the existing short textual similarity models focus on considering semantic similarity while overlooking the importance of syntactic similarity. In this paper, we first propose an enhanced knowledge language representation model based on graph convolutional networks (KEBERT-GCN), which effectively uses fine-grained word relations in the knowledge base to assess semantic similarity and model the relationship between knowledge structure and text structure. To fully leverage the syntactic information of sentences, we also propose a computational model of constituency parse trees based on tree kernels (CPT-TK), which combines syntactic information, semantic features, and attentional weighting mechanisms to evaluate syntactic similarity. Finally, we propose a comprehensive model that integrates both semantic and syntactic information to comprehensively evaluate short-text similarity. The experimental results demonstrate that our proposed short-text similarity model outperforms the models proposed in recent years, achieving a Pearson correlation coefficient of 0.8805 on the STS-B dataset.

Penerapan Machine Learning Untuk Menentukan Tingkat Kepuasan Tamu Hotel Dymens Menggunakan Metode Vader

In Indonesia there are several social media that provide reviews and experiences related to an inn, including Tripadvisor.com, Traveloka, Tiket.com, and many more. Over time, machine learning can replace and improve human abilities in various fields, Machine Learning has been extensively researched and used to solve various problems. Among them is to find out how the assessment and views of hotel guests regarding the services and policies set. With reviews from customers or hotel guests, we can find out how the quality of the product or service is implemented, so that an evaluation can be carried out to increase customer satisfaction. Sentiment analysis, or opinion evaluation is an active area of ​​study in the field of natural language processing that analyzes opinions, sentiments, evaluations, attitudes, and emotions through improving the processing of subjectivity in texts. Sentiment analysis is useful for a wide variety of issues of interest to human-computer interaction practices and researchers, as well as people from fields such as: sociology, marketing and advertising, psychology, eco-economics, and political science. The process of sentiment analysis is often done to find out positive, neutral or negative opinion from the public regarding a certain matter. The Valence Aware Dictionary and Sentiment Reasoner (VADER) method is a sentiment analysis method that can identify a person's emotions based on words. So that the VADER Method can be used to identify hotel guest reviews and hotel management can make decisions based on the results of these assessments or reviews

A weakly supervised method for named entity recognition of Chinese electronic medical records.

The field of Chinese medical natural language processing faces a significant challenge in training accurate entity recognition models due to the limited availability of high-quality labeled data. In response, we propose a joint training model, MCBERT-GCN-CRF, which achieves high performance in identifying medical-related entities in Chinese electronic medical records. Additionally, we introduce CM-NER, a 5-step framework that effectively mitigates the effects of noise in weakly labeled data and establishes a principled connection between supervised and weakly supervised named entity recognition. We demonstrate significant improvements in recall rate and accuracy. Our approach outperforms traditional fully supervised pre-training models and other state-of-the-art methods by suppressing noise in weakly labeled data. Our proposed framework achieves an F1 score of 86.29% on the CCKS-2019 dataset, significantly higher than pre-trained model baselines ranging from 74.17 to 83.06%, and higher than the top-performing named entity recognition supervised learning models in the CCKS-2019 competition. Our results demonstrate the effectiveness of our proposed framework and highlight the potential of leveraging unlabeled data to train accurate models for named entity recognition in Chinese medical natural language processing. This research has significant implications for advancing natural language processing techniques in the medical domain and improving patient care.