NLP Applications Research Articles

Few-Shot Learning for Clinical Natural Language Processing Using Siamese Neural Networks: Algorithm Development and Validation Study

Background Natural language processing (NLP) has become an emerging technology in health care that leverages a large amount of free-text data in electronic health records to improve patient care, support clinical decisions, and facilitate clinical and translational science research. Recently, deep learning has achieved state-of-the-art performance in many clinical NLP tasks. However, training deep learning models often requires large, annotated data sets, which are normally not publicly available and can be time-consuming to build in clinical domains. Working with smaller annotated data sets is typical in clinical NLP; therefore, ensuring that deep learning models perform well is crucial for real-world clinical NLP applications. A widely adopted approach is fine-tuning existing pretrained language models, but these attempts fall short when the training data set contains only a few annotated samples. Few-shot learning (FSL) has recently been investigated to tackle this problem. Siamese neural network (SNN) has been widely used as an FSL approach in computer vision but has not been studied well in NLP. Furthermore, the literature on its applications in clinical domains is scarce. Objective The aim of our study is to propose and evaluate SNN-based approaches for few-shot clinical NLP tasks. Methods We propose 2 SNN-based FSL approaches, including pretrained SNN and SNN with second-order embeddings. We evaluate the proposed approaches on the clinical sentence classification task. We experiment with 3 few-shot settings, including 4-shot, 8-shot, and 16-shot learning. The clinical NLP task is benchmarked using the following 4 pretrained language models: bidirectional encoder representations from transformers (BERT), BERT for biomedical text mining (BioBERT), BioBERT trained on clinical notes (BioClinicalBERT), and generative pretrained transformer 2 (GPT-2). We also present a performance comparison between SNN-based approaches and the prompt-based GPT-2 approach. Results In 4-shot sentence classification tasks, GPT-2 had the highest precision (0.63), but its recall (0.38) and F score (0.42) were lower than those of BioBERT-based pretrained SNN (0.45 and 0.46, respectively). In both 8-shot and 16-shot settings, SNN-based approaches outperformed GPT-2 in all 3 metrics of precision, recall, and F score. Conclusions The experimental results verified the effectiveness of the proposed SNN approaches for few-shot clinical NLP tasks.

Developing and validating a natural language processing algorithm to extract preoperative cannabis use status documentation from unstructured narrative clinical notes.

This study aimed to develop a natural language processing algorithm (NLP) using machine learning (ML) techniques to identify and classify documentation of preoperative cannabis use status. We developed and applied a keyword search strategy to identify documentation of preoperative cannabis use status in clinical documentation within 60 days of surgery. We manually reviewed matching notes to classify each documentation into 8 different categories based on context, time, and certainty of cannabis use documentation. We applied 2 conventional ML and 3 deep learning models against manual annotation. We externally validated our model using the MIMIC-III dataset. The tested classifiers achieved classification results close to human performance with up to 93% and 94% precision and 95% recall of preoperative cannabis use status documentation. External validation showed consistent results with up to 94% precision and recall. Our NLP model successfully replicated human annotation of preoperative cannabis use documentation, providing a baseline framework for identifying and classifying documentation of cannabis use. We add to NLP methods applied in healthcare for clinical concept extraction and classification, mainly concerning social determinants of health and substance use. Our systematically developed lexicon provides a comprehensive knowledge-based resource covering a wide range of cannabis-related concepts for future NLP applications. We demonstrated that documentation of preoperative cannabis use status could be accurately identified using an NLP algorithm. This approach can be employed to identify comparison groups based on cannabis exposure for growing research efforts aiming to guide cannabis-related clinical practices and policies.

Phonetic-Based Forward Online Transliteration Tool from English to Tamil Language

Transliteration is the process of mapping the character of one language to the character of some other language based on its phonetics. India is very much diverse in languages where people speak different languages. Though they speak different languages, it might be difficult for them to read the script of those many languages. In a situation like this, transliteration process plays a major role. It helps in various Natural Language Processing applications such as Information retrieval, Machine translation, Speech recognition. These are NLP applications which make the computer understand the natural language as to how human being interprets. It helps in translating technical terms and proper names from one language to another language. Moreover, transliteration works have been carried out in languages such as Japanese, Chinese and English. But when considering Indian languages, especially Tamil language, very few recognizable works have been carried out. In this paper, transliteration process is carried out on Unicode Tamil characters. The phonetics-based forward list processing is implemented for transliterating from English language to Tamil language which yields promising results.

A Neuro Symbolic Approach for Contradiction Detection in Persian Text

Detection of semantic contradictory sentences is a challenging and fundamental issue for some NLP applications, such as textual entailments recognition. In this study, contradiction means different types of semantic confrontation, such as negation, antonymy, and numerical. Due to the lack of sufficient data to apply precise machine learning and, specifically, deep learning methods to Persian and other low-resource languages, rule-based approaches are of great interest. Also, recently, the emergence of new methods such as transfer learning has opened up the possibility of deep learning for low-resource languages. This paper introduces a hybrid contradiction detection approach for detecting seven categories of contradictions in Persian texts: Antonymy, negation, numerical, factive, structural, lexical and world knowledge. The proposed method consists of 1) a novel data mining method and 2) a transformer-based deep neural method for contradiction detection . Also, a simple baseline is presented for comparison. The data mining method uses frequent rule mining to extract appropriate contradiction detection rules employing a development set. Extracted rules are tested for different categories of contradictory sentences. In the first step, a classifier checks whether the rules work for an input sentence pair. Then, according to the result, rules are used for three categories of negation, numerical, and antonym. In this part, the highest F-measure is obtained for detecting the negation category (90%), the average F-measure for these three categories is 86%, and for the other four categories, in which the rules have a lower F-measure of 62%, the transformer-based method achieved 76%. The proposed hybrid approach has an overall f-measure of higher than 80%. 

Conversational AI for Natural Language Processing: An Review of ChatGPT

ChatGPT is a conversational artificial intelligence model developed by OpenAI, which was introduced in 2019. It employs a transformer-based neural mesh to produce human being responses in real-time, allowing for natural language conversations with a machine. ChatGPT is instructed on huge quantities of data captured using the internet, making it knowledgeable in an extensive span of topics, from news & entertainment to politics and sports. This allows it to generate contextually relevant responses to questions and statements, making the conversation seem more lifelike. The model can be used in various applications, including customer service, personal assistants, and virtual assistants. ChatGPT has also shown promising results in generating creative content, such as jokes and poetry, showcasing its versatility and potential for future applications.This paper provides a comprehensive review of the existing literature on ChatGPT, highlighting its key advantages, such as improved accuracy and flexibility compared to traditional NLP tools, as well as its limitations and the need for further research to address potential ethical concerns. The review also highlights the potential for ChatGPT to be used in NLP applications, including question-answering and dialogue generation, and highlights the need for further research and development in these areas.

ARNLI: ARABIC NATURAL LANGUAGE INFERENCE ENTAILMENT AND CONTRADICTION DETECTION

Natural Language Inference (NLI) is a hot topic research in natural language processing, contradiction detection between sentences is a special case of NLI. This is considered a difficult NLP task which has a big influence when added as a component in many NLP applications, such as Question Answering Systems, text Summarization. Arabic Language is one of the most challenging low-resources languages in detecting contradictions due to its rich lexical, semantics ambiguity. We have created a dataset of more than 12k sentences and named ArNLI, that will be publicly available. Moreover, we have applied a new model inspired by Stanford contradiction detection proposed solutions on English language. We proposed an approach to detect contradictions between pairs of sentences in Arabic language using contradiction vector combined with language model vector as an input to machine learning model. We analyzed results of different traditional machine learning classifiers and compared their results on our created dataset (ArNLI) and on an automatic translation of both PHEME, SICK English datasets. Best results achieved using Random Forest classifier with an accuracy of 99%, 60%, 75% on PHEME, SICK and ArNLI respectively.

TSHD: Topic Segmentation Based on Headings Detection (Case Study: Resumes)

Many unstructured documents contain segments with specific topics. Extracting these segments and identifying their topics helps to access the required information directly. This can improve the quality of many NLP applications such as information extraction, information retrieval, summarization, and question answering. Resumes (CVs) are unstructured documents that have diverse formats. They contain various segments such as personal information, experience, and education. Manually processing resumes to find the most suitable candidates for a particular job is a difficult task. Due to the increased amount of data, it has become very necessary to manipulate resumes by computer to save time and effort. This research presents a new algorithm named TSHD for topic segmentation based on headings detection. We apply the algorithm to extract resume segments and identify their topics. The proposed TSHD algorithm is accurate and addresses many weaknesses in previous studies. Evaluation results show a very high F1 score (about 96%) and a very low segmentation error (about 2%). The algorithm can be easily adapted to deal with other textual domains that contain headings in their segments.

Intelligent Analysis and Application of NLP and OCR Technologies in Power Grid Project Evaluation

Intelligent Analysis and Application of NLP and OCR Technologies in Power Grid Project Evaluation

Applications of AI and NLP to advance Music Recommendations on Voice Assistants

This paper builds on the popular use case of music requests on voice assistants like Siri, Google Assistant, Alexa, and others and explores the different AI and NLP techniques. The paper particularly focuses on how each of these techniques can be applied in the context of musical recommendations and experiences on voice assistants. It enumerates specific problems in the space of music recommendations and illustrates how specific techniques like multi-armed bandits can be applied.

Script Event Prediction via Multilingual Event Graph Networks

Predicting what happens next in text plays a critical role in building NLP applications. Many methods including count-based and neural-network-based have been proposed to tackle the task called script event prediction: predicting the most suitable subsequent event from a candidate list given a chain of narrative events (context). However, two problems including event ambiguity and evidence bias hinder the performance of these monolingual approaches. The former means that some events in the event chain are ambiguous. The latter means that both the wrong and correct candidate events can obtain sufficient support from the event context. In this article, we propose a novel multilingual approach to address two issues simultaneously. Specifically, to alleviate the event ambiguity problem, we project the monolingual event chains to parallel cross-lingual event chains, which can provide complementary information for monolingual event disambiguation. To deal with the evidence bias problem, we construct two monolingual event graphs and a cross-lingual event aligned graph to fully explore connections between events. What’s more, we design a graph attention mechanism to model the confidence of the complement clues, which controls the information integration from various languages. By modeling the events with graphs instead of pairs or chains, the model can compare the candidate subsequent events simultaneously and choose the more suitable subsequent event as the final answer. Extensive experiments were conducted on the widely used New York Times corpus for script event prediction task and experimental results show that our approach outperforms previous models.

BART-IT: An Efficient Sequence-to-Sequence Model for Italian Text Summarization

The emergence of attention-based architectures has led to significant improvements in the performance of neural sequence-to-sequence models for text summarization. Although these models have proved to be effective in summarizing English-written documents, their portability to other languages is limited thus leaving plenty of room for improvement. In this paper, we present BART-IT, a sequence-to-sequence model, based on the BART architecture that is specifically tailored to the Italian language. The model is pre-trained on a large corpus of Italian-written pieces of text to learn language-specific features and then fine-tuned on several benchmark datasets established for abstractive summarization. The experimental results show that BART-IT outperforms other state-of-the-art models in terms of ROUGE scores in spite of a significantly smaller number of parameters. The use of BART-IT can foster the development of interesting NLP applications for the Italian language. Beyond releasing the model to the research community to foster further research and applications, we also discuss the ethical implications behind the use of abstractive summarization models.

Cross-lingual transfer of knowledge in distributional language models: Experiments in Hungarian

AbstractIn this paper, we argue that the very convincing performance of recent deep-neural-model-based NLP applications has demonstrated that the distributionalist approach to language description has proven to be more successful than the earlier subtle rule-based models created by the generative school. The now ubiquitous neural models can naturally handle ambiguity and achieve human-like linguistic performance with most of their training consisting only of noisy raw linguistic data without any multimodal grounding or external supervision refuting Chomsky's argument that some generic neural architecture cannot arrive at the linguistic performance exhibited by humans given the limited input available to children. In addition, we demonstrate in experiments with Hungarian as the target language that the shared internal representations in multilingually trained versions of these models make them able to transfer specific linguistic skills, including structured annotation skills, from one language to another remarkably efficiently.

Revise and Resubmit: An Intertextual Model of Text-based Collaboration in Peer Review

AbstractPeer review is a key component of the publishing process in most fields of science. Increasing submission rates put a strain on reviewing quality and efficiency, motivating the development of applications to support the reviewing and editorial work. While existing NLP studies focus on the analysis of individual texts, editorial assistance often requires modeling interactions between pairs of texts—yet general frameworks and datasets to support this scenario are missing. Relationships between texts are the core object of the intertextuality theory—a family of approaches in literary studies not yet operationalized in NLP. Inspired by prior theoretical work, we propose the first intertextual model of text-based collaboration, which encompasses three major phenomena that make up a full iteration of the review–revise–and–resubmit cycle: pragmatic tagging, linking, and long-document version alignment. While peer review is used across the fields of science and publication formats, existing datasets solely focus on conference-style review in computer science. Addressing this, we instantiate our proposed model in the first annotated multidomain corpus in journal-style post-publication open peer review, and provide detailed insights into the practical aspects of intertextual annotation. Our resource is a major step toward multidomain, fine-grained applications of NLP in editorial support for peer review, and our intertextual framework paves the path for general-purpose modeling of text-based collaboration. We make our corpus, detailed annotation guidelines, and accompanying code publicly available.1

Task-dependent Optimal Weight Combinations for Static Embeddings

A variety of NLP applications use word2vec skip-gram, GloVe, and fastText word embeddings. These models learn two sets of embedding vectors, but most practitioners use only one of them, or alternately an unweighted sum of both. This is the first study to systematically explore a range of linear combinations between the first and second embedding sets. We evaluate these combinations on a set of six NLP benchmarks including IR, POS-tagging, and sentence similarity. We show that the default embedding combinations are often suboptimal and demonstrate 1.0-8.0% improvements. Notably, GloVe’s default unweighted sum is its least effective combination across tasks. We provide a theoretical basis for weighting one set of embeddings more than the other according to the algorithm and task. We apply our findings to improve accuracy in applications of cross-lingual alignment and navigational knowledge by up to 15.2%.

UTILIZING ROOTS AND PATTERNS TO IDENTIFY ARABIC NAMED ENTITIES

Named Entity Recognition NER is a subset of information extraction that seeks to recognize and categorize named things in text data into specified categories, such as people's names, organizations' names, geographic locations, and so on. This task has recently attracted a lot of attention due to the discovery it has the potential to boost the performance of a variety of NLP applications. In the domains of Question Answering and Summarization Systems, Information Retrieval and Extraction, Machine Translation, Video Annotation, Semantic Web Search, and Bioinformatics, the majority of difficulties require named entity recognition. Arabic is an inflectional language, which allows for non-concatenative morphological operations on the root. The purpose of this study is to extract and recognize entity names from Arabic articles. We proposed an algorithm for determining names from roots using patterns. We developed it in Python and leveraged the "pyqt5" visual package to see the results immediately, as well as modify and add patterns easily. To replicate the names, we used a random sample of 400 names and 45 different patterns. The algorithm correctly identified 370 names easily and quickly, yielding a success rate of 93%. All names with the same recognized names will be known in the same way by the method and do not need any manipulation in code or design. The names that are not recognized by our algorithm have no roots in the list of known Arabic roots. Our research shows that the approach can recognize names with roots with high speed and accuracy, but it is not possible to identify nouns that are not in the Arabic language using this method. As a result, we recommend using a hybrid method that incorporates multiple concepts.

Automatic Question Generation Monolingual Multilingual pre-trained Models using RNN and Transformer in Low Resource Indonesian Language

Although Indonesian is the fourth most frequently used language on the internet, the development of NLP in Indonesian has not been studied intensively. One form of NLP application classified as an NLG task is the Automatic Question Generation task. Generally, the task has proven well, using rule-based and cloze tests, but these approaches depend heavily on the defined rules. While this approach is suitable for automated question generation systems on a small scale, it can become less efficient as the scale of the system grows. Many NLG model architectures have recently proven to have significantly improved performance compared to previous architectures, such as generative pre-trained transformers, text-to-text transfer transformers, bidirectional autoregressive transformers, and many more. Previous studies on AQG in Indonesian were built on RNN-based architecture such as GRU, LSTM, and Transformer. The performance of models in previous studies is compared with state-of-the-art models, such as multilingual models mBART and mT5, and monolingual models such as IndoBART and IndoGPT. As a result, the fine-tuned IndoBART performed significantly higher than either BiGRU and BiLSTM on the SQuAD dataset. Fine-tuned IndoBART on most of the metrics also performed better on the TyDiQA dataset only, which has fewer population than the SQuAD dataset.

Learning Chess and NIM with Transformers

Representing a board game’s state space, actions, and transition model by text-based notation enables a wide variety of NLP applications suited to the strengths of language models. These few shot language models can help gain insight into a variety of interesting problems such as learning the rules of a game, detecting player behavior patterns, player attribution, and ultimately learning the game in an unsupervised manner. In this study, we firstly applied the BERT model to the simple combinatorial Nim game to analyze BERT’s performance in the varying presence of noise. We analyzed the model’s performance versus three agents, namely Nim Guru, a Random player, and a Q-learner. Secondly, we applied the BERT model to the game of chess through a large set of high ELO stockfish games with exhaustive encyclopedia openings. Finally, we have shown that model practically learns the rules of the Nim and chess, and have shown that it can competently play against its opponent and in some interesting conditions win.

Employing a Multilingual Transformer Model for Segmenting Unpunctuated Arabic Text

Long unpunctuated texts containing complex linguistic sentences are a stumbling block to processing any low-resource languages. Thus, approaches that attempt to segment lengthy texts with no proper punctuation into simple candidate sentences are a vitally important preprocessing task in many hard-to-solve NLP applications. To this end, we propose a preprocessing solution for segmenting unpunctuated Arabic texts into potentially independent clauses. This solution consists of: (1) a punctuation detection model built on top of a multilingual BERT-based model, and (2) some generic linguistic rules for validating the resulting segmentation. Furthermore, we optimize the strategy of applying these linguistic rules using our suggested greedy-like algorithm. We call the proposed solution PDTS (standing for Punctuation Detector for Text Segmentation). Concerning the evaluation, we showcase how PDTS can be effectively employed as a text tokenizer for unpunctuated documents (i.e., mimicking the transcribed audio-to-text documents). Experimental findings across two evaluation protocols (involving an ablation study and a human-based judgment) demonstrate that PDTS is practically effective in both performance quality and computational cost. In particular, PDTS can reach an average F-Measure score of approximately 75%, indicating a minimum improvement of roughly 13% (i.e., compared to the performance of the state-of-the-art competitor models).

Glottocodes: Identifiers linking families, languages and dialects to comprehensive reference information

Glottocodes constitute the backbone identification system for the language, dialect and family inventory Glottolog (https://glottolog.org). In this paper, we summarize the motivation and history behind the system of glottocodes and describe the principles and practices of data curation, technical infrastructure and update/version-tracking systematics. Since our understanding of the target domain – the dialects, languages and language families of the entire world – is continually evolving, changes and updates are relatively common. The resulting data is assessed in terms of the FAIR (Findable, Accessible, Interoperable, Reusable) Guiding Principles for scientific data management and stewardship. As such the glottocode-system responds to an important challenge in the realm of Linguistic Linked Data with numerous NLP applications.

Sentimental Analysis Applied on Movie Reviews

Nowadays, Natural Language Processing has received the widespread attention from the natural sciences, and sentimental analysis is one of the most widely used NLP applications. In the age of big data, how to find the required information accurately and quickly has become the hotspot of current research. Based on the movie reviews of two movies from the same series, this paper studies the sentimental trend of movies reviews, in order to help the audience obtain a reference for movie choices. Term frequency-Inverse Document Frequency (TF-IDF) algorithm is applied to evaluate the importance of words in the reviews, and TextBlob sentiment analysis library of Python software is used to grade the sentiment scores of the two films. Finally, the sentiment score graph is drawn, which provides a strong support for the further identification of the movie characteristics of two films from the same series. What’s more, Support vector machines (SVM) model is utilized to do the classification of the movie reviews and achieved 85.2% accuracy.