Natural Language Processing Research Research Articles

Evaluation of Impact of Neural Networks in Text Classification

One of the most trending and major areas of research in Natural Language Processing (NLP) is the classification of text data. This necessarily means that the category that the text belongs to is determined by the content of the text. Various algorithms such as Recurrent Neural Network along with its variation which is Long Short-Term Memory, Hierarchical Attention Networks and also Convolutional Neural Network have been used to analyse how the context of the text can be determined from the text data which in available in terms of datasets. These algorithms each have a special characteristic of their own. While Recurrent Neural Network maintains the structural sequence of the contexts, the Convolutional Neural Network manages to obtain the n-gram feature and the Hierarchical Attention Network manages the hierarchy of the documents or data. The above said algorithms have been implemented on the British Broadcasting Corporation News datasets. Various parameters such as recall, precision, accuracy etc. have been considered along with standards such as F1-score, confusion matrix etc. to deduce the impact.

An introduction to Deep Learning in Natural Language Processing: Models, techniques, and tools

Natural Language Processing (NLP) is a branch of artificial intelligence that involves the design and implementation of systems and algorithms able to interact through human language. Thanks to the recent advances of deep learning, NLP applications have received an unprecedented boost in performance. In this paper, we present a survey of the application of deep learning techniques in NLP, with a focus on the various tasks where deep learning is demonstrating stronger impact. Additionally, we explore, describe, and revise the main resources in NLP research, including software, hardware, and popular corpora. Finally, we emphasize the main limits of deep learning in NLP and current research directions.

Improved biomedical word embeddings in the transformer era

Background:Recent natural language processing (NLP) research is dominated by neural network methods that employ word embeddings as basic building blocks. Pre-training with neural methods that capture local and global distributional properties (e.g., skip-gram, GLoVE) using free text corpora is often used to embed both words and concepts. Pre-trained embeddings are typically leveraged in downstream tasks using various neural architectures that are designed to optimize task-specific objectives that might further tune such embeddings. Objective:Despite advances in contextualized language model based embeddings, static word embeddings still form an essential starting point in BioNLP research and applications. They are useful in low resource settings and in lexical semantics studies. Our main goal is to build improved biomedical word embeddings and make them publicly available for downstream applications. Methods:We jointly learn word and concept embeddings by first using the skip-gram method and further fine-tuning them with correlational information manifesting in co-occurring Medical Subject Heading (MeSH) concepts in biomedical citations. This fine-tuning is accomplished with the transformer-based BERT architecture in the two-sentence input mode with a classification objective that captures MeSH pair co-occurrence. We conduct evaluations of these tuned static embeddings using multiple datasets for word relatedness developed by previous efforts. Results:Both in qualitative and quantitative evaluations we demonstrate that our methods produce improved biomedical embeddings in comparison with other static embedding efforts. Without selectively culling concepts and terms (as was pursued by previous efforts), we believe we offer the most exhaustive evaluation of biomedical embeddings to date with clear performance improvements across the board. Conclusion:We repurposed a transformer architecture (typically used to generate dynamic embeddings) to improve static biomedical word embeddings using concept correlations. We provide our code and embeddings for public use for downstream applications and research endeavors: https://github.com/bionlproc/BERT-CRel-Embeddings.

Confronting Abusive Language Online: A Survey from the Ethical and Human Rights Perspective

The pervasiveness of abusive content on the internet can lead to severe psychological and physical harm. Significant effort in Natural Language Processing (NLP) research has been devoted to addressing this problem through abusive content detection and related sub-areas, such as the detection of hate speech, toxicity, cyberbullying, etc. Although current technologies achieve high classification performance in research studies, it has been observed that the real-life application of this technology can cause unintended harms, such as the silencing of under-represented groups. We review a large body of NLP research on automatic abuse detection with a new focus on ethical challenges, organized around eight established ethical principles: privacy, accountability, safety and security, transparency and explainability, fairness and non-discrimination, human control of technology, professional responsibility, and promotion of human values. In many cases, these principles relate not only to situational ethical codes, which may be context-dependent, but are in fact connected to universal human rights, such as the right to privacy, freedom from discrimination, and freedom of expression. We highlight the need to examine the broad social impacts of this technology, and to bring ethical and human rights considerations to every stage of the application life-cycle, from task formulation and dataset design, to model training and evaluation, to application deployment. Guided by these principles, we identify several opportunities for rights-respecting, socio-technical solutions to detect and confront online abuse, including ‘nudging’, ‘quarantining’, value sensitive design, counter-narratives, style transfer, and AI-driven public education applications.evaluation, to application deployment. Guided by these principles, we identify several opportunities for rights-respecting, socio-technical solutions to detect and confront online abuse, including 'nudging', 'quarantining', value sensitive design, counter-narratives, style transfer, and AI-driven public education applications.

Natural Language Processing (NLP) and Its Impact across Industries – Unlocking the True Potential of Digital Healthcare (A Case Study Approach)

During the advent of the 21st century, technical breakthroughs and developments took place. Natural Language Processing or NLP is one of their promising disciplines that has been increasingly dynamic via groundbreaking findings on most computer networks. Because of the digital revolution the amounts of data generated by M2M communication across devices and platforms such as Amazon Alexa, Apple Siri, Microsoft Cortana, etc. were significantly increased. This causes a great deal of unstructured data to be processed that does not fit in with standard computational models. In addition, the increasing problems of language complexity, data variability and voice ambiguity make implementing models increasingly harder. The current study provides an overview of the potential and breadth of the NLP market and its acceptance in industry-wide, in particular after Covid-19. It also gives a macroscopic picture of progress in natural language processing research, development and implementation.

Evotuning protocols for Transformer-based variant effect prediction on multi-domain proteins.

Accurate variant effect prediction has broad impacts on protein engineering. Recent machine learning approaches toward this end are based on representation learning, by which feature vectors are learned and generated from unlabeled sequences. However, it is unclear how to effectively learn evolutionary properties of an engineering target protein from homologous sequences, taking into account the protein's sequence-level structure called domain architecture (DA). Additionally, no optimal protocols are established for incorporating such properties into Transformer, the neural network well-known to perform the best in natural language processing research. This article proposes DA-aware evolutionary fine-tuning, or 'evotuning', protocols for Transformer-based variant effect prediction, considering various combinations of homology search, fine-tuning and sequence vectorization strategies. We exhaustively evaluated our protocols on diverse proteins with different functions and DAs. The results indicated that our protocols achieved significantly better performances than previous DA-unaware ones. The visualizations of attention maps suggested that the structural information was incorporated by evotuning without direct supervision, possibly leading to better prediction accuracy.

Clinically relevant pretraining is all you need.

Clinical notes present a wealth of information for applications in the clinical domain, but heterogeneity across clinical institutions and settings presents challenges for their processing. The clinical natural language processing field has made strides in overcoming domain heterogeneity, while pretrained deep learning models present opportunities to transfer knowledge from one task to another. Pretrained models have performed well when transferred to new tasks; however, it is not well understood if these models generalize across differences in institutions and settings within the clinical domain. We explore if institution or setting specific pretraining is necessary for pretrained models to perform well when transferred to new tasks. We find no significant performance difference between models pretrained across institutions and settings, indicating that clinically pretrained models transfer well across such boundaries. Given a clinically pretrained model, clinical natural language processing researchers may forgo the time-consuming pretraining step without a significant performance drop.

A survey on deploying mobile deep learning applications: A systemic and technical perspective

With the rapid development of mobile devices and deep learning, mobile smart applications using deep learning technology have sprung up. It satisfies multiple needs of users, network operators and service providers, and rapidly becomes a main research focus. In recent years, deep learning has achieved tremendous success in image processing, natural language processing, language analysis and other research fields. Despite the task performance has been greatly improved, the resources required to run these models have increased significantly. This poses a major challenge for deploying such applications on resource-restricted mobile devices. Mobile intelligence needs faster mobile processors, more storage space, smaller but more accurate models, and even the assistance of other network nodes. To help the readers establish a global concept of the entire research direction concisely, we classify the latest works in this field into two categories, which are local optimization on mobile devices and distributed optimization based on the computational position of machine learning tasks. We also list a few typical scenarios to make readers realize the importance and indispensability of mobile deep learning applications. Finally, we conjecture what the future may hold for deploying deep learning applications on mobile devices research, which may help to stimulate new ideas.

A Framework for Indonesian Grammar Error Correction

Grammatical Error Correction (GEC) is a challenge in Natural Language Processing research. Although many researchers have been focusing on GEC in universal languages such as English or Chinese, few studies focus on Indonesian, which is a low-resource language. In this article, we proposed a GEC framework that has the potential to be a baseline method for Indonesian GEC tasks. This framework treats GEC as a multi-classification task. It integrates different language embedding models and deep learning models to correct 10 types of Part of Speech (POS) error in Indonesian text. In addition, we constructed an Indonesian corpus that can be utilized as an evaluation dataset for Indonesian GEC research. Our framework was evaluated on this dataset. Results showed that the Long Short-Term Memory model based on word-embedding achieved the best performance. Its overall macro-average F 0.5 in correcting 10 POS error types reached 0.551. Results also showed that the framework can be trained on a low-resource dataset.

BnVec: Towards the Development of Word Embedding for Bangla Language Processing

Progression in machine learning and statistical inference are facilitating the advancement of domains like computer vision, natural language processing (NLP), automation & robotics, and so on. Among the different persuasive improvements in NLP, word embedding is one of the most used and revolutionary techniques. In this paper, we manifest an open-source library for Bangla word extraction systems named BnVec which expects to furnish the Bangla NLP research community by the utilization of some incredible word embedding techniques. The BnVec is splitted up into two parts, the first one is the Bangla suitable defined class to embed words with access to the six most popular word embedding schemes (CountVectorizer, TF-IDF, Hash Vectorizer, Word2vec, fastText, and Glove). The other one is based on the pre-trained distributed word embedding system of Word2vec, fastText, and GloVe. The pre-trained models have been built by collecting content from the newspaper, social media, and Bangla wiki articles. The total number of tokens used to build the models exceeds 395,289,960. The paper additionally depicts the performance of these models by various hyper-parameter tuning and then analyzes the results.

Person retrieval in surveillance using textual query: a review

Recent advancement of research in biometrics, computer vision, and natural language processing has discovered opportunities for person retrieval from surveillance videos using textual query. The prime objective of a surveillance system is to locate a person using a description, e.g., a short woman with a pink t-shirt and white skirt carrying a black purse. She has brown hair. Such a description contains attributes like gender, height, type of clothing, colour of clothing, hair colour, and accessories. Such attributes are formally known as soft biometrics. They help bridge the semantic gap between a human description and a machine as a textual query contains the person’s soft biometric attributes. It is also not feasible to manually search through huge volumes of surveillance footage to retrieve a specific person. Hence, automatic person retrieval using vision and language-based algorithms is becoming popular. In comparison to other state-of-the-art reviews, the contribution of the paper is as follows: 1. Recommends most discriminative soft biometrics for specific challenging conditions. 2. Integrates benchmark datasets and retrieval methods for objective performance evaluation. 3. A complete snapshot of techniques based on features, classifiers, number of soft biometric attributes, type of the deep neural networks, and performance measures. 4. The comprehensive coverage of person retrieval from handcrafted features based methods to end-to-end approaches based on natural language description.

LeDoCl : A Semantic Model for Legal Documents Classification using Ensemble Methods

NLP is one of the components of Machine Learning. Topic Modeling is a sub component of information retrieval Information Retrieval is a broad domain research in Natural Language Processing (NLP). This downside has been broadly studied in the perspective of cluster algorithms like K–means and K-fold, that tends to converge to at least one of diverse native issue counting on the selection of format method. To overcome the instabilities and assumptions in existing systems such as Vector Space Model (VSM) and SVD, Semantic based topic modeling (SLDA) and ensemble model with generation and integration is proposed. In the case of topic modelling, instability is visible in two distinct aspects. First, when the topic descriptors are examined over multiple runs. During which there will be considerable change in the term rankings and few terms may appear or disappear completely as well. Next, there could be instability due to the extent to which topics have association with document, through several executions. In the proposed system, ensemble learning comprises of algorithms Kernel Support Vector Machine (KSVM) and Random Forest algorithm which overcomes the instability. The first issue of appearance and disappearance of words between multiple runs is overcome by Gibbs Sampling based Semantic LDA (GSLDA). The second issue of alignment of topics with document is aided by using ESLDA. This ensemble SLDA algorithm show increased accuracy in terms of retrieval and reduced time interval compared to conventional models. The accuracy increases up to 98% using ESLDA compared to SLDA (82%) and term frequency methods (78%).

Advancing natural language processing (NLP) applications of morphologically rich languages with bidirectional encoder representations from transformers (BERT): an empirical case study for Turkish

Language model pre-training architectures have demonstrated to be useful to learn language representations. bidirectional encoder representations from transformers (BERT), a recent deep bidirectional self-attention representation from unlabelled text, has achieved remarkable results in many natural language processing (NLP) tasks with fine-tuning. In this paper, we want to demonstrate the efficiency of BERT for a morphologically rich language, Turkish. Traditionally morphologically difficult languages require dense language pre-processing steps in order to model the data to be suitable for machine learning (ML) algorithms. In particular, tokenization, lemmatization or stemming and feature engineering tasks are needed to obtain an efficient data model to overcome data sparsity or high-dimension problems. In this context, we selected five various Turkish NLP research problems as sentiment analysis, cyberbullying identification, text classification, emotion recognition and spam detection from the literature. We then compared the empirical performance of BERT with the baseline ML algorithms. Finally, we found enhanced results compared to base ML algorithms in the selected NLP problems while eliminating heavy pre-processing tasks.

MBA: A Multimodal Bilinear Attention Model with Residual Connection for Abstractive Multimodal Summarization

The combination of vision and natural language modalities has become an important topic in both computer vision and natural language processing research communities. Multimodal summarization has received unprecedented attention with the rapid growth of multimodal information. This paper proposes MBA which consists of pre-trained feature extractors, text encoder, image encoder, multimodal bilinear attention fusion module, and summary decoder to complete abstractive multimodal summarization task. A residual network is added to the model to enhance the textual modality information and alleviate the modality-bias problem. Experiments show that the model is better than the baseline models and performs better than text summarization methods that ignore visual modality.

A panoramic survey of natural language processing in the Arab world

The term natural language refers to any system of symbolic communication (spoken, signed or written) without intentional human planning and design. This distinguishes natural languages such as Arabic and Japanese from artificially constructed languages such as Esperanto or Python. Natural language processing (NLP) is the sub-field of artificial intelligence (AI) focused on modeling natural languages to build applications such as speech recognition and synthesis, machine translation, optical character recognition (OCR), sentiment analysis (SA), question answering, dialogue systems, etc. NLP is a highly interdisciplinary field with connections to computer science, linguistics, cognitive science, psychology, mathematics and others. Some of the earliest AI applications were in NLP (e.g., machine translation); and the last decade (2010-2020) in particular has witnessed an incredible increase in quality, matched with a rise in public awareness, use, and expectations of what may have seemed like science fiction in the past. NLP researchers pride themselves on developing language independent models and tools that can be applied to all human languages, e.g. machine translation systems can be built for a variety of languages using the same basic mechanisms and models. However, the reality is that some languages do get more attention (e.g., English and Chinese) than others (e.g., Hindi and Swahili). Arabic, the primary language of the Arab world and the religious language of millions of non-Arab Muslims is somewhere in the middle of this continuum. Though Arabic NLP has many challenges, it has seen many successes and developments. Next we discuss Arabic's main challenges as a necessary background, and we present a brief history of Arabic NLP. We then survey a number of its research areas, and close with a critical discussion of the future of Arabic NLP.

A Collection of Swedish Diachronic Word Embedding Models Trained on Historical Newspaper Data

This paper describes the creation of several word embedding models based on a large collection of diachronic Swedish newspaper material available through Språkbanken Text, the Swedish language bank. This data was produced in the context of Språkbanken Text’s continued mission to collaborate with humanities and natural language processing (NLP) researchers and to provide freely available language resources, for the development of state-of-the-art NLP methods and tools.

A new Approach of Documents Indexing Using subject modelling and Summarization

Document indexing is a field of research in Natural Language Processing (NLP) that has been rapidly evolving for 70 years. It is an operation that focuses on the synthetic representation of a document according to a model in order to facilitate their subsequent use. This work is concerned with document indexing. Two points are addressed. This work is concerned with document indexing, we are trying to accelerate the indexing process of large document datasets, two points are addressed. The first one concerns the development of a document indexing system using the system's operating process based on three phases namely pre-processing, weighting, and subject modelling. The second point concerns the proposal for a new system that integrates a new developed automatic summary subsystem, the goal of this point is to minimize indexing time.

Aspect Based Multi-Labeling Using SVM Based Ensembler

Sentiment analysis is one of the most prominent sub-areas of research in Natural Language Processing (NLP), where it is important to consider implicit or explicit emotions conveyed by review material. Researchers also recognized that the generic feelings derived from the textual material are insufficient, so the sentiment analysis aspect based was coined to extract the emotions from textual data. Multi-labeling based on aspects data can resolve the issue of extracting emotion aspect based. In this work, a novel approach namely: Evolutionary Ensembler (EEn) is proposed to effectively boost the accuracy and diversity of multi-label learners. Unlike traditional multi-label training methods, EEn emphasizes the accuracy and diversity of multi-label-based models. We have used seven datasets (medical, hotel, movies, automobiles, proteins, birds, emotions, news). At first, we applied a pre-processing step to gain the refined and clean data. Second, we have applied the Vader tool with Bag of Words (BoW) for the feature extraction. Third, the word2vec method is applied to draw an association between words. Moreover, the SVM model (tuned with GA) is trained and tested on the refined data. The accuracy of the aspect-based multi-labeling using the SVM-GA on medical, hotel, movies, automobiles, proteins, birds, emotions, news are 93.13%, 94.32%, 94.0%, 95.10%, 90.20%, 93.22%, 90.0%, and 94.0%, respectively. Our proposed model with different dimensions of multi-label datasets shows that EEn is vastly superior to other popular techniques. Experimental outcomes validate the success of the implemented approach among existing benchmark techniques.

BanglaLM: Bangla Corpus for Language Model Research

Natural language processing (NLP) is an area of machine learning that has garnered a lot of attention in recent days due to the revolution in artificial intelligence, robotics, and smart devices. NLP focuses on training machines to understand and analyze various languages, extract meaningful information from those, translate from one language to another, correct grammar, predict the next word, complete a sentence, or even generate a completely new sentence from an existing corpus. A major challenge in NLP lies in training the model for obtaining high prediction accuracy since training needs a vast dataset. For widely used languages like English, there are many datasets available that can be used for NLP tasks like training a model and summarization but for languages like Bengali, which is only spoken primarily in South Asia, there is a dearth of big datasets which can be used to build a robust machine learning model. Therefore, NLP researchers who mainly work with the Bengali language will find an extensive, robust dataset incredibly useful for their NLP tasks involving the Bengali language. With this pressing issue in mind, we have prepared a dataset whose content is curated from social media, blogs, newspapers, wiki pages, and other similar resources. The amount of samples in this dataset is 19132010, and the length varies from 3 to 512 words. This dataset can easily be used to build any unsupervised machine learning model with an aim to performing necessary NLP tasks involving the Bengali language. We are also releasing two preprocessed version of this dataset that is especially suited for training both core machine learning-based and statistical-based model. As very few attempts have been made in this domain, keeping Bengali language researchers in mind, we believe our dataset will significantly contribute to the Bengali machine learning and NLP community.

User Stories and Natural Language Processing: A Systematic Literature Review

<italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Context:</i> User stories have been widely accepted as artifacts to capture the user requirements in agile software development. They are short pieces of texts in a semi-structured format that express requirements. Natural language processing (NLP) techniques offer a potential advantage in user story applications. <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Objective:</i> Conduct a systematic literature review to capture the current state-of-the-art of NLP research on user stories. <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Method:</i> The search strategy is used to obtain relevant papers from SCOPUS, ScienceDirect, IEEE Xplore, ACM Digital Library, SpringerLink, and Google Scholar. Inclusion and exclusion criteria are applied to filter the search results. We also use the forward and backward snowballing techniques to obtain more comprehensive results. <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Results:</i> The search results identified 718 papers published between January 2009 to December 2020. After applying the inclusion/exclusion criteria and the snowballing technique, we identified 38 primary studies that discuss NLP techniques in user stories. Most studies used NLP techniques to extract aspects of who, what, and why from user stories. The purpose of NLP studies in user stories is broad, ranging from discovering defects, generating software artifacts, identifying the key abstraction of user stories, and tracing links between model and user stories. <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Conclusion:</i> NLP can help system analysts manage user stories. Implementing NLP in user stories has many opportunities and challenges. Considering the exploration of NLP techniques and rigorous evaluation methods is required to obtain quality research. As with NLP research in general, the ability to understand a sentence’s context continues to be a challenge.