Natural Language Processing Problems Research Articles

Research on news text classification based on RCNN

One of the popular directions of natural language processing is text processing, which is an effective method used to manage data, and it has developed rapidly after the rise of artificial intelligence. However, text classification based on traditional machine learning algorithms encountered bottlenecks in the development process, and after deep learning, a multi-level neural network, was introduced, it was soon widely used in natural language processing problems, making many branches of the field develop further, and now deep learning is the mainstream model in the field of text classification. In this paper, we first introduce the general process of text classification, point out the current problems of text classification based on traditional machine learning, and then describe the superiority of deep learning in this aspect of feature extraction. After that, three different deep learning models are experimented, and the differences between the trained models are compared and the relative optimal models are derived. Finally, the future trends and research directions of deep learning in this field are discussed.

NLP-Based Bi-Directional Recommendation System: Towards Recommending Jobs to Job Seekers and Resumes to Recruiters

For more than ten years, online job boards have provided their services to both job seekers and employers who want to hire potential candidates. The provided services are generally based on traditional information retrieval techniques, which may not be appropriate for both job seekers and employers. The reason is that the number of produced results for job seekers may be enormous. Therefore, they are required to spend time reading and reviewing their finding criteria. Reciprocally, recruitment is a crucial process for every organization. Identifying potential candidates and matching them with job offers requires a wide range of expertise and knowledge. This article proposes a reciprocal recommendation based on bi-directional correspondence as a way to support both recruiters’ and job seekers’ work. Recruiters can find the best-fit candidates for every job position in their job postings, and job seekers can find the best-match jobs to match their resumes. We show how machine learning can solve problems in natural language processing of text content and similarity scores depending on job offers in major Saudi cities scraped from Indeed. For bi-directional matching, a similarity calculation based on the integration of explicit and implicit job information from two sides (recruiters and job seekers) has been used. The proposed system is evaluated using a resume/job offer dataset. The performance of generated recommendations is evaluated using decision support measures. Obtained results confirm that the proposed system can not only solve the problem of bi-directional recommendation, but also improve the prediction accuracy.

On the Rate of Convergence of a Classifier Based on a Transformer Encoder

Pattern recognition based on a high-dimensional predictor is considered. A classifier is defined which is based on a Transformer encoder. The rate of convergence of the misclassification probability of the classifier towards the optimal misclassification probability is analyzed. It is shown that this classifier is able to circumvent the curse of dimensionality provided the a posteriori probability satisfies a suitable hierarchical composition model. Furthermore, the difference between the Transformer classifiers theoretically analyzed in this paper and the ones used in practice today is illustrated by means of classification problems in natural language processing.

Quantum Dynamic Optimization Algorithm for Neural Architecture Search on Image Classification

Deep neural networks have proven to be effective in solving computer vision and natural language processing problems. To fully leverage its power, manually designed network templates, i.e., Residual Networks, are introduced to deal with various vision and natural language tasks. These hand-crafted neural networks rely on a large number of parameters, which are both data-dependent and laborious. On the other hand, architectures suitable for specific tasks have also grown exponentially with their size and topology, which prohibits brute force search. To address these challenges, this paper proposes a quantum dynamic optimization algorithm to find the optimal structure for a candidate network using Quantum Dynamic Neural Architecture Search (QDNAS). Specifically, the proposed quantum dynamics optimization algorithm is used to search for meaningful architectures for vision tasks and dedicated rules to express and explore the search space. The proposed quantum dynamics optimization algorithm treats the iterative evolution process of the optimization over time as a quantum dynamic process. The tunneling effect and potential barrier estimation in quantum mechanics can effectively promote the evolution of the optimization algorithm to the global optimum. Extensive experiments on four benchmarks demonstrate the effectiveness of QDNAS, which is consistently better than all baseline methods in image classification tasks. Furthermore, an in-depth analysis is conducted on the searchable networks that provide inspiration for the design of other image classification networks.

Siamese-Based Architecture for Cross-Lingual Plagiarism Detection in English-Hindi Language Pairs.

The cross-lingual plagiarism detection (CLPD) is a challenging problem in natural language processing. Cross-lingual plagiarism is when a text is translated from any other language and used as it is without proper acknowledgment. Most of the existing methods provide good results for monolingual plagiarism detection, whereas the performances of existing methods for the CLPD are very limited. The reason for this is that it is difficult to represent the text from two different languages in a common semantic space. In this article, a novel Siamese architecture-based model is proposed to detect the cross-lingual plagiarism in English-Hindi language pairs. The proposed model combines the convolutional neural network (CNN) and bidirectional long short-term memory (Bi-LSTM) network to learn the semantic similarity among the cross-lingual sentences for the English-Hindi language pairs. In the proposed model, the CNN model learns the local context of words, whereas the Bi-LSTM model learns the global context of sentences in forward and backward directions. The performances of the proposed models are evaluated on the benchmark data set, that is, Microsoft paraphrase corpus, which is converted in the English-Hindi language pairs. The proposed model outperforms other models giving 67%, 72%, and 67% weighted average precision, recall, and F1-measure scores. The experimental results show the effectiveness of the proposed models over the baseline models because the proposed model is very efficient in representing the cross-lingual text very efficiently.

Computing Topological Invariants of Deep Neural Networks.

A deep neural network has multiple layers to learn more complex patterns and is built to simulate the activity of the human brain. Currently, it provides the best solutions to many problems in image recognition, speech recognition, and natural language processing. The present study deals with the topological properties of deep neural networks. The topological index is a numeric quantity associated to the connectivity of the network and is correlated to the efficiency and accuracy of the output of the network. Different degree-related topological indices such as Zagreb index, Randic index, atom-bond connectivity index, geometric-arithmetic index, forgotten index, multiple Zagreb indices, and hyper-Zagreb index of deep neural network with a finite number of hidden layers are computed in this study.

Refining Word Embeddings with Sentiment Information for Sentiment Analysis

Natural Language Processing problems generally require the use of pre-trained distributed word representations to be solved with deep learning models. However, distributed representations usually rely on contextual information which prevents them from learning all the important word characteristics. The task of sentiment analysis suffers from such a problem because sentiment information is ignored during the process of learning word embeddings. The performance of sentiment analysis can be affected since two words with similar vectors may have opposite sentiment orientations. The present paper introduces a novel model called Continuous Sentiment Contextualized Vectors (CSCV) to address this problem. The proposed model can learn word sentiment embedding using its surrounding context words. It uses Continuous Bag-of-Words (CBOW) model to deal with the context and sentiment lexicons to identify sentiment. Existing pre-trained vectors are combined then with the obtained sentiment vectors using Principal component analysis (PCA) to enhance their quality. The experiments show that: (1) CSCV vectors can be used to enhance any pre-trained word vectors; (2) The result vectors strongly alleviate the problem of similar words with opposite polarities; (3) The performance of sentiment classification is improved by applying this approach.

A scoping review of publicly available language tasks in clinical natural language processing.

To provide a scoping review of papers on clinical natural language processing (NLP) shared tasks that use publicly available electronic health record data from a cohort of patients. We searched 6 databases, including biomedical research and computer science literature databases. A round of title/abstract screening and full-text screening were conducted by 2 reviewers. Our method followed the PRISMA-ScR guidelines. A total of 35 papers with 48 clinical NLP tasks met inclusion criteria between 2007 and 2021. We categorized the tasks by the type of NLP problems, including named entity recognition, summarization, and other NLP tasks. Some tasks were introduced as potential clinical decision support applications, such as substance abuse detection, and phenotyping. We summarized the tasks by publication venue and dataset type. The breadth of clinical NLP tasks continues to grow as the field of NLP evolves with advancements in language systems. However, gaps exist with divergent interests between the general domain NLP community and the clinical informatics community for task motivation and design, and in generalizability of the data sources. We also identified issues in data preparation. The existing clinical NLP tasks cover a wide range of topics and the field is expected to grow and attract more attention from both general domain NLP and clinical informatics community. We encourage future work to incorporate multidisciplinary collaboration, reporting transparency, and standardization in data preparation. We provide a listing of all the shared task papers and datasets from this review in a GitLab repository.

An Enhanced Question Pair Similarity Using Machine Learning Approaches

Abstract: Quora is a place to gain and share knowledge—about anything. It’s a platform to ask questions and connect with people who contribute unique insights and quality answers. This empowers people to learn from each other and to better understand the world. Currently, Quora uses a Random Forest model to identify duplicate questions. Tackling this natural language processing problem by applying advanced techniques to classify whether question pairs are duplicates or not, so will make it easier to find high quality answers to questions resulting in an improved experience for Quora writers, seekers, and readers. We by enhancing the features level by level in each system of total 3 got the XG Boost algorithm as the best model in order to solve such problem, not only in the case of Quora but also with the Stack overflow, medium etc

Bio-SODA UX: enabling natural language question answering over knowledge graphs with user disambiguation

The problem of natural language processing over structured data has become a growing research field, both within the relational database and the Semantic Web community, with significant efforts involved in question answering over knowledge graphs (KGQA). However, many of these approaches are either specifically targeted at open-domain question answering using DBpedia, or require large training datasets to translate a natural language question to SPARQL in order to query the knowledge graph. Hence, these approaches often cannot be applied directly to complex scientific datasets where no prior training data is available. In this paper, we focus on the challenges of natural language processing over knowledge graphs of scientific datasets. In particular, we introduce Bio-SODA, a natural language processing engine that does not require training data in the form of question-answer pairs for generating SPARQL queries. Bio-SODA uses a generic graph-based approach for translating user questions to a ranked list of SPARQL candidate queries. Furthermore, Bio-SODA uses a novel ranking algorithm that includes node centrality as a measure of relevance for selecting the best SPARQL candidate query. Our experiments with real-world datasets across several scientific domains, including the official bioinformatics Question Answering over Linked Data (QALD) challenge, as well as the CORDIS dataset of European projects, show that Bio-SODA outperforms publicly available KGQA systems by an F1-score of least 20% and by an even higher factor on more complex bioinformatics datasets. Finally, we introduce Bio-SODA UX, a graphical user interface designed to assist users in the exploration of large knowledge graphs and in dynamically disambiguating natural language questions that target the data available in these graphs.

Stock market index prediction using deep Transformer model

Applications of deep learning in financial market prediction have attracted widespread attention from investors and scholars. From convolutional neural networks to recurrent neural networks, deep learning methods exhibit superior ability to capture the non-linear characteristics of stock markets and, accordingly, achieve a high performance on stock market index prediction. In this paper, we utilize the latest deep learning framework, Transformer, to predict the stock market index. Transformer was initially developed for the natural language processing problem, and has recently been applied to time series forecasting. Through the encoder–decoder architecture and the multi-head attention mechanism, Transformer can better characterize the underlying rules of stock market dynamics. We implement several back-testing experiments on the main stock market indices worldwide, including CSI 300, S&P 500, Hang Seng Index, and Nikkei 225. All the experiments demonstrate that Transformer outperforms other classic methods significantly and can gain excess earnings for investors.

Using Random Perturbations to Mitigate Adversarial Attacks on NLP Models

Deep learning models have excelled in solving many problems in Natural Language Processing, but are susceptible to extensive vulnerabilities. We offer a solution to this vulnerability by using random perturbations such as spelling correction, synonym substitution, or dropping the word. These perturbations are applied to random words in random sentences to defend NLP models against adversarial attacks. Our defense methods are successful in returning attacked models to their original accuracy within statistical significance.

Unsupervised Sentence Representation via Contrastive Learning with Mixing Negatives

Unsupervised sentence representation learning is a fundamental problem in natural language processing. Recently, contrastive learning has made great success on this task. Existing constrastive learning based models usually apply random sampling to select negative examples for training. Previous work in computer vision has shown that hard negative examples help contrastive learning to achieve faster convergency and better optimization for representation learning. However, the importance of hard negatives in contrastive learning for sentence representation is yet to be explored. In this study, we prove that hard negatives are essential for maintaining strong gradient signals in the training process while random sampling negative examples is ineffective for sentence representation. Accordingly, we present a contrastive model, MixCSE, that extends the current state-of-the-art SimCSE by continually constructing hard negatives via mixing both positive and negative features. The superior performance of the proposed approach is demonstrated via empirical studies on Semantic Textual Similarity datasets and Transfer task datasets.

A metaheuristic with a neural surrogate function for Word Sense Disambiguation

Word Sense Disambiguation (WSD) is one of the earliest problems in natural language processing which aims to determine the correct sense of words in context. The semantic information provided by WSD systems is highly beneficial to many tasks such as machine translation, information extraction, and semantic parsing. In this work, a new approach for WSD is proposed which uses a neural network as a surrogate fitness function in a metaheuristic algorithm. Also, a new method for simultaneous training of word and sense embeddings is proposed in this work. Accordingly, the node2vec algorithm is employed on the WordNet graph to generate sequences containing both words and senses. These sequences are then used along with paragraphs from Wikipedia in the word2vec algorithm to generate embeddings for words and senses at the same time. In order to address data imbalance in this task, sense probability distribution data extracted from the training corpus is used in the search process of the proposed simulated annealing algorithm. Furthermore, we introduce a new approach for clustering and mapping senses in the WordNet graph, which considerably improves the accuracy of the proposed method. In this approach, nodes in the WordNet graph are clustered on the condition that no two senses of the same word be present in one cluster. Then, repeatedly, all nodes in each cluster are mapped to a randomly selected node from that cluster, meaning that the representative node can take advantage of the training instances of all the other nodes in the cluster. Training the proposed method in this work is done using the SemCor dataset and the SemEval-2015 dataset has been used as the validation set. The final evaluation of the system is performed on SensEval-2, SensEval-3, SemEval-2007, SemEval-2013, SemEval-2015, and the concatenation of all five mentioned datasets. The performance of the system is also evaluated on the four content word categories, namely, nouns, verbs, adjectives, and adverbs. Experimental results show that the proposed method achieves accuracies in the range of 74.8 to 84.6 percent in the ten aforementioned evaluation categories which are close to and in some cases better than the state of the art in this task.

Cross – Language Information Retrieval

In today‘s growing nation, local database storage and retrieval is crucial for developing countries. A huge amount of information on Internet is available in numerous language that can be access by anybody at any time. Data Retrieval systems are fundamentally linguistic: the content or context of documents must be described, and the inquirers needed for documents must be expressed. These descriptions and expressions are most frequentlyarticulated in free or controlled vocabularies that have some of the same characteristics as natural language. Therefore, the processes of document description or request formulation must be strongly related to the processes of description and inquiry in natural language. Natural language processing is used to (a) preprocess the documents to extract content-carrying terms, (b) discover inter-term dependencies and build a conceptual hierarchy specific to the database domain, and (c) process the user's natural language requests into effective search queries. Since we assume that the information is primarily encoded as text, IR is also a natural language processing problem: to decide if a document is relevant to a given information need, one needs to be able to understand its content. CLIR deals with asking queries in one language and demand for retrieving documents in another language.This paper takes an overview of the new application areas of CLIR and reviews the approaches used in the process of CLIR research for query and document translation.

A Complete Process of Text Classification System Using State-of-the-Art NLP Models.

With the rapid advancement of information technology, online information has been exponentially growing day by day, especially in the form of text documents such as news events, company reports, reviews on products, stocks-related reports, medical reports, tweets, and so on. Due to this, online monitoring and text mining has become a prominent task. During the past decade, significant efforts have been made on mining text documents using machine and deep learning models such as supervised, semisupervised, and unsupervised. Our area of the discussion covers state-of-the-art learning models for text mining or solving various challenging NLP (natural language processing) problems using the classification of texts. This paper summarizes several machine learning and deep learning algorithms used in text classification with their advantages and shortcomings. This paper would also help the readers understand various subtasks, along with old and recent literature, required during the process of text classification. We believe that readers would be able to find scope for further improvements in the area of text classification or to propose new techniques of text classification applicable in any domain of their interest.

Survey on reinforcement learning for language processing

In recent years some researchers have explored the use of reinforcement learning (RL) algorithms as key components in the solution of various natural language processing (NLP) tasks. For instance, some of these algorithms leveraging deep neural learning have found their way into conversational systems. This paper reviews the state of the art of RL methods for their possible use for different problems of NLP, focusing primarily on conversational systems, mainly due to their growing relevance. We provide detailed descriptions of the problems as well as discussions of why RL is well-suited to solve them. Also, we analyze the advantages and limitations of these methods. Finally, we elaborate on promising research directions in NLP that might benefit from RL.

Generating extractive sentiment summaries for natural language user queries on products

Multi-document sentiment analysis is an important natural language processing problem. Summaries generated by these analyzers can greatly reduce the time necessary to read a collection of topically-related documents to locate the desired information needs of a user. With the ever-increasing globalization and technology of the modern day, analysis of online user reviews on different products is an especially pertinent application of the aforementioned problem. At present there are way too many user reviews on popular products for potential buyers to spend adequate time to read and extract the most salient product details and opinions of previous buyers. In solving this problem, we propose a fully-automated summarizer to reduce the workload of online customers. The proposed system takes a user query and extracts the most relevant and essential comments made by individual reviewers. As opposed to existing multi-document summarization approaches, our summarizer compiles comprehensive reviews by extracting important facets and sentiment information based on various sentence features rather than applying complex machine learning algorithms. The design of our summarizer is easy to understand and implement, without the required massive training data and excessive training time. The conducted empirical study shows that the proposed summarization system outperforms current state-of-the-art multi-document sentiment summarization approaches.

Software requirement specific entity extraction using transformer models

Software Requirement Specifications (SRS) documents provide the description of requirements and expectations attributed to software products. The structured text present in the SRS documents serves as a guide for developers in defining various functions in the process of software development. Software specific entity extraction is an important pre-processing step for various Natural Language Processing (NLP) tasks in the requirement engineering domain such as entity-centric search systems, SRS document summmarization, requirement classification, and requirement quality management. Recent advances in transformer-based models have significantly contributed to NLP and information retrieval problems, and achieved state-of-the-art performance for domain specific entity extraction tasks. In this study, we employ the transformer models including BERT, RoBERTa and ALBERT for software specific entity extraction. For this purpose, we annotate three requirement datasets, namely, DOORS, SRE, and RQA with varied sets of software specific entities. Our numerical study shows that transformer models are able to outperform the traditional approaches such as ML-CRF, and we find that BERT variants improve the F1-scores by 4% and 5% on the DOORS and SRE datasets, respectively. We conduct entity level error analysis to examine the partial and exact matching of entities and respective boundaries. Lastly, we experiment with few-shot learning to create sample efficient NER systems with template-based BART model

Deep learning-based approach for Arabic open domain question answering

Open-domain question answering (OpenQA) is one of the most challenging yet widely investigated problems in natural language processing. It aims at building a system that can answer any given question from large-scale unstructured text or structured knowledge-base. To solve this problem, researchers traditionally use information retrieval methods to retrieve the most relevant documents and then use answer extractions techniques to extract the answer or passage from the candidate documents. In recent years, deep learning techniques have shown great success in OpenQA by using dense representation for document retrieval and reading comprehension for answer extraction. However, despite the advancement in the English language OpenQA, other languages such as Arabic have received less attention and are often addressed using traditional methods. In this paper, we use deep learning methods for Arabic OpenQA. The model consists of document retrieval to retrieve passages relevant to a question from large-scale free text resources such as Wikipedia and an answer reader to extract the precise answer to the given question. The model implements dense passage retriever for the passage retrieval task and the AraELECTRA for the reading comprehension task. The result was compared to traditional Arabic OpenQA approaches and deep learning methods in the English OpenQA. The results show that the dense passage retriever outperforms the traditional Term Frequency-Inverse Document Frequency (TF-IDF) information retriever in terms of the top-20 passage retrieval accuracy and improves our end-to-end question answering system in two Arabic question-answering benchmark datasets.