Language Processing Tasks Research Articles

Towards deep understanding of graph convolutional networks for relation extraction

Relation extraction aims at identifying semantic relations between pairs of named entities from unstructured texts and is considered an essential prerequisite for many downstream tasks in natural language processing (NLP). Owing to the ability in expressing complex relationships and interdependency, graph neural networks (GNNs) have been gradually used to solve the relation extraction problem and have achieved state-of-the-art results. However, the designs of GNN-based relation extraction methods are mostly based on empirical intuition, heuristic, and experimental trial-and-error. A clear understanding of why and how GNNs perform well in relation extraction tasks is lacking. In this study, we investigate three well-known GNN-based relation extraction models, CGCN, AGGCN, and SGCN, and aim to understand the underlying mechanisms of the extractions. In particular, we provide a visual analytic to reveal the dynamics of the models and provide insight into the function of intermediate convolutional layers. We determine that entities, particularly subjects and objects in them, are more important features than other words for relation extraction tasks. With various masking strategies, the significance of entity type to relation extraction is recognized. Then, from the perspective of the model architecture, we find that graph structure modeling and aggregation mechanisms in GCN do not significantly affect the performance improvement of GCN-based relation extraction models. The above findings are of great significance in promoting the development of GNNs. Based on these findings, an engineering oriented MLP-based GNN relation extraction model is proposed to achieve a comparable performance and greater efficiency.

Kidney CT Scan Image Classification Using Modified Vision Transformer

With the rising number of kidney-related health issues, early and precise diagnosis is crucial. The study aims to create a reliable method for categorizing kidney CT scan images into four groups: Cyst, Normal, Tumor, and stone. Traditional approaches usually rely on typical Machine Learning (ML) and Convolution Neural Networks (CNNs). However, in this research, the potential of a novel model called Vision Transformer (ViT) is explored. ViT was initially designed for Natural Language Processing (NLP) tasks but shows promise for medical image classification. ViT’s capabilities are enhanced by coupling it with Fully Connected Networks (FCN). This combination helps to merge the feature extraction capability of the ViT and the classification ability of the FCN, which ultimately helps to overcome the challenge of detecting kidney-related issues.

Natural Language Processing Challenges and Issues: A Literature Review

Natural Language Processing (NLP) is the computerized approach to analyzing text using both structured and unstructured data. NLP is a simple, empirically powerful, and reliable approach. It achieves state-of-the-art performance in language processing tasks like Semantic Search (SS), Machine Translation (MT), Text Summarization (TS), Sentiment Analyzer (SA), Named Entity Recognition (NER) and Emotion Detection (ED). NLP is expected to be the technology of the future, based on current technology deployment and adoption. The primary question is: What does NLP have to offer in terms of reality, and what are the prospects? There are several problems to be addressed with this developing method, as it must be compatible with future technology. In this paper, the benefits, challenges and limitations of this innovative paradigm along with the areas open to do research are shown.

Contemporary Approaches in Evolving Language Models

This article provides a comprehensive survey of contemporary language modeling approaches within the realm of natural language processing (NLP) tasks. This paper conducts an analytical exploration of diverse methodologies employed in the creation of language models. This exploration encompasses the architecture, training processes, and optimization strategies inherent in these models. The detailed discussion covers various models ranging from traditional n-gram and hidden Markov models to state-of-the-art neural network approaches such as BERT, GPT, LLAMA, and Bard. This article delves into different modifications and enhancements applied to both standard and neural network architectures for constructing language models. Special attention is given to addressing challenges specific to agglutinative languages within the context of developing language models for various NLP tasks, particularly for Arabic and Turkish. The research highlights that contemporary transformer-based methods demonstrate results comparable to those achieved by traditional methods employing Hidden Markov Models. These transformer-based approaches boast simpler configurations and exhibit faster performance during both training and analysis. An integral component of the article is the examination of popular and actively evolving libraries and tools essential for constructing language models. Notable tools such as NLTK, TensorFlow, PyTorch, and Gensim are reviewed, with a comparative analysis considering their simplicity and accessibility for implementing diverse language models. The aim is to provide readers with insights into the landscape of contemporary language modeling methodologies and the tools available for their implementation.

Language Embeddings Sometimes Contain Typological Generalizations

Abstract To what extent can neural network models learn generalizations about language structure, and how do we find out what they have learned? We explore these questions by training neural models for a range of natural language processing tasks on a massively multilingual dataset of Bible translations in 1,295 languages. The learned language representations are then compared to existing typological databases as well as to a novel set of quantitative syntactic and morphological features obtained through annotation projection. We conclude that some generalizations are surprisingly close to traditional features from linguistic typology, but that most of our models, as well as those of previous work, do not appear to have made linguistically meaningful generalizations. Careful attention to details in the evaluation turns out to be essential to avoid false positives. Furthermore, to encourage continued work in this field, we release several resources covering most or all of the languages in our data: (1) multiple sets of language representations, (2) multilingual word embeddings, (3) projected and predicted syntactic and morphological features, (4) software to provide linguistically sound evaluations of language representations.

Contextualized word senses: from attention to compositionality

Abstract The neural architectures of language models are becoming increasingly complex, especially that of Transformers, based on the attention mechanism. Although their application to numerous natural language processing tasks has proven to be very fruitful, they continue to be models with little or no interpretability and explainability. One of the tasks for which they are best suited is the encoding of the contextual sense of words using contextualized embeddings. In this paper we propose a transparent, interpretable, and linguistically motivated strategy for encoding the contextual sense of words by modeling semantic compositionality. Particular attention is given to dependency relations and semantic notions such as selection preferences and paradigmatic classes. A partial implementation of the proposed model is carried out and compared with Transformer-based architectures for a given semantic task, namely the similarity calculation of word senses in context. The results obtained show that it is possible to be competitive with linguistically motivated models instead of using the black boxes underlying complex neural architectures.

Graphmax for Text Generation

In text generation, a large language model (LM) makes a choice of each new word based only on the former selection of its context using the softmax function. Nevertheless, the link statistics information of concurrent words based on a scene-specific corpus is valuable in choosing the next word, which can help to ensure the topic of the generated text to be aligned with the current task. To fully explore the co-occurrence information, we propose a graphmax function for task-specific text generation. Using the graph-based regularization, graphmax enables the final word choice to be determined by both the global knowledge from the LM and the local knowledge from the scene-specific corpus. The traditional softmax function is regularized with a graph total variation (GTV) term, which incorporates the local knowledge into the LM and encourages the model to consider the statistical relationships between words in a scene-specific corpus. The proposed graphmax is versatile and can be readily plugged into any large pre-trained LM for text generation and machine translation. Through extensive experiments, we demonstrate that the new GTV-based regularization can improve performances in various natural language processing (NLP) tasks in comparison with existing methods. Moreover, through human experiments, we observe that participants can easily distinguish the text generated by graphmax or softmax.

BIR: Biomedical Information Retrieval System for Cancer Treatment in Electronic Health Record Using Transformers

The rapid growth of electronic health records (EHRs) has led to unprecedented biomedical data. Clinician access to the latest patient information can improve the quality of healthcare. However, clinicians have difficulty finding information quickly and easily due to the sheer data mining volume. Biomedical information retrieval (BIR) systems can help clinicians find the information required by automatically searching EHRs and returning relevant results. However, traditional BIR systems cannot understand the complex relationships between EHR entities. Transformers are a new type of neural network that is very effective for natural language processing (NLP) tasks. As a result, transformers are well suited for tasks such as machine translation and text summarization. In this paper, we propose a new BIR system for EHRs that uses transformers for predicting cancer treatment from EHR. Our system can understand the complex relationships between the different entities in an EHR, which allows it to return more relevant results to clinicians. We evaluated our system on a dataset of EHRs and found that it outperformed state-of-the-art BIR systems on various tasks, including medical question answering and information extraction. Our results show that Transformers are a promising approach for BIR in EHRs, reaching an accuracy and an F1-score of 86.46%, and 0.8157, respectively. We believe that our system can help clinicians find the information they need more quickly and easily, leading to improved patient care.

Fintech Key-Phrase: A New Chinese Financial High-Tech Dataset Accelerating Expression-Level Information Retrieval

Expression-level information extraction is a challenging task in natural language processing (NLP), which aims to retrieve crucial semantic information from linguistic documents. However, there is a lack of up-to-date data resources for accelerating expression-level information extraction, particularly in the Chinese financial high technology field. To address this gap, we introduce Fintech Key-Phrase: a human-annotated key-phrase dataset for the Chinese financial high technology domain. This dataset comprises over 12K paragraphs along with annotated domain-specific key-phrases. We extract the publicly released reports, Chinese management’s discussion and analysis (CMD&A), from the renowned Chinese research data services platform (CNRDS) and then filter the reports related to high technology. The high technology key-phrases are annotated following pre-defined philosophy guidelines to ensure annotation quality. In order to better understand the limitations and challenges in the purposed dataset, we conducted comprehensive noise evaluation experiments for the Fintech Key-Phrase, including annotation consistency assessment and absolute annotation quality evaluation. To demonstrate the usefulness of our released Fintech Key-Phrase in retrieving valuable information in the Chinese financial high technology field, we evaluate its significance using several superior information retrieval systems as representative baselines and report corresponding performance statistics. Additionally, we further applied ChatGPT to the text augmentation approach of the Fintech Key-Phrase dataset. Extensive comparative experiments demonstrate that the augmented Fintech Key-Phrase dataset significantly improved the coverage and accuracy of extracting key phrases in the finance and high-tech domains. We believe that this dataset can facilitate scientific research and exploration in the Chinese financial high technology field. We have made the Fintech Key-Phrase dataset and the experimental code of the adopted baselines accessible on Github: https://github.com/albert-jin/Fintech-Key-Phrase. To encourage newcomers to participate in the financial high-tech domain information retrieval research, we have developed a series of tools, including an open website1and corresponding real-time information retrieval APIs.2

Exploring the Chinese Cultural Aphasia Phenomenon and Countermeasures in College English Teaching Based on Semantic Comprehension Models

Abstract The purpose of this paper is to study the phenomenon of Chinese cultural aphasia and analyze the reasons for the aphasia phenomenon. Secondly, the BERT model in the deep learning model is utilized to improve the effect of various natural language processing tasks, and the English dialogue system is set up to improve the students’ English oral expression ability when introducing Chinese culture. Then, a semantic comprehension model is constructed based on the role information, and coping strategies are proposed for the current English cultural aphasia phenomenon. Finally, the results of teaching evaluation in the university English classroom and the effects of learning Chinese culture in AI-assisted English teaching are analyzed, respectively. According to the study, the teaching process stage is where the most noticeable progress is made, with a score 18.66 points higher than the pre-test score. The average score of the learning effect of students who received AI-assisted means to learn Chinese culture was 3.603 points.

Distillation of Large Language Models for Text Simplification

This work presents a comprehensive methodology for harnessing the capabilities of Large Language Models to address specific Natural Language Processing tasks, with a focus on Text Simplification. While LLMs have demonstrated their prowess in tackling a wide range of NLP challenges, their demanding computational requirements can render them impractical for real-time online inference. In response to this limitation, we suggest the concept of text distillation, a technique aimed at effectively transferring the knowledge stored within LLMs to more compact and computationally efficient neural networks.

XAI Transformer based Approach for Interpreting Depressed and Suicidal User Behavior on Online Social Networks

Online social networks can be used for mental healthcare monitoring using Artificial Intelligence and Machine Learning techniques for detecting various mental health disorders and corresponding risk assessment. Recent research in this domain has primarily been focused on leveraging deep neural networks and various Transformer based Large Language Models, which have now become state-of-the-art for most natural language processing and computational linguistic tasks due to their unmatched prediction accuracy. Unlike conventional machine learning algorithms, these deep neural networks are black box architectures, where it is difficult to interpret and explain their predicted outcome. However, a black box classification outcome is insufficient for healthcare applications. Such systems will not be widely adopted and trusted by healthcare practitioners if they are not able to understand and explain the reasoning behind the predicted decisions made by an AI and ML based healthcare diagnostic system. The key objective of our research is to demonstrate the applications of model agnostic, post hoc surrogate XAI techniques for providing explainability to classification decisions of pretrained LLMs (Transformers) based mental healthcare diagnostic systems fine-tuned (or trained) to detect depressive and suicidal behavior using UGC from online social networks. For this, we have used the two most recent and popular techniques, SHAP and LIME. We have conducted extensive and in-depth experiments with four datasets and six pretrained LLMs, three of which have already been domain-adapted using mental health related datasets. We have also performed Few Shot Learning experiments with these three pretrained mental health domain-adapted LLMs. The results of qualitative and descriptive data analysis in this paper demonstrate that in order to build a comprehensive understanding of a person’s psychological state, emotion, and behavior and to discover the causes, symptoms, and triggers of mental health issues, it is essential to utilize eXplAInable (XAI) techniques with Transformer based LLMs (supervised). Alternatively, Transformer based unsupervised topic modeling technique BERTopic may be used for mental health risk monitoring and cause or symptom extraction when supervised training of LLMs is not feasible due to dataset annotation or availability challenges.

Event causality identification via graph contrast-based knowledge augmented networks

Identifying causality between events is a crucial research task in natural language processing. However, existing methods either ignore background knowledge of events or do not consider interference of knowledge graph noise on event representations. In this paper, we propose a novel Graph Contrast-based Knowledge Augmented Network (GCKAN) for event causality identification task, which integrates comprehensive background knowledge of events from knowledge graphs and alleviates the knowledge graph noise problem. First, a descriptive knowledge augmentation module is proposed to aggregate one-hop neighbor information in knowledge graphs and learn meaningful descriptive knowledge of events. Then, a relational knowledge augmentation module is designed to encode multi-hop path information and learn latent reasoning knowledge between event pairs. In addition, trustworthiness-based and degree-based graph contrastive learning schemas are devised in the two modules respectively, which suppress knowledge graph noise during information aggregation and derive more robust knowledge-aware event representations. Extensive experiments on three public datasets demonstrate the consistent superiority of GCKAN over state-of-the-art knowledge-based techniques. Noise interference experiments and cross-topic adaptation experiments further verify the robustness and generalization of GCKAN.

DeepGram: Combining Language Transformer and N-Gram based ML Models for YouTube Spam Comment Detection

Spam comments on YouTube videos are a persistent issue that can negatively impact the user experience and content creator's reputation. In this paper, we propose an algorithm called "DeepGram" for detecting YouTube spam comments using a combination of deep learning-based language transformer models and N-gram based machine learning models. The algorithm leverages the power of language transformers, which have shown significant success in various natural language processing tasks, along with N-gram based models that capture local context and patterns in the text data. The proposed algorithm goes through several stages, including data collection, text preprocessing, feature extraction, and model training. The collected YouTube comments are preprocessed by removing special characters, punctuation, and HTML tags, and converting them to lowercase. Common stop words are also removed, and stemming or lemmatization is applied to reduce dimensionality. The algorithm then extracts features from the preprocessed comments using a combination of language transformer models and N-gram based features. Finally, the features are fed into machine learning models for training and evaluation. Experimental results on a large dataset of YouTube comments show that the DeepGram algorithm achieves high accuracy and robust performance in detecting spam comments. The proposed algorithm can be potentially employed as an effective tool for YouTube content creators and platform moderators to combat spam comments and improve the quality of user interactions on YouTube videos.

An improved Bulgarian natural language processing pipeline

In this paper, we present a language pipeline for processing Bulgarian language data. The pipeline consists of the following steps: tokenization, sentence splitting, part-of-speech tagging, dependency parsing, named entity recognition, lemmatization, and word sense disambiguation. The first two components are based on rules and lists of words specific to the Bulgarian language, while the rest of the components use machine learning algorithms trained on universal dependency data and pretrained word vectors. The pipeline is implemented in the Python library spaCy (https://spacy.io/) and achieves significant results on all the included subtasks. The pipeline is open source and is available on Github (https://github.com/melaniab/spacy-pipeline-bg/) for use by researchers and developers for a variety of natural language processing and text analysis tasks.

Multi-task peer-to-peer learning using an encoder-only transformer model

Peer-to-peer (P2P) learning is a decentralized approach to organizing the collaboration between end devices known as agents. Agents contain heterogeneous data, and that heterogeneity is disrupting the convergence and accuracy of the collectively learned models. A common technique to mitigate the negative impact of heterogeneous data is to arrange the learning process in a multi-task setting where each task, although it has the same learning objective, is learned separately. However, the multi-task technique can also be applied to solve distinct learning tasks. This paper presents and evaluates a novel approach that utilizes an encoder-only transformer model to enable collaboration between agents learning two distinct Natural Language Processing (NLP) tasks. The evaluation of the approach studied revealed that collaboration among agents, even when working towards separate objectives, can result in mutual benefits, mainly when the connections between agents are carefully considered. The multi-task collaboration led to a statistically significant increase of 11.6% in the mean relative accuracy compared to the baseline results for individual tasks. To our knowledge, this is the first study demonstrating a successful and beneficial collaboration between two distinct NLP tasks in a peer-to-peer setting.

Named entity recognition of rice genes and phenotypes based on BiGRU neural networks

Named Entity Recognition (NER) is a fundamental but crucial task in natural language processing (NLP) and big data analysis, with wide application range. NER for rice genes and phenotypes is a technique to identify genes and phenotypes from a large amount of text. NER for rice genes and phenotypes can facilitate the acquisition of information in the field of crops and provide references for our research on higher quality crops. At the same time, named entity recognition still faces many challenges. In this paper, we propose an improved bidirectional gated recurrent unit neural network (BI-GRU) method, which is used to automatically identify the required entities (i.e. gene names, rice phenotypes) from relevant rice literature and patents. The neural network model is combined with the Softmax function to directly output the probabilities of labels, forming the BI-GRU-SF model. With the ability of deep learning methods, the semantic information in the context can be learned without the need for feature engineering. Finally, we conducted experiments, and the results showed that our proposed model provided better performance compared to other models. All datasets and resource codes of BI-GRU-SF are available at https://github.com/qqeeqq/NER for academic use.

A framework-based transformer and knowledge distillation for interior style classification

Interior style classification is an interesting problem which has potential applications both commercial and academic communities. This task aims to devise interior design styles automatically. Thus, interior designers will explore customers’ tastes and then precisely provide suggestions for decor inspiration based on their preferences. Recently, Convolutional Neural Networks (CNNs) have been considered the de-facto standard in computer vision tasks. Therefore, several current works have tended to address interior style classification using CNN-based architectures. Moreover, transformer-based architectures and attention-based encoder–decoder models have been proven successfully in computer vision and natural language processing tasks. Sequentially, more studies have been arguing the efficiency of combining CNN-based architectures and transformer-based architectures for normal image classification problems. In this project, we focus on finding an architecture network that is suitable for the interior style classification problem. We propose a robustness method to address interior style design classification, named ISC-DeIT. The proposed method is based on Data-efficient image transformer architectures and knowledge distillation, which can be trained on small datasets effectively. Especially, a proposed additional module is plugged to leverage learning feature representations for improving predictive accuracy. Experiments were carried out on a new curated dataset with five interior styles including Art-Decor, Hitech, Indochina, Industrial, and Scandinavian. Empirical results of ISC- DeiT indicated that the ability of prediction for interior style classification of the proposed method has been increased significantly, compared with other state-of-the-art methods.

Efficient utilization of pre-trained models: A review of sentiment analysis via prompt learning

Sentiment analysis is one of the traditional well-known tasks in Natural Language Processing (NLP) research. In recent years, Pre-trained Models (PMs) have become one of the frontiers of NLP, and the knowledge in PMs is usually leveraged to improve machine learning models' performance for a variety of downstream NLP tasks including sentiment analysis. However, there are also some shortcomings in PM-based approaches. For example, many studies pointed out there are gaps between pre-training and fine-tuning. In addition, because of the time-consuming and high-cost data annotation process, the labeled training data are usually precious and scarce, which often leads to the over-fitting of models. The recent advent of prompt learning technology provides a promising solution to the above challenges. In this paper, we first discussed the background of prompt learning and its basic principle. Prompt learning changes the model input by adding templates, allowing learning tasks to adapt actively to pre-trained models, and therefore can promote the innovation and applicability of pre-trained models. Then we investigated the evolution of sentiment analysis and explored the application of prompt learning to different sentiment analysis tasks. Our research and review show that prompt learning is more suitable for sentiment analysis tasks and can achieve good performance. Finally, we also provided some future research directions on prompt-based sentiment analysis. Our survey demonstrated that prompt learning can facilitate the efficient utilization of pre-trained models in sentiment analysis and other tasks, which makes it a new paradigm worthy of further exploration.

Human behavior recognition based on sparse transformer with channel attention mechanism.

Human activity recognition (HAR) has recently become a popular research field in the wearable sensor technology scene. By analyzing the human behavior data, some disease risks or potential health issues can be detected, and patients' rehabilitation progress can be evaluated. With the excellent performance of Transformer in natural language processing and visual tasks, researchers have begun to focus on its application in time series. The Transformer model models long-term dependencies between sequences through self-attention mechanisms, capturing contextual information over extended periods. In this paper, we propose a hybrid model based on the channel attention mechanism and Transformer model to improve the feature representation ability of sensor-based HAR tasks. Extensive experiments were conducted on three public HAR datasets, and the results show that our network achieved accuracies of 98.10%, 97.21%, and 98.82% on the HARTH, PAMAP2, and UCI-HAR datasets, respectively, The overall performance is at the level of the most advanced methods.