Contextualized Word Representations Research Articles

Channel-Aware Decoupling Network for Multiturn Dialog Comprehension.

Training machines to understand natural language and interact with humans is one of the major goals of artificial intelligence. Recent years have witnessed an evolution from matching networks to pretrained language models (PrLMs). In contrast to the plain-text modeling as the focus of the PrLMs, dialog texts involve multiple speakers and reflect special characteristics, such as topic transitions and structure dependencies, between distant utterances. However, the related PrLM models commonly represent dialogs sequentially by processing the pairwise dialog history as a whole. Thus, the hierarchical information on either utterance interrelation or speaker roles coupled in such representations is not well addressed. In this work, we propose compositional learning for holistic interaction across the utterances beyond the sequential contextualization from PrLMs, in order to capture the utterance-aware and speaker-aware representations entailed in a dialog history. We decouple the contextualized word representations by masking mechanisms in transformer-based PrLM, making each word only focus on the words in the current utterance, other utterances, and two speaker roles (i.e., utterances of the sender and utterances of the receiver), respectively. In addition, we employ domain-adaptive training strategies to help the model adapt to the dialog domains. Experimental results show that our method substantially boosts the strong PrLM baselines in four public benchmark datasets, achieving new state-of-the-art performance over previous methods.

ALBERT4Spam: A Novel Approach for Spam Detection on Social Networks

Engaging in social media browsing stands out as one of the most prevalent online activities. As social media increasingly integrates into our daily routines, it opens up numerous opportunities for spammers seeking to target individuals through these platforms. Given the concise and sporadic nature of messages exchanged on social networks, they fall within the realm of short text classification challenges. Effectively addressing such issues requires appropriately representing the text to enhance classifier efficiency.Accordingly, this study utilizes robust representations derived from contextualized models as a component of the feature extraction process within our deep neural network model, which is built upon the Bidirectional Long Short-Term Memory neural network (BLSTM). Introducing ALBERT4Spam, the study presents a deep learning methodology aimed at identifying spam on social networking platforms. It harnesses the proven ALBERT model to acquire contextualized word representations, thereby elevating the effectiveness of the suggested neural network framework.The random search method was used to fine-tune the ALBERT4Spam model's hyperparameters, which included the number of BLSTM layers, neuron count, layer count, activation function, weight initializer, learning rate, optimizer, and dropout, in order to obtain optimal performance. The experiments conducted on three benchmark datasets demonstrate that our innovative model surpasses widely used methods in social network spam detection. The precision results stand at 0.98, 0.96, and 0.98 for Twitter, YouTube, and SMS datasets, respectively, showcasing superior performance outcomes.

The Impact of Word Splitting on the Semantic Content of Contextualized Word Representations

Abstract When deriving contextualized word representations from language models, a decision needs to be made on how to obtain one for out-of-vocabulary (OOV) words that are segmented into subwords. What is the best way to represent these words with a single vector, and are these representations of worse quality than those of in-vocabulary words? We carry out an intrinsic evaluation of embeddings from different models on semantic similarity tasks involving OOV words. Our analysis reveals, among other interesting findings, that the quality of representations of words that are split is often, but not always, worse than that of the embeddings of known words. Their similarity values, however, must be interpreted with caution.

Code Comments: A Way of Identifying Similarities in the Source Code

This study investigates whether analyzing the code comments available in the source code can effectively reveal functional similarities within software. The authors explore how both machine-readable comments (such as linter instructions) and human-readable comments (in natural language) can contribute towards measuring the code similarity. For the former, the work is relying on computing the cosine similarity over the one-hot encoded representation of the machine-readable comments, while for the latter, the focus is on detecting similarities in English comments, using threshold-based computations against the similarity measurements obtained using models based on Levenshtein distances (for form-based matches), Word2Vec (for contextual word representations), as well as deep learning models, such as Sentence Transformers or Universal Sentence Encoder (for semantic similarity). For evaluation, this research has analyzed the similarities between different source code versions of the open-source code editor, VSCode, based on existing ESlint-specific directives, as well as applying natural language processing techniques on incremental releases of Kubernetes, an open-source system for automating containerized application management. The experiments outlines the potential for detecting code similarities solely based on comments, and observations indicate that models like Universal Sentence Encoder are providing a favorable balance between recall and precision. This research is integrated into Project Martial, an open-source project for automatic assistance in detecting plagiarism in software.

On the Role of Morphological Information for Contextual Lemmatization

Abstract Lemmatization is a natural language processing (NLP) task that consists of producing, from a given inflected word, its canonical form or lemma. Lemmatization is one of the basic tasks that facilitate downstream NLP applications, and is of particular importance for high-inflected languages. Given that the process to obtain a lemma from an inflected word can be explained by looking at its morphosyntactic category, including fine-grained morphosyntactic information to train contextual lemmatizers has become common practice, without considering whether that is the optimum in terms of downstream performance. In order to address this issue, in this article we empirically investigate the role of morphological information to develop contextual lemmatizers in six languages within a varied spectrum of morphological complexity: Basque, Turkish, Russian, Czech, Spanish, and English. Furthermore, and unlike the vast majority of previous work, we also evaluate lemmatizers in out-of-domain settings, which constitutes, after all, their most common application use. The results of our study are rather surprising. It turns out that providing lemmatizers with fine-grained morphological features during training is not that beneficial, not even for agglutinative languages. In fact, modern contextual word representations seem to implicitly encode enough morphological information to obtain competitive contextual lemmatizers without seeing any explicit morphological signal. Moreover, our experiments suggest that the best lemmatizers out-of-domain are those using simple UPOS tags or those trained without morphology and, lastly, that current evaluation practices for lemmatization are not adequate to clearly discriminate between models.

ALBERT-Based Personalized Educational Recommender System: Enhancing Students’ Learning Outcomes in Online Learning

Online learners must navigate vast educational resources to find materials that meet their needs. This study introduces an ALBERT-based personalized educational recommender system to improve student learning. ALBERT (A Lite BERT), an optimized variant of the BERT algorithm, captures contextualized word representations and understands the semantic meaning of learning resources, student profiles, and interactions. This study evaluates the ALBERT-based recommender system’s personalized learning recommendations. To assess learning outcomes, a diverse group of students from different educational domains is evaluated. Before and after the recommender system, academic performance, knowledge retention, and engagement are assessed. User satisfaction surveys assess recommendation quality, relevance, and user experience. The recommender system uses ALBERT’s model optimization to improve recommendation accuracy, learner engagement, and personalized learning. The evaluation shows the ALBERT-based personalized recommender system improves online learning outcomes. System-generated recommendations boost student engagement, knowledge retention, and academic performance. User satisfaction surveys show that the ALBERT-based system meets learners’ needs by providing relevant and high-quality recommendations. This research shows how advanced deep learning algorithms like ALBERT can improve personalized online learning. ALBERT’s optimized training and inference speeds up the recommender system’s scalability. This empowers learners to access tailored and high-quality educational resources, maximizing their learning outcomes and potential in online learning.

Unmasking deception: a CNN and adaptive PSO approach to detecting fake online reviews.

Online reviews play a critical role in modern word-of-mouth communication, influencing consumers' shopping preferences and purchase decisions, and directly affecting a company's reputation and profitability. However, the credibility and authenticity of these reviews are often questioned due to the prevalence of fake online reviews that can mislead customers and harm e-commerce's credibility. These fake reviews are often difficult to identify and can lead to erroneous conclusions in user feedback analysis. This paper proposes a new approach to detect fake online reviews by combining convolutional neural network (CNN) and adaptive particle swarm optimization with natural language processing techniques. The approach uses datasets from popular online review platforms like Ott, Amazon, Yelp, TripAdvisor, and IMDb and applies feature selection techniques to select the most informative features. The paper suggests using attention mechanisms like bidirectional encoder representations from transformers and generative pre-trained transformer, as well as other techniques like Deep contextualized word representation, word2vec, GloVe, and fast Text, for feature extraction from online review datasets. The proposed method uses a multimodal approach based on a CNN architecture that combines text data to achieve a high accuracy rate of 99.4%. This outperforms traditional machine learning classifiers in terms of accuracy, recall, and F measure. The proposed approach has practical implications for consumers, manufacturers, and sellers in making informed product choices and decision-making processes, helping maintain the credibility of online consumer reviews. The proposed model shows excellent generalization abilities and outperforms conventional discrete and existing neural network benchmark models across multiple datasets. Moreover, it reduces the time complexity for both training and testing.

WITHDRAWN: Visual interpretation for contextualized word representation

WITHDRAWN: Visual interpretation for contextualized word representation

Automatic text summarization using deep reinforced model coupling contextualized word representation and attention mechanism

Automatic text summarization using deep reinforced model coupling contextualized word representation and attention mechanism

Creating CREATE queries with multi-task deep neural networks

Text-to-SQL is the task of mapping natural language utterances to structured query language (SQL). Prior studies focus on information retrieval aspect of this task. In this paper, we demonstrate a new use case for the text-to-SQL studies where a user can create database models from natural language and introduce the first dataset for this task. Furthermore, we propose a framework that consists of three modular components: (1) classifier component which predicts the data type and constraints of a column, (2) constraint component which establishes foreign key relationships between tables, (3) query component which generates a series of CREATE queries through a slot-filling approach. We propose various baseline models to evaluate the classifier component in different aspects. Each model is based on a state-of-the-art pre-trained language model that allows us to assess contextualized word representations in the table creation task. The obtained results showed that such representations play a vital role in classifying column data types and constraints correctly. One of the downsides of pre-trained models is the training time and the model size. Our experiments revealed that a multi-task BERT model achieving 75% and 96% accuracy for the data type and constraint prediction tasks, respectively, effectively addresses both problems.

Neural POS tagging of shahmukhi by using contextualized word representations

Part of Speech (POS) tagging has a preliminary role in building natural language processing applications. This paper presents the development and evaluation of the first POS tagged corpus along with a Bi-directional long-short memory (BiLSTM) network based POS tagger for Shahmukhi (Western Punjabi) at this scale. A balanced corpus of 0.13 million words has been annotated which contains text from 14 different text domains. A Shahmukhi POS tagset has been devised by studying the applicability of the CLE Urdu POS tagset and tagging guidelines have also been designed for annotation. A multi-step corpus evaluation process has been employed for tagged corpus including grammar-based and n-gram based consistency evaluations. The average inter-annotator agreement for all domains is 95.35% along with an average Kappa coefficient of 0.94. The performance of the BiLSTM POS tagger has been compared with the well-known language independent TreeTagger and the Stanford POS tagger. The accuracy of the tagger has been further improved by employing transfer learning by training context-free (Word2Vec) and contextualized (ELMo) word representations on a corpus of 14.9 Shahmukhi words which has been collected from World Wide Web. The tagger performed with an f-score of 96.11 and the accuracy of 96.12%. For a morphologically-rich and low-resourced language, these POS tagging results are quite promising.

RadioBERT: A deep learning-based system for medical report generation from chest X-ray images using contextual embeddings

Background:Increasing number of chest X-ray (CXR) examinations in radiodiagnosis departments burdens radiologists’ and makes the timely generation of accurate radiological reports highly challenging. An automatic radiological report generation (ARRG) system is envisaged to generate radiographic reports with minimal human intervention, ease radiologists’ burden, and smoothen the clinical workflow. The success of an ARRG system depends on two critical factors: i) quality of the features extracted by the ARRG system from the CXR images, and ii) quality of the linguistic expression generated by the ARRG system describing the normalities and abnormalities as indicated by the extracted features. Most of the existing ARRG systems miserably fail due to the latter factor and do not generate clinically acceptable reports because they ignore the contextual importance of the medical terms. Objectives:The advent of contextual word embeddings, like ELMo and BERT, has revolutionized several natural language processing (NLP) tasks. A contextual embedding represents a word based on its context. The main objective of this work is to develop an ARRG system that uses contextual word embeddings to generate clinically accurate reports from CXR images. Methods:We present an end-to-end deep neural network that uses contextual word representations for generating clinically useful radiological reports from CXR images. The proposed network, termed as RadioBERT, uses DistilBERT for contextual word representation and leverages transfer learning. Additionally, due to the importance of abnormal observations over the normal ones, the network reorders the generated sentences by applying sentiment analysis to keep abnormal descriptions on the top of the generated report. Results:The empirical study consisting of several experiments performed on the OpenI dataset indicates that CNN+Hierarchical LSTM with DistilBERT embedding improves the benchmark performance. We have been able to achieve the following performance scores: BLEU-1 = 0.772, BLEU-2 = 0.770, BLEU-3 = 0.768, BLEU-4 = 0.767, CIDEr = 0.5563, and ROUGE = 0.897. Conclusion:The proposed method improves the state-of-the-art performance scores by a substantial margin. It is concluded that the use of word embeddings generated by DistilBERT enhances the performance of hierarchical LSTM for producing clinical reports by significant margin.

Context-Aware Sentiment Analysis using Tweet Expansion Method

The large source of information space produced by the plethora of social media platforms in general and microblogging in particular has spawned a slew of new applications and prompted the rise and expansion of sentiment analysis research. We propose a sentiment analysis technique that identifies the main parts to describe tweet intent and also enriches them with relevant words, phrases, or even inferred variables. We followed a state-of-the-art hybrid deep learning model to combine Convolutional Neural Network (CNN) and the Long Short-Term Memory network (LSTM) to classify tweet data based on their polarity. To preserve the latent relationships between tweet terms and their expanded representation, sentence encoding and contextualized word embeddings are utilized. To investigate the performance of tweet embeddings on the sentiment analysis task, we tested several context-free models (Word2Vec, Sentence2Vec, Glove, and FastText), a dynamic embedding model (BERT), deep contextualized word representations (ELMo), and an entity-based model (Wikipedia). The proposed method and results prove that text enrichment improves the accuracy of sentiment polarity classification with a notable percentage.

RastrOS Project: Natural Language Processing contributions to the development of an eye-tracking corpus with predictability norms for Brazilian Portuguese.

This article presents RastrOS, a new eye-tracking corpus of eye movement data from university students during silent reading of paragraphs of texts in Brazilian Portuguese (BP). The article shows the potential of the corpus for natural language processing (NLP) using it to evaluate the sentence complexity prediction task in BP and it also focuses on the description of NLP resources and methods developed to create the corpus. Specifically, we present: (i) the method used to select the corpus paragraphs from large corpora, using linguistic metrics and clustering algorithms; (ii) the platform for collecting the Cloze test, which is also responsible for creating the project datasets, and (iii) the hybrid semantic similarity method, based on word embedding models and contextualised word representations, used to generate semantic predictability norms. RastrOS can be downloaded from the open science framework repository with the computational infrastructure mentioned above. Datasets with predictability norms of 393 participants and eye-tracking data of 37 participants are available in the OSF repository for this work (https://osf.io/9jxg3/).

Sequential verb metaphor detection with linguistic theories

Metaphor plays an indispensable role in human life. Although sequence tagging models took advantage of linguistic theories of metaphor identification, the usage of metaphor in common words is not considered, when choosing the literal meaning of the target verbs. We present a novel approach to express the literal meaning subtly, combining the common usage and the inherent visualizability properties of words, termed GloVe embedding and visual embedding. Meanwhile, we import position information of the target verbs to gain the contextual meaning more accurately. Both two DNN models use these embeddings as inputs in this paper, which are inspired by two human metaphor identification procedures augmented with contextualized word representations (ELMo embedding). By testing on two public datasets, the results show improvement over previous state-of-the-art approaches. In addition, we also verify the universality of the approach by testing the examples that the target words were adjectives, adverbs, and nouns, and the results show the approach is applicable to the above three parts of speech.

Spam detection on social networks using deep contextualized word representation

Spam detection on social networks using deep contextualized word representation

Exploring deep learning methods for recognizing rare diseases and their clinical manifestations from texts

Background and objectiveAlthough rare diseases are characterized by low prevalence, approximately 400 million people are affected by a rare disease. The early and accurate diagnosis of these conditions is a major challenge for general practitioners, who do not have enough knowledge to identify them. In addition to this, rare diseases usually show a wide variety of manifestations, which might make the diagnosis even more difficult. A delayed diagnosis can negatively affect the patient’s life. Therefore, there is an urgent need to increase the scientific and medical knowledge about rare diseases. Natural Language Processing (NLP) and Deep Learning can help to extract relevant information about rare diseases to facilitate their diagnosis and treatments.MethodsThe paper explores several deep learning techniques such as Bidirectional Long Short Term Memory (BiLSTM) networks or deep contextualized word representations based on Bidirectional Encoder Representations from Transformers (BERT) to recognize rare diseases and their clinical manifestations (signs and symptoms).ResultsBioBERT, a domain-specific language representation based on BERT and trained on biomedical corpora, obtains the best results with an F1 of 85.2% for rare diseases. Since many signs are usually described by complex noun phrases that involve the use of use of overlapped, nested and discontinuous entities, the model provides lower results with an F1 of 57.2%.ConclusionsWhile our results are promising, there is still much room for improvement, especially with respect to the identification of clinical manifestations (signs and symptoms).

Interpretable Privacy Preservation of Text Representations Using Vector Steganography

Contextual word representations generated by language models learn spurious associations present in the training corpora. Adversaries can exploit these associations to reverse-engineer the private attributes of entities mentioned in the training corpora. These findings have led to efforts towards minimizing the privacy risks of language models. However, existing approaches lack interpretability, compromise on data utility and fail to provide privacy guarantees. Thus, the goal of my doctoral research is to develop interpretable approaches towards privacy preservation of text representations that maximize data utility retention and guarantee privacy. To this end, I aim to study and develop methods to incorporate steganographic modifications within the vector geometry to obfuscate underlying spurious associations and retain the distributional semantic properties learnt during training.

Embedding arabic questions by feature-level fusion of word representations for questions classification: It is worth doing?

Questions classification remains one of the most critical phases of Question Answering Systems. It aims to reduce the answers search space by assigning a predefined class label to each question. Recently, contextualized word representations methods based on deep learning approach have achieved state of the art performance in various fields of Natural Language Processing. However, few works have applied these representations to classify Arabic questions. In this research, we propose an Arabic question classification method based on a sentence transformers-based representation. Besides, we investigate the fusion of various word representations including Bidirectional Encoder Representations from Transformers (BERT), Embeddings from Language Models (ELMo), and word embeddings enriched by subwords information (W2V). Our contribution is threefold. First, our method handles out of vocabulary words. Second, we apply the BERT representation to extract the most valuable features from words and then construct better question representation. Third, we study the impact of fusing various word embeddings on Arabic question classification. To evaluate the proposed models, we perform stratified 5-folds cross-validation on a dataset containing 3173 questions labeled with Arabic and Li & Roth taxonomies. The experimental results show that all our models surpassed previous works related to Arabic question classification task. In the Arabic taxonomy case, we scored the maximum accuracy of 94.20% with our AraBERT-based model. As for Li & Roth taxonomy, the model based on the concatenation of AraBERT and W2V scored the highest overall accuracy of 93.51%.

Detecting Alzheimer's Disease Using Natural Language Processing of Referential Communication Task Transcripts.

People with Alzheimer's disease (AD) often demonstrate difficulties in discourse production. Referential communication tasks (RCTs) are used to examine a speaker's capability to select and verbally code the characteristics of an object in interactive conversation. In this study, we used contextualized word representations from Natural language processing (NLP) to evaluate how well RCTs are able to distinguish between people with AD and cognitively healthy older adults. We adapted machine learning techniques to analyze manually transcribed speech transcripts in an RCT from 28 older adults, including 12 with AD and 16 cognitively healthy older adults. Two approaches were applied to classify these speech transcript samples: 1) using clinically relevant linguistic features, 2) using machine learned representations derived by a state-of-art pretrained NLP transfer learning model, Bidirectional Encoder Representation from Transformer (BERT) based classification model. The results demonstrated the superior performance of AD detection using a designed transfer learning NLP algorithm. Moreover, the analysis showed that transcripts of a single image yielded high accuracies in AD detection. The results indicated that RCT may be useful as a diagnostic tool for AD, and that the task can be simplified to a subset of images without significant sacrifice to diagnostic accuracy, which can make RCT an easier and more practical tool for AD diagnosis. The results also demonstrate the potential of RCT as a tool to better understand cognitive deficits from the perspective of discourse production in people with AD.