Contextualized Word Embeddings Research Articles

Fine-grained hate speech detection in Arabic using transformer-based models

With the proliferation of social media platforms, characterized by features such as anonymity, user-friendly access, and the facilitation of online community building and discourse, the matter of detecting and monitoring hate speech has emerged as an increasingly formidable challenge for society, individuals, and researchers. Despite the crucial importance of hate speech detection task, the majority of work in this field has been conducted in English, with insufficient focus on other languages, particularly Arabic. Furthermore, most existing studies on Arabic hate speech detection have addressed this task as a binary classification problem, which is unreliable. Therefore, the aim of this study is to provide an enhanced model for detecting fine-grained hate speech in Arabic. To this end, three transformer-based models were evaluated to generate contextualized word embeddings from input sequence. Additionally, these models were combined with a bidirectional gated recurrent unit (BiGRU) layer to further improve the extracted semantic and context features. The experiments were conducted on an Arabic reference dataset provided by the open-source Arabic corpora and processing tools (OSACT-5) shared task. A comparative analysis indicates the efficiency of the proposed model over the baseline and related work models by achieving a macro F1-score of 61.68%.

Named entity recognition in the perovskite field based on convolutional neural networks and MatBERT

Due to the significant increase in publications in the field of materials science, there has been a bottleneck in organizing material science knowledge and discovering new materials. The number of literature in the emerging field of perovskite materials has grown to a massive scale. It is necessary to compile information on the structure, properties, synthesis methods, characterization techniques, and applications of perovskite materials. To address this issue, we employed named entity recognition, a natural language processing technique, to extract important entities from perovskite material texts. In this paper, we propose a method based on convolutional neural networks (CNN) and MatBERT. Firstly, we utilized MatBERT, which has been pre-trained on a large amount of material science text, to generate contextualized word embeddings. Next, we extracted feature information using a CNN model. Finally, a conditional random field (CRF) layer was used for decoding sequences in addition to calculating the training and validation loss. Experimental results demonstrated that the performance of our model on perovskite material dataset was improved by 1 %∼6% compared with BERT, SciBERT and MatBERT models. Through this model, we extracted the entities of 2389 abstracts to obtain knowledge of perovskite materials.

Fake news detection in Dravidian languages using multiscale residual CNN_BiLSTM hybrid model

Fake news detection is the process of identifying news that contain purposeful misinformation disseminated through traditional news sources or social media platforms. Detecting fake news in Dravidian languages poses unique challenges due to the linguistic diversity and limited resources available for these languages. In this study, we propose a novel approach for fake news detection in Dravidian languages, leveraging contextualized word embeddings from MuRIL. MuRIL provides contextual embeddings that capture nuanced linguistic nuances, making it well-suited for the complexities of Dravidian languages. The proposed hybrid model combines multiscale residual CNN and BiLSTM layers to capture local and global dependencies in textual data effectively. The multiscale architecture allows the model to extract features at various levels of granularity while the BiLSTM layer captures long-range dependencies and contextual information. We performed extensive experiments on diverse datasets in Dravidian languages to evaluate the proposed approach. The results demonstrate the effectiveness of our hybrid model in discerning between genuine and fake news articles.

Beyond Word-Based Model Embeddings: Contextualized Representations for Enhanced Social Media Spam Detection

As social media platforms continue their exponential growth, so do the threats targeting their security. Detecting disguised spam messages poses an immense challenge owing to the constant evolution of tactics. This research investigates advanced artificial intelligence techniques to significantly enhance multiplatform spam classification on Twitter and YouTube. The deep neural networks we use are state-of-the-art. They are recurrent neural network architectures with long- and short-term memory cells that are powered by both static and contextualized word embeddings. Extensive comparative experiments precede rigorous hyperparameter tuning on the datasets. Results reveal a profound impact of tailored, platform-specific AI techniques in combating sophisticated and perpetually evolving threats. The key innovation lies in tailoring deep learning (DL) architectures to leverage both intrinsic platform contexts and extrinsic contextual embeddings for strengthened generalization. The results include consistent accuracy improvements of more than 10–15% in multisource datasets, unlocking actionable guidelines on optimal components of neural models, and embedding strategies for cross-platform defense systems. Contextualized embeddings like BERT and ELMo consistently outperform their noncontextualized counterparts. The standalone ELMo model with logistic regression emerges as the top performer, attaining exceptional accuracy scores of 90% on Twitter and 94% on YouTube data. This signifies the immense potential of contextualized language representations in capturing subtle semantic signals vital for identifying disguised spam. As emerging adversarial attacks exploit human vulnerabilities, advancing defense strategies through enhanced neural language understanding is imperative. We recommend that social media companies and academic researchers build on contextualized language models to strengthen social media security. This research approach demonstrates the immense potential of personalized, platform-specific DL techniques to combat the continuously evolving threats that threaten social media security.

From unstructured accident reports to a hybrid decision support system for occupational risk management: The consensus converging approach

Introduction: Workplace accidents in the petroleum industry can cause catastrophic damage to people, property, and the environment. Earlier studies in this domain indicate that the majority of the accident report information is available in unstructured text format. Conventional techniques for the analysis of accident data are time-consuming and heavily dependent on experts’ subject knowledge, experience, and judgment. There is a need to develop a machine learning-based decision support system to analyze the vast amounts of unstructured text data that are frequently overlooked due to a lack of appropriate methodology. Method: To address this gap in the literature, we propose a hybrid methodology that uses improved text-mining techniques combined with an un-bias group decision-making framework to combine the output of objective weights (based on text mining) and subjective weights (based on expert opinion) of risk factors to prioritize them. Based on the contextual word embedding models and term frequencies, we extracted five important clusters of risk factors comprising more than 32 risk sub-factors. A heterogeneous group of experts and employees in the petroleum industry were contacted to obtain their opinions on the extracted risk factors, and the best-worst method was used to convert their opinions to weights. Conclusions and Practical Applications: The applicability of our proposed framework was tested on the data compiled from the accident data released by the petroleum industries in India. Our framework can be extended to accident data from any industry, to reduce analysis time and improve the accuracy in classifying and prioritizing risk factors.

Assessing the Impact of Static, Contextual and Character Embeddings for Arabic Machine Translation

Word embeddings/representations are an important component of Natural Language Processing (NLP) tasks. Most Neural Machine Translation (NMT) systems that use such representations disregard word morphology by assigning a unique vector to each unique word in the used vocabulary and thus cannot handle Out-Of-Vocabulary (OOV) words. In some languages, such as Arabic, the meaning of words is associated with the meaning of the individual characters that constitute them, as these characters embody internal information. In this study, a combination of character- and word- level models is used to determine the most effective approaches to semantically and morphologically representing affective Arabic words. Furthermore, this work examines the strategy of combining static, character and contextual word embeddings to obtain richer representations for the Arabic Machine Translation (MT) task. To the best of our knowledge, we are the first to investigate the combination of static word embeddings, contextual ones and character-level representation in Arabic MT. Furthermore, a Deep Learning (DL) architecture is employed on data preprocessed by various prominent preprocessing techniques. Various experiments were conducted and the findings indicate that the integration of various models for word embedding and character-level representation is feasible and more effective than the state-of-the-art Arabic MT systems.

Contextual Word Embedding for Biomedical Knowledge Extraction: a Rapid Review and Case Study.

The online version contains supplementary material available at 10.1007/s41666-023-00157-y.

Exploring the roles of AI-Assisted ChatGPT in the field of data science

In this study, we explore the roles of AI-assisted ChatGPT (Generative Pre-trained Transformer) in the field of data science. AI-assisted ChatGPT, a powerful language model, is fine-tuned using domain-specific data for specialised data science tasks, such as sentiment analysis and named entity recognition (NER). The results reveal significant reductions in model size and memory usage with minor trade-offs in inference time, providing valuable resource-efficient deployment. Various data augmentation methods, including back-translation, synonym replacement, and contextual word embeddings, are employed to augment the training dataset. The study's results are subjected to rigorous statistical analysis, including paired t-tests and ANOVA tests, to determine the significance of the findings. The research concludes with insightful suggestions and future scope, including advanced fine-tuning strategies, model optimization techniques, and ethical considerations.

Lost in Context? On the Sense-Wise Variance of Contextualized Word Embeddings

Lost in Context? On the Sense-Wise Variance of Contextualized Word Embeddings

A Comparative Study of Different Dimensionality Reduction Techniques for Arabic Machine Translation

Word embeddings are widely deployed in a tremendous range of fundamental natural language processing applications and are also useful for generating representations of paragraphs, sentences, and documents. In some contexts involving constrained memory, it may be beneficial to reduce the size of word embeddings since they represent a core component of several natural language processing tasks. By reducing the dimensionality of word embeddings, their usefulness in memory-limited devices can be significantly improved, yielding gains in many real-world applications. This article aims to provide a comparative study of different dimensionality reduction techniques to generate efficient lower-dimensional word vectors. Based on empirical experiments carried out on the Arabic machine translation task, we found that the post-processing algorithm combined with independent component analysis provides optimal performance over the considered dimensionality reduction techniques. Therefore, we arrive at a new combination of the post-processing algorithm and dimensionality reduction (independent component analysis) techniques, which has not been investigated before. The latter was applied to both contextual and non-contextual word embeddings to reduce the size of the vectors while achieving a better translation quality than the original ones.

A Novel Knowledge-augmented Model Customization Approach for Arabic Offensive Language Detection

Multiple attempts to develop systems for detecting online Arabic offensive language have been explored in previous studies. However, most of these attempts do not consider the variation of Arabic dialects, cultures, and offensive phrases. In contrast, this study aims to extract knowledge from multiple offensive language datasets to build a cross-dialect and culture knowledge-based repository. This knowledge-based repository is utilized to develop novel system architecture based on customizing the AraBERT model in a unique method to preserve dialectal knowledge and offensive cultural knowledge within the contextual word embedding of BERT architecture. Performance evaluation procedures consist of statistical evaluation metrics and a behavioral checklist. Results report more effective predictions by the customized model than the uncustomized one, particularly for offensive text. The customization process allows the model to gain more knowledge of informal text in general, and a better understanding of dialectal Arabic.

Personalized Query Expansion with Contextual Word Embeddings

Personalized Query Expansion, the task of expanding queries with additional terms extracted from the user-related vocabulary, is a well-known solution to improve the retrieval performance of a system w.r.t. short queries. Recent approaches rely on word embeddings to select expansion terms from user-related texts. Although promising results have been delivered with former word embedding techniques, we argue that these methods are not suited for contextual word embeddings, which produce a unique vector representation for each term occurrence. In this article, we propose a Personalized Query Expansion method designed to solve the issues arising from the use of contextual word embeddings with the current Personalized Query Expansion approaches based on word embeddings. Specifically, we employ a clustering-based procedure to identify the terms that better represent the user interests and to improve the diversity of those selected for expansion, achieving improvements of up to 4% w.r.t. the best-performing baseline in terms of MAP@100. Moreover, our approach outperforms previous ones in terms of efficiency, allowing us to achieve sub-millisecond expansion times even in data-rich scenarios. Finally, we introduce a novel metric to evaluate the expansion terms’ diversity and empirically show the unsuitability of previous approaches based on word embeddings when employed along with contextual word embeddings, which cause the selection of semantically overlapping expansion terms.

Effectiveness of ELMo embeddings, and semantic models in predicting review helpfulness

Online product reviews (OPR) are a commonly used medium for consumers to communicate their experiences with products during online shopping. Previous studies have investigated the helpfulness of OPRs using frequency-based, linguistic, meta-data, readability, and reviewer attributes. In this study, we explored the impact of robust contextual word embeddings, topic, and language models in predicting the helpfulness of OPRs. In addition, the wrapper-based feature selection technique is employed to select effective subsets from each type of features. Five feature generation techniques including word2vec, FastText, Global Vectors for Word Representation (GloVe), Latent Dirichlet Allocation (LDA), and Embeddings from Language Models (ELMo), were employed. The proposed framework is evaluated on two Amazon datasets (Video games and Health & personal care). The results showed that the ELMo model outperformed the six standard baselines, including the fine-tuned Bidirectional Encoder Representations from Transformers (BERT) model. In addition, ELMo achieved Mean Square Error (MSE) of 0.0887 and 0.0786 respectively on two datasets and MSE of 0.0791 and 0.0708 with the wrapper method. This results in the reduction of 1.43% and 1.63% in MSE as compared to the fine-tuned BERT model on respective datasets. However, the LDA model has a comparable performance with the fine-tuned BERT model but outperforms the other five baselines. The proposed framework demonstrated good generalization abilities by uncovering important factors of product reviews and can be evaluated on other voting platforms.

Shahmukhi named entity recognition by using contextualized word embeddings

Named Entity Recognition (NER) is an imperative Natural Language Processing (NLP) task which intents to identify and classify predefined named entities in a given span of text. For many Western and Asian languages, NER is a systematically premeditated and established task, however, a little work has been done for Shahmukhi. This paper presents Shahmukhi NER with four key contributions. First, a Bi-directional Long-Short Term Memory (BiLSTM) network based NER model has been developed by incorporating various features including character and word embeddings and Part of Speech (POS) tagging. Second, transfer learning has been employed by training context-free Word2Vec and contextualized Embeddings from Language Models (ELMo) word representations. The word representations have been trained using a Shahmukhi corpus of 14.9 million words. Third, we prepared a cleaner version of an existing Shahmukhi NER corpus by performing Unicode normalization and tokenization errands. The corpus has been deduplicated and results are reported on an unseen evaluation set which produced valid results. Fourth, we have studied the impact of two annotation schemes; Inside-Outside (IO) and Inside-Outside-Beginning (IOB) for Shahmukhi. Transfer learning was quite helpful to enhance the performance of NER models especially ELMo embeddings significantly improved the results by prompting contextualized embedding vectors. This is the first study to use character embeddings, POS tagging and transfer learning for Shahmukhi named entity recognition. The IO scheme based model achieved an accuracy of 98.60% with an f-score of 83.75. The IOB scheme based model performed with an accuracy of 98.43% and an f-score of 75.55. These scores are quite promising for an under-resourced morphologically-rich language.

Natural language processing (NLP) aided qualitative method in health research

Qualitative data analysis is produced frequently in healthcare settings, which is a time-consuming and skilled analytic task. The use of qualitative research findings in clinical settings takes years, which is sometimes obsolete knowledge as the health context is dynamic. Artificial Intelligence (AI)-based qualitative data analysis might present with rapid analysis of text-based data in real-time, thereby empowering qualitative researchers to expedite their analysis and facilitate timely use of the research findings. We tested an AI-based method to complement the manual analysis of text-based data from the verbatim transcripts of seven mall managers’ interviews. First, we prepared text data into a machine-calculable format and employed BERT model to extract sentence-level features in our case. Second, we implement TF-IDF-based keywords mining techniques to extract the main candidate themes from the interview transcripts to support text-based analysis, including: 1) primary cluster detection algorithm, and 2) keyword extraction algorithm. The extracted core themes provide qualitative researchers with a more comprehensive overview of the qualitative data. Most of the sentences clustered in meaningful short topics or sentences carrying independent and clear information. The extracted topics and clustered sentences reduced qualitative researchers’ workload by condensing and identifying meaningful concepts and naming them. This method combining contextualized word embeddings, unsupervised clustering, and keyword extraction techniques can significantly reduce the overall workload and time consumed in qualitative research using conventional methods.

Multilabel Classification for Keyword Determination of Scientific Articles

In writing scientific articles, there are provisions regarding the structure or parts of writing that must be fulfilled. One part of the scientific article that must be included is keywords. The process of determining keywords manually can cause discrepancies with the specific themes discussed in the article. Thus, causing readers to be unable to reach the scientific article. The process of determining the keywords of scientific articles is determined automatically by the classification method. The classification process is carried out by determining the set of keywords owned by each scientific article data based on the abstract and title. Therefore, the classification process applied is multi-feature and multi-label. Classification is done by applying the Contextualized Word Embedding Method. The implementation of Contextualized Word Embedding Method is done by applying BERT Model. By applying the BERT Model, it is expected to provide good performance in determining the keywords of scientific articles. The evaluation results by applying the BERT Model to the case of multi-label classification on abstract data for keyword determination resulted in a loss value of Training Data is 0.514, loss value of Validation Data is 0.511, and an accuracy value of 0.71, a precision value of 0.71, a recall value of 0.71, an error value of 0.29 and f-1 score of 0.83. Based on the results of the evaluation, it shows that the BERT Classification Model can carry out a classification process to determine a set of keywords from each abstract data in scientific articles.

The Contribution of Selected Linguistic Markers for Unsupervised Arabic Verb Sense Disambiguation

Word sense disambiguation (WSD) is the task of automatically determining the meaning of a polysemous word in a specific context. Word sense induction is the unsupervised clustering of word usages in a different context to distinguish senses and perform unsupervised WSD. Most studies consider function words as stop words and delete them in the pre-processing step. However, function words can encode meaningful information that can help to improve the performance of WSD approaches. We propose in this work a novel approach to solve Arabic verb sense disambiguation that is based on a preposition-based classification that is used in an automatic word sense induction step to build sense inventories to disambiguate Arabic verbs. However, in the wake of the success of neural language models, recent works obtained encouraging results using BERT pre-trained models for English-language WSD approaches. Hence, we use contextualized word embeddings for an unsupervised Arabic WSD that is based on linguistic markers and uses sentence-BERT Transformer pre-trained models, which yields encouraging results that outperform other existing unsupervised neural AWSD approaches.

ADscreen: A speech processing-based screening system for automatic identification of patients with Alzheimer's disease and related dementia

Alzheimer's disease and related dementias (ADRD) present a looming public health crisis, affecting roughly 5 million people and 11 % of older adults in the United States. Despite nationwide efforts for timely diagnosis of patients with ADRD, >50 % of them are not diagnosed and unaware of their disease. To address this challenge, we developed ADscreen, an innovative speech-processing based ADRD screening algorithm for the protective identification of patients with ADRD. ADscreen consists of five major components: (i) noise reduction for reducing background noises from the audio-recorded patient speech, (ii) modeling the patient's ability in phonetic motor planning using acoustic parameters of the patient's voice, (iii) modeling the patient's ability in semantic and syntactic levels of language organization using linguistic parameters of the patient speech, (iv) extracting vocal and semantic psycholinguistic cues from the patient speech, and (v) building and evaluating the screening algorithm. To identify important speech parameters (features) associated with ADRD, we used the Joint Mutual Information Maximization (JMIM), an effective feature selection method for high dimensional, small sample size datasets. Modeling the relationship between speech parameters and the outcome variable (presence/absence of ADRD) was conducted using three different machine learning (ML) architectures with the capability of joining informative acoustic and linguistic with contextual word embedding vectors obtained from the DistilBERT (Bidirectional Encoder Representations from Transformers). We evaluated the performance of the ADscreen on an audio-recorded patients' speech (verbal description) for the Cookie-Theft picture description task, which is publicly available in the dementia databank. The joint fusion of acoustic and linguistic parameters with contextual word embedding vectors of DistilBERT achieved F1-score = 84.64 (standard deviation [std] = ±3.58) and AUC-ROC = 92.53 (std = ±3.34) for training dataset, and F1-score = 89.55 and AUC-ROC = 93.89 for the test dataset. In summary, ADscreen has a strong potential to be integrated with clinical workflow to address the need for an ADRD screening tool so that patients with cognitive impairment can receive appropriate and timely care.

A benchmark for evaluating Arabic contextualized word embedding models

Word embeddings, which represent words as numerical vectors in a high-dimensional space, are contextualized by generating a unique vector representation for each sense of a word based on the surrounding words and sentence structure. They are typically generated using such deep learning models as BERT and trained on large amounts of text data and using self-supervised learning techniques. Resulting embeddings are highly effective at capturing the nuances of language, and have been shown to significantly improve the performance of numerous NLP tasks. Word embeddings represent textual records of human thinking, with all the mental relations that we utilize to produce the succession of sentences that make up texts and discourses. Consequently, the distributed representation of words within embeddings ought to capture the reasoning relations that hold texts together. This paper makes its contribution to the field by proposing a benchmark for the assessment of contextualized word embeddings that probes into their capability for true contextualization by inspecting how well they capture resemblance, contrariety, comparability, identity, relations in time and space, causation, analogy, and sense disambiguation. The proposed metrics adopt a triangulation approach, so they use (1) Hume’s reasoning relations, (2) standard analogy, and (3) sense disambiguation. The benchmark has been evaluated against 22 Arabic contextualized embeddings and has proven to be capable of quantifying their differential performance in terms of these reasoning relations. Results of evaluation of the target embeddings revealed that they do take context into account and that they do reasonably well in sense disambiguation but have weakness in their identification of converseness, synonymy, complementarity, and analogy. Results also show that size of an embedding has diminishing returns because the highly frequent language patterns swamp low frequency patterns. Furthermore, the suggest that future research endeavors should not be concerned with the quantity of data as much as its quality, and that it should focus more on the representativeness of data, and on model architecture, design, and training.

ADEPT: A DEbiasing PrompT Framework

Several works have proven that finetuning is an applicable approach for debiasing contextualized word embeddings. Similarly, discrete prompts with semantic meanings have shown to be effective in debiasing tasks. With unfixed mathematical representation at the token level, continuous prompts usually surpass discrete ones at providing a pre-trained language model (PLM) with additional task-specific information. Despite this, relatively few efforts have been made to debias PLMs by prompt tuning with continuous prompts compared to its discrete counterpart. Furthermore, for most debiasing methods that alter a PLM's original parameters, a major problem is the need to not only decrease the bias in the PLM but also to ensure that the PLM does not lose its representation ability. Finetuning methods typically have a hard time maintaining this balance, as they tend to violently remove meanings of attribute words (like the words developing our concepts of "male" and "female" for gender), which also leads to an unstable and unpredictable training process. In this paper, we propose ADEPT, a method to debias PLMs using prompt tuning while maintaining the delicate balance between removing biases and ensuring representation ability. To achieve this, we propose a new training criterion inspired by manifold learning and equip it with an explicit debiasing term to optimize prompt tuning. In addition, we conduct several experiments with regard to the reliability, quality, and quantity of a previously proposed attribute training corpus in order to obtain a clearer prototype of a certain attribute, which indicates the attribute's position and relative distances to other words on the manifold. We evaluate ADEPT on several widely acknowledged debiasing benchmarks and downstream tasks, and find that it achieves competitive results while maintaining (and in some cases even improving) the PLM's representation ability. We further visualize words' correlation before and after debiasing a PLM, and give some possible explanations for the visible effects.