Pre-trained Word Embeddings Research Articles

Application of BiLSTM-CRF model with different embeddings for product name extraction in unstructured Turkish text

Named entity recognition (NER) plays a pivotal role in Natural Language Processing by identifying and classifying entities within textual data. While NER methodologies have seen significant advancements, driven by pretrained word embeddings and deep neural networks, the majority of these studies have focused on text with well-defined grammar and structure. A significant research gap exists concerning NER in informal or unstructured text, where traditional grammar rules and sentence structure are absent. This research addresses this crucial gap by focusing on the detection of product names within unstructured Turkish text. To accomplish this, we propose a deep learning-based NER model which combines a Bidirectional Long Short-Term Memory (BiLSTM) architecture with a Conditional Random Field (CRF) layer, further enhanced by FastText embeddings. To comprehensively evaluate and compare our model’s performance, we explore different embedding approaches, including Word2Vec and Glove, in conjunction with the Bidirectional Long Short-Term Memory and Conditional Random Field (BiLSTM-CRF) model. Furthermore, we conduct comparisons against BERT to assess the efficacy of our approach. Our experimentation utilizes a Turkish e-commerce dataset gathered from the internet, where traditional grammatical and structural rules may not apply. The BiLSTM-CRF model with FastText embeddings achieved an F1 score value of 57.40%, a precision value of 55.78%, and a recall value of 59.12%. These results indicate promising performance in outperforming other baseline techniques. This research contributes to the field of NER by addressing the unique challenges posed by unstructured Turkish text and opens avenues for improved entity recognition in informal language settings, with potential applications across various domains.

Thematic classification of ancient Chinese poetry using TwinEmbedAttentionNet

Ancient Chinese poetry, a reflection of Chinas rich cultural and philosophical fabric, encapsulates the evolving socio-cultural nuances of its historical epochs. Despite its cultural significance, there remains an evident lacuna in comprehensively classifying its recurring themes, due in part to the conciseness and polysemy intrinsic to the language and the essentiality of embedded cultural and historical contexts. Addressing this challenge, this study introduces TwinEmbedAttentionNet, a pioneering method tailored for the thematic classification of ancient Chinese poetry. This approach synergistically integrates pretrained word and sentence embeddings with an attention mechanism, ensuring the nuanced representation of the poetrys intricate details. Our results showcase its superior performance over existing models. Furthermore, an in-depth examination of model components offers insights into their respective thematic categorization efficacies. This research not only advances the academic understanding of ancient Chinese poetry but also underscores the potential of innovative neural networks in processing historically rich textual data.

Comparative Analysis of Word Embeddings for Multiclass Cyberbullying Detection

Cyberbullying has emerged as a pervasive concern in modern society, particularly within social media platforms. This phenomenon encompasses employing digital communication to instill fear, threaten, harass, or harm individuals. Given the prevalence of social media in our lives, there is an escalating need for effective methods to detect and combat cyberbullying. This paper aims to explore the utilization of word embeddings and to discern the comparative effectiveness of trainable word embeddings, pre-trained word embeddings, and fine-tuned language models in multiclass cyberbullying detection. Distinguishing from previous binary classification methods, our research delves into nuanced multiclass detection. The exploration of word embeddings holds significant promise due to its ability to transform words into dense numerical vectors within a high-dimensional space. This transformation captures intricate semantic and syntactic relationships inherent in language, enabling machine learning (ML) algorithms to discern patterns that might signify cyberbullying. In contrast to previous research, this work delves beyond primary binary classification and centers on the nuanced realm of multiclass cyberbullying detection. The research employs diverse techniques, including convolutional neural networks and bidirectional long short-term memory, alongside well-known pre-trained models such as word2vec and bidirectional encoder representations from transformers (BERT). Moreover, traditional ML algorithms such as K-nearest neighbors, Random Forest, and Naïve Bayes are integrated to evaluate their performance vis-à-vis deep learning models. The findings underscore the promise of a fine-tuned BERT model on our dataset, yielding the most promising results in multiclass cyberbullying detection, and achieving the best-recorded accuracy of 85% on the dataset.

Comparative Study for Sentiment Analysis of Financial Tweets with Deep Learning Methods

Nowadays, Twitter is one of the most popular social networking services. People post messages called “tweets”, which may contain photos, videos, links and text. With the vast amount of interaction on Twitter, due to its popularity, analyzing Twitter data is of increasing importance. Tweets related to finance can be important indicators for decision makers if analyzed and interpreted in relation to stock market. Financial tweets containing keywords from the BIST100 index were collected and the tweets were tagged as “POSITIVE”, “NEGATIVE” and “NEUTRAL”. Binary and multi-class datasets were created. Word embedding and pre-trained word embedding were used for tweet representation. As classifiers, Neural Network, Convolutional Neural Network (CNN), Long Short-Term Memory (LSTM), Gated Recurrent Units (GRU) and GRU-CNN models were used in this study. The best results for binary and multi-class datasets were observed with pre-trained word embedding with the CNN model (83.02%, 72.73%). When word embedding was employed, the Neural Network model had the best results on the multi-class dataset (63.85%) and GRU-CNN had the best results on the binary dataset (80.56%).

A new word embedding model integrated with medical knowledge for deep learning-based sentiment classification

The development of intelligent systems that use social media data for decision-making processes in numerous domains such as politics, business, marketing, and finance, has been made possible by the popularity of social media platforms. However, the utilization of textual data from social media in the healthcare management industry is still somewhat limited when it is compared to other industries. Investigating how current machine learning and natural language processing technologies can be used in the healthcare industry to gauge public sentiment is an important study. Earlier works on healthcare sentiment analysis have utilized traditional word embedding models trained on the general and medical corpus. However, integration of medical knowledge to pre-trained word embedding models has not been considered yet. Word embedding models trained on the general corpus led to the problem of lacking medical knowledge and the models trained on the small size of the medical corpus have limitations in capturing semantic and syntactic properties. This research proposes a new word embedding model named Word Embedding Integrated with Medical Knowledge Vector (WE-iMKVec). The proposed model integrates sentiment lexicons and medical knowledgebases into the pre-trained word embedding to enrich the properties of word embedding. A new medical-aware sentiment polarity score is proposed for the utilization in learning neural-network sentiment and these vectors incorporate with the original pre-trained word vectors. The resulting vectors are enriched with lexicon vectors and the medical knowledge vectors: Adverse Drug Reaction (ADR) vector and Unified Medical Language System (UMLS) vector are used to build the proposed WE-iMKVec model. WE-iMKVec is validated on the five different social media healthcare review datasets and the empirical results showed its superiority over traditional word embedding models in medical sentiment analysis. The highest improvement can be found in the patients.info medical condition dataset where the proposed model outperforms three conventional word2vec models (Google-News, PubMed-PMC, and Drug Reviews) by 12.7 %, 31.4 %, and 25.4 % respectively in terms of F1 score.

Leveraging Pre-Trained Word Embedding Models for Fake Review Identification

Leveraging Pre-Trained Word Embedding Models for Fake Review Identification

How direct is the link between words and images?

Abstract Günther et al. (2022) investigated the relationship between words and images in which they concluded the possibility of a direct link between words and embodied experience. In their study, participants were presented with a target noun and a pair of images, one chosen by their model and another chosen randomly. Participants were asked to select the image that best matched the target noun. Building upon their work, we addressed the following questions. 1. Apart from utilizing visually embodied simulation, what other strategies subjects might have used? How much does this setup rely on visual information? Can it be solved using textual representations? 2. Do current visually-grounded embeddings explain subjects’ selection behavior better than textual embeddings? 3. Does visual grounding improve the representations of both concrete and abstract words? For this aim, we designed novel experiments based on pre-trained word embeddings. Our experiments reveal that subjects’ selection behavior is explained to a large extend on text-based embeddings and word-based similarities. Visually grounded embeddings offered modest advantages over textual embeddings in certain cases. These findings indicate that the experiment by Günther et al. (2022) may not be well suited for tapping into the perceptual experience of participants, and the extent to which it measures visually grounded knowledge is unclear.

Comparative Analysis of Long Short-Term Memory Architecture for Text Classification

Text classification which is a part of NLP is a grouping of objects in the form of text based on certain characteristics that show similarities between one document and another. One of methods used in text classification is LSTM. The performance of the LSTM method itself is influenced by several things such as datasets, architecture, and tools used to classify text. On this occasion, researchers analyse the effect of the number of layers in the LSTM architecture on the performance generated by the LSTM method. This research uses IMDB movie reviews data with a total of 50,000 data. The data consists of positive, negative data and there is data that does not yet have a label. IMDB Movie Reviews data go through several stages as follows: Data collection, data pre-processing, conversion to numerical format, text embedding using the pre-trained word embedding model: Fastext, train and test classification model using LSTM, finally validate and test the model so that the results are obtained from the stages of this research. The results of this study show that the one-layer LSTM architecture has the best accuracy compared to two-layer and three-layer LSTM with training accuracy and testing accuracy of one-layer LSTM which are 0.856 and 0.867. While the training accuracy and testing accuracy on two-layer LSTM are 0.846 and 0.854, the training accuracy and testing accuracy on three layers are 0.848 and 864.

Language with vision: A study on grounded word and sentence embeddings

Grounding language in vision is an active field of research seeking to construct cognitively plausible word and sentence representations by incorporating perceptual knowledge from vision into text-based representations. Despite many attempts at language grounding, achieving an optimal equilibrium between textual representations of the language and our embodied experiences remains an open field. Some common concerns are the following. Is visual grounding advantageous for abstract words, or is its effectiveness restricted to concrete words? What is the optimal way of bridging the gap between text and vision? To what extent is perceptual knowledge from images advantageous for acquiring high-quality embeddings? Leveraging the current advances in machine learning and natural language processing, the present study addresses these questions by proposing a simple yet very effective computational grounding model for pre-trained word embeddings. Our model effectively balances the interplay between language and vision by aligning textual embeddings with visual information while simultaneously preserving the distributional statistics that characterize word usage in text corpora. By applying a learned alignment, we are able to indirectly ground unseen words including abstract words. A series of evaluations on a range of behavioral datasets shows that visual grounding is beneficial not only for concrete words but also for abstract words, lending support to the indirect theory of abstract concepts. Moreover, our approach offers advantages for contextualized embeddings, such as those generated by BERT (Devlin et al, 2018), but only when trained on corpora of modest, cognitively plausible sizes. Code and grounded embeddings for English are available at (https://github.com/Hazel1994/Visually_Grounded_Word_Embeddings_2).

Stance-Based Fake News Identification on Social Media with Hybrid CNN and RNN-LSTM Models

Today, fake news can be readily generated and disseminated via social media platforms. Misinformation and hoaxes propagated via online social media or traditional news media are commonly referred to as fake news. Stance-based fake news is based on the opinions of an audience rather than providing correct facts. This study presents a hybrid model focusing on the CNN model and RNN-LSTM model to identify fake news. A balanced dataset of 216k news items called ‘SherLock-FakeNewsNet’ is explored throughout the study resulting in the proposed hybrid model. The NLTK toolkit is utilized to perform some noise reduction. After that, the Keras tokenizer and pre-processor are used for the tokenization and pre-processing steps. Next, the text data is fitted and represented using pre-trained GloVe word embeddings. CNN model is applied with Conv1D layers and MaxPooling layers to extract higher-level features of the text. Following this, to detect the longer context of the given text and to capture interdependencies among word sequences LSTM units are employed in the RNN model. Dropout layers are carefully chosen to reduce overfitting in the final hybrid model. The proposed hybrid model achieves the highest accuracy rate of 92% by outperforming most of the conventional models today with an average Precision of 91%, Recall of 91%, and F1-Score of 91% using an Adam optimizer and a Binary Cross-Entropy loss function. Based on the five experiments performed in the study, we have presented the results by comparing our proposed hybrid model with five related datasets.

Unleashing the power of 2D CNN with attention and pre-trained embeddings for enhanced online review analysis

Online reviews play a significant role in the success of a business. Deep learning models have emerged as crucial tools in this domain, with one-dimensional Convolutional Neural Network (1D CNN) being commonly used. However, this paper proposes a novel approach utilizing a Two-Dimensional Convolutional Neural Network (Att + 2D CNN) with attention mechanism, which effectively captures the dimensionality of the input text, resembling a 2D matrix. To further enhance the model’s performance, we employ pretrained word embeddings, specifically GloVe and Word2Vec. We thoroughly analyze the performance of these embeddings in conjunction with deep learning models. Remarkably, our proposed method, leveraging 2D CNN with attention, consistently achieves superior accuracy when compared to other models, specifically on Amazon Cell Phone reviews and Amazon Kindle reviews datasets, for both balanced and unbalanced natures. By employing this novel methodology, we demonstrate the ability to extract valuable insights from online reviews, enabling businesses to make informed decisions.

Evaluating Familiarity Ratings of Domain Concepts with Interpretable Machine Learning: A Comparative Study

Psycholinguistic properties such as concept familiarity and concreteness have been investigated in relation to technological innovations in teaching and learning. Due to ongoing advances in semantic representation and machine learning technologies, the automatic extrapolation of lexical psycholinguistic properties has received increased attention across a number of disciplines in recent years. However, little attention has been paid to the reliable and interpretable assessment of familiarity ratings for domain concepts. To address this gap, we present a regression model grounded in advanced natural language processing and interpretable machine learning techniques that can predict domain concepts’ familiarity ratings based on their lexical features. Each domain concept is represented at both the orthographic–phonological level and semantic level by means of pretrained word embedding models. Then, we compare the performance of six tree-based regression models (adaptive boosting, gradient boosting, extreme gradient boosting, a light gradient boosting machine, categorical boosting, and a random forest) on domain concepts’ familiarity rating prediction. Experimental results show that categorical boosting with the lowest MAPE (0.09) and the highest R2 value (0.02) is best suited to predicting domain concepts’ familiarity. Experimental results also revealed the prospect of integrating tree-based regression models and interpretable machine learning techniques to expand psycholinguistic resources. Specifically, findings showed that the semantic information of raw words and parts of speech in domain concepts are reliable indicators when predicting familiarity ratings. Our study underlines the importance of leveraging domain concepts’ familiarity ratings; future research should aim to improve familiarity extrapolation methods. Scholars should also investigate the correlation between students’ engagement in online discussions and their familiarity with domain concepts.

Comparison of Pre-trained vs Custom-trained Word Embedding Models for Word Sense Disambiguation

The prime objective of word sense disambiguation (WSD) is to develop such machines that can automatically recognize the actual meaning (sense) of ambiguous words in a sentence. WSD can improve various NLP and HCI challenges. Researchers explored a wide variety of methods to resolve this issue of sense ambiguity. However, majorly, their focus was on English and some other well-reputed languages. Urdu with more than 300 million users and a large amount of electronic text available on the web is still unexplored. In recent years, for a variety of Natural Language Processing tasks, word embedding methods have proven extremely successful. This study evaluates, compares, and applies a variety of word embedding approaches to Urdu Word embedding (both Lexical Sample and All-Words), including pre-trained (Word2Vec, Glove, and FastText) as well as custom-trained (Word2Vec, Glove, and FastText trained on the Ur-Mono corpus). Two benchmark corpora are used for the evaluation in this study: (1) the UAW-WSD-18 corpus and (2) the ULS-WSD-18 corpus. For Urdu All-Words WSD tasks, top results have been achieved (Accuracy=60.07 and F1=0.45) using pre-trained FastText. For the Lexical Sample, WSD has been achieved (Accuracy=70.93 and F1=0.60) using custom-trained GloVe word embedding method.

Protein-Protein Interaction Network Extraction Using Text Mining Methods Adds Insight into Autism Spectrum Disorder.

Text mining methods are being developed to assimilate the volume of biomedical textual materials that are continually expanding. Understanding protein-protein interaction (PPI) deficits would assist in explaining the genesis of diseases. In this study, we designed an automated system to extract PPIs from the biomedical literature that uses a deep learning sentence classification model, a pretrained word embedding, and a BiLSTM recurrent neural network with additional layers, a conditional random field (CRF) named entity recognition (NER) model, and shortest-dependency path (SDP) model using the SpaCy library in Python. The automated system ensures that it targets sentences that contain PPIs and not just these proteins mentioned in the framework of disease discovery or other context. Our first model achieved 13% greater precision on the Aimed/BioInfr benchmark corpus than the previous state-of-the-art BiLSTM neural network models. The NER model presented in this study achieved 98% precision on the Aimed/BioInfr corpus over previous models. In order to facilitate the production of an accurate representation of the PPI network, the processes were developed to systematically map the protein interactions in the texts. Overall, evaluating our system through the use of 6027 abstracts pertaining to seven proteins associated with Autism Spectrum Disorder completed the manually curated PPI network for these proteins. When it comes to complicated diseases, these networks would assist in understanding how PPI deficits contribute to disease development while also emphasizing the influence of interactions on protein function and biological processes.

Trajectories through semantic spaces in schizophrenia and the relationship to ripple bursts

Human cognition is underpinned by structured internal representations that encode relationships between entities in the world (cognitive maps). Clinical features of schizophrenia-from thought disorder to delusions-are proposed to reflect disorganization in such conceptual representations. Schizophrenia is also linked to abnormalities in neural processes that support cognitive map representations, including hippocampal replay and high-frequency ripple oscillations. Here, we report a computational assay of semantically guided conceptual sampling and exploit this to test a hypothesis that people with schizophrenia (PScz) exhibit abnormalities in semantically guided cognition that relate to hippocampal replay and ripples. Fifty-two participants [26 PScz (13 unmedicated) and 26 age-, gender-, and intelligence quotient (IQ)-matched nonclinical controls] completed a category- and letter-verbal fluency task, followed by a magnetoencephalography (MEG) scan involving a separate sequence-learning task. We used a pretrained word embedding model of semantic similarity, coupled to a computational model of word selection, to quantify the degree to which each participant's verbal behavior was guided by semantic similarity. Using MEG, we indexed neural replay and ripple power in a post-task rest session. Across all participants, word selection was strongly influenced by semantic similarity. The strength of this influence showed sensitivity to task demands (category > letter fluency) and predicted performance. In line with our hypothesis, the influence of semantic similarity on behavior was reduced in schizophrenia relative to controls, predicted negative psychotic symptoms, and correlated with an MEG signature of hippocampal ripple power (but not replay). The findings bridge a gap between phenomenological and neurocomputational accounts of schizophrenia.

Statement Recognition of Access Control Policies in IoT Networks.

Access Control Policies (ACPs) are essential for ensuring secure and authorized access to resources in IoT networks. Recognizing these policies involves identifying relevant statements within project documents expressed in natural language. While current research focuses on improving recognition accuracy through algorithm enhancements, the challenge of limited labeled data from individual clients is often overlooked, which impedes the training of highly accurate models. To address this issue and harness the potential of IoT networks, this paper presents FL-Bert-BiLSTM, a novel model that combines federated learning and pre-trained word embedding techniques for access control policy recognition. By leveraging the capabilities of IoT networks, the proposed model enables real-time and distributed training on IoT devices, effectively mitigating the scarcity of labeled data and enhancing accessibility for IoT applications. Additionally, the model incorporates pre-trained word embeddings to leverage the semantic information embedded in textual data, resulting in improved accuracy for access control policy recognition. Experimental results substantiate that the proposed model not only enhances accuracy and generalization capability but also preserves data privacy, making it well-suited for secure and efficient access control in IoT networks.

Text Classification of Patient Experience Comments in Saudi Dialect Using Deep Learning Techniques

Improving the quality of healthcare services is of the utmost importance in healthcare systems. Patient experience is a key aspect that should be gauged and monitored continuously. However, the measurement of such a vital indicator typically cannot be carried out directly, instead being derived from the opinions of patients who usually express their experience in free text. When it comes to patient comments written in the Arabic language, the currently used strategy to classify Arabic comments is totally reliant on human annotation, which is time-consuming and prone to subjectivity and error. Thus, fully using the value of patient feedback in a timely manner is difficult. This paper addresses the problem of classifying patient experience (PX) comments written in Arabic into 25 classes by using deep learning- and BERT-based models. A real-world data set of patient comments is obtained from the Saudi Ministry of Health for this purpose. Features are extracted from the data set, then used to train deep learning-based classifiers—including BiLSTM and BiGRU—for which pre-trained static word embedding and pre-training vector word embeddings are utilized. Furthermore, we utilize several Arabic pre-trained BERT models, in addition to building PX_BERT, a customized BERT model using the PX unlabeled database. From the experimental results for the 28 classifiers built in this study, the best-performing models (based on the F1 score) are found to be PX_BERT and AraBERTv02. To the best of our knowledge, this is the first study to tackle PX comment classification for the Arabic language.

Text2Color Networks: Deep Learning Models for Color Generation from Compositional Color Descriptions

Color serves as an important cue in graphics, arts and in many computer vision applications. We reliably and effortlessly name colors and communicate them with their names. However, many applications such as graphics, color design, palette generation and color selection tools demand numerical values of colors. Predicting and communicating colors by their numerical values is less intuitive and difficult task as it is a mapping of millions of colors for a specific color space. To bridge the gap between linguistic color names and numerical color values, in this paper, we present neural network architectures that predict a point in color space for a given color name. Proposed models provide user interface between colors and names by learning the language semantics and mimic human level comprehension of color descriptions to predict colors and modify them with respect to linguistic adjectives of color names. We consider color prediction as a regression problem and solve it as a language modeling task. Color descriptions are taken as text sequences and each sentence is represented with word-level tokenization. Each token is transformed into a word vector in the latent space using CBOW word embeddings model. Word vectors representing color names are fed as input to neural networks and trained with normalized R, G, B values as supervision information. Trained models are capable of predicting color for a given color name and modify colors for different nouns and adjectives associated with color names. We also built color generation models based on pre-trained word embeddings to overcome the limited availability of large linguistic color name datasets. These pre-trained models perform well with datasets containing few thousand color names. We then present two recommendation engines that suggest similar color palette to user given color name. These recommendation engines enhance the color vocabulary and assist users in the color selection process.

SentiUrdu-1M: A large-scale tweet dataset for Urdu text sentiment analysis using weakly supervised learning.

Low-resource languages are gaining much-needed attention with the advent of deep learning models and pre-trained word embedding. Though spoken by more than 230 million people worldwide, Urdu is one such low-resource language that has recently gained popularity online and is attracting a lot of attention and support from the research community. One challenge faced by such resource-constrained languages is the scarcity of publicly available large-scale datasets for conducting any meaningful study. In this paper, we address this challenge by collecting the first-ever large-scale Urdu Tweet Dataset for sentiment analysis and emotion recognition. The dataset consists of a staggering number of 1, 140, 821 tweets in the Urdu language. Obviously, manual labeling of such a large number of tweets would have been tedious, error-prone, and humanly impossible; therefore, the paper also proposes a weakly supervised approach to label tweets automatically. Emoticons used within the tweets, in addition to SentiWordNet, are utilized to propose a weakly supervised labeling approach to categorize extracted tweets into positive, negative, and neutral categories. Baseline deep learning models are implemented to compute the accuracy of three labeling approaches, i.e., VADER, TextBlob, and our proposed weakly supervised approach. Unlike the weakly supervised labeling approach, the VADER and TextBlob put most tweets as neutral and show a high correlation between the two. This is largely attributed to the fact that these models do not consider emoticons for assigning polarity.

A Visually Enhanced Neural Encoder for Synset Induction

The synset induction task is to automatically cluster semantically identical instances, which are often represented by texts and images. Previous works mainly consider textual parts, while ignoring the visual counterparts. However, how to effectively employ the visual information to enhance the semantic representation for the synset induction is challenging. In this paper, we propose a Visually Enhanced NeUral Encoder (i.e., VENUE) to learn a multimodal representation for the synset induction task. The key insight lies in how to construct multimodal representations through intra-modal and inter-modal interactions among images and text. Specifically, we first design the visual interaction module through the attention mechanism to capture the correlation among images. To obtain the multi-granularity textual representations, we fuse the pre-trained tags and word embeddings. Second, we design a masking module to filter out weakly relevant visual information. Third, we present a gating module to adaptively regulate the modalities’ contributions to semantics. A triplet loss is adopted to train the VENUE encoder for learning discriminative multimodal representations. Then, we perform clustering algorithms on the obtained representations to induce synsets. To verify our approach, we collect a multimodal dataset, i.e., MMAI-Synset, and conduct extensive experiments. The experimental results demonstrate that our method outperforms strong baselines on three groups of evaluation metrics.