Word Embedding Technique Research Articles

Deep-WET: a deep learning-based approach for predicting DNA-binding proteins using word embedding techniques with weighted features

DNA-binding proteins (DBPs) play a significant role in all phases of genetic processes, including DNA recombination, repair, and modification. They are often utilized in drug discovery as fundamental elements of steroids, antibiotics, and anticancer drugs. Predicting them poses the most challenging task in proteomics research. Conventional experimental methods for DBP identification are costly and sometimes biased toward prediction. Therefore, developing powerful computational methods that can accurately and rapidly identify DBPs from sequence information is an urgent need. In this study, we propose a novel deep learning-based method called Deep-WET to accurately identify DBPs from primary sequence information. In Deep-WET, we employed three powerful feature encoding schemes containing Global Vectors, Word2Vec, and fastText to encode the protein sequence. Subsequently, these three features were sequentially combined and weighted using the weights obtained from the elements learned through the differential evolution (DE) algorithm. To enhance the predictive performance of Deep-WET, we applied the SHapley Additive exPlanations approach to remove irrelevant features. Finally, the optimal feature subset was input into convolutional neural networks to construct the Deep-WET predictor. Both cross-validation and independent tests indicated that Deep-WET achieved superior predictive performance compared to conventional machine learning classifiers. In addition, in extensive independent test, Deep-WET was effective and outperformed than several state-of-the-art methods for DBP prediction, with accuracy of 78.08%, MCC of 0.559, and AUC of 0.805. This superior performance shows that Deep-WET has a tremendous predictive capacity to predict DBPs. The web server of Deep-WET and curated datasets in this study are available at https://deepwet-dna.monarcatechnical.com/. The proposed Deep-WET is anticipated to serve the community-wide effort for large-scale identification of potential DBPs.

Optimizing aspect-based sentiment analysis using sentence embedding transformer, bayesian search clustering, and sparse attention mechanism

The aspect-based sentiment analysis of hotel reviews represents a critical yet underexplored area of research. Such reviews encompass a range of expressions that extend beyond simple positive, negative, or neutral sentiments, delving into specific aspects discussed within the review. For instance, a statement like 'the location is comfortable, the food is good, but unfortunately, the service is bad' encapsulates multiple aspects, including location, food, and service. Existing research in Aspect-Based Sentiment Analysis (ABSA) frequently employs word embedding techniques for feature extraction, utilizing both Term and Transformer models. This study introduces an innovative approach by transitioning from word to sentence embedding, employing the SBERT transformer model for this transformation. This study further innovates label creation and topic modeling by integrating Bayesian search clustering with Inverse Document Frequency (IDF) calculations. This approach enables the identification of relevant topics, categorizing IDF scores based on occurrence frequencies. Scores ranging between 4–6 are marked as having low value but high informative potential, making them suitable candidates for label-specific datasets, particularly those without pre-existing labels. Additionally, this study incorporates the BigBird model to enhance this classification methodology. Applying this comprehensive framework to hotel review datasets, the result of the ABSA evaluation achieved an impressive accuracy score of 99%, indicating the efficacy of the proposed methodology in this nuanced field.

Diverse ensemble classifier driven Email spam classification using multiple word embedding’s with COCOB optimizer

Spam Email is a serious concern which can steal user’s personal information and cause huge financial loss due to the increasing rate of internet users. Therefore, the demand for accurate spam filtering has become more sophisticated for the Email spam detection. In the existing techniques, it is difficult to intricate the relationship between words in the Email using certain word embedding techniques and learning rate tuning is one of the greatest challenges of stochastic optimization. To overcome this difficulty, the proposed framework uses diverse ensemble based Email spam classification by incorporating multiple word embedding’s with Continuous Coin Betting optimizer. Word2Vec is used to produce the first set of 200D, next set of 200D word embedding is produced by Glove and 768D is produced by using Bidirectional Encoder Representations from Transformers (BERT) respectively. After generating word embedding, then it is classified through diverse ensemble based classifier with base level classifier consists of Long Short Term Memory (LSTM) Networks, Gated Recurrent Unit (GRU) and Bi-directional Gated Recurrent Unit (Bi-GRU) and LSTM as Meta-classifier using COCOB optimizer. Experiments were conducted on 3 benchmark Email dataset and result shows that the proposed system outperforms well with a low false positive rate.

An empirical assessment of different word embedding and deep learning models for bug assignment

Bug assignment, or bug triage, focuses on identifying the appropriate developers to repair newly discovered bugs, thereby managing them more effectively. Several deep learning-based approaches have been proposed for automated bug assignment. These approaches view automated bug assignment as a text classification task — the textual description of a bug report is utilized as the input and the potential fixers are regarded as the output labels. Such approaches typically depend on the classification performance of natural language processing and machine learning techniques. Various word embedding and deep learning models have emerged continuously. The effectiveness of those approaches depends on the chosen deep learning model, used for classification, and the word embedding model, used for representing bug reports. However, prior research does not empirically evaluate the impacts of various word embedding and deep learning models for automated bug assignment. In this paper, we conduct an empirical study to analyze the performance variations among 35 deep learning-based automated bug assignment approaches. These approaches are based on five word embedding techniques, i.e., Word2Vec, GloVe, NextBug, ELMo, and BERT, and seven text classification models, i.e., TextCNN, LSTM, Bi-LSTM, LSTM with attention, Bi-LSTM with attention, MLP, and Naive Bayes. We evaluated these combinations across three benchmark datasets, namely Eclipse JDT, GCC, and Firefox, and their mergence i.e., a cross-project dataset. Our main observations are: (1) Bi-LSTM with attention and Bi-LSTM using ELMo are significantly superior to other deep learning models on bug assignment tasks in terms of top-k (k = 1, 5, 10) accuracy and MRR; (2) Both the summary and description of bug reports are useful for bug assignment, but the description is more useful than the summary; (3) The training corpus for word embedding models has a significant impact on the performance of deep learning-based bug assignment methods. Our results show the importance of tuning different components (e.g. word embedding model, classification model, and textual input) in deep learning-based automated bug assignment methods and provide important insights for practitioners and researchers.

Word Embedding as a Semantic Feature Extraction Technique in Arabic Natural Language Processing: An Overview

Feature extraction has transformed the field of Natural Language Processing (NLP) by providing an effective way to represent linguistic features. Various techniques are utilised for feature extraction, such as word embedding. This latter has emerged as a powerful technique for semantic feature extraction in Arabic Natural Language Processing (ANLP). Notably, research on feature extraction in the Arabic language remains relatively limited compared to English. In this paper, we present a review of recent studies focusing on word embedding as a semantic feature extraction technique applied in Arabic NLP. The review primarily includes studies on word embedding techniques applied to the Arabic corpus. We collected and analysed a selection of journal papers published between 2018 and 2023 in this field. Through our analysis, we categorised the different feature extraction techniques, identified the Machine Learning (ML) and/or Deep Learning (DL) algorithms employed, and assessed the performance metrics utilised in these studies. We demonstrate the superiority of word embeddings as a semantic feature representation in ANLP. We compare their performance with other feature extraction techniques, highlighting the ability of word embeddings to capture semantic similarities, detect contextual associations, and facilitate a better understanding of Arabic text. Consequently, this article provides valuable insights into the current state of research in word embedding for Arabic NLP.

Bidirectional Gated Recurrent Unit with Glove Embedding and Attention Mechanism for Movie Review Classification

During the current Internet era, enormous volume of structured and unstructured textual data is generated and exchanged online that made the text classification more crucial. With the massive amount of user-generated content on social media platforms, businesses and organizations can leverage sentiment analysis to understand customer opinions and attitudes towards the products, services, or brand. Text classification is a natural language processing task in which a machine learning model is trained to categorize the text into predefined classes or categories. Due to the advent of promising Deep Learning techniques, intelligent and accurate text classification system can be developed. In movie review classification, algorithms are used to automatically categorize movie reviews into positive, negative, or sometimes neutral sentiments based on the opinions expressed in the text. The goal is to automate the process of determining whether a review reflects a positive or negative sentiment, helping users quickly understand the overall reception of a movie. This work proposes an automated movie review classification system based on Bidirectional Gated Recurrent Unit encoder with attention module. This method encodes the words using GloVe word embedding technique. Context information is captured in an efficient manner and the encoding is more reliable. Bidirectional GRU layer captures long-term dependencies in sentences, followed by a custom Attention layer that assigns higher weights to key components. The attention mechanism allows the model to selectively focus on important parts of the input sequence, by dynamically determining attention weights based on task relevance. The model was individually trained and evaluated using popular benchmark IMDb dataset. Experimental studies have proved the better performance of proposed movie review classification system that obtained accuracy of 98%.

Word Embedding as a Semantic Feature Extraction Technique in Arabic Natural Language Processing: An Overview

Feature extraction has transformed the field of Natural Language Processing (NLP) by providing an effective way to represent linguistic features. Various techniques are utilised for feature extraction, such as word embedding. This latter has emerged as a powerful technique for semantic feature extraction in Arabic Natural Language Processing (ANLP). Notably, research on feature extraction in the Arabic language remains relatively limited compared to English. In this paper, we present a review of recent studies focusing on word embedding as a semantic feature extraction technique applied in Arabic NLP. The review primarily includes studies on word embedding techniques applied to the Arabic corpus. We collected and analysed a selection of journal papers published between 2018 and 2023 in this field. Through our analysis, we categorised the different feature extraction techniques, identified the Machine Learning (ML) and/or Deep Learning (DL) algorithms employed, and assessed the performance metrics utilised in these studies. We demonstrate the superiority of word embeddings as a semantic feature representation in ANLP. We compare their performance with other feature extraction techniques, highlighting the ability of word embeddings to capture semantic similarities, detect contextual associations, and facilitate a better understanding of Arabic text. Consequently, this article provides valuable insights into the current state of research in word embedding for Arabic NLP.

Test

Feature extraction has transformed the field of Natural Language Processing (NLP) by providing an effective way to represent linguistic features. Various techniques are utilised for feature extraction, such as word embedding. This latter has emerged as a powerful technique for semantic feature extraction in Arabic Natural Language Processing (ANLP). Notably, research on feature extraction in the Arabic language remains relatively limited compared to English. In this paper, we present a review of recent studies focusing on word embedding as a semantic feature extraction technique applied in Arabic NLP. The review primarily includes studies on word embedding techniques applied to the Arabic corpus. We collected and analysed a selection of journal papers published between 2018 and 2023 in this field. Through our analysis, we categorised the different feature extraction techniques, identified the Machine Learning (ML) and/or Deep Learning (DL) algorithms employed, and assessed the performance metrics utilised in these studies. We demonstrate the superiority of word embeddings as a semantic feature representation in ANLP. We compare their performance with other feature extraction techniques, highlighting the ability of word embeddings to capture semantic similarities, detect contextual associations, and facilitate a better understanding of Arabic text. Consequently, this article provides valuable insights into the current state of research in word embedding for Arabic NLP.

Suicidal Thought Detection using Max Voting Ensemble Technique

Online social networking (SN) data presents a data stream that is rich in context and temporal information. It holds promise for predicting suicidal thoughts and behaviors. The fusion of SN data with machine learning algorithms offers a potential path forward. This research proposes a Max Voting Ensemble classifier model applied to a Reddit dataset for the identification of suicidal ideation. The preprocessing involves data cleansing, tokenization, and lemmatization. Additionally, TF-IDF and Word2Vec word embedding techniques are applied. Diverse machine learning algorithms, including Support Vector Machines (SVM), Logistic Regression (LR), Random Forest (RF), Multinomial Naive Bayes (MNB), AdaBoost, and XGBoost, are implemented. The results of selected Machine Learning Classifiers (MLCs) are amalgamated using a Max Voting Ensemble classifier. The research findings clearly indicate that the Max Voting Ensemble classifier yields improved precision of 91.39% coupled with a substantial accuracy of 87.5%. The application of Ensembling Techniques (ET) to SN data holds the potential to address the complexities and modeling challenges inherent in predicting acute Suicidal Ideation within these dynamic time scales.

A Comparative Analysis of Word Embeddings Techniques for Italian News Categorization

A Comparative Analysis of Word Embeddings Techniques for Italian News Categorization

Aspect Based Sentiment Analysis: Feature Extraction using Latent Dirichlet Allocation (LDA) and Term Frequency - Inverse Document Frequency (TF-IDF) in Machine Learning (ML)

The growth and development of social networks, blogs, forums, and e-commerce websites has produced a number of data, notably textual data, which has increased tremendously. Twitter is one of the most popular media social platforms; during the COVID-19 pandemic, people all around the world use social media to share their opinions or concerns about the pandemic that has changed their lives. It revealed a significant rise in tweets on coronavirus, including positive, negative, and neutral tweets about the virus's impact. Sentiment analysis faces challenges: sparse data limits understanding, while topic coherence and interpretability demand improvement for clearer insights. The primary goal of this paper is to improve the accuracy and effectiveness of sentiment analysis during the COVID-19 pandemic through the application of advanced techniques and classifiers. In this article, we experiment with such Support Vector Machines (SVM) and Naive Bayes (NB) on Twitter data for high-accuracy machine learning models. Using Latent Dirichlet Allocation (LDA)for feature extraction, we aim to capture comprehensive aspects and topics for sentiment analysis. Additionally, we explore Count Vectorizer and Term Frequency - Inverse Document Frequency (TF-IDF) as word embedding techniques. The main objectives are to extract topics, understand public concerns about Covid-19, and compare classifier performance in Aspect-Based Sentiment Analysis on Covid-19 tweets. This paper introduces advanced sentiment analysis techniques, such as LDA, Count Vectorizer, and SVM, enhancing nuanced sentiment analysis during the COVID-19 pandemic with notable 85% accuracy in SVM classification.

Boosting Prediction of Protein-Protein Interactions using Word Embedding Techniques

Understanding protein-protein interactions (PPIs) helps to identify protein functions and develop other important applications such as drug preparation, protein-disease relationship identification. Machine learning methods have been developed for the PPI prediction task in order to reduce the cost and time of previous experimental methods. In this paper, we study a method for determining PPIs using deep learning and protein sequence representation learning. In our method, an word embedding technique is utilized for protein sequence representation learning. This technique captures the semantic relationship between amino acids in protein sequences. The semantic relationship is then used as the input information, which is fed into a neural network to help recognize the interaction signature of the input protein pair. Different from previous studies, we integrate the protein sequence embedding mechanism into a neural network model. Thereby, the protein sequence embedding is better controlled for PPI prediction by our neural network model. We evaluate our method on benchmark datasets including Yeast, Human, and eight different independent sets. In addition, we also conduct an extensive comparison with the other existing methods. Our results show that the proposed method is superior to other existing methods and achieves high efficiency in predicting cross-species PPIs. The dataset and our source code are available at https://github.com/thnhub/BoostPPIP.git.

Employing word-embedding for schema matching in standard lifecycle management

Today, businesses rely on numerous information systems to achieve their production goals and improve their global competitiveness. Semantically integrating those systems is essential for businesses to achieve both. To do so, businesses must rely on standards, the most important of which are data exchange standards (DES). DES focus on technical and business semantics that are needed to deliver quality and timely products and services. Consequently, the ability for businesses to quickly use and adapt DES to their innovations and processes is crucial.Traditionally, information standards are managed and used 1) in a platform-specific form and 2) usually with standalone and file-based applications. These traditional approaches no longer meet today's business and information agility needs. For example, businesses now must deal with companies and suppliers that use heterogeneous syntaxes for their information. Syntaxes that are optimized for individual but have different objectives. Moreover, file-based standards and the usage specifications derived from the standards cause inconsistencies since there is neither a single standard format for each usage specification nor a single source of truth for all of them.As the number and types of information systems grow, developing, maintaining, reviewing, and approving standards and their derived usage specifications are becoming more difficult and time consuming. Each file-based usage specification is typically based on a different syntax than the standard syntax. As a result, each usage specification must be manually updated as the standard evolves; this can cause significant delays and costs in adopting the new and better standard versions. National Institute of Standards and Technology (NIST) in collaboration with the Open Application Groups Inc. (OAGi) has developed a web-based standard lifecycle management tool called SCORE to address these problems. The objective of this paper is to introduce the SCORE tool and discuss its particular functionality where a word-embedding technique has been employed along with other schema-matching approaches. Together they can assist standard users in updating the usage specification due to the release of new version of a standard leading to faster adaptations of DES to new processes.

Personalized Query Expansion with Contextual Word Embeddings

Personalized Query Expansion, the task of expanding queries with additional terms extracted from the user-related vocabulary, is a well-known solution to improve the retrieval performance of a system w.r.t. short queries. Recent approaches rely on word embeddings to select expansion terms from user-related texts. Although promising results have been delivered with former word embedding techniques, we argue that these methods are not suited for contextual word embeddings, which produce a unique vector representation for each term occurrence. In this article, we propose a Personalized Query Expansion method designed to solve the issues arising from the use of contextual word embeddings with the current Personalized Query Expansion approaches based on word embeddings. Specifically, we employ a clustering-based procedure to identify the terms that better represent the user interests and to improve the diversity of those selected for expansion, achieving improvements of up to 4% w.r.t. the best-performing baseline in terms of MAP@100. Moreover, our approach outperforms previous ones in terms of efficiency, allowing us to achieve sub-millisecond expansion times even in data-rich scenarios. Finally, we introduce a novel metric to evaluate the expansion terms’ diversity and empirically show the unsuitability of previous approaches based on word embeddings when employed along with contextual word embeddings, which cause the selection of semantically overlapping expansion terms.

Sentiment Analysis of Roman Urdu Text Using Machine Learning Techniques

Social media has attained popularity during the last few decades due to the rapid growth of online businesses and social interaction. People can interact with one another and communicate their sentiments by expressing their ideas and points of view on social media. Businesses involved in manufacturing, sales, and marketing increasingly focus on social media to get feedback on their goods and services from people worldwide. Businesses must process and analyze this feedback in the form of sentiments to gain business insights. Every day, millions of Urdu and Roman Urdu sentences are posted on social media platforms. The critical loss of this massive amount of data results from ignoring the thoughts and opinions in language with limited resources, such as Urdu and Roman Urdu in the favor of resource-rich languages, such as English. The current study focused on sentiment analysis of Roman Urdu text. Bag of Words (BoW) and Term Frequency-Inverse Document Frequency (TF-IDF) word embedding techniques were deployed to conduct the current study. Support Vector Machine (SVM), Linear Support Vector Machine (LSVC), Logistic Regression (LR), and Random Forest (RF) classifiers were deployed. The experiments showed that SVM showed 94.74%, while RF showed 93.13% accuracy using BoW word embedding technique

Authorship Verification in software engineering education

The prevalence of academic misconduct, specifically contract cheating, is a rising concern in higher education institutions globally. Among the recent advancements, Generative Artificial Intelligence (genAI) has exacerbated the situation by offering authentically generated writings, making detection through traditional plagiarism tools ineffective. This paper explores the development and application of students' academic writing profiles, using a combination of word embedding (Word2Vec) and stylistic feature extraction techniques. By leveraging a Siamese neural network, our method focuses on recognising distinctive writing styles, a concept rooted in Authorship Verification (AV). Our approach's efficacy evaluates favourably against other AV methods and is tested against AI-generated texts deliberately designed to mimic student writing. The study emphasises the importance of understanding individual academic writing styles to identify outsourcing or AI-generated work effectively.

A Natural-Language-Processing-Based Method for the Clustering and Analysis of Movie Reviews and Classification by Genre

Reclassification of massive datasets acquired through different approaches, such as web scraping, is a big challenge to demonstrate the effectiveness of a machine learning model. Notably, there is a strong influence of the quality of the dataset used for training those models. Thus, we propose a threshold algorithm as an efficient method to remove stopwords. This method employs an unsupervised classification technique, such as K-means, to accurately categorize user reviews from the IMDb dataset into their most suitable categories, generating a well-balanced dataset. Analysis of the performance of the algorithm revealed a notable influence of the text vectorization method used concerning the generation of clusters when assessing various preprocessing approaches. Moreover, the algorithm demonstrated that the word embedding technique and the removal of stopwords to retrieve the clustered text significantly impacted the categorization. The proposed method involves confirming the presence of a suggested stopword within each review across various genres. Upon satisfying this condition, the method assesses if the word’s frequency exceeds a predefined threshold. The threshold algorithm yielded a mapping genre success above 80% compared to precompiled lists and a Zipf’s law-based method. In addition, we employed the mini-batch K-means method for the clustering formation of each differently preprocessed dataset. This approach enabled us to reclassify reviews more coherently. Summing up, our methodology categorizes sparsely labeled data into meaningful clusters, in particular, by using a combination of the proposed stopword removal method and TF-IDF. The reclassified and balanced datasets showed a significant improvement, achieving 94% accuracy compared to the original dataset.

A content-based metric for social media influencer marketing

PurposeThe study aims to propose an instrument for measuring product-centeredness (i.e. the extent to which comment content is related to a product) using word embedding techniques as well as explore its determinants.Design/methodology/approachThe study collected branded posts from 205 Instagram influencers and empirically examined how four factors (i.e. authenticity, vividness, coolness and influencer–product congruence) influence the content of the comments on branded posts.FindingsPost authenticity and congruence are shown to have positive effects on product-centeredness. The interaction between coolness and authenticity is also significant. The number of comments or likes on branded posts is not correlated with product-centeredness.Originality/valueIn social media influencer marketing, volume-based metrics such as the numbers of likes and comments have been researched and applied extensively. However, content-based metrics are urgently needed, as fans may ignore brands and focus on influencers. The proposed instrument for assessing comment content enables marketers to construct content-based metrics. Additionally, the authors' findings enhance the understanding of social media users' engagement behaviors.

Representation-centric approach for classification of Consumer Health Questions

Question Classification is the very first and pivotal step in Question Answering(QA) systems. It maps a given question into a predefined class. Classification of Consumer Health Questions contributes to answering medical queries of non-expert users. It plays a principal role in the performance of Consumer Health Question Answering systems. There are several approaches to the classification of questions. However, they do not consider the nature of Consumer Health Questions. In this study, we perform experiments focusing on the question representation methods to classify Consumer Health Questions accurately. These experiments provide an intuition of practical considerations for Consumer Health Question classification. The questions were represented through certain manually-designed features, word embedding, sentence embedding, and finetuning-based techniques. We create question classifiers that involve designing models for representing and categorizing questions. Based on the result, the fine-tuned Bidirectional Encoder Representations from Transformers(BERT)-large-based model with an accuracy of 86.00% on Genetic and Rare Diseases (GARD) and 80.40% on Yahoo! Answers-based datasets outperform previous works. In addition, the method based on A Robustly Optimized BERT Pretraining Approach(RoBERTa)-large model with 86.20% accuracy surpasses fine-tuned BERT-large model on the GARD dataset. Examining the outcomes of the models gives insight into decisive considerations for representing and classifying Consumer Health Questions.

Topic Modeling based Text Classification Regarding Islamophobia using Word Embedding and Transformers Techniques

Islamophobia is a rising area of concern in the current era where Muslims face discrimination and receive negative perspectives towards their religion, Islam. Islamophobia is a type of racism that is being practiced by individuals, groups, and organizations worldwide. Moreover, the ease of access to social media platforms and their augmented usage has also contributed to spreading hate speech, false information, and negative opinions about Islam. In this research study, we focused to detect Islamophobic textual content shared on various social media platforms. We explored the state-of-the-art techniques being followed in text data mining and Natural Language Processing (NLP). Topic modelling algorithm Latent Dirichlet Allocation is used to find top topics. Then, word embedding approaches such as Word2Vec and Global Vectors for word representation (GloVe) are used as feature extraction techniques. For text classification, we utilized modern text analysis techniques of transformers-based Deep Learning algorithms named Bidirectional Encoders Representation from Transformers (BERT) and Generative Pre-Trained Transformer (GPT). For results comparison, we conducted an extensive empirical analysis of Machine Learning algorithms and Deep Learning using conventional textual features such as the Term Frequency-Inverse Document Frequency, N-gram, and Bag of words (BoW). The empirical based results evaluated using standard performance evaluation measures show that the proposed approach effectively detects the textual content related to Islamophobia. In the corpus of the study under Machine Learning models Support Vector Machine (SVM) performed best with an F1 score of 91%. The Transformer based core NLP models and the Deep Learning model Convolutional Neural Network (CNN) when combined with GloVe performed best among all the techniques except SVM with BoW. GPT, SVM when combined with BoW and BERT yielded the best F1 score of 92%, 92% and 91.9% respectively, while CNN performed slightly poor with an F1 score of 91%.