Continuous Bag Of Words Research Articles

Enhanced Arabic Sentiment Analysis Using a Novel Stacking Ensemble of Hybrid and Deep Learning Models

Sentiment analysis (SA) is a machine learning application that drives people’s opinions from text using natural language processing (NLP) techniques. Implementing Arabic SA is challenging for many reasons, including equivocation, numerous dialects, lack of resources, morphological diversity, lack of contextual information, and hiding of sentiment terms in the implicit text. Deep learning models such as convolutional neural networks (CNN) and long short-term memory (LSTM) have significantly improved in the Arabic SA domain. Hybrid models based on CNN combined with long short-term memory (LSTM) or gated recurrent unit (GRU) have further improved the performance of single DL models. In addition, the ensemble of deep learning models, especially stacking ensembles, is expected to increase the robustness and accuracy of the previous DL models. In this paper, we proposed a stacking ensemble model that combined the prediction power of CNN and hybrid deep learning models to predict Arabic sentiment accurately. The stacking ensemble algorithm has two main phases. Three DL models were optimized in the first phase, including deep CNN, hybrid CNN-LSTM, and hybrid CNN-GRU. In the second phase, these three separate pre-trained models’ outputs were integrated with a support vector machine (SVM) meta-learner. To extract features for DL models, the continuous bag of words (CBOW) and the skip-gram models with 300 dimensions of the word embedding were used. Arabic health services datasets (Main-AHS and Sub-AHS) and the Arabic sentiment tweets dataset were used to train and test the models (ASTD). A number of well-known deep learning models, including DeepCNN, hybrid CNN-LSTM, hybrid CNN-GRU, and conventional ML algorithms, have been used to compare the performance of the proposed ensemble model. We discovered that the proposed deep stacking model achieved the best performance compared to the previous models. Based on the CBOW word embedding, the proposed model achieved the highest accuracy of 92.12%, 95.81%, and 81.4% for Main-AHS, Sub-AHS, and ASTD datasets, respectively.

Contextual Word2Vec Model for Understanding Chinese Out of Vocabularies on Online Social Media

In this chapter, the authors propose to use contextual Word2Vec model for understanding OOV (out of vocabulary). The OOV is extracted by using left-right entropy and point information entropy. They choose to use Word2Vec to construct the word vector space and CBOW (continuous bag of words) to obtain the contextual information of the words. If there is a word that has similar contextual information to the OOV, the word can be used to understand the OOV. They chose the Weibo corpus as the dataset for the experiments. The results show that the proposed model achieves 97.10% accuracy, which is better than Skip-Gram by 8.53%.

A Dynamic Strategy for Classifying Sentiment From Bengali Text by Utilizing Word2vector Model

In today's world, around 230 million people used the Bengali or Bangla language to communicate. These individuals are progressively associated with online exercises on famous micro-blogging and long-range interpersonal communication locales, imparting insights what's more, musings, and also the vast majority of articles are in the Bengali language. Thus, Bengali people express their emotions using the Bangla language by reviewing, commenting, or recommendations. Sentiment analysis helps determine the people's emotions expressed on social media or several online platforms. Therefore, this study focused on extracting their emotion from a Bengali text by utilizing Word2vector, Skip-Gram, and Continuous Bag of Words (CBOW) with a new Word to Index model by focusing on three individual classes happy, angry, and excited. The authors achieved the highest accuracy of 75% by utilizing the skip-gram model to classify those three types of emotions. This study also outperformed other existing works with LSTM, CNN model with existing datasets.

SFN: A Novel Scalable Feature Network for Vulnerability Representation of Open-Source Codes.

Vulnerability detection technology has become a hotspot in the field of software security, and most of the current methods do not have a complete consideration during code characterizing, which leads to problems such as information loss. Therefore, this paper proposes one class of Scalable Feature Network (SFN), a composite feature extraction method based on Continuous Bag of Words and Convolutional Neural Network. In addition, to characterize the source code more comprehensively, we construct multiscale code metrics in terms of semantic-, line-, and function granularity. In order to verify the effectiveness of the SFN, this paper builds a Scalable Vulnerability Detection Model (SVDM) by combining SFN with Bi-LSTM. The experimental results show that the proposed SVDM can obtain precision over 84.3% and recall at 83.4%, respectively, while both FNR and FPR are less than 17%.

Refining Word Embeddings with Sentiment Information for Sentiment Analysis

Natural Language Processing problems generally require the use of pre-trained distributed word representations to be solved with deep learning models. However, distributed representations usually rely on contextual information which prevents them from learning all the important word characteristics. The task of sentiment analysis suffers from such a problem because sentiment information is ignored during the process of learning word embeddings. The performance of sentiment analysis can be affected since two words with similar vectors may have opposite sentiment orientations. The present paper introduces a novel model called Continuous Sentiment Contextualized Vectors (CSCV) to address this problem. The proposed model can learn word sentiment embedding using its surrounding context words. It uses Continuous Bag-of-Words (CBOW) model to deal with the context and sentiment lexicons to identify sentiment. Existing pre-trained vectors are combined then with the obtained sentiment vectors using Principal component analysis (PCA) to enhance their quality. The experiments show that: (1) CSCV vectors can be used to enhance any pre-trained word vectors; (2) The result vectors strongly alleviate the problem of similar words with opposite polarities; (3) The performance of sentiment classification is improved by applying this approach.

Military Chain: Construction of Domain Knowledge Graph of Kill Chain Based on Natural Language Model

With the advent of the Big Data era, the specialized data in the kill chain domain has increased dramatically, and the engine-based method of retrieving information can hardly meet the users' need for more accurate answers. The kill chain domain includes four components: control equipment, sensor equipment, strike equipment (weapon and platform), and evaluator equipment, as well as related data which contain a large amount of valuable information such as the parameter information contained in each component. If these fragmented and confusing data are integrated and effective query methods are established, they can help professionals complete the military kill chain knowledge system. The knowledge system constructed in this paper is based on the Neo4j graph database and the US Command simulation system to establish a target-oriented knowledge map of kill chain, aiming to provide data support for the Q&A system. Secondly, in order to facilitate the query, this paper establishes entity and relationship/attribute mining based on the continuous bag-of-words (CBOW) encoding model, bidirectional long short-term memory–conditional random field (BiLSTM-CRF) named entity model, and bidirectional gated recurrent neural network (BiGRU) intent recognition model for Chinese kill chain question and answer; returns the corresponding entity or attribute values in combination with the knowledge graph triad form; and finally constructs the answer return. The constructed knowledge map of the kill chain contains 2767 items (including sea, land, and air), and the number of parameters involved is 30124. The number of model parameters of the deep learning network is 27.9 M for the Q&A system built this time, and the accuracy rate is 85.5% after 200 simulated queries.

Augmented language model with deep learning adaptation on sentiment analysis for E-learning recommendation

Sentiment Analysis is considered as an important research field in text mining, and is significant in recommendation systems and e-learning environments. This research proposes a new methodology of e-learning hybrid Recommendation System Based on Sentiment Analysis (RSBSA) by leveraging tailored Natural Language Processing (NLP) and Convolutional Neural Network (CNN) techniques, to recommend appropriate e-learning materials based on learner’s preferences. Integration is done on fine-grained sentiment analysis models, to classify text reviews of e-content posted on e-learning platform. Two enhanced language models based on ‘Continuous Bag of Word’ and ‘Skip-Gram’ are introduced. Moreover, three resilient language modelsbased on the hybrid language techniques are developed to produce a superior vocabulary representation. These models were trained using various CNN models to predict ratings of resources from online reviews provided by learners. To accomplish this, a customizable dataset ‘ABHR-1′ is used, which is derived from e-content' reviews with corresponding ratings labeled [1–5]. The proposed models are evaluated and tested using ABHR-1 and two public datasets. According to the simulation results, Multiplication-Several-Channels-CNNmodel outperformed other models with an accuracy of 90.37 % for fine-grained sentiment classification on 5 discrete classes and the empirical results are compared.

Dynamic Data Infrastructure Security for Interoperable e-Healthcare Systems: A Semantic Feature-Driven NoSQL Intrusion Attack Detection Model.

The exponential rise in advanced software computing and low-cost hardware has broadened the horizon for the Internet of Medical Things (IoMT), interoperable e-Healthcare systems serving varied purposes including electronic healthcare records (EHRs) and telemedicine. However, being heterogeneous and dynamic in nature, their database security remains a challenge forever. Numerous intrusion attacks including bot-attack and malware have confined major classical databases towards e-Healthcare. Despite the robustness of NoSQL over the structured query language databases, the dynamic data nature over a heterogeneous environment makes it vulnerable to intrusion attacks, especially over interoperable e-Healthcare systems. Considering these challenges, this work proposed a first of its kind semantic feature-driven NoSQL intrusion attack (NoSQL-IA) detection model for interoperable e-Healthcare systems. This work assessed the efficacy of the different semantic feature-extraction methods like Word2Vec, Continuous Bag of Words, N-Skip Gram (SKG), Count Vectorizer, TF-IDF, and GLOVE towards NoSQL-IA prediction. Subsequently, to minimize computational exhaustion, different feature selection methods including Wilcoxon Rank Sum Test (WRST), significant predictor test, principal component analysis, Select K-Best, and variance threshold feature selection algorithms were employed. To alleviate the data imbalance problem, it applied different resampling methods including upsampling, downsampling, and synthetic minority oversampling technique (SMOTE) over the selected features. Later, Min–Max normalization was performed over the input feature vectors to alleviate any possibility of overfitting. Towards NoSQL-IA prediction, different machine learning methods like Multinomial Naïve Bayes, decision tree, logistic regression, support vector machine, k-NN, AdaBoost, Extra Tree Classifier, random forest ensemble, and XG-Boost were applied, which classified each input query as the regular query or the NoSQL-IA attack query. The depth performance assessment revealed that the use of Word2Vec features SKG in sync with VTFS feature selection and SMOTE resampling processed with the bootstrapped random forest classifier can provide the best performance in terms of high accuracy (98.86%), F-Measure (0.974), and area under the curve (AUC) (0.981), thus enabling it suitable for interoperable e-Healthcare database security.

Semantic Query-Featured Ensemble Learning Model for SQL-Injection Attack Detection in IoT-Ecosystems

Structured query language (SQL) has emerged as one of the most used databases, serving an array of Internet-of-Things (IoTs)-enabled services including web-transactions, grid networks, industrial activity log and proactive decision systems, smart-home, financial transactions, business communication etc. With high pace increase in SQL-driven IoT applications, the threat of SQL-injection attacks (SQLIAs) at the middleware layer has increased significantly. To address such issues, machine learning-based SQLIA-prediction systems are proposed; however, majority of the existing methods are found limited in terms of intrusion detection accuracy because of their complete-reliance on structural features and inferior learning model(s). On the contrary, intruders these days intrude the system by mimicking the normal queries and hence confuses most of the classical learning-based methods. To alleviate such problems, this article emphasizes on exploiting semantic features along with the state-of-art highly robust computing environment. We proposed a robust semantic query-featured ensemble learning model for SQLIA prediction. Unlike classical (query's) template-matching or term-assessment-based methods, our proposed SQLIA-prediction model exploits latent semantic features from large SQL-queries to train an ensemble learning model that classifies each query as the normal query or the SQLIA query. Functionally, it performs preprocessing over large set of SQL-queries using count-vectorizer and stopping word removal. Subsequently, it applies Word2Vec feature extraction method over each query using continuous bag of words (CBOW) and N-skip gram (SKG) algorithms, which obtained CBOW and SKG semantic features from each SQL-query. The extracted features were processed for data resampling so as to alleviate the problem of class-imbalance and skewness. To alleviate redundant computation, two feature selection algorithms named Mann-Whitney significance predictor test and principal component analysis were applied over the resampled features. Moreover, to eliminate over-fitting and convergence problem, Min-Max normalization was performed over the selected features which were later processed for learning using a state-of-art robust heterogeneous ensemble learning model. Unlike standalone classifier-based SQLIA, the proposed learning-model employed a set of nine base classifiers designed to serve maximum voting ensemble-based prediction. The proposed ensemble-learning method classified each SQL-query as the normal-query or the SQLIA-query. Simulation results affirmed superiority of the proposed SQLIA prediction model in terms of accuracy (98%), F-Score (0.989), AUC (0.999) signifying its efficacy toward real-world SQL-driven IoT-ecosystems.

A Hybrid Approach for Network Rumor Detection Based on Attention Mechanism and Bidirectional GRU Model in Big Data Environment

In view of the low accuracy of Internet rumor detection in traditional machine learning detection methods due to the influence of comment directivity and incomplete features caused by too long sequences, an Internet rumor detection method combining attention mechanism and bidirectional gated recurrent unit (GRU) model in big data environment is proposed. First, the continuous bag of words (CBOW) model is modified by defining the context window and introducing the weighting module, and it is used to obtain the word vectors of the text. Then, the word vectors are used as the word embedding layer which is fed into a bidirectional GRU combined with an attention mechanism through the convolutional layer and pooling layer of the convolutional neural network (CNN) to fully extract the features of the text. Finally, the text feature vectors are input into the Softmax classifier for classification so as to realize the detection of Internet rumors. Based on the selected microblog dataset, the experimental results of the proposed model show that when the learning rate is set to 0.015, Accuracy, Precision, Recall, and F1 values are about 95.8%, 93.1%, 92.9% and 93.5%, respectively, which are better than other comparison models.

Sentiment analysis using global vector and long short-term memory

Tweet sentiment analysis is a Deep Learning study that is beneficial for automatically determining public opinion on a certain topic. Using the Long Short-Term Memory (LSTM) algorithm, this paper aims to proposes a Twitter analysis technique that divides Tweets into two categories (positive and negative). The Global Vector (GloVe) word embedding score is used to rate many selected words as network input. GloVe converts words into vectors by building a corpus matrix. The GloVe outperforms its prior model, owing to its smaller vector and corpora sizes. GloVe has a higher accuracy than the model word embedding word2vec, Continuous Bag of Word(CBoW), and word2vec Skip-gram. The preprocessed term variation was conducted to test the performance of sentiment classification. The test results show that this proposed method has succeeded in classifying with the best results with an accuracy of 95.61%.

Deep Sentiment Analysis Using CNN-LSTM Architecture of English and Roman Urdu Text Shared in Social Media

Sentiment analysis (SA) has been an active research subject in the domain of natural language processing due to its important functions in interpreting people’s perspectives and drawing successful opinion-based judgments. On social media, Roman Urdu is one of the most extensively utilized dialects. Sentiment analysis of Roman Urdu is difficult due to its morphological complexities and varied dialects. The purpose of this paper is to evaluate the performance of various word embeddings for Roman Urdu and English dialects using the CNN-LSTM architecture with traditional machine learning classifiers. We introduce a novel deep learning architecture for Roman Urdu and English dialect SA based on two layers: LSTM for long-term dependency preservation and a one-layer CNN model for local feature extraction. To obtain the final classification, the feature maps learned by CNN and LSTM are fed to several machine learning classifiers. Various word embedding models support this concept. Extensive tests on four corpora show that the proposed model performs exceptionally well in Roman Urdu and English text sentiment classification, with an accuracy of 0.904, 0.841, 0.740, and 0.748 against MDPI, RUSA, RUSA-19, and UCL datasets, respectively. The results show that the SVM classifier and the Word2Vec CBOW (Continuous Bag of Words) model are more beneficial options for Roman Urdu sentiment analysis, but that BERT word embedding, two-layer LSTM, and SVM as a classifier function are more suitable options for English language sentiment analysis. The suggested model outperforms existing well-known advanced models on relevant corpora, improving the accuracy by up to 5%.

Seeds: Sampling-Enhanced Embeddings.

Finding a desirable sampling estimator has a profound impact on the development of static word embedding models, such as continue-bag-of-words (CBOW) and skip gram (SG), which have been generally accepted as popular low-resource algorithms to generate task-agnostic word representations. Due to the prevalence of large-scale pretrained models, less attention has been paid to these static models in the recent years. However, compared with the dynamic embedding models (e.g., BERT), these static models are straightforward to interpret, cost effective to train, and out-of-box to deploy, thus are still widely used in various downstream models until now. Therefore, it is still of considerable significance to study and improve them, especially the crucial components shared by these static models. In this article, we focus on negative sampling (NS), a key component shared by the sampling-based static models, by investigating and mitigating some critical problems of the sampling core. Concretely, we propose Seeds, a sampling enhanced embedding framework, to learn static word embeddings by a new algorithmic innovation for replacing the NS estimator, in which multifactor global priors are considered dynamically for different training pairs. Then, we implement this framework by four concrete models. For the first two implementations, namely CBOW-GP and SG-GP, both negative words and positive auxiliaries are sampled. And for the other two implementations, CBOW-GN and SG-GN, estimations are simplified by sampling only the negative instances. Extensive experimental results across a variety of standard intrinsic and extrinsic tasks demonstrate that embeddings learned by the proposed models outperform their NS-based counterparts, such as CBOW-NS and SG-NS, as well as other strong baselines.

Ensemble Classifiers for Arabic Sentiment Analysis of Social Network (Twitter Data) towards COVID-19-Related Conspiracy Theories

Sentiment analysis has recently become increasingly important with a massive increase in online content. It is associated with the analysis of textual data generated by social media that can be easily accessed, obtained, and analyzed. With the emergence of COVID-19, most published studies related to COVID-19’s conspiracy theories were surveys on the people's sentiments and opinions and studied the impact of the pandemic on their lives. Just a few studies utilized sentiment analysis of social media using a machine learning approach. These studies focused more on sentiment analysis of Twitter tweets in the English language and did not pay more attention to other languages such as Arabic. This study proposes a machine learning model to analyze the Arabic tweets from Twitter. In this model, we apply Word2Vec for word embedding which formed the main source of features. Two pretrained continuous bag-of-words (CBOW) models are investigated, and Naïve Bayes was used as a baseline classifier. Several single-based and ensemble-based machine learning classifiers have been used with and without SMOTE (synthetic minority oversampling technique). The experimental results show that applying word embedding with an ensemble and SMOTE achieved good improvement on average of F1 score compared to the baseline classifier and other classifiers (single-based and ensemble-based) without SMOTE.

Abstractive Arabic Text Summarization Based on Deep Learning.

Text summarization (TS) is considered one of the most difficult tasks in natural language processing (NLP). It is one of the most important challenges that stand against the modern computer system's capabilities with all its new improvement. Many papers and research studies address this task in literature but are being carried out in extractive summarization, and few of them are being carried out in abstractive summarization, especially in the Arabic language due to its complexity. In this paper, an abstractive Arabic text summarization system is proposed, based on a sequence-to-sequence model. This model works through two components, encoder and decoder. Our aim is to develop the sequence-to-sequence model using several deep artificial neural networks to investigate which of them achieves the best performance. Different layers of Gated Recurrent Units (GRU), Long Short-Term Memory (LSTM), and Bidirectional Long Short-Term Memory (BiLSTM) have been used to develop the encoder and the decoder. In addition, the global attention mechanism has been used because it provides better results than the local attention mechanism. Furthermore, AraBERT preprocess has been applied in the data preprocessing stage that helps the model to understand the Arabic words and achieves state-of-the-art results. Moreover, a comparison between the skip-gram and the continuous bag of words (CBOW) word2Vec word embedding models has been made. We have built these models using the Keras library and run-on Google Colab Jupiter notebook to run seamlessly. Finally, the proposed system is evaluated through ROUGE-1, ROUGE-2, ROUGE-L, and BLEU evaluation metrics. The experimental results show that three layers of BiLSTM hidden states at the encoder achieve the best performance. In addition, our proposed system outperforms the other latest research studies. Also, the results show that abstractive summarization models that use the skip-gram word2Vec model outperform the models that use the CBOW word2Vec model.

Bengali paper classification using ensemble machine learning algorithms

Text classification is one of the most challenging problems in natural language processing (NLP). Language models are at the heart of NLP. The ability to represent texts as numbers has given rise to many NLP tasks, for example, text categorisation, translation, and summarisation. Unfortunately, NLP for Bengali texts has not reached the state-of-art level of other Languages like English yet, mostly due to the scarcity of resources and the complexities seen in Bengali grammar. Therefore, not much work has been done in this field. In this paper, we have studied one of the word embedding methods, Word2vec, based on continuous bag of words (CBOW) with several ensemble machine learning algorithms, e.g., Adaptive Boosting Classifiers, Light Gradient Boosting Machine, XGboost, and random forest classifiers (RFC). The model is trained on a large corpus of Bengali newspapers of a considerable size that has 99283949 words and 8284804 sentences in 392772 documents. In our experiment, Word2vec CBOW model with XGboost algorithm performed much better than other models and achieved 92.24% accuracy.

A Simple and Effective Usage of Word Clusters for CBOW Model

We propose a simple and effective method for incorporating word clusters into the Continuous Bag-of-Words (CBOW) model. Specifically, we propose to replace infrequent input and output words in CBOW model with their clusters. The resulting cluster-incorporated CBOW model produces embeddings of frequent words and a small amount of cluster embeddings, which will be fine-tuned in downstream tasks. We empirically show our replacing method works well on several downstream tasks. Through our analysis, we show that our method might be also useful for other similar models which produce word embeddings.

A hybrid E-learning recommendation integrating adaptive profiling and sentiment analysis

This research proposes a novel framework named Enhanced e-Learning Hybrid Recommender System (ELHRS) that provides an appropriate e-content with the highest predicted ratings corresponding to the learner’s particular needs. To accomplish this, a new model is developed to deduce the Semantic Learner Profile automatically. It adaptively associates the learning patterns and rules depending on the learner’s behavior and the semantic relations computed in the semantic matrix that mutually links e-learning materials and terms. Here, a semantic-based approach for term expansion is introduced using DBpedia and WordNet ontologies. Further, various sentiment analysis models are proposed and incorporated as a part of the recommender system to predict ratings of e-learning resources from posted text reviews utilizing fine-grained sentiment classification on five discrete classes. Qualitative Natural Language Processing (NLP) methods with tailored-made Convolutional Neural Network (CNN) are developed and evaluated on our customized dataset collected for a specific domain and a public dataset. Two improved language models are introduced depending on Skip-Gram (S-G) and Continuous Bag of Words (CBOW) techniques. In addition, a robust language model based on hybridization of these couple of methods is developed to derive better vocabulary representation, yielding better accuracy 89.1% for the CNN-Three-Channel-Concatenation model. The suggested recommendation methodology depends on the learner’s preferences, other similar learners’ experience and background, deriving their opinions from the reviews towards the best learning resources. This assists the learners in finding the desired e-content at the proper time.

Reusable Component Retrieval from a Large Repository Using Word2Vec with Continuous Bag of Words

A reusable code component is the one which can be easily used with a little or no adaptation to fit in to the application being developed. The major concern in such process is the maintenance of these reusable components in one place called ‘Repository’, so that those code components can be effectively identified as well as reused. Word embedding allows us to numerically represent our textual information. They have become so pervasive that almost all Natural Language Processing projects make use of them. In this work, we considered to use Word2Vec concept to find vector representation of features of a reusable component. The features of a reusable component in the form of sequence of words are input to Word2Vec network. Our method using Word2Vec with Continuous Bag of Words out performs existing method in the market. The proposed methodology has shown an accuracy of 94.8% in identifying the existing reusable component.

Applying Sentiment Product Reviews and Visualization for BI Systems in Vietnamese E-Commerce Website: Focusing on Vietnamese Context

Product reviews become more important in the buying decision-making process of customers. Exploiting and analyzing customer product reviews in sentiments also become an advantage for businesses and researchers in e-commerce platforms. This study proposes a sentiment evaluation model of customer reviews by extracting objects, emotional words for emotional level analysis, using machine learning algorithms. The research object is the Vietnamese language, which has special semantic structures and characteristics. In this research model, emotional dictionaries and sets of extract rules are combined to build a data training data set based on the semantic dependency relationship between words in sentences of the given Vietnamese context. The recurrent neural network model (RNN) solves the emotional analysis issue, specifically, the long short-term memory neural network (LSTMs). This analysis model combines the vector representations of words with a continuous bag-of-words (CBOW) architecture. Our system is designed to crawl realistic data in an e-commerce website and automatically aggregate them. These data will be stored in MongoDB before processing and input into our model on the server. Then, the system can exploit the features in products reviews and classify customer reviews. These features extracted from different feedback on each shopping step and depending on the kinds of products. Finally, there is a web-app to connect to a server and visualize all the research results. Based on the research results, enterprises can follow up their customers in real-time and receive recommendations to understand their customers. From there, they can improve their services and provide sustainable consumer service.