Skip-gram Word2vec Research Articles

Whoosh User Sentiment Analysis on Social Media Using Word2Vec and the Best Naïve Bayes Probability Model

By using the Twitter microblogging feature, users can post short tweets with limited characters that express their thoughts and opinions regarding a matter. The newest transportation in Indonesia, a high-speed train namely Whoosh is one of the things that Twitter users responded to. This latest transportation has led to the emergence of opinions from the Indonesian people which are shared publicly in various media, one of which is social media. Therefore, to make it easier for business people or companies to understand public opinion regarding service improvements in the future, sentiment analysis on social media is needed to determine user opinions regarding high-speed train transportation. In this research, sentiment analysis of high-speed train users will be carried out on social media Twitter using Word2Vec and Naïve Bayes as classification methods. In this research, a comparison of Naïve Bayes models will also be carried out to find out the best Naïve Bayes method opportunity model. Simultaneously, the Word2vec feature extraction method was chosen because Word2Vec can be used to improve model performance and increase the accuracy of sentiment classification. This research found that the Word2Vec Skip-Gram model outperformed the Word2Vec CBOW model. The best model obtained was the use of the Gaussian Naïve Bayes and Word2Vec Skip-Gram models with an accuracy score of 77.18%, precision 70.35%, recall 76.09%, and f1-score 73.10%.

Depression Detection of Users in Social-Media Twitter Using Decision Tree with Word2Vec

Social media is a medium or place on the Internet that allows users to be themselves. Interact, cooperate, share, and communicate with other users virtually. Not only do users share happy feelings, but they also share their emotions and sentiments towards a particular issue. Which sometimes makes users look depressed when they deliver it. Depression itself is the most commonly encountered mental illness, which makes the sufferer feel sad, lonely, inferior, and disconnected from the people around them. And even worse, depression can make the sufferer have suicidal thoughts. Therefore, we need to know whether the user indicated being depressed or not to prevent unwanted things by using a depression measurement tool scale called DASS 42 for data labeling. To detect depression, we can use the sufferer's Twitter account to take data based on tweets from the user and change the entire dataset to a vector using both the architectures of Word 2, Vec Skip-Gram, and CBOW. In this research, we utilize a decision tree to detect depression. The best results were obtained from the Word2Vec Skip-Gram model with a data ratio of 90:10 using the Gini criterion parameter and a maximum depth value of 20, resulting in an accuracy of 93% and a f1-score of 94%.

Comparison of the Word2vec Skipgram Model Method Linkaja Application Review using Bidirectional LSTM Algorithm and Support Vector Machine

Comparison of the Word2vec Skipgram Model Method Linkaja Application Review using Bidirectional LSTM Algorithm and Support Vector Machine

Arabic Idioms Detection by Utilizing Deep Learning and Transformer-based Models

Arabic language resources and natural language processing technologies have seen significant advancements in recent years. The detection of idiomatic expressions is a crucial problem in Arabic natural language processing. Unfortunately, there have been few advancements in this field due to the scarcity of datasets and computational models that still struggle to handle Arabic idiomatic phrases. Although idioms have significance in Arabic because of their distinctive relation to history and cultural knowledge, they also offer a language tool for conveying concepts and thoughts that may diverge from their literal interpretation. Hence, to accurately detect their incongruent language, a model requires the ability to interpret the idioms’ contextual meaning. To tackle this issue, this study entails comprehensive work on detecting Arabic idiomatic and literal statements. For that, we introduce a large Arabic idiomatic and literal statements. The dataset includes idioms from several Arabic idiomatic platforms balanced with literal statements from titles in news platforms with similar keywords found in the idioms. This study adopts four deep learning models, namely, CNN, LSTM, Bi-LSTM, and GRU trained with word embeddings, FastText and Skip-Gram Word2Vec, and three transformer-based models, BERT, RoBERTa, and DistilBERT were compiled and assessed via empirical evaluations. Our experiments demonstrated that transformer-based models outperformed deep learning models trained by word embedding to detect Arabic idioms and literal statements, reaching 97% in favor of DistilBERT. We also enhanced the accuracy through the ensemble stacking method, boosting it by 0.8 points. Furthermore, we propose a novel Arabic idiom interpreter that offers simple explanations for selected idiomatic contents within the literal text. Our work enriches the challenge of detecting incongruities found in similar figurative genres, specifically in the rich morphological language, Arabic.

Task-dependent Optimal Weight Combinations for Static Embeddings

A variety of NLP applications use word2vec skip-gram, GloVe, and fastText word embeddings. These models learn two sets of embedding vectors, but most practitioners use only one of them, or alternately an unweighted sum of both. This is the first study to systematically explore a range of linear combinations between the first and second embedding sets. We evaluate these combinations on a set of six NLP benchmarks including IR, POS-tagging, and sentence similarity. We show that the default embedding combinations are often suboptimal and demonstrate 1.0-8.0% improvements. Notably, GloVe’s default unweighted sum is its least effective combination across tasks. We provide a theoretical basis for weighting one set of embeddings more than the other according to the algorithm and task. We apply our findings to improve accuracy in applications of cross-lingual alignment and navigational knowledge by up to 15.2%.

Sentiment analysis using global vector and long short-term memory

Tweet sentiment analysis is a Deep Learning study that is beneficial for automatically determining public opinion on a certain topic. Using the Long Short-Term Memory (LSTM) algorithm, this paper aims to proposes a Twitter analysis technique that divides Tweets into two categories (positive and negative). The Global Vector (GloVe) word embedding score is used to rate many selected words as network input. GloVe converts words into vectors by building a corpus matrix. The GloVe outperforms its prior model, owing to its smaller vector and corpora sizes. GloVe has a higher accuracy than the model word embedding word2vec, Continuous Bag of Word(CBoW), and word2vec Skip-gram. The preprocessed term variation was conducted to test the performance of sentiment classification. The test results show that this proposed method has succeeded in classifying with the best results with an accuracy of 95.61%.

Abstractive Arabic Text Summarization Based on Deep Learning.

Text summarization (TS) is considered one of the most difficult tasks in natural language processing (NLP). It is one of the most important challenges that stand against the modern computer system's capabilities with all its new improvement. Many papers and research studies address this task in literature but are being carried out in extractive summarization, and few of them are being carried out in abstractive summarization, especially in the Arabic language due to its complexity. In this paper, an abstractive Arabic text summarization system is proposed, based on a sequence-to-sequence model. This model works through two components, encoder and decoder. Our aim is to develop the sequence-to-sequence model using several deep artificial neural networks to investigate which of them achieves the best performance. Different layers of Gated Recurrent Units (GRU), Long Short-Term Memory (LSTM), and Bidirectional Long Short-Term Memory (BiLSTM) have been used to develop the encoder and the decoder. In addition, the global attention mechanism has been used because it provides better results than the local attention mechanism. Furthermore, AraBERT preprocess has been applied in the data preprocessing stage that helps the model to understand the Arabic words and achieves state-of-the-art results. Moreover, a comparison between the skip-gram and the continuous bag of words (CBOW) word2Vec word embedding models has been made. We have built these models using the Keras library and run-on Google Colab Jupiter notebook to run seamlessly. Finally, the proposed system is evaluated through ROUGE-1, ROUGE-2, ROUGE-L, and BLEU evaluation metrics. The experimental results show that three layers of BiLSTM hidden states at the encoder achieve the best performance. In addition, our proposed system outperforms the other latest research studies. Also, the results show that abstractive summarization models that use the skip-gram word2Vec model outperform the models that use the CBOW word2Vec model.

Complex methods to assess complex problems: using Natural Language Processing to explore Health Related Qualitiy of Life in endometriosis

Background and Objectives: Endometriosis is a benign, although chronic painful condition that represents a major burden in women’s life, affecting all the domains of their health related quality of life (HRQL). The objective of this work is to conduct a Systematic Review of Systematic Reviews assessing HRQL of women with endometriosis, and to test a methodology based on Natural Language Processing (NLP) to assess factors associated with losses of HRQL linked with the most relevant clinical symptoms that women with endometriosis experience: pain and dyspareunia.Methods:We extracted free text from articles included in the Systematic Reviews with 'Pain' and 'dyspareunia' as keywords, and we used NLP to identify their occurrences, grammatical variants and meaningful context words. Our model used a word2vec skip-gram approach to predict context following our keywords.Results: Text data included 6,430 context words corresponding to 1,655 unique words in the 17 different systematic reviews that were included for review.Conclusions: The method we have tested using NLP has demonstrated to be valid to capture the complexity of issues that surround endometriosis and affects women´s HRQL. Among the diversity of symptoms that women with endometriosis experience, chronic pelvic pain has the highest impact on their HRQL. Pain affects their physical health, but also the social and functioning dimensions of women’s life. Painful intercourse has also a significant impact on the sexual life of women with endometriosis, and disturbing the emotional wellbeing of women. HRQL is also affected by the clinical complexities of endometriosis, the difficulties to reach to a diagnosis as well as to provide women with effective medical care. Different measurements, instruments and approaches have been proposed to measure HRQL in endometriosis, but it seems necessary the use of common instruments to facilitate comparisons across studies.

Word2vec Skip-Gram Dimensionality Selection via Sequential Normalized Maximum Likelihood.

In this paper, we propose a novel information criteria-based approach to select the dimensionality of the word2vec Skip-gram (SG). From the perspective of the probability theory, SG is considered as an implicit probability distribution estimation under the assumption that there exists a true contextual distribution among words. Therefore, we apply information criteria with the aim of selecting the best dimensionality so that the corresponding model can be as close as possible to the true distribution. We examine the following information criteria for the dimensionality selection problem: the Akaike’s Information Criterion (AIC), Bayesian Information Criterion (BIC), and Sequential Normalized Maximum Likelihood (SNML) criterion. SNML is the total codelength required for the sequential encoding of a data sequence on the basis of the minimum description length. The proposed approach is applied to both the original SG model and the SG Negative Sampling model to clarify the idea of using information criteria. Additionally, as the original SNML suffers from computational disadvantages, we introduce novel heuristics for its efficient computation. Moreover, we empirically demonstrate that SNML outperforms both BIC and AIC. In comparison with other evaluation methods for word embedding, the dimensionality selected by SNML is significantly closer to the optimal dimensionality obtained by word analogy or word similarity tasks.

Big Data Analytics Performance Enhancement For Covid-19 Data Using Machine Learning And Cloud

The exponential rise in software computing, internet and web-services has broadened the horizon for BigData that demands robust and highly efficient analytics system to serve timely and accurate distributed data support. The distributed frameworks with parallelized computing have been found key driving force behind the contemporary BigData analytics systems; however, the lack of optimal data pre-processing, feature sensitive computation and more importantly feature learning makes major at-hand solutions inferior, especially in terms of time and accuracy. Unlike major at hand methods employing machine learning for BigData analytics, in this paper the key emphasis was made on improving pre-processing, low-dimensional semantic feature extraction and lightweight improved machine learning based feature learning for BigData analytics. Noticeably, the proposed model hypothesizes that an analytics solution with BigData characteristics must have the potential to process humongous, heterogenous, unstructured and multi-dimensional features to yield time-efficient and accuracy analytical outputs. In this reference, we proposed a state-of-art new and robust BigData analytics model, specially designed for Spark distributed framework. To process analytical task our proposed model at first employs tokenization, followed by Word2Vec based semantic feature extraction using CBOW and N-Skip-Gram methods. Our proposed model was found more effective with Skip-Gram Word2Vec feature extraction. Simulation results with a publicly available COVID-19 data exhibited better performance than existing K-Means based MapReduce distributed data frameworks.

Sentiment Analysis On YouTube Comments Using Word2Vec and Random Forest

Purpose: This study aims to determine the accuracy of sentiment classification using the Random-Forest, and Word2Vec Skip-gram used for features extraction. Word2Vec is one of the effective methods that represent aspects of word meaning and, it helps to improve sentiment classification accuracy.Methodology: The research data consists of 31947 comments downloaded from the YouTube channel for the 2019 presidential election debate. The dataset consists of 23612 positive comments and 8335 negative comments. To avoid bias, we balance the amount of positive and negative data using oversampling. We use Skip-gram to extract features word. The Skip-gram will produce several features around the word the context (input word). Each of these features contains a weight. The feature weight of each comment is calculated by an average-based approach. Random Forest is used to building a sentiment classification model. Experiments were carried out several times with different epoch and window parameters. The performance of each model experiment was measured by cross-validation.Result: Experiments using epochs 1, 5, and 20 and window sizes of 3, 5, and 10, obtain the average accuracy of the model is 90.1% to 91%. However, the results of testing reach an accuracy between 88.77% and 89.05%. But accuracy of the model little bit lower than the accuracy model also was not significant. In the next experiment, it recommended using the number of epochs and the window size greater than twenty epochs and ten windows, so that accuracy increasing significantly.Value: The number of epoch and window sizes on the Skip-Gram affect accuracy. More and more epoch and window sizes affect increasing the accuracy.

WITHDRAWN: Improving feature engineering by fine tuning the parameters of Skip gram model

Word Embedding is the primary task in text mining, a subfield of Natural Language Processing. Feature Engineering also referred as feature extraction deals with building features out of existing data. This process involves constructing right features from the dataset and feeding relevant and best features into the Machine learning model for training. Feature engineering leads to greater improvements on many NLP tasks. The various aspects of feature engineering includes feature selection, feature addition, feature filtering and feature scaling. The Skip-gram architecture works better than other feature extraction models like CBOW, TD-IDF etc. This unsupervised learning technique used to find context word for a given target word. The skip-gram word2vec model outclasses the shortcomings with word-embedding matrix like loss of information from former cell, noisy information and over-fitting problems. The Skip gram algorithm trains the neural network and each time the output will be a softmax vector and it computes the cross entropy loss between output vector and true one-hot encoder. Skip-gram word2vec model shows good representation over finding out rare words or phrases. To improve the computational speed and accuracy, the parameters of skip gram model are fine tuned. Experimental results proves the efficiency of the model.

HOAX DETECTION IN INDONESIA LANGUAGE USING LONG SHORT-TERM MEMORY MODEL

Nowadays, the internet and social media grow fast. This condition has positive and negative effects on society. They become media to communicate and share information without limitation. However, many people use that easiness to broadcast news or information which do not accurate with the facts and gather people's opinions to get benefits or we called a hoax. Therefore, we need to develop a system that can detect hoax. This research uses the neural network method with Long Short-Term Memory (LSTM) model. The process of the LSTM model to identify hoax has several steps, including dataset collection, pre-processing data, word embedding using pre-trained Word2Vec, built the LSTM model. Detection model performance measurement using precision, recall, and f1-measure matrix. This research results the highest average score of precision is 0.819, recall is 0.809, and f1-measure is 0.807. These results obtained from the combination of the following parameters, i.e., Skip-gram Word2Vec Model Architecture, Hierarchical Softmax, 100 as vector dimension, max pooling, 0.5 as dropout value, and 0.001 of learning rate.

A deep neural network-based model for named entity recognition for Hindi language

The aim of this work is to develop efficient named entity recognition from the given text that in turn improves the performance of the systems that use natural language processing (NLP). The performance of IoT-based devices such as Alexa and Cortana significantly depends upon an efficient NLP model. To increase the capability of the smart IoT devices in comprehending the natural language, named entity recognition (NER) tools play an important role in these devices. In general, the NER is a two-step process that initially the proper nouns are identified from text and then classify them into predefined categories of entities such as person, location, measure, organization and time. NER is often performed as a subtask while processing natural languages which increases the accuracy level of a NLP task. In this paper, we propose deep neural network architecture for named entity recognition for the resource-scarce language Hindi, based on convolutional neural network (CNN), bidirectional long short-term memory (Bi-LSTM) neural network and conditional random field (CRF). In the proposed approach, initially, we use skip-gram word2vec model and GloVe model to represent words in semantic vectors which are further used in different deep neural network-based architectures. In the proposed approach, we use character- and word-level embedding to represent the text that includes information at fine-grained level. Due to the use of character-level embeddings, the proposed model is robust for the out-of-vocabulary words. Experimental results show that the combination of Bi-LSTM, CNN and CRF algorithms performs better as compared to the other baseline methods such as recurrent neural network, long short-term memory and Bi-LSTM individually.

Automatic classification of social media reports on violent incidents in South Africa using machine learning

With the growing amount of data available in the digital age, it has become increasingly important to use automated methods to extract useful information from data. One such application is the extraction of events from news sources for the purpose of a quantitative analysis that does not rely on someone needing to read through thousands of news articles. Overseas, projects such as the Integrated Crisis Early Warning System (ICEWS) monitor news stories and extract events using automated coding. However, not all violent events are reported in the news, and while monitoring only news agencies is sufficient for projects such as ICEWS which have a global focus, more news sources are required when assessing a local situation. We used WhatsApp as a news source to identify the occurrence of violent incidents in South Africa. Using machine learning, we have shown how violent incidents can be coded and recorded, allowing for a local level recording of these events over time. Our experimental results show good performance on both training and testing data sets using a logistic regression classifier with unigrams and Word2vec feature models. Future work will evaluate the inclusion of pre-trained word embedding for both Afrikaans and English words to improve the performance of the machine learning classifier.
 Significance:
 
 The logistic regression classifier using TFIDF unigram, CBOW and skip-gram Word2Vec models were successfully implemented to automatically analyse and classify WhatsApp messages from groups that share information on protests and mass violence in South Africa. At the time of publishing, messages were collected from 26 WhatsApp groups across South Africa and automatically classified on an hourly basis.

Learning variable-length representation of words

A standard word embedding algorithm, such as ‘word2vec’, embeds each word as a dense vector of a preset dimensionality, the components of which are learned by maximizing the likelihood of predicting the context around it. However, as an inherent linguistic phenomenon, it is evident that there is a varying degree of difficulty in identifying words from their contexts. This suggests that a variable granularity in word vector representation may be useful to obtain sparser and more compressed word representations, requiring less storage space. To that end, in this paper, we propose a word vector training algorithm that uses a variable number of components to represent words. Given a text collection of documents, our algorithm, similar to the skip-gram approach of word2vec, learns to predict the context of a word given the current instance of a word. However, in contrast to skip-gram, which uses a static number of dimensions for each word vector, we propose to dynamically increase the dimensionality as a stochastic function of the prediction error. Our experiments with standard test collections demonstrate that our word representation method is able to achieve comparable (and sometimes even better) effectiveness than skip-gram word2vec, using a significantly smaller number of parameters (achieving compression ratio of around 65%).

Application of Machine Learning and Word Embeddings in the Classification of Cancer Diagnosis Using Patient Anamnesis

Currently, one of the main challenges for information systems in healthcare is focused on support for health professionals regarding disease classifications. This work presents an innovative method for a recommendation system for the diagnosis of breast cancer using patient medical histories. In this proposal, techniques of natural language processing (NLP) were implemented on real datasets: one comprised 160, 560 medical histories of anonymous patients from a hospital in Chile for the following categories: breast cancer, cysts and nodules, other cancer, breast cancer surgeries and other diagnoses; and the other dataset was obtained from the MIMIC III dataset. With the application of word-embedding techniques, such as word2vec's skip-gram and BERT, and machine learning techniques, a recommendation system as a tool to support the physician's decision-making was implemented. The obtained results demonstrate that using word embeddings can define a good-quality recommendation system. The results of 20 experiments with 5-fold cross-validation for anamnesis written in Spanish yielded an F1 of 0.980 ± 0.0014 on the classification of `cancer' versus `not cancer' and 0.986 ± 0.0014 for `breast cancer' versus `other cancer'. Similar results were obtained with the MIMIC III dataset.

A hybrid structured deep neural network with Word2Vec for construction accident causes classification

According to the latest fatal work injury rates reported by the Bureau of Labors Statistics, construction sites remain the most hazardous workplaces. In the construction sector, fatality investigation summary reports are available for past accidents and by investigating such reports, valuable insights can be gained. In this study, text mining algorithms are explored for automatic construction accident causes classification. To be more specific, Word2Vec skip-gram model is utilized to learn word embedding from a domain-specific corpus and a hybrid structured deep neural network is proposed by incorporating the learned word embedding for accident reports classification. Dataset from Occupational Safety and Health Administration (OSHA) is employed in the experiment to evaluate the performance of the proposed approach. Besides, five baseline models: support vector machine (SVM), linear regression (LR), K-nearest neighbor (KNN), decision tree (DT), Naive Bayes (NB) are employed to compare with the proposed approach. Experiment results show that the proposed model achieves the highest average weighted F1 score among all models considered in this study. The result also proves the effectiveness of applying Word2Vec skip-gram algorithm for semantic information augmentation. As a result, robustness of the model is improved when classifying cases of low support values.

Character convolutions for Arabic Named Entity Recognition with Long Short-Term Memory Networks

Named Entity Recognition (NER) is a significant information extraction task since it is an important component of many natural language processing applications, such as Information Retrieval, Question Answering and Speech Recognition. The complexity and morphological richness of the Arabic language is the main reason why most existing Arabic NER systems rely strongly on hand-crafted feature engineering. In this paper, we propose to augment the existing LSTM neural tagging model for Arabic NER with a Convolutional Neural Network (CNN) for the extraction of relevant character-level features. By operating on the character-level, the proposed model is able to handle out-of-vocabulary words. Our results show that character CNN is able to outperform the previously used character-level Bi-directional Long Short-Term Memory Networks (BiLSTM) in many settings. Moreover, our observations indicate that CNNs tend to perform better than BiLSTM on relatively longer tokens. In addition, we conduct a comparison of four different pre-trained word vector models for Arabic NER and results show that a Skip-Gram Word2vec model, pre-trained on a subset of the Arabic Gigaword corpus, is generally sufficient to obtain acceptable Arabic NER performance.

From Motion Activity to Geo-Embeddings: Generating and Exploring Vector Representations of Locations, Traces and Visitors through Large-Scale Mobility Data

The rapid growth of positioning technology allows tracking motion between places, making trajectory recordings an important source of information about place connectivity, as they map the routes that people commonly perform. In this paper, we utilize users’ motion traces to construct a behavioral representation of places based on how people move between them, ignoring geographical coordinates and spatial proximity. Inspired by natural language processing techniques, we generate and explore vector representations of locations, traces and visitors, obtained through an unsupervised machine learning approach, which we generically named motion-to-vector (Mot2vec), trained on large-scale mobility data. The algorithm consists of two steps, the trajectory pre-processing and the Word2vec-based model building. First, mobility traces are converted into sequences of locations that unfold in fixed time steps; then, a Skip-gram Word2vec model is used to construct the location embeddings. Trace and visitor embeddings are finally created combining the location vectors belonging to each trace or visitor. Mot2vec provides a meaningful representation of locations, based on the motion behavior of users, defining a direct way of comparing locations’ connectivity and providing analogous similarity distributions for places of the same type. In addition, it defines a metric of similarity for traces and visitors beyond their spatial proximity and identifies common motion behaviors between different categories of people.