Word2vec Model Research Articles

NLP for SDGs: Measuring Corporate Alignment with the Sustainable Development Goals

This article uses advanced natural language processing (NLP) methods to identify companies that are aligned with the UN Sustainable Development Goals (SDGs) based on the text in their sustainability disclosures. Using the Corporate Social Responsibility (CSR) reports of Russell 1000 companies between 2010–2019, we apply a logistic classifier, support vector machines (SVM), and a fully-connected neural network to predict alignment with the SDGs. Specifically, we use word embeddings to augment dictionary-based input features, as well as the embeddings as features themselves, based on Word2Vec and Doc2Vec models to classify companies’ alignment with the SDGs over time. Notably, the Doc2Vec embedding inputs to the SVM classifier result in high average accuracy of above 80% for predicting alignment.

Public Reaction to Scientific Research via Twitter Sentiment Prediction

Abstract Purpose Social media users share their ideas, thoughts, and emotions with other users. However, it is not clear how online users would respond to new research outcomes. This study aims to predict the nature of the emotions expressed by Twitter users toward scientific publications. Additionally, we investigate what features of the research articles help in such prediction. Identifying the sentiments of research articles on social media will help scientists gauge a new societal impact of their research articles. Design/methodology/approach Several tools are used for sentiment analysis, so we applied five sentiment analysis tools to check which are suitable for capturing a tweet's sentiment value and decided to use NLTK VADER and TextBlob. We segregated the sentiment value into negative, positive, and neutral. We measure the mean and median of tweets’ sentiment value for research articles with more than one tweet. We next built machine learning models to predict the sentiments of tweets related to scientific publications and investigated the essential features that controlled the prediction models. Findings We found that the most important feature in all the models was the sentiment of the research article title followed by the author count. We observed that the tree-based models performed better than other classification models, with Random Forest achieving 89% accuracy for binary classification and 73% accuracy for three-label classification. Research limitations In this research, we used state-of-the-art sentiment analysis libraries. However, these libraries might vary at times in their sentiment prediction behavior. Tweet sentiment may be influenced by a multitude of circumstances and is not always immediately tied to the paper's details. In the future, we intend to broaden the scope of our research by employing word2vec models. Practical implications Many studies have focused on understanding the impact of science on scientists or how science communicators can improve their outcomes. Research in this area has relied on fewer and more limited measures, such as citations and user studies with small datasets. There is currently a critical need to find novel methods to quantify and evaluate the broader impact of research. This study will help scientists better comprehend the emotional impact of their work. Additionally, the value of understanding the public's interest and reactions helps science communicators identify effective ways to engage with the public and build positive connections between scientific communities and the public. Originality/value This study will extend work on public engagement with science, sociology of science, and computational social science. It will enable researchers to identify areas in which there is a gap between public and expert understanding and provide strategies by which this gap can be bridged.

A BiLSTM cardinality estimator in complex database systems based on attention mechanism

An excellent cardinality estimation can make the query optimiser produce a good execution plan. Although there are some studies on cardinality estimation, the prediction results of existing cardinality estimators are inaccurate and the query efficiency cannot be guaranteed as well. In particular, they are difficult to accurately obtain the complex relationships between multiple tables in complex database systems. When dealing with complex queries, the existing cardinality estimators cannot achieve good results. In this study, a novel cardinality estimator is proposed. It uses the core techniques with the BiLSTM network structure and adds the attention mechanism. First, the columns involved in the query statements in the training set are sampled and compressed into bitmaps. Then, the Word2vec model is used to embed the word vectors about the query statements. Finally, the BiLSTM network and attention mechanism are employed to deal with word vectors. The proposed model takes into consideration not only the correlation between tables but also the processing of complex predicates. Extensive experiments and the evaluation of BiLSTM-Attention Cardinality Estimator (BACE) on the IMDB datasets are conducted. The results show that the deep learning model can significantly improve the quality of cardinality estimation, which is a vital role in query optimisation for complex databases.

Methodology of Mediametric Information Analysis with the Use of Machine Learning Algorithms

The qualitative analysis of information messages and the assessment of publications on the Internet are becoming more urgent than ever. A large number of materials are published on the Internet on various events in the world; the nature of these publications can affect the political and social life of society. In order to ensure the safety of the population of the Russian Federation and meet the requirements of regulatory documents, a methodology for mediametric information analysis with the use of machine learning algorithms is proposed. Based on the results of research in this area, the main approaches to mediametric information analysis are determined. An approach is proposed for determining the sentiment of publications using the Word2Vec model and machine learning algorithms for natural language processing. A methodology is formulated that takes into account the technical features of a publication source and the existing methods of mediametric information analysis. On the basis of real information publications, the results of the implementation of the methodology of mediametric analysis and determination of the sentiment of messages are presented.

SENTIMENT ANALYSIS PROTOKOL KESEHATAN VIRUS CORONA DARI TWEET MENGGUNAKAN WORD2VEC MODEL DAN RECURRENT NEURAL NETWORK LEARNING

Sentiment analysis is a computational study of opinion from various opinions, which is part of the work that conducts a review related to the computational treatment of opinions, sentiments, and perceptions of the text. To solve various problems in sentiment analysis, needed a good text representation method. In this study, a deep learning analysis was carried out using the Recurrent Neural Network (RNN) method and the Word2Vec Model as word embedding in sentiment classification. The sentiment dataset used comes from user reviews on Twitter (tweets) on the health protocols implemented by the public from the government's appeal. The results showed that the RNN model using sigmoid activation resulted in the greatest accuracy of 66%. The training process in this test uses 10 epochs and 32 batch sizes so that the precision value for negative sentiment is 54% and for positive sentiment is 67%.

Performance Analysis of Sense Embeddings in Multilingual WSD Framework

Execution of Word Sense Disambiguation (WSD) is one of the difficult undertakings in the space of Natural language processing (NLP). Age of sense clarified corpus for multilingual WSD is far off for most languages regardless of whether assets are accessible. In this paper we propose a solo technique utilizing word and sense embeddings for working on the presentation of WSD frameworks utilizing untagged corpora and make two bags to be specific context bag and wiki sense bag to create the faculties with most noteworthy closeness. Wiki sense bag gives outer information to the framework needed to help the disambiguation exactness. We investigate Word2Vec model to produce the sense back and notice huge execution acquire for our dataset.

Sentiment analysis of Indonesian hotel reviews: from classical machine learning to deep learning

Currently, there are a large number of hotel reviews on the Internet that need to be evaluated to turn the data into practicable information. Deep learning has excellent capabilities for recognizing this type of data. With the advances in deep learning paradigms, many algorithms have been developed that can be used in sentiment analysis tasks. In this study, we aim to compare the performance of classical machine learning algorithms—logistic regression (LR), naïve Bayes (NB), and support vector machine (SVM) using the Word2Vec model in conjunction with deep learning algorithms such as a convolutional neural network (CNN) to classify hotel reviews on the Traveloka website into positive or negative classes. Both learning methods apply hyperparameter tuning to determine the parameters that produce the best model. Furthermore, the Word2Vec model parameters use the skip-gram model, hierarchical softmax evaluation, and the value of 100 vector dimensions. The highest average accuracy obtained was 98.08% by using the CNN with a dropout of 0.2, Tanh as convolution activation, softmax as output activation, and Adam as the optimizer. The findings from the study demonstrate that the integration of the Word2Vec model and the CNN model obtains significantly better accuracy than other classical machine learning methods.

Which environmental features contribute to positive and negative perceptions of urban parks? A cross-cultural comparison using online reviews and Natural Language Processing methods

Assessing the perceptions of urban parks and understanding the relationships between environmental features and park perceptions are critical to the design and management of urban parks. However, it is challenging to quantify park perceptions at a large spatial-temporal scale. In addition, little is known about which environmental features contribute to different perceptions in cross-cultural comparison. This study collected 21462 and 28398 online reviews from 2015 to 2020 and applied Natural Language Processing (NLP) methods to understand how people perceive urban parks and which environmental features contribute to positive and negative park perceptions in Shanghai and Brussels. We quantified park perceptions through sentiment analysis and transformed unstructured high-frequency words into structured environmental feature clusters corresponding to positive and negative perceptions using the word2vec models and machine learning. In addition, we created Chinese-based and English-based environmental feature lexicons for standardized measurement of the perceived frequencies of different environmental features and their associations with park perceptions. The results of our sentiment analysis showed that urban parks were generally perceived positively in both cities. We also identified 15 environmental feature clusters that were related to positive and negative perceptions. Beta regression results showed that the perceived frequencies of water bodies and colorfulness could be used as positive predictors for park perceptions in Shanghai. In contrast, the perceived frequency of tranquility had a positive impact on park perceptions in Brussels. We found that the perceived frequency of crowdedness was also positively associated with park perceptions in Brussels, suggesting the potential use conflicts between different user groups in urban parks. This study can provide insights into park design and management from cross-cultural comparison through systematic identification and standardized assessment of landscape preferences using online reviews and NLP methods.

A Hybrid Method of Long Short-Term Memory and Auto-Encoder Architectures for Sarcasm Detection

Sarcasm detection is considered one of the most challenging tasks in sentiment analysis and opinion mining applications in the social media. Sarcasm identification is therefore essential for a good public opinion decision. There are some studies on sarcasm detection that apply standard word2vec model and have shown great performance with word-level analysis. However, once a sequence of terms is being tackled, the performance drops. This is because averaging the embedding of each term in a sentence to get the general embedding would discard the important embedding of some terms. LSTM showed significant improvement in terms of document embedding. However, within the classification LSTM requires adding additional information in order to precisely classify the document into sarcasm or not. This study aims to propose two technique based on LSTM and Auto-Encoder for improving the sarcasm detection. A benchmark dataset has been used in the experiments along with several pre-processing operations that have been applied. These include stop word removal, tokenization and special character removal with LSTM which can be represented by configuring the document embedding and using Auto-Encoder the classifier that was trained on the proposed LSTM. Results showed that the proposed LSTM with Auto-Encoder outperformed the baseline by achieving 84% of f-measure for the dataset. The main reason behind the superiority is that the proposed auto encoder is processing the document embedding as input and attempt to output the same embedding vector. This will enable the architecture to learn the interesting embedding that have significant impact on sarcasm polarity.

Off-Topic Detection of Business English Essay Based on Deep Learning Model

The automatic scoring system of business English essay has been widely used in the field of education, and it is indispensable for the task of off-topic detection of essay. Most of the traditional off-topic detection methods convert text into vector representation of vector space and then calculate the similarity between the text and the correct text to get the off-topic result. However, those methods only focus on the structure of the text, but ignore the semantic association. In addition, the traditional detection method has a low off-topic detection effect for essays with high divergence. In view of the above problems, this paper proposes an off-topic detection method for business English essay based on the deep learning model. Firstly, the word2vec model is used to represent words in sentences as word vectors. And, LDA is used to extract the vector of topic and text, respectively. Then, word vector and topic word vector are spliced together as the input of the convolutional neural network (CNN). CNN is used to extract and screen the features of sentences and perform similarity calculation. When the similarity is less than the threshold, the paper also maps the topic and the subject words in the coupling space and calculates their relevance. Finally, unsupervised off-topic detection is realized by the clustering method. The experimental results show that the off-topic detection method based on the deep learning model can improve the detection accuracy of both the essays with low divergence and the essays with high divergence to a certain extent, especially the essays with high divergence.

Introducing Various Semantic Models for Amharic: Experimentation and Evaluation with Multiple Tasks and Datasets

The availability of different pre-trained semantic models has enabled the quick development of machine learning components for downstream applications. However, even if texts are abundant for low-resource languages, there are very few semantic models publicly available. Most of the publicly available pre-trained models are usually built as a multilingual version of semantic models that will not fit well with the need for low-resource languages. We introduce different semantic models for Amharic, a morphologically complex Ethio-Semitic language. After we investigate the publicly available pre-trained semantic models, we fine-tune two pre-trained models and train seven new different models. The models include Word2Vec embeddings, distributional thesaurus (DT), BERT-like contextual embeddings, and DT embeddings obtained via network embedding algorithms. Moreover, we employ these models for different NLP tasks and study their impact. We find that newly-trained models perform better than pre-trained multilingual models. Furthermore, models based on contextual embeddings from FLAIR and RoBERTa perform better than word2Vec models for the NER and POS tagging tasks. DT-based network embeddings are suitable for the sentiment classification task. We publicly release all the semantic models, machine learning components, and several benchmark datasets such as NER, POS tagging, sentiment classification, as well as Amharic versions of WordSim353 and SimLex999.

NLP-MeTaxa: A Natural Language Processing Approach for Metagenomic Taxonomic Binning Based on Deep Learning

Background: Metagenomics is the study of genomic content in mass from an environment of interest such as the human gut or soil. Taxonomy is one of the most important fields of metagenomics, which is the science of defining and naming groups of microbial organisms that share the same characteristics. The problem of taxonomy classification is the identification and quantification of microbial species or higher-level taxa sampled by high throughput sequencing. Objective: Although many methods exist to deal with the taxonomic classification problem, assignment to low taxonomic ranks remains an important challenge for binning methods as is scalability to Gbsized datasets generated with deep sequencing techniques. Methods: In this paper, we introduce NLP-MeTaxa, a novel composition-based method for taxonomic binning, which relies on the use of words embeddings and deep learning architecture. The new proposed approach is word-based, where the metagenomic DNA fragments are processed as a set of overlapping words by using the word2vec model to vectorize them in order to feed the deep learning model. NLP-MeTaxa output is visualized as NCBI taxonomy tree, this representation helps to show the connection between the predicted taxonomic identifiers. NLP-MeTaxa was trained on large-scale data from the NCBI RefSeq, more than 14,000 complete microbial genomes. The NLP-MeTaxa code is available at the website: https://github.com/padriba/NLP_MeTaxa/. Results: We evaluated NLP-MeTaxa with a real and simulated metagenomic dataset and compared our results to other tools' results. The experimental results have shown that our method outperforms the other methods especially for the classification of low-ranking taxonomic class such as species and genus. Conclusion: In summary, our new method might provide novel insight for understanding the microbial community through the identification of the organisms it might contain.

A Cluster Analysis Method of Software Development Activities Based on Event Log

Background: Event log data generated in the software development process contains historical information and future trends in software development activities. The mining and analysis of event log data contribute to identify and discover software development activities and provide effective support for software development process mining and modeling. Methods: Firstly, a deep learning model (Word2vec) was used for feature extraction and vectorization of software development process event logs. Then, the K-means clustering algorithm and measure of silhouette coefficient and intra-cluster SSE were used for clustering and clustering effect evaluation of vectorized software development process event logs. Results: This paper obtained the mapping relationship between software development activities and events, and realized the identification and discovery of software development activities. Conclusion: Two practical software development projects (jEdit and Argouml) are given to prove the feasibility, rationality and effectiveness of our proposed method. This work provides effective support for software development process mining and software development behavior guidance.

Analysis of social media data for public emotion on the Wuhan lockdown event during the COVID-19 pandemic

BackgroundWith outbreaks of COVID-19 around the world, lockdown restrictions are routinely imposed to limit the spread of the virus. During periods of lockdown, social media has become the main channel for citizens to exchange information with others. Public emotions are being generated and shared rapidly online with citizens using internet platforms to reduce anxiety and stress, and stay connected while isolated. ObjectivesThis study aims to explore the regularity of emotional evolution by examining public emotions expressed in online discussions about the Wuhan lockdown event in January 2020. MethodsData related to the Wuhan lockdown was collected from Sina Weibo by web crawler. In this study, the Ortony, Clore, and Collins (OCC) model, Word2Vec, and Bi-directional Long Short-Term Memory model were employed to determine emotional types, train vectorization of words, and identify each text emotion for the training set. Latent Dirichlet Allocation models were also employed to mine the various topic categories, while topic emotional evolution was visualized. ResultsSeven types of emotions and four phases were categorized to describe emotional evolution on the Wuhan lockdown event. The study found that negative emotions such as blame and fear dominated in the early days, and public attitudes towards the lockdown gradually alleviated and reached a balance as the situation improved. Emotional expression about Wuhan lockdown event were significantly related to users’ gender, location, and whether or not their account was verified. There were statistically significant correlations between different emotions within the subtle emotional categories. In addition, the evolution of emotions presented a different path due to different topics. ConclusionsMultiple emotional categories were determined in our study, providing a detailed and explainable emotion analysis to explored emotional appeal of citizen. The public emotions were gradually easing related to the Wuhan lockdown event, there yet exists regional discrimination and post-traumatic stress disorder in this process, which would lead us to pay continuous attention to citizens lives and psychological status post-pandemic. In addition, this study provided an appropriate method and reference case for the government's public opinion control and emotional appeasement.

NLP Methods for Extraction of Symptoms from Unstructured Data for Use in Prognostic COVID-19 Analytic Models

Statistical modeling of outcomes based on a patient's presenting symptoms (symptomatology) can help deliver high quality care and allocate essential resources, which is especially important during the COVID-19 pandemic. Patient symptoms are typically found in unstructured notes, and thus not readily available for clinical decision making. In an attempt to fill this gap, this study compared two methods for symptom extraction from Emergency Department (ED) admission notes. Both methods utilized a lexicon derived by expanding The Center for Disease Control and Prevention's (CDC) Symptoms of Coronavirus list. The first method utilized a word2vec model to expand the lexicon using a dictionary mapping to the Uni ed Medical Language System (UMLS). The second method utilized the expanded lexicon as a rule-based gazetteer and the UMLS. These methods were evaluated against a manually annotated reference (f1-score of 0.87 for UMLS-based ensemble; and 0.85 for rule-based gazetteer with UMLS). Through analyses of associations of extracted symptoms used as features against various outcomes, salient risks among the population of COVID-19 patients, including increased risk of in-hospital mortality (OR 1.85, p-value < 0.001), were identified for patients presenting with dyspnea. Disparities between English and non-English speaking patients were also identified, the most salient being a concerning finding of opposing risk signals between fatigue and in-hospital mortality (non-English: OR 1.95, p-value = 0.02; English: OR 0.63, p-value = 0.01). While use of symptomatology for modeling of outcomes is not unique, unlike previous studies this study showed that models built using symptoms with the outcome of in-hospital mortality were not significantly different from models using data collected during an in-patient encounter (AUC of 0.9 with 95% CI of [0.88, 0.91] using only vital signs; AUC of 0.87 with 95% CI of [0.85, 0.88] using only symptoms). These findings indicate that prognostic models based on symptomatology could aid in extending COVID-19 patient care through telemedicine, replacing the need for in-person options. The methods presented in this study have potential for use in development of symptomatology-based models for other diseases, including for the study of Post-Acute Sequelae of COVID-19 (PASC).

Performance comparison of TF-IDF and Word2Vec models for emotion text classification

Emotion is the human feeling when communicating with other humans or reaction to everyday events. Emotion classification is needed to recognize human emotions from text. This study compare the performance of the TF-IDF and Word2Vec models to represent features in the emotional text classification. We use the support vector machine (SVM) and Multinomial Naïve Bayes (MNB) methods for classification of emotional text on commuter line and transjakarta tweet data. The emotion classification in this study has two steps. The first step classifies data that contain emotion or no emotion. The second step classifies data that contain emotions into five types of emotions i.e. happy, angry, sad, scared, and surprised. This study used three scenarios, namely SVM with TF-IDF, SVM with Word2Vec, and MNB with TF-IDF. The SVM with TF-IDF method generate the highest accuracy compared to other methods in the first dan second steps classification, then followed by the MNB with TF-IDF, and the last is SVM with Word2Vec. Then, the evaluation using precision, recall, and F1-measure results that the SVM with TF-IDF provides the best overall method. This study shows TF-IDF modeling has better performance than Word2Vec modeling and this study improves classification performance results compared to previous studies.

An Empirical Study on Knowledge Aggregation in Academic Virtual Community Based on Deep Learning

Academic virtual community provides an environment for users to exchange knowledge, so it gathers a large amount of knowledge resources and presents a trend of rapid and disorderly growth. We learn how to organize the scattered and disordered knowledge of network community effectively and provide personalized service for users. We focus on analyzing the knowledge association among titles in an all-round way based on deep learning, so as to realize effective knowledge aggregation in academic virtual community. We take ResearchGate (RG) “online community” resources as an example and use Word2Vec model to realize deep knowledge aggregation. Then, principal component analysis (PCA) is used to verify its scientificity, and Wide & Deep learning model is used to verify its running effect. The empirical results show that the knowledge aggregation system of “online community” works well and has scientific rationality.

Query bot for retrieving patients’ clinical history: A COVID-19 use-case

ObjectiveWith increasing patient complexity whose data are stored in fragmented health information systems, automated and time-efficient ways of gathering important information from the patients' medical history are needed for effective clinical decision making. Using COVID-19 as a case study, we developed a query-bot information retrieval system with user-feedback to allow clinicians to ask natural questions to retrieve data from patient notes. Materials and methodsWe applied clinicalBERT, a pre-trained contextual language model, to our dataset of patient notes to obtain sentence embeddings, using K-Means to reduce computation time for real-time interaction. Rocchio algorithm was then employed to incorporate user-feedback and improve retrieval performance. ResultsIn an iterative feedback loop experiment, MAP for final iteration was 0.93/0.94 as compared to initial MAP of 0.66/0.52 for generic and 1./1. compared to 0.79/0.83 for COVID-19 specific queries confirming that contextual model handles the ambiguity in natural language queries and feedback helps to improve retrieval performance. User-in-loop experiment also outperformed the automated pseudo relevance feedback method. Moreover, the null hypothesis which assumes identical precision between initial retrieval and relevance feedback was rejected with high statistical significance (p ≪ 0.05). Compared to Word2Vec, TF-IDF and bioBERT models, clinicalBERT works optimally considering the balance between response precision and user-feedback. DiscussionOur model works well for generic as well as COVID-19 specific queries. However, some generic queries are not answered as well as others because clustering reduces query performance and vague relations between queries and sentences are considered non-relevant. We also tested our model for queries with the same meaning but different expressions and demonstrated that these query variations yielded similar performance after incorporation of user-feedback. ConclusionIn conclusion, we develop an NLP-based query-bot that handles synonyms and natural language ambiguity in order to retrieve relevant information from the patient chart. User-feedback is critical to improve model performance.

Text Analysis and Visualization Research on the Hetu Dangse During the Qing Dynasty of China

In traditional historical research, interpreting historical documents subjectively and manually causes problems such as one-sided understanding, selective analysis, and one-way knowledge connection. In this study, we aim to use machine learning to automatically analyze and explore historical documents from a text analysis and visualization perspective. This technology solves the problem of large-scale historical data analysis that is difficult for humans to read and intuitively understand. In this study, we use the historical documents of the Qing Dynasty Hetu Dangse,preserved in the Archives of Liaoning Province, as data analysis samples. China’s Hetu Dangse is the largest Qing Dynasty thematic archive with Manchu and Chinese characters in the world. Through word frequency analysis, correlation analysis, co-word clustering, word2vec model, and SVM (Support Vector Machines) algorithms, we visualize historical documents, reveal the relationships between functions of the government departments in the Shengjing area of the Qing Dynasty, achieve the automatic classification of historical archives, improve the efficient use of historical materials as well as build connections between historical knowledge. Through this, archivists can be guided practically in historical materials’ management and compilation.

Sino-Japanese Relations from the Perspective of Chinese Mainstream Media During the Pandemic: Based on Word2vec and DTM Model

We conducted sentiment classification for news reports during the COVID-19 pandemic on Sino-Japanese relations from Chinese and Japanese media. Based on the Word2vec and Dynamic Topic Models, we analyzed the macroscopic dimension of theme distribution and the semantic change process of “Sino-Japanese” related words. The results indicate that Chinese media reports are mainly neutral, but Japanese ones are mainly negative. The meaning of the terms related to “Sino-Japanese” has not changed significantly, but there are minor differences in the content of reports on the same subject in different periods. During Suga’s administration, the overall stability of Sino-Japanese relations and the continued advancement of pragmatic cooperation are expected to continue.