Word2vec Model Research Articles

Ancient Text Translation Model Optimized with GujiBERT and Entropy-SkipBERT

To cope with the challenges posed by the complex linguistic structure and lexical polysemy in ancient texts, this study proposes a two-stage translation model. First, we combine GujiBERT, GCN, and LSTM to categorize ancient texts into historical and non-historical categories. This categorization lays the foundation for the subsequent translation task. To improve the efficiency of word vector generation and reduce the limitations of the traditional Word2Vec model, we integrated the entropy weight method in the hopping lattice training process and spliced the word vectors with GujiBERT. This improved method improves the efficiency of word vector generation and enhances the model’s ability to accurately represent lexical polysemy and grammatical structure in ancient documents through dependency weighting. In training the translation model, we used a different dataset for each text category, significantly improving the translation accuracy. Experimental results show that our categorization model improves the accuracy by 5% compared to GujiBERT. In contrast, the Entropy-SkipBERT improves the BLEU scores by 0.7 and 0.4 on historical and non-historical datasets. Ultimately, the proposed two-stage model improves the BLEU scores by 2.7 over the baseline model.

Sustainable sentiment analysis on E-commerce platforms using a weighted parallel hybrid deep learning approach for smart cities applications

Sentiment analysis (SA) of several user evaluations on e-commerce platforms can be used to increase customer happiness. This method automatically extracts and identifies subjective data from product evaluations using natural language processing (NLP) and machine learning (ML) methods. These statistics may eventually reveal information on the favourable, neutral, or negative attitudes of the consumer base. Due to its capacity to grasp the complex links between words and phrases in reviews as well as the emotions they imply, deep learning (DL) is very useful for SA tasks. A unique approach termed Weighted Parallel Hybrid Deep Learning-based Sentiment Analysis on E-Commerce Product Reviews (WPHDL-SAEPR) is introduced by the proposed system. Accurately distinguishing between distinct sentiments found in online store reviews is the aim of the WPHDL-SAEPR technique. Additional data pre-processing processes are implemented within the WPHDL-SAEPR architecture to guarantee compatibility. Words are embedded into the paper using the word2vec model, while sentiment is classified using the WPHDL model. The Restricted Boltzmann Machine (RBM) and Singular Value Decomposition (SVD) models are combined in this model. The results of the WPHDL-SAEPR approach’s simulation were assessed using a consumer review database, with the results being emphasized at each stage.

The construction of a Chinese fine-grained sentiment dictionary for Chinese domestic investors (CN-FSD) and its application

Stock bar comment texts are used to forecast stock market trends because they contain sentiment data provided by investors. However, previous research has relied on generic or polar sentiment dictionaries to construct sentiment indicators, which has limited progress in this field. This paper crawls nearly 1.8 million stock bar comment texts from the Eastmoney (www.eastmoney.com) Stock Bar Forum to perform sentiment analysis in this specialized area of the financial market. It reclassifies existing Chinese sentiment dictionaries, and utilizes Word2Vec and CNN models and word frequency statistics with a penalty mechanism to extract polar sentiment words. Additionally, the improved PMI (Pointwise Mutual Information) algorithm was utilized for fine-grained segmentation of negative emotion words. Finally, a fine-grained sentiment dictionary (four categories: happy, anger, sad and fear; with a total of 6155 words) was established for Chinese domestic investors (denoted by CN-FSD). We found that the improved PMI algorithm increased accuracy by 14%. In addition, we introduce the investor sentiment quantification indicators from two aspects, i.e., fine-grained sentiments and on the emotion theory dimension of psychology from the valence and arousal dimensions. This is done to verify the validity and robustness of the constructed dictionary. The case study from 2019 to 2022 discovered that under the influence of extreme occurrences like COVID-19, the fine-grained sentiment indicators constructed in this study are found to be significantly more strongly correlated with market trend indicators like index return and trading volume than the existing polar or fine-grained sentiment dictionaries. ‘fear’ emotions in fine-grained emotions dominate the stock market. Just as this case study emphasize the importance of ‘fear’ sentiment, studying the indicators of high arousal sentiment is also important. This case study further validates the efficacy of CN-FSD.

AI-Generated Spam Review Detection Framework with Deep Learning Algorithms and Natural Language Processing

Spam reviews pose a significant challenge to the integrity of online platforms, misleading consumers and undermining the credibility of genuine feedback. This paper introduces an innovative AI-generated spam review detection framework that leverages Deep Learning algorithms and Natural Language Processing (NLP) techniques to identify and mitigate spam reviews effectively. Our framework utilizes multiple Deep Learning models, including Convolutional Neural Networks (CNNs), Long Short-Term Memory (LSTM) networks, Gated Recurrent Unit (GRU), and Bidirectional LSTM (BiLSTM), to capture intricate patterns in textual data. The system processes and analyzes large volumes of review content to detect deceptive patterns by utilizing advanced NLP and text embedding techniques such as One-Hot Encoding, Word2Vec, and Term Frequency-Inverse Document Frequency (TF-IDF). By combining three embedding techniques with four Deep Learning algorithms, a total of twelve exhaustive experiments were conducted to detect AI-generated spam reviews. The experimental results demonstrate that our approach outperforms the traditional machine learning models, offering a robust solution for ensuring the authenticity of online reviews. Among the models evaluated, those employing Word2Vec embeddings, particularly the BiLSTM_Word2Vec model, exhibited the strongest performance. The BiLSTM model with Word2Vec achieved the highest performance, with an exceptional accuracy of 98.46%, a precision of 0.98, a recall of 0.97, and an F1-score of 0.98, reflecting a near-perfect balance between precision and recall. Its high F2-score (0.9810) and F0.5-score (0.9857) further highlight its effectiveness in accurately detecting AI-generated spam while minimizing false positives, making it the most reliable option for this task. Similarly, the Word2Vec-based LSTM model also performed exceptionally well, with an accuracy of 97.58%, a precision of 0.97, a recall of 0.96, and an F1-score of 0.97. The CNN model with Word2Vec similarly delivered strong results, achieving an accuracy of 97.61%, a precision of 0.97, a recall of 0.96, and an F1-score of 0.97. This study is unique in its focus on detecting spam reviews specifically generated by AI-based tools rather than solely detecting spam reviews or AI-generated text. This research contributes to the field of spam detection by offering a scalable, efficient, and accurate framework that can be integrated into various online platforms, enhancing user trust and the decision-making processes.

Preservation of emotional context in tweet embeddings on social networking sites

In communication, emotional information is crucial, yet its preservation in tweet embeddings remains a challenge. This study aims to address this gap by exploring three distinct methods for generating embedding vectors of tweets: word2vec models, pre-trained BERT models, and fine-tuned BERT models. We conducted an analysis to assess the degree to which emotional information is conserved in the resulting embedding vectors. Our findings indicate that the fine-tuned BERT model exhibits a higher level of preservation of emotional information compared to other methods. These results underscore the importance of utilizing advanced natural language processing techniques for preserving emotional context in text data, with potential implications for enhancing sentiment analysis and understanding human communication in social media contexts.

Unveiling Similarities in the Code of Life: A Detailed Exploration of DNA Sequence Matching Algorithm

Identifying similar DNA sequences is crucial in various biological research endeavors. This paper delves into the intricate workings of a specific algorithm designed for this purpose. We provide a systematic explanation, exploring how the algorithm handles user input, reads stored DNA sequences, utilizes the Word2Vec model for vector representation, and calculates sequence similarity using diverse metrics like Cosine Similarity and Neutrosophic Distance. Additionally, the paper explores the incorporation of neutrosophic values to account for uncertainty in the comparisons. Finally, we discuss the extraction of results, including matched sequences, similarity scores, and accuracy measures. This in-depth exploration provides a clear understanding of the algorithm's capabilities and fosters its effective application in DNA sequence analysis.

Exploring Deep Learning Models for Lyric Generation and Addressing Biases in Word Embeddings

The aim of this project is to explore the performance of different model architectures (such as RNN, LSTM, GRU) by generating lyrics using deep learning models, and to use the Word2Vec model for distributed semantic analysis to understand semantic phenomena and potential biases in word embedding models. The experimental results show that LSTM and GRU perform better than traditional RNN models when processing long sequence data. In addition, by analyzing word embeddings, we revealed potential gender and racial biases and proposed corresponding solutions.

Green public procurement and corporate environmental performance: An empirical analysis based on data from green procurement contracts

Green public procurement (GPP) is regarded as a vital tool for achieving energy conservation and emission reduction. This policy aims to guide enterprises towards cleaner production by creating market demand for green products on the demand side. However, it remains unclear whether GPP can genuinely improve corporate environmental performance. This study reviews relevant policy documents and constructs a wordlist of “green procurement contract" using the Word2vec model. By employing text analysis, we identify public procurement contracts that meet green standards. Subsequently, we match these contracts with data from China's A-share-listed companies to empirically test the impact of GPP contracts on corporate environmental performance. The findings reveal that GPP significantly enhances corporate environmental performance, primarily through conceptual governance effects, source governance effects, terminal governance effects, and supervision governance effects. Heterogeneity analysis indicates that this environmental incentive effect is particularly significant among non-state-owned enterprises and small and medium-sized enterprises. Additionally, the effect is more pronounced in regions with high government environmental awareness and moderate environmental enforcement intensity. Further analysis shows that GPP also improves corporate social responsibility performance and promotes corporate ESG performance, thereby fostering sustainable development. Moreover, the study points out that increasing the transparency of information disclosure, enhancing procurement efficiency, and maintaining the stability of procurement relationships all contribute to strengthening the environmental performance incentive effect of GPP. This article helps to deepen the understanding of the “proactive government” promoting the development of green economy and also provides theoretical reference for the breakthrough of corporate green transformation under the target of “dual carbon.”

ANALISIS SENTIMEN KEPUASAN PELAYANAN TRANSPORTASI ONLINE GOJEK MENGGUNAKAN ALGORITMA EXTREME LEARNING MACHINE

With the rapid advancement of technology, online transportation has become the main solution for many people in Indonesia to travel easily and efficiently. Companies such as GOJEK are constantly innovating to improve their services, resulting in many responses and reviews from users. This research aims to analyze customer satisfaction with these online transportation services by analyzing the sentiment of user opinions on the Twitter platform. Sentiment analysis plays a very important role in decision making by classifying user reviews. Data was retrieved through a crawling process using specific keywords related to each service. The data preprocessing process includes case folding, tokenizing, normalization, stemming, filtering, and convert negation. This aims to clean and prepare the data so that it can be processed using the algorithm better. This process includes removing irrelevant elements from the text data, converting the text into a consistent or more standardized form, reducing the number of features in the data by stemming, and converting the text into numbers or vectors so that it can be processed by the algorithm. Feature extraction is performed using the Word2Vec model to convert text into a numerical vector representation that can later be processed by ELM. Converts words into numeric vectors in a high-dimensional space, where words that have the same context in the text are close to each other in that space. The ELM (Extreme Learning Machine) algorithm is used as a classification model due to its high training speed and good generalization ability. Model evaluation is done using confusion matrix which measures classification performance through accuracy, precision, recall matrix. The results of this study show that the ELM algorithm with Word2Vec feature extraction is able to classify user sentiment with a high level of accuracy. This research provides insight into user satisfaction with online transportation services and can be a reference for companies to improve their service quality

Exploiting web content semantic features to detect web robots from weblogs

Nowadays, web robots are predominantly used for auto-accessing web content, sharing almost one-third of the total web traffic and often posing threats to various web applications’ security, privacy, and performance. Detecting these robots is essential, and both online and offline methods are employed. One popular offline method is the use of weblog feature-based automated learning. However, this method alone cannot accurately identify web robots that continuously evolve and camouflage. Web content features combined with weblog features are used to detect such robots based on the assumption that human users exhibit specific interests while robots randomly navigate web pages. State-of-the-art web content-based feature methods lack the ability to generate coherent topics, which can confound the performance of classification models. Therefore, we propose a new content semantic feature extraction method that uses the LDA2Vec topic model, combining the strengths of LDA and the Word2Vec model to produce more semantically coherent topics by exploiting website content for a web session. To effectively detect web robots, web resource content semantic features are combined with log-based features in the proposed web robot detection approach. The proposed approach is evaluated in an e-commerce website access logs and content data. The F-score, balanced accuracy, G-mean, and Jaccard similarity are used for performance measures, and the coherence score metric is used to determine the number of topics for a session. Experimental results demonstrate that a combination of weblogs and content semantic features is effective in web robot detection.

Optimizing word embeddings for small datasets: a case study on patient portal messages from breast cancer patients

Patient portal messages often relate to specific clinical phenomena (e.g., patients undergoing treatment for breast cancer) and, as a result, have received increasing attention in biomedical research. These messages require natural language processing and, while word embedding models, such as word2vec, have the potential to extract meaningful signals from text, they are not readily applicable to patient portal messages. This is because embedding models typically require millions of training samples to sufficiently represent semantics, while the volume of patient portal messages associated with a particular clinical phenomenon is often relatively small. We introduce a novel adaptation of the word2vec model, PK-word2vec (where PK stands for prior knowledge), for small-scale messages. PK-word2vec incorporates the most similar terms for medical words (including problems, treatments, and tests) and non-medical words from two pre-trained embedding models as prior knowledge to improve the training process. We applied PK-word2vec in a case study of patient portal messages in the Vanderbilt University Medical Center electric health record system sent by patients diagnosed with breast cancer from December 2004 to November 2017. We evaluated the model through a set of 1000 tasks, each of which compared the relevance of a given word to a group of the five most similar words generated by PK-word2vec and a group of the five most similar words generated by the standard word2vec model. We recruited 200 Amazon Mechanical Turk (AMT) workers and 7 medical students to perform the tasks. The dataset was composed of 1389 patient records and included 137,554 messages with 10,683 unique words. Prior knowledge was available for 7981 non-medical and 1116 medical words. In over 90% of the tasks, both reviewers indicated PK-word2vec generated more similar words than standard word2vec (p = 0.01).The difference in the evaluation by AMT workers versus medical students was negligible for all comparisons of tasks’ choices between the two groups of reviewers (p=0.774\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$${\ ext{p}} = 0.774$$\\end{document} under a paired t-test). PK-word2vec can effectively learn word representations from a small message corpus, marking a significant advancement in processing patient portal messages.

Advancing equity in breast cancer care: natural language processing for analysing treatment outcomes in under-represented populations

ObjectiveThe study aimed to develop natural language processing (NLP) algorithms to automate extracting patient-centred breast cancer treatment outcomes from clinical notes in electronic health records (EHRs), particularly for women from...

Semantic Similarity Measure Using a Combination of Word2Vec and WordNet Models

Semantic Similarity Measure Using a Combination of Word2Vec and WordNet Models

Алгоритмы поиска вербальных маркеров идентичности в современном научном дискурсе

The article is devoted to the study of identity verbalization specifics via Data Mining. The research material consists of English texts from Internet scientific repositories and e-libraries devoted to various concepts of youth identity. A methodology based on the use of modern natural language processing and machine learning tools was developed and tested as part of the research. The analysis was carried out using the Natural Language Toolkit library for tokenization and POS-tagging procedures for calculating the frequency of tokens from the «identity» environment. Word Embeddings, pre-trained Word2Vec model and K-means algorithm were used for the subsequent analysis and clustering of words based on their semantic proximity. Gensim library and Scikit-learn library were used to work with the Word2Vec model. As a result, it was proved that in English scientific discourse young person’s identity is verbalized within 9 semantic categories: behavior, communities, communication, education, identity, language, practice, complexity, science, the most common of which are education (33%), language (21%) and communities (18%). N-grams analysis made it possible to identify semantic fields, establish their attributes, and evaluate texts’ similarity, which provided the most accurate vector space search for semantically close n-grams. Optimization made it possible to establish a similarity measure to rank phrases according to the query, as well as assign each n-gram a certain ranking weight. Improvements can be achieved by adding statistical word weighting, such as TF-IDF. The proposed system is capable of searching in a large text array of related phrases with a similar meaning.

Research on the evolution of cross-platform online public opinion for public health emergencies considering stakeholders.

To explore the different processes of the themes and emotional evolution of various stakeholders in the network public opinion of sudden public health emergencies at different stages of the public opinion evolution lifecycle. This paper proposes a cross-platform analysis method for online public opinion during the public health emergencies based on stakeholders. Firstly, data from multiple platforms are collected and integrated. Secondly, stakeholders are categorized and the stages of public opinion evolution are determined based on stakeholder theory and lifecycle theory. Finally, the Latent Dirichlet Allocation (LDA)+Word2vec model and Convolutional Neural Network (CNN) model are used to analyze the themes and emotional evolution of stakeholders during different stages of public opinion evolution. There are differences in the evolution patterns of different types of stakeholders. The evolution process of stakeholders' focus points exhibits a two-stage transition from concentration to divergence. The focus points of stakeholders are closely associated with their respective social domains. The emotions of the public undergo a three-stage process of positive-negative-positive change. This study can provide a reference for the government to have a more comprehensive understanding of the development trend of public opinion and reduce the negative impact of public opinion.

Corporate financial distress prediction using the risk-related information content of annual reports

This study presents a financial distress prediction model focusing on the linguistic analysis of risk-related sections of corporate annual reports. Here, we introduce a novel methodology that leverages BERT-based contextualized embedding models for nuanced extraction of financial sentiment and topic coherence. This stands in contrast to existing research, which predominantly relies on dictionary-based or non-contextual word embeddings and addresses their limitations in context sensitivity. Furthermore, we apply an innovative financial distress prediction model that combines the robust XGBoost algorithm with unsupervised outlier detection techniques. This hybrid model is specifically designed to tackle the issue of class imbalance, a persistent challenge in financial distress prediction. The efficacy of the proposed model is empirically validated using a comprehensive dataset of 2545 companies listed on major global stock exchanges. Our findings indicate that the introduced model not only significantly outperforms most existing state-of-the-art financial distress prediction models in terms of predictive accuracy, but also significantly outperforms the Loughran & McDonald dictionary-based approach and the Word2Vec model, underlining its potential as a superior analytical tool for financial distress prediction.

Values That Are Explicitly Present in Fairy Tales: Comparing Samples from German, Italian and Portuguese Traditions

Looking at how social values are represented in fairy tales can give insights about the variations in communication of values across cultures. We study how values are communicated in fairy tales from Portugal, Italy and Germany using a technique called word embedding with a compass to quantify vocabulary differences and commonalities. We study how these three national traditions differ in their explicit references to values. To do this, we specify a list of value-charged tokens, consider their word stems and analyse the distance between these in a bespoke pre-trained Word2Vec model. We triangulate and critically discuss the validity of the resulting hypotheses emerging from this quantitative model. Our claim is that this is a reusable and reproducible method for the study of the values explicitly referenced in historical corpora. Finally, our preliminary findings hint at a shared cultural understanding and the expression of values such as Benevolence, Conformity, and Universalism across the studied cultures, suggesting the potential existence of a pan-European cultural memory.

Optimizing Word Embeddings for Patient Portal Message Datasets with a Small Number of Samples.

Patient portal messages often relate to specific clinical phenomena (e.g., patients undergoing treatment for breast cancer) and, as a result, have received increasing attention in biomedical research. These messages require natural language processing and, while word embedding models, such as word2vec, have the potential to extract meaningful signals from text, they are not readily applicable to patient portal messages. This is because embedding models typically require millions of training samples to sufficiently represent semantics, while the volume of patient portal messages associated with a particular clinical phenomenon is often relatively small. We introduce a novel adaptation of the word2vec model, PK-word2vec, for small-scale messages. PK-word2vec incorporates the most similar terms for medical words (including problems, treatments, and tests) and non-medical words from two pre-trained embedding models as prior knowledge to improve the training process. We applied PK-word2vec on patient portal messages in the Vanderbilt University Medical Center electric health record system sent by patients diagnosed with breast cancer from December 2004 to November 2017. We evaluated the model through a set of 1000 tasks, each of which compared the relevance of a given word to a group of the five most similar words generated by PK-word2vec and a group of the five most similar words generated by the standard word2vec model. We recruited 200 Amazon Mechanical Turk (AMT) workers and 7 medical students to perform the tasks. The dataset was composed of 1,389 patient records and included 137,554 messages with 10,683 unique words. Prior knowledge was available for 7,981 non-medical and 1,116 medical words. In over 90% of the tasks, both reviewers indicated PK-word2vec generated more similar words than standard word2vec (p=0.01).The difference in the evaluation by AMT workers versus medical students was negligible for all comparisons of tasks' choices between the two groups of reviewers (p = 0.774 under a paired t-test). PK-word2vec can effectively learn word representations from a small message corpus, marking a significant advancement in processing patient portal messages.

On Embedding Implementations in Text Ranking and Classification Employing Graphs

This paper aims to enhance the Graphical Set-based model (GSB) for ranking and classification tasks by incorporating node and word embeddings. The model integrates a textual graph representation with a set-based model for information retrieval. Initially, each document in a collection is transformed into a graph representation. The proposed enhancement involves augmenting the edges of these graphs with embeddings, which can be pretrained or generated using Word2Vec and GloVe models. Additionally, an alternative aspect of our proposed model consists of the Node2Vec embedding technique, which is applied to a graph created at the collection level through the extension of the set-based model, providing edges based on the graph’s structural information. Core decomposition is utilized as a method for pruning the graph. As a byproduct of our information retrieval model, we explore text classification techniques based on our approach. Node2Vec embeddings are generated by our graphs and are applied in order to represent the different documents in our collections that have undergone various preprocessing methods. We compare the graph-based embeddings with the Doc2Vec and Word2Vec representations to elaborate on whether our approach can be implemented on topic classification problems. For that reason, we then train popular classifiers on the document embeddings obtained from each model.

Mapping of Alzheimer’s disease related data elements and the NIH Common Data Elements

BackgroundAlzheimer’s Disease (AD) is a devastating disease that destroys memory and other cognitive functions. There has been an increasing research effort to prevent and treat AD. In the US, two major data sharing resources for AD research are the National Alzheimer’s Coordinating Center (NACC) and the Alzheimer’s Disease Neuroimaging Initiative (ADNI); Additionally, the National Institutes of Health (NIH) Common Data Elements (CDE) Repository has been developed to facilitate data sharing and improve the interoperability among data sets in various disease research areas.MethodTo better understand how AD-related data elements in these resources are interoperable with each other, we leverage different representation models to map data elements from different resources: NACC to ADNI, NACC to NIH CDE, and ADNI to NIH CDE. We explore bag-of-words based and word embeddings based models (Word2Vec and BioWordVec) to perform the data element mappings in these resources.ResultsThe data dictionaries downloaded on November 23, 2021 contain 1,195 data elements in NACC, 13,918 in ADNI, and 27,213 in NIH CDE Repository. Data element preprocessing reduced the numbers of NACC and ADNI data elements for mapping to 1,099 and 7,584 respectively. Manual evaluation of the mapping results showed that the bag-of-words based approach achieved the best precision, while the BioWordVec based approach attained the best recall. In total, the three approaches mapped 175 out of 1,099 (15.92%) NACC data elements to ADNI; 107 out of 1,099 (9.74%) NACC data elements to NIH CDE; and 171 out of 7,584 (2.25%) ADNI data elements to NIH CDE.ConclusionsThe bag-of-words based and word embeddings based approaches showed promise in mapping AD-related data elements between different resources. Although the mapping approaches need further improvement, our result indicates that there is a critical need to standardize CDEs across these valuable AD research resources in order to maximize the discoveries regarding AD pathophysiology, diagnosis, and treatment that can be gleaned from them.