Word Embedding Models Research Articles

A Hybrid Semantic Representation Method Based on Fusion Conceptual Knowledge and Weighted Word Embeddings for English Texts

The accuracy of traditional topic models may be compromised due to the sparsity of co-occurring vocabulary in the corpus, whereas conventional word embedding models tend to excessively prioritize contextual semantic information and inadequately capture domain-specific features in the text. This paper proposes a hybrid semantic representation method that combines a topic model that integrates conceptual knowledge with a weighted word embedding model. Specifically, we construct a topic model incorporating the Probase concept knowledge base to perform topic clustering and obtain topic semantic representation. Additionally, we design a weighted word embedding model to enhance the contextual semantic information representation of the text. The feature-based information fusion model is employed to integrate the two textual representations and generate a hybrid semantic representation. The hybrid semantic representation model proposed in this study was evaluated based on various English composition test sets. The findings demonstrate that the model presented in this paper exhibits superior accuracy and practical value compared to existing text representation methods.

Demystifying the dynamics of China’s overseas image with a large-scale social media dataset

ABSTRACT The COVID-19 pandemic has resulted in a noticeable decline in global attitudes toward China. It remains unclear whether this negative shift is a temporary phenomenon linked to the pandemic or a reflection of a more enduring trend in China’s overseas image. By analyzing approximately 1.7 billion representative English-language tweets worldwide from 2012 to 2018, this research investigates popular attitudes toward China using word embedding models. The findings indicate a general decline in global attitudes toward China, particularly in relation to its political system. At the same time, however, China has increasingly been recognized as an economic powerhouse, a rising global power, and a more eco-friendly society.

Analyzing Semantic Properties of Word Embeddings Using Eigenvectors

Dense word vectors have demonstrated their efficacy in several downstream natural language processing (NLP) tasks in recent years. Nevertheless, the interpretability of these embeddings' dimensions remains challenging. In this paper, we investigate how eigenvectors can reveal different semantic properties captured by word embedding models. Hence, we train word embeddings (e.g., Word2Vec) on English Wikipedia corpus to analyze the top eigenvectors, identify specific semantic properties (e.g., sentiment, formality) associated with each, and explore how these properties are encoded in the embedding space. This paper also discussed the limitations and potential benefits of this approach compared to other methods for analyzing word embeddings.

Improvement for deep learning for suicide and depression identification with unsupervised label correction

Abstract. Our study try to address the challenge of accurately identifying depression and suicidal ideation on social media platforms by introducing an enhanced novel methodology for unsupervised feature selection and label correction. Utilizing advanced word embedding models like BERT and the Universal Sentence Encoder, we transform textual content into dense numerical vectors that capture the nuanced emotional context of online discussions. Our approach enhances these embeddings with a deep neural network (DNN) to extract distinctive features, reducing the dimensionality of the data through Principal Component Analysis (PCA). For label correction, we employ clustering techniques including OPTICS, K-medoids, and hierarchical clustering, which are robust against noisy data points. We then train classifiers using CNN, DNN, logistic regression, and random forest algorithms, evaluated with metrics such as accuracy, precision, recall, F1 score, and AUC. This methodology improves the accuracy of classifying depressive and suicidal sentiments to some extent, assisting to utilize the vast data available on social media to advance mental health diagnostics and interventions.

Human–robot interaction: predicting research agenda by long short-term memory

The article addresses the identification and prediction of research topics in human–robot interaction (HRI), fundamental in Industry 4.0 (I4.0) and future Industry 5.0 (I5.0). In the absence of research agendas in the scientific literature, the study proposes a multilayered model to create a precise agenda to guide the scientific community in new developments in collaborative robotics and HRI technologies. The methodology is divided into four stages, which make up the three layers of the model. In the first two stages, scientific articles on HRI for the period 2020–2021 were collected and analyzed using data mining techniques together with VantagePoint and Gephi software to identify keywords and their relationships. These initial stages form layer 1 of the model, where the main scientific themes are recognized. In the third stage, article titles and abstracts are cleaned and processed using natural language processing (NLP) techniques, generating word embeddings models that highlight relevant HRI-related terms, forming layer 2. The fourth and final stage uses Recurrent Neural Networks (RNN) with long short-term memory (LSTM) architecture to predict future topics, consolidating the previously identified terms and forming layer 3 of the model. The results show that in layer 1 HRI has intensive application in various sectors through advanced computational algorithms, with trust as a key feature. In layer 2, terms such as vision, sensors, communication, collaboration and anthropomorphic aspects are fundamental, while layer 3 anticipates future topics such as design, performance, method and controllers, essential to improve robot interaction. The study concludes that the methodology is effective in defining a robust and relevant research agenda. By identifying future trends and needs, this work fills a gap in the scientific literature, providing a valuable tool for the research community in the field of HRI.

Leveraging Temporal Trends for Training Contextual Word Embeddings to Address Bias in Biomedical Applications: Development Study.

Women have been underrepresented in clinical trials for many years. Machine-learning models trained on clinical trial abstracts may capture and amplify biases in the data. Specifically, word embeddings are models that enable representing words as vectors and are the building block of most natural language processing systems. If word embeddings are trained on clinical trial abstracts, predictive models that use the embeddings will exhibit gender performance gaps. We aim to capture temporal trends in clinical trials through temporal distribution matching on contextual word embeddings (specifically, BERT) and explore its effect on the bias manifested in downstream tasks. We present TeDi-BERT, a method to harness the temporal trend of increasing women's inclusion in clinical trials to train contextual word embeddings. We implement temporal distribution matching through an adversarial classifier, trying to distinguish old from new clinical trial abstracts based on their embeddings. The temporal distribution matching acts as a form of domain adaptation from older to more recent clinical trials. We evaluate our model on 2 clinical tasks: prediction of unplanned readmission to the intensive care unit and hospital length of stay prediction. We also conduct an algorithmic analysis of the proposed method. In readmission prediction, TeDi-BERT achieved area under the receiver operating characteristic curve of 0.64 for female patients versus the baseline of 0.62 (P<.001), and 0.66 for male patients versus the baseline of 0.64 (P<.001). In the length of stay regression, TeDi-BERT achieved a mean absolute error of 4.56 (95% CI 4.44-4.68) for female patients versus 4.62 (95% CI 4.50-4.74, P<.001) and 4.54 (95% CI 4.44-4.65) for male patients versus 4.6 (95% CI 4.50-4.71, P<.001). In both clinical tasks, TeDi-BERT improved performance for female patients, as expected; but it also improved performance for male patients. Our results show that accuracy for one gender does not need to be exchanged for bias reduction, but rather that good science improves clinical results for all. Contextual word embedding models trained to capture temporal trends can help mitigate the effects of bias that changes over time in the training data.

LongT5Rank: A Novel Integrated Hybrid Approach for Text Summarisation

Text summarisation reduces text length while retaining important information, helping individuals, especially students, in managing information overload during research or assignments. However, existing text summarisation methods often lose important details, generate irrelevant or redundant sentences, or produce incoherent summaries. This study introduces a hybrid approach, LongT5Rank (coined in this study), which combines TextRank, an extractive summarisation algorithm, with LongT5, an abstractive summarisation algorithm, to automate the summarisation process. TextRank utilizes GloVe, a pre-trained word embedding model, and PageRank, a graph-based ranking algorithm, to select representative sentences. LongT5, an encoder-decoder transformer model, condenses extracted sentences into a concise and coherent summary, handling input sequences up to 16,384 tokens, for long-range sequence-to-sequence tasks. The LongT5Rank approach has shown significant achievements, including a minimum 60% compression rate, a minimum 0.6 semantic textual similarity score, and an improved F-measure compared to employing TextRank alone. Furthermore, it received positive feedback from Human Level Performance (HLP), underlining the importance of evaluating results directly from human users. This emphasizes the belief that the performance of the proposed solution should be assessed natively by humans. By combining both extractive and abstractive methods, LongT5Rank excels in generating accurate and coherent summaries.

Exploring Deep Learning Models for Lyric Generation and Addressing Biases in Word Embeddings

The aim of this project is to explore the performance of different model architectures (such as RNN, LSTM, GRU) by generating lyrics using deep learning models, and to use the Word2Vec model for distributed semantic analysis to understand semantic phenomena and potential biases in word embedding models. The experimental results show that LSTM and GRU perform better than traditional RNN models when processing long sequence data. In addition, by analyzing word embeddings, we revealed potential gender and racial biases and proposed corresponding solutions.

Advancements in Word Embeddings: A Comprehensive Survey and Analysis

In recent years, the field of Natural Language Processing (NLP) has seen significant growth in the study of word representation, with word embeddings proving valuable for various NLP tasks by providing representations that encapsulate prior knowledge. We reviewed word embedding models, their applications, cross-lingual embeddings, model analyses, and techniques for model compression. We offered insights into the evolving landscape of word representations in NLP, focusing on the models and algorithms used to estimate word embeddings and their analysis strategies. To address this, we conducted a detailed examination and categorization of these evaluations and models, highlighting their significant strengths and weaknesses. We discussed a prevalent method of representing text data to capture semantics, emphasizing how different techniques can be effectively applied to interpret text data. Unlike traditional word representations, such as Word to Vector (word2vec), newer contextual embeddings, like Bidirectional Encoder Representations from Transformers (BERT) and Embeddings from Language Models (ELMo), have pushed the boundaries by capturing the use of words through diverse contexts and encoding information transfer across different languages. These embeddings leverage context to represent words, leading to innovative applications in various NLP tasks.

Litigation risk and the cost of debt financing in M&As

In this paper, we use a semi-supervised machine learning technique, the word2vec word embedding model, to measure litigation risk for fixed-income issuers that use bond financing in primary market for mergers and acquisitions (M&As) in 28 countries. We investigate the relationship between the litigation risk and the offering yield of these securities, demonstrating that increased litigation risk increases financing costs. We analyze several ways to mitigate adverse effects, including the employment of more M&A advisors and assessing the legal environment in the issuing country. Our results are robust to an instrumental variable approach and alternative measures.

Global Evolution of Environmental and Social Disclosure in Annual Reports

ABSTRACTWe study environmental and social (E&amp;S) disclosures in annual reports. Using the word embedding model to examine over 210,000 annual reports from 24,271 public firms in 30 international countries/regions between 2001 and 2020, we create an E&amp;S dictionary that allows us to document trends in annual report E&amp;S disclosure. Specifically, we find: (1) increases in length and boilerplate language and (2) decreases in specificity. Our results also suggest that E&amp;S disclosure quality improves after the adoption of voluntary ESG reporting frameworks but deteriorates after disclosure mandates. Our findings provide systematic evidence on the evolution of E&amp;S disclosure in annual reports over the past two decades and shed light on how voluntary and mandatory standards have shaped global E&amp;S reporting practices.

A workflow for analyzing cultural schemas in texts

ABSTRACT Concept class analysis (CoCA) is a method for recovering cultural schemas in texts using a combination of word embedding and community detection models. Like survey-based forms of schematic class analysis (SCA), however, interpreting results can be difficult. Some of these interpretive difficulties are applicable across types of SCA, while others are unique to CoCA. In this paper, we propose a complete workflow for interpreting and analyzing CoCA output. We use the case of social identity schemas in a collection of over 13,000 U.S. political blog posts to outline a number of interpretive and analytical strategies and a robustness check to make sense of the cultural schemas recovered from texts.

Learning Joint Topic Representation for Detecting Drift in Social Media Text

Social media texts like tweets and blogs are collaboratively created by human interaction. Rapidly changing trends are leading to topic drift in the social media text. This drift is usually associated with words and hashtags. However, geotags play an essential part in determining topic distribution with location context. The rate of change in the distribution of words, hashtags and geotags cannot be considered uniform and must be handled accordingly. This paper builds a topic model that associates the topic with a mixture of distributions of words, hashtags and geotags. Stochastic gradient Langevin dynamic model with varying mini-batch sizes is used to capture the changes due to the asynchronous distribution of words and tags. Topic representations with co-occurrence and location contexts are specified as hashtag context vector and geotag context vector respectively. These two vectors are jointly learned to yield topical word embedding vectors over time conditioned on hashtags and geotags that can predict location-based topical variations effectively. When evaluated with Chennai and UK geolocated Twitter data, the proposed joint topical word embedding model enhanced by the social tags context, outperforms other methods.

Integrating word embedding and topic modeling for sentiment analysis: A case study on the social mood on economy

In recent years, textual analysis and embedding spaces have become essential and complementary tools for sentiment analysis in National Statistics Institutes’ research, owing to their ability to summarize discussed topics effectively. Istat has developed an innovative tool, wordembox, which allows external users to explore the outputs of popular word embedding algorithms, such as Word2Vec and FastText. This tool enriches the analysis with a novel graph functionality, enabling users to discover clusters of words and facilitating implicit topic modeling. This article focuses on Social Mood on Economy (SME) posts over a period in which the index recorded a strong downward trend: the first month of the Russia-Ukraine conflict at the beginning of 2022. We compare findings from wordembox with standard topic modeling techniques, including Bayesian Latent Dirichlet Allocation (LDA), Top2Vec, and BERTopic, recent methods that extract clusters from word embedding spaces. These techniques show coherent results, and their combined use in textual analysis may create a synergy that enhances the informative content of synthetic indexes such as ‘Social Mood on Economy Index (SMEI)’.

Moral attitudes towards effort and efficiency: a comparison between American and Chinese history

In some cultures, merely exerting effort is considered virtuous, even when the effort is inefficient. Our study examines how this moral attitude towards effort (relative to efficiency) has evolved historically across two distinct sociopolitical and linguistic contexts: the People’s Republic of China and the United States, using natural language processing techniques. Specifically, two formal political corpora were used—the People’s Daily (1950–2021) and the Congressional speeches for the U.S. (1873–2011). We developed dictionaries for each concept based on pre-trained word embedding models in both languages. Moral attitudes towards effort and efficiency were calculated on a year-by-year basis as the cosine similarity between the dictionaries of these concepts and an existing dictionary of morality. We benchmarked the fluctuations of moral attitude towards inefficient effort against critical historical events in the two countries. Additional time series analysis and Granger tests revealed the association and potential directionality between the evolution of moral attitude towards inefficient effort and critical socio-cultural variables such as collectivism and cultural looseness. Our research sheds light on the historical and socio-cultural roots of moralization of effort and has implications for historical psychology research on moral attitudes.

The principal components of meaning, revisited.

Osgood, Suci, and Tannebaum were the first to attempt to identify the principal components of semantics using dimensional reduction of a high-dimensional model of semantics constructed from human judgments of word relatedness. Modern word-embedding models analyze patterns of words to construct higher dimensional models of semantics that can be similarly subjected to dimensional reduction. Hollis and Westbury characterized the first eight principal components (PCs) of a word-embedding model by correlating them with several well-known lexical measures, such as logged word frequency, age of acquisition, valence, arousal, dominance, and concreteness. The results show some clear differentiation of interpretation between the PCs. Here, we extend this work by analyzing a larger word-embedding matrix using semantic measures initially derived from subjective inspection of the PCs. We then use quantitative analysis to confirm the utility of these subjective measures for predicting PC values and cross-validate them on two word-embedding matrices developed on distinct corpora. Several semantic and word class measures are strongly predictive of early PC values, including first-person and second-person verbs, personal relevance of abstract and concrete words, affect terms, and names of places and people. The predictors of the lowest magnitude PCs generalized well to word-embedding matrices constructed from separate corpora, including matrices constructed using different word-embedding methods. The predictive categories we describe are consistent with Wittgenstein's argument that an autonomous level of social interaction grounds linguistic meaning.

Cumulative rib fracture risk after stereotactic body radiotherapy in patients with localized non-small cell lung cancer

IntroductionRib fracture is a known complication after stereotactic body radiotherapy (SBRT). Patient-related parameters are essential to provide patient-tailored risk estimation, however, their impact on rib fracture is less documented compared to dosimetric parameters. This study aimed to predict the risk of rib fractures in patients with localized non-small cell lung cancer (NSCLC) post-SBRT based on both patient-related and dosimetric parameters with death as a competing risk. Materials and methodsIn total, 602 patients with localized NSCLC treated with SBRT between 2010–2020 at Odense University Hospital, Denmark were included. All patients received SBRT with 45–66 Gray (Gy)/3 fractions. Rib fractures were identified in CT-scans using a word embedding model. The cumulative incidence function was based on cause-specific Cox hazard models with variable selection based on cross-validation model likelihood performed using 50 bootstraps. ResultsIn total, 19 % of patients experienced a rib fracture. The cumulative risk of rib fracture increased rapidly from 6-54 months post-SBRT. Female gender, bone density, near max dose to the rib, V30 and V40 to the rib, gross tumor volume, and mean lung dose were significantly associated with rib fracture risk in univariable analysis. The final multi-variable model consisted of V20 and V30 to the rib and mean lung dose. ConclusionFemale gender and low bone density in male patients are significant predictors of rib fracture risk. The final model predicting cumulative rib fracture risk of 19 % in patients with localized NSCLC treated with SBRT contained no patient-related parameters, suggesting that dosimetric parameters are the primary drivers.

Artificial Intelligence-Based Co-Facilitator (AICF) for Detecting and Monitoring Group Cohesion Outcomes in Web-Based Cancer Support Groups: Single-Arm Trial Study.

Commonly offered as supportive care, therapist-led online support groups (OSGs) are a cost-effective way to provide support to individuals affected by cancer. One important indicator of a successful OSG session is group cohesion; however, monitoring group cohesion can be challenging due to the lack of nonverbal cues and in-person interactions in text-based OSGs. The Artificial Intelligence-based Co-Facilitator (AICF) was designed to contextually identify therapeutic outcomes from conversations and produce real-time analytics. The aim of this study was to develop a method to train and evaluate AICF's capacity to monitor group cohesion. AICF used a text classification approach to extract the mentions of group cohesion within conversations. A sample of data was annotated by human scorers, which was used as the training data to build the classification model. The annotations were further supported by finding contextually similar group cohesion expressions using word embedding models as well. AICF performance was also compared against the natural language processing software Linguistic Inquiry Word Count (LIWC). AICF was trained on 80,000 messages obtained from Cancer Chat Canada. We tested AICF on 34,048 messages. Human experts scored 6797 (20%) of the messages to evaluate the ability of AICF to classify group cohesion. Results showed that machine learning algorithms combined with human input could detect group cohesion, a clinically meaningful indicator of effective OSGs. After retraining with human input, AICF reached an F1-score of 0.82. AICF performed slightly better at identifying group cohesion compared to LIWC. AICF has the potential to assist therapists by detecting discord in the group amenable to real-time intervention. Overall, AICF presents a unique opportunity to strengthen patient-centered care in web-based settings by attending to individual needs. RR2-10.2196/21453.

Optimizing word embeddings for small datasets: a case study on patient portal messages from breast cancer patients

Patient portal messages often relate to specific clinical phenomena (e.g., patients undergoing treatment for breast cancer) and, as a result, have received increasing attention in biomedical research. These messages require natural language processing and, while word embedding models, such as word2vec, have the potential to extract meaningful signals from text, they are not readily applicable to patient portal messages. This is because embedding models typically require millions of training samples to sufficiently represent semantics, while the volume of patient portal messages associated with a particular clinical phenomenon is often relatively small. We introduce a novel adaptation of the word2vec model, PK-word2vec (where PK stands for prior knowledge), for small-scale messages. PK-word2vec incorporates the most similar terms for medical words (including problems, treatments, and tests) and non-medical words from two pre-trained embedding models as prior knowledge to improve the training process. We applied PK-word2vec in a case study of patient portal messages in the Vanderbilt University Medical Center electric health record system sent by patients diagnosed with breast cancer from December 2004 to November 2017. We evaluated the model through a set of 1000 tasks, each of which compared the relevance of a given word to a group of the five most similar words generated by PK-word2vec and a group of the five most similar words generated by the standard word2vec model. We recruited 200 Amazon Mechanical Turk (AMT) workers and 7 medical students to perform the tasks. The dataset was composed of 1389 patient records and included 137,554 messages with 10,683 unique words. Prior knowledge was available for 7981 non-medical and 1116 medical words. In over 90% of the tasks, both reviewers indicated PK-word2vec generated more similar words than standard word2vec (p = 0.01).The difference in the evaluation by AMT workers versus medical students was negligible for all comparisons of tasks’ choices between the two groups of reviewers (p=0.774\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$${\ ext{p}} = 0.774$$\\end{document} under a paired t-test). PK-word2vec can effectively learn word representations from a small message corpus, marking a significant advancement in processing patient portal messages.

A Comment Aspect-Level User Preference Transfer Model for Cross-Domain Recommendations

Traditional cross-domain recommendation models make it difficult to deeply mine users' aspect-level preferences from comment information due to existing problems such as polysemy of comment text, sparse comment data, and user cold start. A Cross-Domain Recommender (CDR) model that integrates comment knowledge enhancement and aspect-level user preference transfer (C-KE-AUT) was proposed to address the above issues. Firstly, an aspect-level user preference extraction model was constructed by combining the RoBERTa word embedding model, high-level feature representation based on Transformer, and aspect-level attention-learning methods. Then, a user aspect-level preference cross-domain transfer model was constructed based on a two-stage generative adversarial network that can transfer the aspect-level interest preferences of users in the source domain to the target domain with sparse data. The experimental results on the Amazon 2018 comment dataset indicated that the recommendation performance of the proposed C-KE-AUT model was significantly superior to other advanced comparative models.