Word Embeddings Research Articles

Hyperbolic Graph Convolutional Network Relation Extraction Model Combining Dependency Syntax and Contrastive Learning

In current relation extraction tasks, when the input sentence structure is complex, the performance of in-context learning methods based on large language model is still lower than that of traditional pre-train fine-tune models. For complex sentence structures, dependency syntax information can provide effective prior text structure information for relation extraction. However, most studies are affected by the noise in the syntactic information automatically extracted by natural language processing toolkits. Additionally, traditional pre-training encoders have issues such as an overly centralized representation of word embedding for high-frequency words, which adversely affects the model to learn contextual semantic information. To address proposed problem, the paper proposes a Hyperbolic Graph Convolutional Network Relation Extraction Model Combine Dependency Syntax and Contrastive Learning. Based on the hyperbolic graph neural network, dependent syntactic information and information optimization strategies are introduced to solve the problem of word embedding concentration. Simultaneously, to mitigate the impact of noise in dependency syntax information on the relation extraction task, a contrastive learning approach is employed. After the model learns context semantics separately in the original dependency syntax information and dependency syntax information with added random noise, it maximizes the mutual information between entity words to assist the model in distinguishing noise in dependency syntax. The experiments indicate that the proposed model in this paper can effectively enhance the performance of relation extraction on public datasets, especially achieving significantly higher precision on datasets with complex sentence structures compared to in-context learning.

XGBoost-enhanced ensemble model using discriminative hybrid features for the prediction of sumoylation sites

Posttranslational modifications (PTMs) are essential for regulating protein localization and stability, significantly affecting gene expression, biological functions, and genome replication. Among these, sumoylation a PTM that attaches a chemical group to protein sequences—plays a critical role in protein function. Identifying sumoylation sites is particularly important due to their links to Parkinson’s and Alzheimer’s. This study introduces XGBoost-Sumo, a robust model to predict sumoylation sites by integrating protein structure and sequence data. The model utilizes a transformer-based attention mechanism to encode peptides and extract evolutionary features through the PsePSSM-DWT approach. By fusing word embeddings with evolutionary descriptors, it applies the SHapley Additive exPlanations (SHAP) algorithm for optimal feature selection and uses eXtreme Gradient Boosting (XGBoost) for classification. XGBoost-Sumo achieved an impressive accuracy of 99.68% on benchmark datasets using 10-fold cross-validation and 96.08% on independent samples. This marks a significant improvement, outperforming existing models by 10.31% on training data and 2.74% on independent tests. The model’s reliability and high performance make it a valuable resource for researchers, with strong potential for applications in pharmaceutical development.

Intent-Bert and Universal Context Encoders: A Framework for Workload and Sensor Agnostic Human Intention Prediction

Determining human intention is a challenging task. Many existing techniques seek to address it by combining many forms of data, such as images, point clouds, poses, and others, creating multi-modal models. However, these techniques still often require significant foreknowledge in the form of known potential activities and objects in the environment, as well as specific types of data to collect. To address these limitations, we propose Intent-BERT and Universal Context Encoders, which combine to form workload-agnostic framework that can be used to predict the next activity that a human performs as an Open Vocabulary Problem and the time until that switch, along with the time the current activity ends. Universal Context Encoders utilize the distances between the embeddings of words to extract relationships between Human-Readable English descriptions of both the current task and the origin of various multi-modal inputs to determine how to weigh the values themselves. We examine the effectiveness of this approach by creating a multi-modal model using it and training it on the InHARD dataset. It is able to return a completely accurate description of the next Action performed by a human working alongside a robot in a manufacturing task in ∼42% of test cases and has a 95% Top-3 accuracy, all from a single time point, outperforming multi-modal gpt4o by about 50% on a token by token basis.

Enhancing Semantic Search Precision through the CBOW Algorithm in the Semantic Web

The Semantic Web enhances data interoperability and enables intelligent information retrieval through structured data representation. However, challenges remain in achieving high precision in semantic search. This paper uses the Continuous Bag of Words (CBOW) model to enhance semantic search precision. By generating rich word embeddings, CBOW enables a better understanding of contextual relationships among terms within semantic queries. Our approach has been evaluated using the websites intended to be used as a sample for testing the efficiency of semantic information retrieval, demonstrating significant improvements in search precision compared to traditional methods. The findings indicate that integrating CBOW into semantic search frameworks can lead to more relevant and accurate search results, paving the way for future advancements in Semantic Web technologies.

Event Detection and Classification in Tweets using Deep Learning

Online social networks have become important sources of information and contextual data in all areas of life, including finance, elections, social events, health, sports, etc. Recently, the detection and classification of useful events presented in tweets has attracted a lot of interest. However, due to the inherent challenges associated with the nature of the events to be detected or classified, traditional approaches have not yielded satisfactory results. The use of deep learning-based text word embedding representations, such as Word2Vec, GloVe, FastText, and BERT, has shown significant efficacy in improving detection performance by considering the semantic context. This study proposes a model that uses an LSTM stacked on top of BERT representations to effectively detect and classify events in tweets. To this end, a dataset of about 310,000 event-related tweets has been collected and categorized into 50 event types based on a selected set of representative keywords. Multiple experiments were carried out on the collected dataset to evaluate the performance of the proposed model. The proposed model attained an overall accuracy greater than 94.3% and an F1 score of more than 90%, achieving state-of-the-art results in the classification of most of the event categories.

Classification of poetry text into the emotional state using Deep Learning

Poetry can elicit powerful emotions on the basis of elevated language and literary devices, and placing these emotions becomes problematic owing to the subjectivity of human emotions and the complexity entwined in poetic expression. This research puts forward a deep learning-based approach towards automatic classification of poetry into emotions like happiness, sadness, anger, fear, and surprise. The proposed approach also engages advanced NLP techniques that aid in discovering the semantics and emotional hinterlands of poems. For the purpose of this study, we represent poetic texts in higher-dimensional space using word embeddings such as Word2Vec and GloVe as reliable representations of contextual meanings. Recurrent Neural Networks (RNNs), particularly Long Short-Term Memory (LSTM) networks, have been employed to cater to the sequential nature of poems, thus recognizing complex patterns of emotions. Furthermore, we explore Transformer-based models such as BERT because of their superior power to understand context and nuances of emotion in text. The dataset was a curated collection of poems with annotations for emotions. Then, to ensure robust results, the performance of the models was evaluated in terms of accuracy, precision, recall, and F1-score. Initial results showed promising accuracy for such emotional identification techniques, emphasizing the aptitude of the proposed deep learning techniques to understand and elucidate poetry's emotional content. The offered study adds a new light on the conjunction of computational linguistics and emotional analysis, bettering the understanding of automated literary analysis. Keywords: Poetry, Emotion Analytics, Sentiment Analytics, Creative Writing, Listening AI, Natural Language Processing, Textual Analysis, Expressive AI, Mood Poetry, Emotional Intelligence, Machine Training.

Contrastive Learning for Enhanced Retrieval on Online Medical Platforms

Online medical platforms have rapidly developed in recent years. Due to search inaccuracies and incomplete information provided by doctors, some suitable doctors are excluded during the first-stage retrieval. To enhance doctor retrieval and reduce user search costs, obtaining disease word embeddings is essential. Existing methods primarily rely on external corpora and sentence context, which may not align with downstream tasks, and many languages lack standardized medical corpora. Therefore, we integrate contrastive learning with medical knowledge to generate and augment data using simulations and propose a simple network structure for training on the constructed samples. We created two task datasets based on the platform’s data. Experimental results demonstrate that our framework achieves superior outcomes with lower embedding dimensions. In the similarity task, our framework attains an accuracy of 0.854, and in the retrieval task, it achieves an F1 score of 0.853, surpassing the current best results. Our framework has been successfully implemented on Baidu Health, one of the China’s largest online medical platforms, serving over 10 million users. This framework effectively simulates a doctor retrieval system, optimizing the process to ensure more accurate and comprehensive retrieval of suitable doctors.

An improved reptile search algorithm-based machine learning for sentiment analysis

The rapid growth of mobile technologies has transformed social media, making it crucial for expressing emotions and thoughts. When making significant decisions, businesses and governments can benefit from understanding public opinion. This information makes sentiment analysis vital for understanding public sentiment polarity. This study develops a hyper tuned deep learning model with swarm intelligence and many approaches for sentiment analysis. convolutional neural network (CNN), bidirectional encoder representations from transformers (BERT), long short-term memory (LSTM), CNN-LSTM, BERT-LSTM, and BERT-CNN are the six deep learning models of the sentiment analysis using deep learning with reinforced learning based on reptile search algorithm (SA-DLRLRSA) model. The reptile search algorithm, an enhanced swarm intelligence algorithm (SIA), optimizes deep learning model hyper parameters. Word2Vec word embedding is used to convert textual input sequences to representative embedding spaces. Pre-trained Word2Vec embedding is also used to address issue of unbalanced datasets. Experimental results demonstrate that the SA-DLRLRSA model works best with accuracies of 93.1%, 94.7%, 96.8%, 96.3%, 97.2%, and 98.3% utilizing CNN, LSTM, BERT, CNN-LSTM, BERT-CNN, and BERT-LSTM.

EXPRESS: A generative approach to extrapolate word concreteness ratings.

A wealth of psycholinguistic and clinical research is supported by normative ratings of lexicosemantic properties, e.g., word concreteness, word valence, age-of-acquisition, etc. Collecting such ratings for a sufficiently large number of words is however notoriously labor-intensive. This study utilized the mixture density network (MDN), a generative approach, to implement a computational expansion of the concreteness ratings for simplified Chinese words. Based on different word embeddings, the MDN was trained to generate the probability density of a word's trial-level ratings, allowing us to predict not only the word's mean concreteness rating (con.mean), but also the potential variability (con.var) in people's perceptions about the word's concreteness. The resulting estimates were shown to largely converge with human ratings in both central tendency and variability, and to precisely reflect the important representational features of the construct. Apart from these internal validations, we also examined the contributions of con.mean to Chinese lexical processing. The results revealed the concreteness effect on event-related potentials associated with visual word recognition. To assist and enhance future research, we released the extrapolated concreteness ratings, along with degrees of variability, for over 78,000 Chinese words in the Open Science Framework.

Sentiment Classification from Social Media Data with Gaussian Conditional Rule-Based Deep Convolutional Neural Network

Social media platforms share several emotions in the form of text, audio, and images, where the text-shared data are less convincing to understand the emotions. The classification of the sentiments should be accurate to achieve a better understanding of the values in the commercial industry. Twitter has proven to be an effective instrument for text-based sentiment analysis when compared to other traditional assessments like surveys and stock market performance records. Though the existing research on sentiment analysis achieved good results of performance, the techniques had drawbacks such as long-term sentence prediction, less robustness, high computational resources, lack of complementary feature extraction, and so on. In this research, the Gaussian conditional rule-based deep convolutional neural network (DCNN) (Gaussian conditional rule-based DCNN) is proposed to tackle the limitation of the existing techniques, where the Gaussian distribution is applied for the selection of the kernel filters in the formation of the feature maps, and the classifier parameters are adaptively trained using the proposed adaptive nature optimizer (ANO). Through the integration of the conditionality of the random forest algorithm into the developed architecture, the model attains the potential to adapt the processing concerning the supplementary contextual information or features. Utilizing the prominent word embedding approach maximizes the input quality, converts words into continuous vector representations in a high-dimensional space, and captures both syntactic and semantic associations between words. In addition, the Graph embedding preserves some of the graph’s structural characteristics and effectively captures the syntactic dependencies, co-occurrences, semantic similarities, as well as pertinent linguistic correlations among the texts. Specifically, the proposed ANO algorithm and the conditional rule exploited in the model provide stable as well as robust performance in the classification of sentiments. The experimental results demonstrate that the Gaussian conditional rule-based DCNN model achieves 98.36 %, 98.80%, and 97.44% for accuracy, specificity, and sensitivity, respectively.

Identifying causes of aviation safety events using wW2V-tCNN with data augmentation

Identifying the causes of these safety events is crucial for safety agencies to create recommendations and for airlines to enhance procedures and mitigate hazards. This paper proposes a model to identify the causes of civil aviation safety events using a weighted Word2Vec-based Text-CNN (wW2V-tCNN) algorithm and data augmentation techniques. A corpus is built by matching narrative texts from investigation reports with cause labels from the Aviation Safety Network database. This corpus is transformed into Text-CNN inputs using a weighted sentence vector method based on word embeddings, considering word frequency and part-of-speech weighting. Additionally, a novel document balancing method is introduced for data augmentation. The proposed identification model achieves Macro-F1 and Macro-accuracy scores of 0.9803 and 0.9699, outperforming traditional methods and showing significant improvement over models like Doc2vec and SBERT. This model provides an accurate tool for safety agencies and airlines to analyze and effectively mitigate civil aviation safety events.

Biomedical named entity recognition using improved green anaconda-assisted Bi-GRU-based hierarchical ResNet model

BackgroundBiomedical text mining is a technique that extracts essential information from scientific articles using named entity recognition (NER). Traditional NER methods rely on dictionaries, rules, or curated corpora, which may not always be accessible. To overcome these challenges, deep learning (DL) methods have emerged. However, DL-based NER methods may need help identifying long-distance relationships within text and require significant annotated datasets.ResultsThis research has proposed a novel model to address the challenges in natural language processing. The Improved Green anaconda-assisted Bi-GRU based Hierarchical ResNet BNER model (IGa-BiHR BNERM) is the model. IGa-BiHR BNERM model has shown promising results in accurately identifying named entities. The MACCROBAT dataset was obtained from Kaggle and underwent several pre-processing steps such as Stop Word Filtering, WordNet processing, Removal of non-alphanumeric characters, stemming Segmentation, and Tokenization, which is standardized and improves its quality. The pre-processed text was fed into a feature extraction model like the Robustly Optimized BERT –Whole Word Masking model. This model provides word embeddings with semantic information. Then, the BNER process utilized an Improved Green Anaconda-assisted Bi-GRU-based Hierarchical ResNet BNER model (IGa-BiHR BNERM).ConclusionTo improve the training phase of the IGa-BiHR BNERM, the Improved Green Anaconda Optimization technique was used to select optimal weight parameter coefficients for training the model parameters. After the model was tested using the MACCROBAT dataset, it outperformed previous models with a tremendous accuracy rate of 99.11%. This model effectively and accurately identifies biomedical names within the text, significantly advancing this field.

Word embedding factor based multi-head attention

The natural language processing (NLP) field has made significant progress using deep learning models based on multi-head attention mechanisms, such as Transformer and BERT. However, there are two major limitations to this approach. First, the number of heads is often manually set based on empirical experience, and second, it is not clear enough in semantic understanding and interpretation. In this study, we propose a novel attention mechanism called Factor Analysis-based Multi-head (FAM) Attention, which combines the theory of explorative factor analysis and word embedding. The experimental results demonstrate that FAM Attention achieves better performance and requires fewer parameters compared to traditional methods while also having better semantic understanding ability and interpretability at the token level. This also has significant implications for current Large Language Models (LLMs), particularly in terms of effectively reducing parameter counts and enhancing performance.

Discovering emergent connections in quantum physics research via dynamic word embeddings

Abstract As the field of quantum physics evolves, researchers naturally form subgroups focusing on specialized problems. 
While this encourages in-depth exploration, it can limit the exchange of ideas across structurally similar problems in different subfields. 
To encourage cross-talk among these different specialized areas, data-driven approaches using machine learning have recently shown promise to uncover meaningful connections between research concepts, promoting cross-disciplinary innovation.
Current state-of-the-art approaches represent concepts using knowledge graphs and frame the task as a link prediction problem, where connections between concepts are explicitly modeled.
In this work, we introduce a novel approach based on dynamic word embeddings for concept combination prediction. 
Unlike knowledge graphs, our method captures implicit relationships between concepts, can be learned in a fully unsupervised manner, and encodes a broader spectrum of information. 
We demonstrate that this representation enables accurate predictions about the co-occurrence of concepts within research abstracts over time.
To validate the effectiveness of our approach, we provide a comprehensive benchmark against existing methods and offer insights into the interpretability of these embeddings, particularly in the context of quantum physics research. 
Our findings suggest that this representation offers a more flexible and informative way of modeling conceptual relationships in scientific literature.

Social media network public opinion emotion classification method based on multi-feature fusion and multi-scale hybrid neural network.

With the rapid development of the internet, an increasing number of users express their subjective opinions on social media platforms. By analyzing the sentiment of these texts, we can gain insights into public sentiment, industry changes, and market trends, enabling timely adjustments and preemptive strategies. This article initially constructs vectors using semantic fusion and word order features. Subsequently, it develops a lexicon vector based on word similarity and leverages supervised corpora training to obtain a more pronounced transfer weight vector of sentiment intensity. A multi-feature fused emotional word vector is ultimately formed by concatenating and fusing these weighted transfer vectors. Experimental comparisons on two multi-class microblog comment datasets demonstrate that the multi-feature fusion (WOOSD-CNN) word vector model achieves notable improvements in sentiment polarity accuracy and categorization effectiveness. Additionally, for aspect-level sentiment analysis of user generated content (UGC) text, a unified learning framework based on an information interaction channel is proposed, which enables the team productivity center (TPC) task. Specifically, an information interaction channel is designed to assist the model in leveraging the latent interactive characteristics of text. An in-depth analysis addresses the label drift phenomenon between aspect term words, and a position-aware module is constructed to mitigate the local development plan (LDP) issue.

Evolving techniques in sentiment analysis: a comprehensive review

With the rapid expansion of social media and e-commerce platforms, an unprecedented volume of user-generated content has emerged, offering organizations, governments, and researchers invaluable insights into public sentiment. Yet, the vast and unstructured nature of this data challenges traditional analysis methods. Sentiment analysis, a specialized field within natural language processing, has evolved to meet these challenges by automating the detection and categorization of opinions and emotions in text. This review comprehensively examines the evolving techniques in sentiment analysis, detailing foundational processes such as data gathering and feature extraction. It explores a spectrum of methodologies, from classical word embedding techniques and machine learning algorithms to recent contextual embedding and advanced transformer models like Generative Pre-trained Transformer (GPT), Bidirectional Encoder Representations from Transformers (BERT), and T5. With a critical comparison of these methods, this article highlights their appropriate uses and limitations. Additionally, the review provides a thorough overview of current trends, insights into future directions, and a critical exploration of unresolved challenges. By synthesizing these developments, this review equips researchers with a solid foundation for assessing the current state of sentiment analysis and guiding future advancements in this dynamic field.

An Optimized Weighted-Voting-Based Ensemble Learning Approach for Fake News Classification

The emergence of diverse content-sharing platforms and social media has rendered the dissemination of fake news and misinformation increasingly widespread. This misinformation can cause extensive confusion and fear throughout the populace. Confronting this dilemma necessitates an effective and accurate approach to identifying misinformation, an intrinsically intricate process. This research introduces an automated and efficient method for detecting false information. We evaluated the efficacy of various machine learning and deep learning models on two separate fake news datasets of differing sizes via holdout cross-validation. Furthermore, we evaluated the efficacy of three distinct word vectorization methods. Additionally, we employed an enhanced weighted voting ensemble model that enhances fake news detection by integrating logistic regression (LR), support vector machine (SVM), gated recurrent unit (GRU), and long short-term memory (LSTM) networks. This method exhibits enhanced performance relative to previous techniques: 98.76% for the PolitiFact dataset and 97.67% for the BuzzFeed dataset. Furthermore, the model outperforms individual components, resulting in superior accuracy, precision, recall, and F1 scores. The enhancements in performance result from the ensemble method’s capacity to use the advantages of each base model, hence providing robust generalization across datasets. Cross-validation was employed to enhance the model’s trustworthiness, validating its capacity to generalize effectively to novel data.

Big Data and Machine Learning in ESG Research*

AbstractThe wide applications of machine learning techniques to big data allow researchers to dig deep into novel large‐scale data sets, such as job postings, earnings calls, and news reports. They also equip researchers with powerful tools to study important but subtle/challenging topics that are impossible to explore before on a large scale, such as corporate culture and climate risk exposure. In this review, I survey various applications of different machine learning techniques in ESG research, beginning with foundational methods such as bag‐of‐words, progressing through topic modeling, word embedding, and BERT, and culminating with generative artificial intelligence (AI) and other advanced machine learning approaches. I conclude by outlining future directions for using big data and machine learning in ESG research.

Tibetan Judicial Event Extraction Based on Deep Word Representation and Hybrid Neural Networks

For the extraction of judicial events for Tibetan, a low-resource language, traditional simple neural network approaches struggle to adequately capture the deep semantics and features of the texts because Tibetan texts are usually lengthy and contain numerous judicial-related entities. To overcome this limitation, this research presents an event extraction model combining deep word representation with hybrid neural networks for the Tibetan judicial domain. The model introduces the Chinese minority pre-trained language model (CINO), which generates dynamic word vector representations, addressing the challenge of modeling the deep semantics inherent in Tibetan texts. During feature extraction, a bidirectional long short-term memory network (BiLSTM) is applied to extract the temporal and contextual dependencies, while a convolutional neural network (CNN) is utilized to capture the local semantic features to construct a comprehensive global semantic representation. Finally, the sequences are decoded through conditional random field (CRF) to generate optimal prediction results, thus achieving the efficient extraction of Tibetan judicial events. The experimental findings indicate that the model outperforms the baselines by achieving F1 scores of 70.47% for trigger detection and 62.99% for argument recognition, with improvements of 16.6% and 16.42%, respectively. These results confirm the effectiveness and superiority of the proposed model.

CSpredR: A Multi-Site mRNA Subcellular Localization Prediction Method Based on Fusion Encoding and Hybrid Neural Networks

Current research widely acknowledges that the subcellular localization of mRNA is crucial for understanding its biological functions. However, current methods for mRNA subcellular localization based on k-mer frequency features may overlook the sequential information of the sequence, and a single encoding method may not adequately extract the sequence’s features. This paper proposes a novel deep learning prediction method, CSpredR, specifically designed for predicting the subcellular localization of multi-site mRNAs. Unlike previous methods, CSpredR first employs k-mer to tokenize the mRNA sequences, then converts the tokenized sequences into de Bruijn graphs, thereby enabling a more precise capture of the structural information within the sequences. To mitigate the impact of lost sequential information and better capture sequence features, we combine word2vec and fasttext models to extract the features of each node in the graph and retain the sequence order. They can encode the k-mer units in the sequence into word vectors, thus serving as the node feature vectors of the graph. In this way, each node in the graph is assigned a feature vector containing rich semantic information. Subsequently, we utilize multi-scale convolutional neural networks and bidirectional long short-term memory networks to capture sequence features, respectively, and fuse the results as input for a multi-head attention mechanism model. The information from these heads is integrated into the node representations, and finally, the attention-processed data are fed into an MLP (Multi-Layer Perceptron) for prediction tasks. Extensive experiments reveal that CSpredR achieves a 2% improvement over the best existing predictors, offering a more effective tool for mRNA subcellular localization prediction.