Vector Representations Of Words Research Articles

Neurocomputer System of Semantic Analysis of the Text in the Kazakh Language

The purpose of the study is to solve an extreme mathematical problem—semantic analysis of natural language, which can be used in various fields, including marketing research, online translators, and search engines. When training the neural network, data training methods based on the latent Dirichlet allocation model and vector representation of words were used. This study presents the development of a neurocomputer system used for the purpose of semantic analysis of the text in the Kazakh language, based on machine learning and the use of the latent Dirichlet allocation model. In the course of the study, the stages of system development were considered, regarding the text recognition algorithm. The Python programming language was used as a tool using libraries that greatly simplify the process of creating neural networks, including the Keras library. An experiment was conducted with the involvement of experts to test the effectiveness of the system, the results of which confirmed the reliability of the data provided by the system. The papers of modern computer linguists dealing with the problems of natural language processing using various technologies and methods are considered.

Scoring alignments by embedding vector similarity.

Sequence similarity is of paramount importance in biology, as similar sequences tend to have similar function and share common ancestry. Scoring matrices, such as PAM or BLOSUM, play a crucial role in all bioinformatics algorithms for identifying similarities, but have the drawback that they are fixed, independent of context. We propose a new scoring method for amino acid similarity that remedies this weakness, being contextually dependent. It relies on recent advances in deep learning architectures that employ self-supervised learning in order to leverage the power of enormous amounts of unlabelled data to generate contextual embeddings, which are vector representations for words. These ideas have been applied to protein sequences, producing embedding vectors for protein residues. We propose the E-score between two residues as the cosine similarity between their embedding vector representations. Thorough testing on a wide variety of reference multiple sequence alignments indicate that the alignments produced using the new $E$-score method, especially ProtT5-score, are significantly better than those obtained using BLOSUM matrices. The new method proposes to change the way alignments are computed, with far-reaching implications in all areas of textual data that use sequence similarity. The program to compute alignments based on various $E$-scores is available as a web server at e-score.csd.uwo.ca. The source code is freely available for download from github.com/lucian-ilie/E-score.

Meta heuristic approaches for sentiment analysis

AbstractOn the Internet, online microblogging and social networks have experienced tremendous growth. Millions of individuals share opinions on social networking platforms including Twitter, Facebook, YouTube, and Microblogging sites. Millions of individuals express their thoughts on social platforms namely Twitter, Facebook, YouTube, and microblogging sites. As user‐generated content is growing to a huge extent on the internet, the analysis, extraction identification and classification of opinion and sentiment polarity of different aspects has now become a challenging issue in sentiment analysis. In sentiment analysis, aspect level opinion mining, opinion spam review detection, extraction and representation of most relevant accessed data from the user‐generated data is a difficult process. The word vector representation using existing deep learning techniques affects the performance in aspect level opinion mining. Spammers propagate false opinion for payment exchange, which leads to degradation of the financial growth, business, and fame of the organization. To overcome the limitations aspect level opinion mining is used to recognize sentiment and polarity of aspects in the specified text. This work presents various input feature vectors of deep learning approaches aiming to improve performance in aspect level opinion mining. Next, three meta‐heuristic algorithms with k‐means clustering approach for classification of opinion spam reviews on social media transit tweets is presented. The proposed approaches are evaluated and compared with other existing approaches as well as other benchmark datasets namely Restaurant and Amazon reviews.

A novel joint extraction model based on cross-attention mechanism and global pointer using context shield window

Relational triple extraction is a critical step in knowledge graph construction. Compared to pipeline-based extraction, joint extraction is gaining more attention because it can better utilize entity and relation information without causing error propagation issues. Yet, the challenge with joint extraction lies in handling overlapping triples. Existing approaches adopt sequential steps or multiple modules, which often accumulate errors and interfere with redundant data. In this study, we propose an innovative joint extraction model with cross-attention mechanism and global pointers with context shield window. Specifically, our methodology begins by inputting text data into a pre-trained RoBERTa model to generate word vector representations. Subsequently, these embeddings are passed through a modified cross-attention layer along with entity type embeddings to address missing entity type information. Next, we employ the global pointer to transform the extraction problem into a quintuple extraction problem, which skillfully solves the issue of overlapping triples. It is worth mentioning that we design a context shield window on the global pointer, which facilitates the identification of correct entities within a limited range during the entity extraction process. Finally, the capability of our model against malicious samples is improved by adding adversarial training during the training process. Demonstrating superiority over mainstream models, our approach achieves impressive results on three publicly available datasets.

A Study on the Emotional Tendency of Aquatic Product Quality and Safety Texts Based on Emotional Dictionaries and Deep Learning

A Study on the Emotional Tendency of Aquatic Product Quality and Safety Texts Based on Emotional Dictionaries and Deep Learning

Identifying the Risk Factors of Allergic Rhinitis Based on Zhihu Comment Data Using a Topic-Enhanced Word-Embedding Model: Mixed Method Study and Cluster Analysis.

Allergic rhinitis (AR) is a chronic disease, and several risk factors predispose individuals to the condition in their daily lives, including exposure to allergens and inhalation irritants. Analyzing the potential risk factors that can trigger AR can provide reference material for individuals to use to reduce its occurrence in their daily lives. Nowadays, social media is a part of daily life, with an increasing number of people using at least 1 platform regularly. Social media enables users to share experiences among large groups of people who share the same interests and experience the same afflictions. Notably, these channels promote the ability to share health information. This study aims to construct an intelligent method (TopicS-ClusterREV) for identifying the risk factors of AR based on these social media comments. The main questions were as follows: How many comments contained AR risk factor information? How many categories can these risk factors be summarized into? How do these risk factors trigger AR? This study crawled all the data from May 2012 to May 2022 under the topic of allergic rhinitis on Zhihu, obtaining a total of 9628 posts and 33,747 comments. We improved the Skip-gram model to train topic-enhanced word vector representations (TopicS) and then vectorized annotated text items for training the risk factor classifier. Furthermore, cluster analysis enabled a closer look into the opinions expressed in the category, namely gaining insight into how risk factors trigger AR. Our classifier identified more comments containing risk factors than the other classification models, with an accuracy rate of 96.1% and a recall rate of 96.3%. In general, we clustered texts containing risk factors into 28 categories, with season, region, and mites being the most common risk factors. We gained insight into the risk factors expressed in each category; for example, seasonal changes and increased temperature differences between day and night can disrupt the body's immune system and lead to the development of allergies. Our approach can handle the amount of data and extract risk factors effectively. Moreover, the summary of risk factors can serve as a reference for individuals to reduce AR in their daily lives. The experimental data also provide a potential pathway that triggers AR. This finding can guide the development of management plans and interventions for AR.

How Fast Do Distribution and Semantics of Polysemic Words Change?

Creation of large diachronic text corpora triggered studies of evolution of natural languages using quantitative methods in the last decade. For the first time, it became possible to raise questions about general patterns of lexical semantic change. In the work by W. Hamilton et al., 2016, a hypothesis was formulated about a more rapid change in the meanings of polysemic words. In this paper, we consider how frequency of words influence variations of semantic metrics. We use explicit word vector representations and Jensen-Shannon divergence as a measure of change in the distribution and semantics of a word. The obtained results show that changes in the frequency ratio of a word in different meanings can themselves lead to a change in a word distribution, even in the absence of any changes in its meaning. Therefore, the observations made in the work by W. Hamilton et al about correlation between the rate of change of words and their local cluster coefficient in the semantic connection network (considered as a quantitative measure of polysemy) can be fully explained by the effect described in the article.

Short Text Classification Model based on Pre-trained Language Model with Feature Fusion

In response to the low accuracy of Chinese short text classification in the current data mining field and the defects of existing deep learning models with more model parameters and higher time complexity, this paper proposes a new text classification model - short text classification model (ACBSM) based on pre-trained language model with feature expansion. In ACBSM, to address the problem of high dimensionality of text data without accurate text representation, the Bert model is used to train word vector representation to solve the problem of multiple meanings of a word. From the parallelization acceleration level, a parallel acceleration strategy of two-channel neural network is designed to improve the efficiency of the algorithm in processing massive data. To address the sparsity of text data and the more complex semantics, an attention mechanism is introduced and a CNN model is used to enhance the extraction of keyword information; secondly, BiSRU is used to capture the contextual features of the text, and finally, experimental validation is conducted on a news dataset. The experimental results show that ACBSM improves the accuracy of text classification to 95.83% under the same environment and dataset, and its classification performance is better than other text classification methods.

News Classification Using Bidirectional Long Short Term Memory and GloVe

The dissemination of information and news via online media encompasses not only established news platforms but also contributions from internet users, lacking oversight. News constitutes fact-grounded insights into ongoing occurrences. This research employed Bidirectional Long- and Short-Term Memory with Hyperparameter tuning on GloVe for news classification. This research aims to optimize news categorization through hyperparameter tuning on GloVe. GloVe facilitated the transformation of words into vector matrices, exploring its efficacy in news classification with hyperparameter tuning and Bi-LSTM for text analysis. Experiments encompassed untuned and hyperparameter-tuned approaches, employing GloVe's hyperparameters using Gridsearch and manual methods. GloVe's hyperparameter tuning reveals the potential for enhancing word vector representations. Surprisingly, non-hyperparameter tuned news classification yielded superior evaluation results compared to the hyperparameter approach. The untuned experiment achieved an accuracy of 0.98, while the gridsearch method yielded 0.85 accuracy, and hyperparameter tuning generated a 0.88 precision in the -11 model. These findings underscore the nuanced interplay of hyperparameters in optimizing text classification models like GloVe.

Construction of Online Interactive Teaching Platform for College English Based on Multilayer Perceptual Machine Modeling

Abstract Most of the existing English online teaching platforms are just digital replicas of offline teaching, which cannot fully utilize the advantages of information technology. Based on the learning theory of constructivism, this paper builds the functional architecture of a personalized English interactive teaching platform and completes the design of the main functional modules. According to the quantitative results of the platform’s user proposition requirements, a matrix description method for English test papers is proposed, and an algorithm is designed to realize the automatic generation of test papers. Meanwhile, the resource recommendation mechanism of the English platform is optimized using a multilayer perceptual machine. Unstructured English learning resources are processed based on the word vector representation method, and for the problem of insufficient processing of nonlinear data in the DN-CBR model, multilayer training is carried out using the MLP to improve the perceptual degree of this part of the data, and the design goals are verified through the test of the teaching platform. The correlation coefficient between self-assessment difficulty and topic difficulty is 0.368, and the correlation coefficient between self-assessment difficulty and positive error is 0.163, indicating that there is a moderate correlation between learners’ self-assessment difficulty and the difficulty of the system topics and between learners’ self-assessment difficulty and positive error. The MLP-based pushed questions’ difficulty is adapted to the learner’s personalization level by dynamically revising it based on their learning process data.

Entity recognition method for airborne products metrological traceability knowledge graph construction

The airborne system, as one of the complex and extensive subsystems of an aircraft, primarily performs critical flight assurance functions. The quality of its components has a direct impact on the aircraft's safety and reliability. Metrology documents comprehensively document the performance parameters throughout the entire product life cycle. The Metrological Traceability Knowledge Graph (MTKG) for airborne products offers decision support to engineers engaged in metrological tasks, ensuring the continuous high quality of the products. This paper introduces an entity recognition method for airborne product metrological traceability knowledge graph construction. First, the ontology for MTKG is developed. Next, a fine-tuned multi-network model is proposed. Named entities in the field of metrology are recognized through three stages: word vector representation, sentence feature extraction, and optimal label assignment. Meanwhile, active learning methods are incorporated to reduce the expense of data annotation. The proposed model is validated using an actual metrology corpus, and the experimental results demonstrate its superior performance compared to the other four baseline methods. Finally, the MTKG is developed using this approach, offering engineers intelligent applications, including metrological traceability analysis and traceability path reasoning within the process of product metrology. This enhances the metrology capabilities of airborne products and demonstrates the extensive potential of knowledge graphs in metrology.

Boosting Lightweight Sentence Embeddings with Knowledge Transfer from Advanced Models: A Model-Agnostic Approach

In this study, we investigate knowledge transfer between two distinct sentence embedding models: a computationally demanding, highly performant model and a lightweight model derived from word vector averaging. Our objective is to augment the representational power of the lightweight model by exploiting the sophisticated features of the robust model. Diverging from traditional knowledge distillation methods that align logits or hidden states of teacher and student models, our approach uses only the output sentence vectors of the teacher model for the alignment with the student models’s word vector representations. We implement two minimization techniques for this purpose: distance minimization and distance and perplexity minimization Our methodology uses WMT datasets for training, and the enhanced embeddings are validated via Google’s Analogy tasks and Meta’s SentEval datasets. We found that our proposed models intriguingly retained and conveyed information in a model-specific fashion.

A Named Entity Recognition Approach for Electronic Medical Records Using BERT Semantic Enhancement and BiLSTM

Aiming at the problems of missing local context features, single word vector representation, and low entity recognition accuracy, a method for e-medical recording with named entity recognition, which is based on BERT and model fusion, is proposed. First, with the model of BERT for pre-training, the preceding and following contextual information is fused for the enhancement of word semantic representation and alleviation of the problem of polysemy; second, the network of bi-directional long-short term memory is for obtaining the sequence feature matrix, generation of optimal sequence in global sense achieved through the conditional random field model; finally, data enhancement is used to alleviate the class imbalance and improve the model ability in generalization. Results of the experiments find model proposal measured by F1 on CCKS21 data set reaches 0.8548, which is 0.51% and 0.08% higher than models with ID-CNNs-CRF and multi-task RNN. This demonstrates the excellent performance of the method proposed in this paper in improving named entity recognition.

Aspect-based sentiment score and star rating prediction for travel destination using Multinomial Logistic Regression with fuzzy domain ontology algorithm

Due to its significant advantages for downstream applications, such as recommender systems, In the context of travel and tourism, user reviews on TripAdvisor have an impact on other travellers' judgments regarding a variety of travel-related issues, including the choice of a vacation spot, lodging, and places to visit. In graded user reviews, the model must specifically forecast the user's review score after receiving the textual review. The purpose of this article is to present a predictive outline for aspect-based extraction and classification that can estimate the users' optimal travel destination. This impression helps travellers in a variety of ways, such as by recommending better locations that also include expensive destinations. The underlying classification algorithm's processing time is significantly impacted by more dimensions. The term “curse of dimensionality” is often used in statistics and machine learning to describe these issues. By projecting the high-dimensional input data into the low-dimensional subspace while roughly maintaining the distance between the data points with a higher probability, the Random Projection (RP) ensemble classifier decreases the complexity of multivariate data. Extract the crucial information from the reviews, then use Glove word vector representation to categorise the relevant emotions. Further, the article proposed Multinomial Logistic Regression (MNLR) with a Fuzzy Domain Ontology (FDO) algorithm for aspect-based sentiment analysis. More intricate aspects than just the products themselves impact how satisfied people are with tourist destinations. The most important factor in evaluating how convenient a tourist route will be is typically the weather. The combined form of predicted sentiment score, start ratings and environment factor has to be calculated to predict the travel destination based on the measurement of personalized search results. The simulation was processed in Python software. The presented work has utilized some performance measures to evaluate the classification model such as F1-score, Recall, Precision, Mean Absolute Error (MAE), Mean Squared Error (MSE), Cohen Score and Matthew Score. The accuracy with GloVe word vector representation was 90% and after the GloVe representation, the classification model accuracy was about 94%. The proposed strategy outperforms in terms of classification accuracy, according to simulated results and analyses of real-world data.

Research of methods based on neural networks for the analysis of the tonality of the corps of the texts

The object of the study is methods based on neural networks for analyzing the tonality of a corpus of texts. To achieve the goal set in the work, it is necessary to solve the following tasks: study the theoretical material for learning deep neural networks and their features in relation to natural language processing; study the documentation of the Tensorflow library; develop models of convolutional and recurrent neural networks; to develop the implementation of linear and non-linear classification methods on bag of words and Word2Vec models; to compare the accuracy and other quality indicators of implemented neural network models with classical methods. Tensorboard is used for learning visualization. The work shows the superiority of classifiers based on deep neural networks over classical classification methods, even if the Word2Vec model is used for vector representations of words. The model of recurrent neural network with LSTM blocks has the highest accuracy for this corpus of texts.

Gender Classification of Twitter Users Using Convolutional Neural Network

Social media has become a place for social media analysts to obtain data to gain deeper insights and understanding of user behavior, trends, public opinion, and patterns associated with social media usage. Twitter is one of the most popular social media platforms where users can share messages or ”tweets” in a short text format. However, on Twitter, user information such as gender is not shown, but without realizing it or not, there is information about it in an unstructured manner. In social media analytics, gender is one of the important data that someone likes, so this research was conducted to determine the best accuracy for gender classification. The purpose of this study was to determine whether using combined data can improve the accuracy of gender classification using data from Twitter, tweets, and descriptions. The method used was word vector representation using word2vec and the application of a 2D Convolutional Neural Network (CNN) model. Word2vec was used to generate word vector representations that take into account the context and meaning of words in the text. The 2D CNN model extracted features from the word vector representation and performed gender classification. The research aimed to compare tweet data, descriptions, and a combination of tweets and descriptions to find the most accurate. The result of this study was that combined data between tweets and

Towards a Method to Enable the Selection of Physical Models within the Systems Engineering Process: A Case Study with Simulink Models

The use of different techniques and tools is a common practice to cover all stages in the development life-cycle of systems generating a significant number of work products. These artefacts are frequently encoded using diverse formats, and often require access through non-standard protocols and formats. In this context, Model-Based Systems Engineering (MBSE) emerges as a methodology to shift the paradigm of Systems Engineering practice from a document-oriented environment to a model-intensive environment. To achieve this major goal, a formalised application of modelling is employed throughout the life-cycle of systems to generate various system artefacts represented as models, such as requirements, logical models, and multi-physics models. However, the mere use of models does not guarantee one of the main challenges in the Systems Engineering discipline, namely, the reuse of system artefacts. Considering the fact that models are becoming the main type of system artefact, it is necessary to provide the capability to properly and efficiently represent and retrieve the generated models. In light of this, traditional information retrieval techniques have been widely studied to match existing software assets according to a set of capabilities or restrictions. However, there is much more at stake than the simple retrieval of models or even any piece of knowledge. An environment for model reuse must provide the proper mechanisms to (1) represent any piece of data, information, or knowledge under a common and shared data model, and (2) provide advanced retrieval mechanisms to elevate the meaning of information resources from text-based descriptions to concept-based ones. This need has led to novel methods using word embeddings and vector-based representations to semantically encode information. Such methods are applied to encode the information of physical models while preserving their underlying semantics. In this study, a text corpus from MATLAB Simulink models was preprocessed using Natural Language Processing (NLP) techniques and trained to generate word vector representations. Then, the presented method was validated using a testbed of MATLAB Simulink physical models in which verbalisations of models are transformed into vectors. The effectiveness of the proposed solution was assessed through a use case study. Evaluation of the results demonstrates a precision value of 0.925, a recall value of 0.865, and an F1 score of 0.884.

Fusion of XLNet and BiLSTM-TextCNN for Weibo Sentiment Analysis in Spark Big Data Environment

This article proposes a Weibo sentiment analysis method to improve traditional algorithms' analysis efficiency and accuracy. The proposed algorithm uses deep learning in the Spark big data environment. First, the input data are converted into dynamic word vector representations using the Chinese version of the XLNet model. Then, dual-channel feature extraction is performed on the data using TextCNN and BiLSTM. The proposed algorithm uses an attention mechanism to allocate computing resources efficiently and realizes feature fusion and data classification. Comparative experiments are conducted on two public datasets under identical experimental conditions. In the NLPCC2014 and NLPCC2015 datasets, the proposed model improves the precision and F1 metrics by at least 4.26% and 2.64%, respectively. In the weibo_senti_100k dataset, the proposed model improves the precision and F1 metrics by at least 4.66% and 2.69%, respectively. The results indicate that the proposed method has better sentiment analysis and prediction abilities than existing methods.

Can language models be used for real-world urban-delivery route optimization?

Language models have contributed to breakthroughs in interdisciplinary research, such as protein design and molecular dynamics understanding. In this study, we reveal that beyond language, representations of other entities, such as human behaviors, that are mappable to learnable sequences can be learned by language models. One compelling example is the real-world delivery route optimization problem. We here propose a novel approach based on the language model to optimize delivery routes on the basis of drivers' historical experiences. Although a broad range of optimization-based approaches have been designed to optimize delivery routes, they do not capture the implicit knowledge of complex delivery operating environments. The model we propose integrates this knowledge in the route optimization process by learning from driving behaviors in experienced drivers. A real-world delivery route that preserves drivers' implicit behavioral patterns is first analogized to a sentence in natural language. Through unsupervised learning, we then learn the vector representations of words and infer the drivers' delivery chains on the basis of the tailored chain-reaction-based algorithm. We also provide insights into the fusion of language models and operations research methods. In our approach, language models are applied to learn drivers' delivery behaviors and infer new deliveries at the delivery zone level, while the classic traveling salesman problem (TSP) model is embedded into the hybrid framework for intra-zone optimization. Numerical experiments performed on real-world data from Amazon's delivery service demonstrate that the proposed approach outperforms pure optimization, supporting the effectiveness, efficiency, and extensibility of our model. As a versatile approach, the proposed framework can easily be extended to various disciplines in which the data follow certain grammar rules. We anticipate that our work will serve as a stepping stone toward the understanding and application of language models in tackling interdisciplinary research problems.

Research on Emotional Analysis of Online Book Reviews Based on Word2Vec Method

At present, emotional analysis in the field of natural language processing is gradually becoming a popular research direction on social media. With the rise of artificial intelligence technology and deep learning algorithms, word vector representation methods based on neural networks have emerged and emerged in the field of sentiment analysis. According to the current research results of experts, the Word2Vec model has become the mainstream word vector representation method. The algorithm model is based on continuous optimization and iteration of random initialization word vectors, ultimately obtaining a stable performance word vector representation. If the research goal is to obtain sentence vectors, weighting processing is also required. Therefore, after systematically summarizing and analyzing the representation techniques of initialized word vectors and solving sentence vectors through word vector weighting, this article proposes corresponding improvement methods. This article improves the initialization algorithm of its word vector based on the Word2Vec model. On the basis of the original randomly generated word vector, the correlation information of words is added to change the distribution characteristics of the initial word vector, thereby enhancing the performance of the word vector. In the comparative experiment of the user comment dataset, the improved model showed a 1% to 3% improvement in sentiment polarity classification accuracy compared to the other model.