Vector Representations Of Words Research Articles

Improving Word Embedding on Malayalam Corpus

Abstract: NLP is natural language processing or neuro linguis-tic programming.Natural languages like malayalam are highly inflectional and agglutinative in nature.This is problematic whendealing with nlp based malayalam applications.So that inorder toimprove performance of malayalam nlp based applications, wordembedding improvement on malayalam corpus is needed.The improvement is based on converting the words contained inthe malayalam corpus into a standardised means removingall inflectional parts in the words in the existing malayalamcorpus ie taking root words only.All that needed is a stemmer.Inthis project i have used a malayalam morphological analyserfor taking root words of all words in the existing malayalam corpus.The advanatge of removing inflectional parts from allwords is that we can reduce the sparsity in the existing malayalam corpus.Also there will be a high hike in frequency of wordsin the resulting corpus,then the space and time complexity of wordembedding representation of the existing corpus willdecreases.According to zipfs law by increasing frequency ofwords performance of neural word embedding will increases. Zipfs Law is a discrete probability distribution that tells you the probability of encountering a word in a given corpus.By applying zipfs law am proposing there will be improvement on malyalam wordembedding.Here using fasttext, word embeddingsare performed and capture dense word vector representation ofthe malayalam corpus with dimensionality reduction from thesparse word co-occurence matrix.The improvement is mainly used for wordnet,analogy,ontology based malayalam applications.Index Terms—Morphological Analyzer,Zipfs law, Preprocessing, Testing, Training

A text sentiment analysis method based on BiGRU and capsule network

Most of the existing sentiment analysis methods based on product review texts rarely consider the aspect features of the review texts, and the relevant analysis models do not consider the long-term contextual dependency features and local text features at the same time, thus affecting the accuracy of sentiment analysis. A text sentiment analysis method based on a bidirectional gated recurrent network (BiGRU) and a capsule network is proposed. This method first uses a word frequency statistics-based method to extract the aspect features of the review text and integrates them into the word vector representation, thereby effectively improving the expressive power of the word vector. Then, BiGRU is used to extract the long-term contextual dependency features of the text, and the capsule network is used to extract the local features of the text, thereby achieving high-precision text sentiment analysis based on aspects. Experimental results on real datasets show that the proposed method is superior to existing sentiment analysis models such as bidirectional long short-term memory network (BiLSTM), CNN-LSTM, and TextCNN in terms of evaluation indicators such as accuracy, precision, recall, and score.

Microblog Negative Comments Data Analysis Model Based on Multi-scale Convolutional Neural Network and Weighted Naive Bayes Algorithm

As a form of public supervision, Microblog’s negative reviews allow people to share their opinions and experiences and express dissatisfaction with unfair and unreasonable phenomena. This form of supervision has the potential to promote social fairness, drive governments, businesses, and individuals to correct mistakes and enhance transparency. To characterize the sentiment trend and determine the influence of Microblog negative reviews, we propose a multi-scale convolutional neural network and weighted naive bayes algorithm (MCNN–WNB). We define the feature vector characterization index for Microblog negative review data and preprocess the data accordingly. We quantify the relationship between attributes and categories using the weighted Naive Bayes method and use the quantification value as the weighting coefficient for the attributes, addressing the issue of decreased classification performance in traditional methods. We introduce a sentiment classification model based on word vector representation and a multi-scale convolutional neural networks to filter out Microblog negative review data. We conduct simulation experiments using real data, analyzing key influencing parameters such as convergence time, training set sample size, and number of categories. By comparing with K-means, Naive Bayes algorithm, Spectral Clustering algorithm and Autoencoder algorithm, we validate the effectiveness of our proposed method. We discover that the convergence time of the MCNN–WNB algorithm increases as the number of categories increases. The average classification accuracy of the algorithm remains relatively stable with varying test iterations. The algorithm’s precision increases with the number of training set samples and eventually stabilizes.

A multi-model attention based CNN-BiLSTM model for personality traits prediction based on user behavior on social media

The prediction of an individual’s personality traits through the analysis of their online social media activities is an area of research that has gained considerable attention in the digital era. The statistical data derived from people’s thoughts that are conveyed through their status updates on social media serves as an essential resource for understanding the various aspects of human behavior and personality. The present study is motivated by the various applications associated with personality prediction, such as targeted advertising, personalized entertainment, and customized recommendations. A multi-model attention-based convolutional neural network-bidirectional long short-term memory (CNN+BiLSTM) is proposed in the present work for personality traits prediction. The proposed model utilizes pre-trained language models, such as Global Vectors for Word Representation (GloVe) and Bidirectional Encoder Representations from Transformers (BERT), to create vector representations of words, which effectively captures word semantics. Additionally, network features such as size, betweenness, transitivity, and density are integrated with text features to enhance personality traits prediction. The results highlight the contextual understanding of the BERT model and emphasize the effectiveness of the proposed model, particularly the fusion of the attention layer with the CNN+BiLSTM architecture, which significantly improves information extraction and prediction capabilities. The integration of social network features strengthens the classifier’s ability to predict personality traits, resulting in improved overall performance.

Prediction of protein secondary structure by the improved TCN-BiLSTM-MHA model with knowledge distillation

Secondary structure prediction is a key step in understanding protein function and biological properties and is highly important in the fields of new drug development, disease treatment, bioengineering, etc. Accurately predicting the secondary structure of proteins helps to reveal how proteins are folded and how they function in cells. The application of deep learning models in protein structure prediction is particularly important because of their ability to process complex sequence information and extract meaningful patterns and features, thus significantly improving the accuracy and efficiency of prediction. In this study, a combined model integrating an improved temporal convolutional network (TCN), bidirectional long short-term memory (BiLSTM), and a multi-head attention (MHA) mechanism is proposed to enhance the accuracy of protein prediction in both eight-state and three-state structures. One-hot encoding features and word vector representations of physicochemical properties are incorporated. A significant emphasis is placed on knowledge distillation techniques utilizing the ProtT5 pretrained model, leading to performance improvements. The improved TCN, achieved through multiscale fusion and bidirectional operations, allows for better extraction of amino acid sequence features than traditional TCN models. The model demonstrated excellent prediction performance on multiple datasets. For the TS115, CB513 and PDB (2018–2020) datasets, the prediction accuracy of the eight-state structure of the six datasets in this paper reached 88.2%, 84.9%, and 95.3%, respectively, and the prediction accuracy of the three-state structure reached 91.3%, 90.3%, and 96.8%, respectively. This study not only improves the accuracy of protein secondary structure prediction but also provides an important tool for understanding protein structure and function, which is particularly applicable to resource-constrained contexts and provides a valuable tool for understanding protein structure and function.

Impact of Effective Word Vectors on Deep Learning Based Subjective Classification of Online Reviews

Sentiment Analysis tasks are made considerably simpler by extracting subjective statements from online reviews, thereby reducing the overhead of the classifiers. The review dataset encompasses both subjective and objective sentences, where subjective writing expresses the author's opinions, and objective text presents factual information. Assessing the subjectivity of review statements involves categorizing them as objective or subjective. The effectiveness of word vectors plays a crucial role in this process, as they capture the semantics and contextual cues of a subjective language. This study investigates the significance of employing sophisticated word vector representations to enhance the detection of subjective reviews. Several methodologies for generating word vectors have been investigated, encompassing both conventional approaches, such as Word2Vec and Global Vectors for word representation, and recent innovations, such as like Bidirectional Encoder Representations from Transformers (BERT), ALBERT, and Embeddings from Language Models. These neural word embeddings were applied using Keras and Scikit-Learn. The analysis focuses on Cornell subjectivity review data within the restaurant domain, and metrics evaluating performance, such as accuracy, F1-score, recall, and precision, are assessed on a dataset containing subjective reviews. A wide range of conventional vector models and deep learning-based word embeddings are utilized for subjective review classification, frequently in combination with deep learning architectures like Long Short-Term Memory (LSTM). Notably, pre-trained BERT-base word embeddings exhibited exceptional accuracy of 96.4%, surpassing the performance of all other models considered in this study. It has been observed that BERT-base is expensive because of its larger structure.

Neurocomputer System of Semantic Analysis of the Text in the Kazakh Language

The purpose of the study is to solve an extreme mathematical problem—semantic analysis of natural language, which can be used in various fields, including marketing research, online translators, and search engines. When training the neural network, data training methods based on the latent Dirichlet allocation model and vector representation of words were used. This study presents the development of a neurocomputer system used for the purpose of semantic analysis of the text in the Kazakh language, based on machine learning and the use of the latent Dirichlet allocation model. In the course of the study, the stages of system development were considered, regarding the text recognition algorithm. The Python programming language was used as a tool using libraries that greatly simplify the process of creating neural networks, including the Keras library. An experiment was conducted with the involvement of experts to test the effectiveness of the system, the results of which confirmed the reliability of the data provided by the system. The papers of modern computer linguists dealing with the problems of natural language processing using various technologies and methods are considered.

Scoring alignments by embedding vector similarity.

Sequence similarity is of paramount importance in biology, as similar sequences tend to have similar function and share common ancestry. Scoring matrices, such as PAM or BLOSUM, play a crucial role in all bioinformatics algorithms for identifying similarities, but have the drawback that they are fixed, independent of context. We propose a new scoring method for amino acid similarity that remedies this weakness, being contextually dependent. It relies on recent advances in deep learning architectures that employ self-supervised learning in order to leverage the power of enormous amounts of unlabelled data to generate contextual embeddings, which are vector representations for words. These ideas have been applied to protein sequences, producing embedding vectors for protein residues. We propose the E-score between two residues as the cosine similarity between their embedding vector representations. Thorough testing on a wide variety of reference multiple sequence alignments indicate that the alignments produced using the new $E$-score method, especially ProtT5-score, are significantly better than those obtained using BLOSUM matrices. The new method proposes to change the way alignments are computed, with far-reaching implications in all areas of textual data that use sequence similarity. The program to compute alignments based on various $E$-scores is available as a web server at e-score.csd.uwo.ca. The source code is freely available for download from github.com/lucian-ilie/E-score.

Meta heuristic approaches for sentiment analysis

AbstractOn the Internet, online microblogging and social networks have experienced tremendous growth. Millions of individuals share opinions on social networking platforms including Twitter, Facebook, YouTube, and Microblogging sites. Millions of individuals express their thoughts on social platforms namely Twitter, Facebook, YouTube, and microblogging sites. As user‐generated content is growing to a huge extent on the internet, the analysis, extraction identification and classification of opinion and sentiment polarity of different aspects has now become a challenging issue in sentiment analysis. In sentiment analysis, aspect level opinion mining, opinion spam review detection, extraction and representation of most relevant accessed data from the user‐generated data is a difficult process. The word vector representation using existing deep learning techniques affects the performance in aspect level opinion mining. Spammers propagate false opinion for payment exchange, which leads to degradation of the financial growth, business, and fame of the organization. To overcome the limitations aspect level opinion mining is used to recognize sentiment and polarity of aspects in the specified text. This work presents various input feature vectors of deep learning approaches aiming to improve performance in aspect level opinion mining. Next, three meta‐heuristic algorithms with k‐means clustering approach for classification of opinion spam reviews on social media transit tweets is presented. The proposed approaches are evaluated and compared with other existing approaches as well as other benchmark datasets namely Restaurant and Amazon reviews.

A novel joint extraction model based on cross-attention mechanism and global pointer using context shield window

Relational triple extraction is a critical step in knowledge graph construction. Compared to pipeline-based extraction, joint extraction is gaining more attention because it can better utilize entity and relation information without causing error propagation issues. Yet, the challenge with joint extraction lies in handling overlapping triples. Existing approaches adopt sequential steps or multiple modules, which often accumulate errors and interfere with redundant data. In this study, we propose an innovative joint extraction model with cross-attention mechanism and global pointers with context shield window. Specifically, our methodology begins by inputting text data into a pre-trained RoBERTa model to generate word vector representations. Subsequently, these embeddings are passed through a modified cross-attention layer along with entity type embeddings to address missing entity type information. Next, we employ the global pointer to transform the extraction problem into a quintuple extraction problem, which skillfully solves the issue of overlapping triples. It is worth mentioning that we design a context shield window on the global pointer, which facilitates the identification of correct entities within a limited range during the entity extraction process. Finally, the capability of our model against malicious samples is improved by adding adversarial training during the training process. Demonstrating superiority over mainstream models, our approach achieves impressive results on three publicly available datasets.

A Study on the Emotional Tendency of Aquatic Product Quality and Safety Texts Based on Emotional Dictionaries and Deep Learning

A Study on the Emotional Tendency of Aquatic Product Quality and Safety Texts Based on Emotional Dictionaries and Deep Learning

Identifying the Risk Factors of Allergic Rhinitis Based on Zhihu Comment Data Using a Topic-Enhanced Word-Embedding Model: Mixed Method Study and Cluster Analysis.

Allergic rhinitis (AR) is a chronic disease, and several risk factors predispose individuals to the condition in their daily lives, including exposure to allergens and inhalation irritants. Analyzing the potential risk factors that can trigger AR can provide reference material for individuals to use to reduce its occurrence in their daily lives. Nowadays, social media is a part of daily life, with an increasing number of people using at least 1 platform regularly. Social media enables users to share experiences among large groups of people who share the same interests and experience the same afflictions. Notably, these channels promote the ability to share health information. This study aims to construct an intelligent method (TopicS-ClusterREV) for identifying the risk factors of AR based on these social media comments. The main questions were as follows: How many comments contained AR risk factor information? How many categories can these risk factors be summarized into? How do these risk factors trigger AR? This study crawled all the data from May 2012 to May 2022 under the topic of allergic rhinitis on Zhihu, obtaining a total of 9628 posts and 33,747 comments. We improved the Skip-gram model to train topic-enhanced word vector representations (TopicS) and then vectorized annotated text items for training the risk factor classifier. Furthermore, cluster analysis enabled a closer look into the opinions expressed in the category, namely gaining insight into how risk factors trigger AR. Our classifier identified more comments containing risk factors than the other classification models, with an accuracy rate of 96.1% and a recall rate of 96.3%. In general, we clustered texts containing risk factors into 28 categories, with season, region, and mites being the most common risk factors. We gained insight into the risk factors expressed in each category; for example, seasonal changes and increased temperature differences between day and night can disrupt the body's immune system and lead to the development of allergies. Our approach can handle the amount of data and extract risk factors effectively. Moreover, the summary of risk factors can serve as a reference for individuals to reduce AR in their daily lives. The experimental data also provide a potential pathway that triggers AR. This finding can guide the development of management plans and interventions for AR.

How Fast Do Distribution and Semantics of Polysemic Words Change?

Creation of large diachronic text corpora triggered studies of evolution of natural languages using quantitative methods in the last decade. For the first time, it became possible to raise questions about general patterns of lexical semantic change. In the work by W. Hamilton et al., 2016, a hypothesis was formulated about a more rapid change in the meanings of polysemic words. In this paper, we consider how frequency of words influence variations of semantic metrics. We use explicit word vector representations and Jensen-Shannon divergence as a measure of change in the distribution and semantics of a word. The obtained results show that changes in the frequency ratio of a word in different meanings can themselves lead to a change in a word distribution, even in the absence of any changes in its meaning. Therefore, the observations made in the work by W. Hamilton et al about correlation between the rate of change of words and their local cluster coefficient in the semantic connection network (considered as a quantitative measure of polysemy) can be fully explained by the effect described in the article.

Short Text Classification Model based on Pre-trained Language Model with Feature Fusion

In response to the low accuracy of Chinese short text classification in the current data mining field and the defects of existing deep learning models with more model parameters and higher time complexity, this paper proposes a new text classification model - short text classification model (ACBSM) based on pre-trained language model with feature expansion. In ACBSM, to address the problem of high dimensionality of text data without accurate text representation, the Bert model is used to train word vector representation to solve the problem of multiple meanings of a word. From the parallelization acceleration level, a parallel acceleration strategy of two-channel neural network is designed to improve the efficiency of the algorithm in processing massive data. To address the sparsity of text data and the more complex semantics, an attention mechanism is introduced and a CNN model is used to enhance the extraction of keyword information; secondly, BiSRU is used to capture the contextual features of the text, and finally, experimental validation is conducted on a news dataset. The experimental results show that ACBSM improves the accuracy of text classification to 95.83% under the same environment and dataset, and its classification performance is better than other text classification methods.

News Classification Using Bidirectional Long Short Term Memory and GloVe

The dissemination of information and news via online media encompasses not only established news platforms but also contributions from internet users, lacking oversight. News constitutes fact-grounded insights into ongoing occurrences. This research employed Bidirectional Long- and Short-Term Memory with Hyperparameter tuning on GloVe for news classification. This research aims to optimize news categorization through hyperparameter tuning on GloVe. GloVe facilitated the transformation of words into vector matrices, exploring its efficacy in news classification with hyperparameter tuning and Bi-LSTM for text analysis. Experiments encompassed untuned and hyperparameter-tuned approaches, employing GloVe's hyperparameters using Gridsearch and manual methods. GloVe's hyperparameter tuning reveals the potential for enhancing word vector representations. Surprisingly, non-hyperparameter tuned news classification yielded superior evaluation results compared to the hyperparameter approach. The untuned experiment achieved an accuracy of 0.98, while the gridsearch method yielded 0.85 accuracy, and hyperparameter tuning generated a 0.88 precision in the -11 model. These findings underscore the nuanced interplay of hyperparameters in optimizing text classification models like GloVe.

Improving Cross-Domain Aspect-Based Sentiment Analysis using Bert-BiLSTM Model and Dual Attention Mechanism

Data across different domains can be influenced by variations in language styles and expressions, making it challenging to migrate specialized words, particularly when focusing on aspectual words. This complexity poses difficulties in conducting cross-domain aspectbased sentiment analysis. The article begins by introducing BERT for generating word vectors as representations of training texts, enhancing text semantics in the word vector representation stage. To capture more nuanced interaction information and context-related details, the paper proposes the Bert-BiLSTM model with a dual attention mechanism(BBDAM), which divides the original input sequence into three parts: above, aspectual words, and below. A dual attention mechanism was used to assess the interaction of aspect words with the three aspects (above, below, and neighboring words) in the three discourse segments. This mechanism allows for the comprehensive extraction of interaction information. By comparing with other modeling approaches, the experimental results show that the BB-DAM model produces good results in fine-grained cross-domain sentiment analysis.

Construction of Online Interactive Teaching Platform for College English Based on Multilayer Perceptual Machine Modeling

Abstract Most of the existing English online teaching platforms are just digital replicas of offline teaching, which cannot fully utilize the advantages of information technology. Based on the learning theory of constructivism, this paper builds the functional architecture of a personalized English interactive teaching platform and completes the design of the main functional modules. According to the quantitative results of the platform’s user proposition requirements, a matrix description method for English test papers is proposed, and an algorithm is designed to realize the automatic generation of test papers. Meanwhile, the resource recommendation mechanism of the English platform is optimized using a multilayer perceptual machine. Unstructured English learning resources are processed based on the word vector representation method, and for the problem of insufficient processing of nonlinear data in the DN-CBR model, multilayer training is carried out using the MLP to improve the perceptual degree of this part of the data, and the design goals are verified through the test of the teaching platform. The correlation coefficient between self-assessment difficulty and topic difficulty is 0.368, and the correlation coefficient between self-assessment difficulty and positive error is 0.163, indicating that there is a moderate correlation between learners’ self-assessment difficulty and the difficulty of the system topics and between learners’ self-assessment difficulty and positive error. The MLP-based pushed questions’ difficulty is adapted to the learner’s personalization level by dynamically revising it based on their learning process data.

Entity recognition method for airborne products metrological traceability knowledge graph construction

The airborne system, as one of the complex and extensive subsystems of an aircraft, primarily performs critical flight assurance functions. The quality of its components has a direct impact on the aircraft's safety and reliability. Metrology documents comprehensively document the performance parameters throughout the entire product life cycle. The Metrological Traceability Knowledge Graph (MTKG) for airborne products offers decision support to engineers engaged in metrological tasks, ensuring the continuous high quality of the products. This paper introduces an entity recognition method for airborne product metrological traceability knowledge graph construction. First, the ontology for MTKG is developed. Next, a fine-tuned multi-network model is proposed. Named entities in the field of metrology are recognized through three stages: word vector representation, sentence feature extraction, and optimal label assignment. Meanwhile, active learning methods are incorporated to reduce the expense of data annotation. The proposed model is validated using an actual metrology corpus, and the experimental results demonstrate its superior performance compared to the other four baseline methods. Finally, the MTKG is developed using this approach, offering engineers intelligent applications, including metrological traceability analysis and traceability path reasoning within the process of product metrology. This enhances the metrology capabilities of airborne products and demonstrates the extensive potential of knowledge graphs in metrology.

Boosting Lightweight Sentence Embeddings with Knowledge Transfer from Advanced Models: A Model-Agnostic Approach

In this study, we investigate knowledge transfer between two distinct sentence embedding models: a computationally demanding, highly performant model and a lightweight model derived from word vector averaging. Our objective is to augment the representational power of the lightweight model by exploiting the sophisticated features of the robust model. Diverging from traditional knowledge distillation methods that align logits or hidden states of teacher and student models, our approach uses only the output sentence vectors of the teacher model for the alignment with the student models’s word vector representations. We implement two minimization techniques for this purpose: distance minimization and distance and perplexity minimization Our methodology uses WMT datasets for training, and the enhanced embeddings are validated via Google’s Analogy tasks and Meta’s SentEval datasets. We found that our proposed models intriguingly retained and conveyed information in a model-specific fashion.

A Named Entity Recognition Approach for Electronic Medical Records Using BERT Semantic Enhancement and BiLSTM

Aiming at the problems of missing local context features, single word vector representation, and low entity recognition accuracy, a method for e-medical recording with named entity recognition, which is based on BERT and model fusion, is proposed. First, with the model of BERT for pre-training, the preceding and following contextual information is fused for the enhancement of word semantic representation and alleviation of the problem of polysemy; second, the network of bi-directional long-short term memory is for obtaining the sequence feature matrix, generation of optimal sequence in global sense achieved through the conditional random field model; finally, data enhancement is used to alleviate the class imbalance and improve the model ability in generalization. Results of the experiments find model proposal measured by F1 on CCKS21 data set reaches 0.8548, which is 0.51% and 0.08% higher than models with ID-CNNs-CRF and multi-task RNN. This demonstrates the excellent performance of the method proposed in this paper in improving named entity recognition.