Word Representations Research Articles

Exploring Latent Themes-Analysis of various Topic Modelling Algorithms

This research explores the effectiveness of four common topic modelling methods for identifying latent themes and topics in unstructured text data: Latent Dirich- let Allocation(LDA), Non-Negative Matrix Factorization(NMF), Top2Vec, and BERTopic. Topic modelling is an essential method for gaining insights from massive amounts of textual data. Top2Vec and BERTopic are recent approaches that use unsupervised neural networks to develop distributed representations of texts and words, whereas NMF and LDA are traditional techniques frequently utilised for topic modelling. This document gives a timeline of important advances in topic modelling, including the development of NMF and LDA, as well as many refinements and additions to LDA. According to the study’s findings, BERTopic surpasses the other approaches, particularly in recognising overlapping and fine- grained subjects. This work emphasises the significance of text processing quality, the variety of subjects in the text, and the right selection of topic modelling methods in efficiently breaking down topics.

Texts Classification with the usage of Neural Network based on the Word2vec's Words Representation

Texts Classification with the usage of Neural Network based on the Word2vec's Words Representation

Learning Sentiment-Enhanced Word Representations by Fusing External Hybrid Sentiment Knowledge

Learning Sentiment-Enhanced Word Representations by Fusing External Hybrid Sentiment Knowledge

CSR-SVM: Compositional semantic representation for intelligent identification of engineering change documents based on SVM

Timely and accurate identification of engineering change documents plays an important role in the document management system of the Architecture, Engineering, and Construction (AEC) industry, which facilitates decision making. The current way of manual review and analysis may cause a delay or omission in information transmission, which may severely affect the project schedule and cost. This paper adopts text classification to explore the intelligent identification of unstructured engineering change documents. However, the paucity of available corpus and the limited number of engineering change texts are challenges in practice. Additionally, the semantic information of texts cannot be elegantly exploited. Especially there are unregistered words that interfere with semantics in engineering change texts. To tackle these problems, we propose a compositional semantic representation (CSR) and develop an SVM-based method named CSR-SVM. We introduce a language model to produce word embeddings; here, a domain dictionary is established for unregistered words. The embeddings are then exploited in CSR so that CSR can incorporate both key and global semantic representations. The former is obtained based on dependency parsing and word embeddings, and the latter is obtained according to all the word representations of a text. The CSR provides sufficient semantic representation for SVM in an efficient way. The advantages of CSR-SVM have been validated by experiments on a real-world dataset.

Sentiment Analysis of Social Network Comment Text Based on LSTM and Bert

This paper has the aim of solving problems in research studies on the analysis tasks of text emotion; the problems are the low utilization of text, the difficulty of effective information extraction, the failure of recognizing word polysemy with effectiveness. Thus, based on LSTM and Bert, the method of sentiment analysis on text is adopted. To be precise, word embedding of dataset in view of the skip-gram model is used for training course. In each sample, the word embeddings combine matric with the two-dimensional feature to be neural network input. Next, construction of analysis model for text sentiment combines Bert pre-training language model and long short-term memory (LSTM) network, using the word vector pre-trained by Bert instead of that trained in the traditional way to dynamically generate the semantic vector according to the word context. Finally, the semantic representation of words from text is improved by effectively identifying the polysemy of words, and the semantic vector is input into the LSTM to capture the semantic dependencies, thereby enhancing the ability to extract valid information. The Accuracy, Precision, Recall and F-Measure for the method of Bert–LSTM based on analysis of text sentiment are 0.89, 0.9, 0.84 and 0.87, indicating high value than the compared ones. Thus, the proposed method significantly outperforms the comparison methods in text sentiment analysis.

Deep Learning-based Fuzzy Translation Problem in Chinese-English Epidemic News Reporting

Abstract To smoothly realize the information conversion from the original language information to the target language, this paper constructs a deep learning-based fuzzy translation model for news reports so that the translated text can faithfully convey the meaning of the original language text information and achieve natural semantic equivalence. A neural probabilistic language model is used to construct objective functions in speech recognition and lexical annotation so that the translated text can provide a more appropriate linguistic representation of the polysemantic words in the original language text according to the differences in contextual morphology. A deep learning occurrence mechanism model is constructed through fuzzy semantic reasoning and fuzzy translation logic, and learning state indicators such as emotional interaction are designed to evaluate the occurrence status of fuzzy translation accurately. The simulation results show that the natural language processing (GLUE) test score of the deep learning-based fuzzy translation model for news reports is 89.8, 9.2, and 6.9 points higher than 80.6 and 82.9 for the other two models, respectively. The average error discrimination ability of the model designed in this paper is 93.57, and the average training set, development set, and test set values are 98.425, 10.16, and 45.95, respectively. Thus, it can be seen that the deep learning-based fuzzy translation model for news reports can more naturally and accurately respond to the dynamic changes in language, which promotes the rapid development of translation theory and practice.

Microsynt: Exploring the syntax of EEG microstates

Microstates represent electroencephalographic (EEG) activity as a sequence of switching, transient, metastable states. Growing evidence suggests the useful information on brain states is to be found in the higher-order temporal structure of these sequences. Instead of focusing on transition probabilities, here we propose “Microsynt”, a method designed to highlight higher-order interactions that form a preliminary step towards understanding the syntax of microstate sequences of any length and complexity. Microsynt extracts an optimal vocabulary of “words” based on the length and complexity of the full sequence of microstates. Words are then sorted into classes of entropy and their representativeness within each class is statistically compared with surrogate and theoretical vocabularies. We applied the method on EEG data previously collected from healthy subjects undergoing propofol anesthesia, and compared their “fully awake” (BASE) and “fully unconscious” (DEEP) conditions. Results show that microstate sequences, even at rest, are not random but tend to behave in a more predictable way, favoring simpler sub-sequences, or “words”. Contrary to high-entropy words, lowest-entropy binary microstate loops are prominent and favored on average 10 times more than what is theoretically expected. Progressing from BASE to DEEP, the representation of low-entropy words increases while that of high-entropy words decreases. During the awake state, sequences of microstates tend to be attracted towards “A – B – C” microstate hubs, and most prominently A – B binary loops. Conversely, with full unconsciousness, sequences of microstates are attracted towards “C – D – E” hubs, and most prominently C – E binary loops, confirming the putative relation of microstates A and B to externally-oriented cognitive processes and microstate C and E to internally-generated mental activity. Microsynt can form a syntactic signature of microstate sequences that can be used to reliably differentiate two or more conditions.

A Novel Fake-News Dataset and Detection System to Mitigate Cyber War with Emphasis on Nigerian News Events

Fake-news refers to a cyber-weapon launched through the social media, as, its consequence can result to the breakdown of law and order in the society both physically and on the cyber-social-space. In Nigeria, there is currently no established law that guides the use of social media. Therefore, the rate at which fake-news propagates is alarming. This paper presents a new dataset, with focus on Nigeria’s trending news such as EndSARS and Herdsmen attacks, which was further used to simulate Convolutional Neural Network (CNN) and Recurrent Neural Network (RNN) machine learning models to detect fake-news. The data were extracted from twitter using twitter Application Package Interface (API) and from facebook using a scraping tool. The dataset was encoded using Unicode escape function in python to make all characters accessible by the algorithm and tokenised using Global Vectors for Word Representation. The dataset was used to train CNN and RNN models built in python on google colab platform to detect fake-news using accuracy, sensitivity, recall and F1 score as evaluation metrics. Results showed that RNN performed better in terms of accuracy and precision, at 82.34% and 93.19% compared to 81.96% and 79.65% for CNN, F1 scores are approximately the same for both models and CNN performed better than RNN in terms of recall at 98.03% to 50.61% for RNN.

Unmasking deception: a CNN and adaptive PSO approach to detecting fake online reviews.

Online reviews play a critical role in modern word-of-mouth communication, influencing consumers' shopping preferences and purchase decisions, and directly affecting a company's reputation and profitability. However, the credibility and authenticity of these reviews are often questioned due to the prevalence of fake online reviews that can mislead customers and harm e-commerce's credibility. These fake reviews are often difficult to identify and can lead to erroneous conclusions in user feedback analysis. This paper proposes a new approach to detect fake online reviews by combining convolutional neural network (CNN) and adaptive particle swarm optimization with natural language processing techniques. The approach uses datasets from popular online review platforms like Ott, Amazon, Yelp, TripAdvisor, and IMDb and applies feature selection techniques to select the most informative features. The paper suggests using attention mechanisms like bidirectional encoder representations from transformers and generative pre-trained transformer, as well as other techniques like Deep contextualized word representation, word2vec, GloVe, and fast Text, for feature extraction from online review datasets. The proposed method uses a multimodal approach based on a CNN architecture that combines text data to achieve a high accuracy rate of 99.4%. This outperforms traditional machine learning classifiers in terms of accuracy, recall, and F measure. The proposed approach has practical implications for consumers, manufacturers, and sellers in making informed product choices and decision-making processes, helping maintain the credibility of online consumer reviews. The proposed model shows excellent generalization abilities and outperforms conventional discrete and existing neural network benchmark models across multiple datasets. Moreover, it reduces the time complexity for both training and testing.

Conceptual knowledge of writing words and numbers: Developmental data from preschool children

Children’s conceptual knowledge of writing words and numbers is an important aspect of their cognitive development. Children use notations as representations that have a communicative value and begin to learn about formal differences between writing words and writing numbers at an early age before the onset of formal schooling. The aim of the present study was to examine preschool children’s conceptual knowledge of writing words and numbers in an ecologically valid task with communicative value. One hundred and twenty Greek-speaking preschool children between the ages of 3 and 6 years old were assessed on the identity card task, which examined their production of notations for words and numbers during their effort to communicate personal information to others. The results demonstrated a developmental trend in the type of notation children produced with the younger children providing more “ambiguous” notations, for both tasks implying writing words and numbers, while as they got older, they provided more “writing-like” notations for the tasks implying writing words and more “number-like” notations for the tasks implying writing numbers. Understanding when and how children differentiate their symbolic representations for words and numbers can inform both theory and practice by expanding our understanding of whether certain constraints characterize the developmental course of a specific notational system. The findings of the present study could be incorporated in educational practice and enhance children’s emerging literacy and numeracy skills.

Sentiment analysis in SemEval: a review of sentiment identification approaches

<div class="page" title="Page 1"><div class="layoutArea"><div class="column"><div class="page" title="Page 1"><div class="layoutArea"><div class="column"><p>ocial media platforms are becoming the foundations of social interactions including messaging and opinion expression. In this regard, sentiment analysis techniques focus on providing solutions to ensure the retrieval and analysis of generated data including sentiments, emotions, and discussed topics. International competitions such as the International Workshop on Semantic Evaluation (SemEval) have attracted many researchers and practitioners with a special research interest in building sentiment analysis systems. In our work, we study top-ranking systems for each SemEval edition during the 2013-2021 period, a total of 658 teams participated in these editions with increasing interest over years. We analyze the proposed systems marking the evolution of research trends with a focus on the main components of sentiment analysis systems including data acquisition, preprocessing, and classification. Our study shows an active use of preprocessing techniques, an evolution of features engineering and word representation from lexicon-based approaches to word embeddings, and the dominance of neural networks and transformers over the classification phasefostering the use of ready-to-use models. Moreover, we provide researchers with insights based on experimented systems which will allow rapid prototyping of new systems and help practitioners build for future SemEval editions.</p></div></div></div></div></div></div>

The Arabic Diglossia Reality: The Effect of Specific Story Reading in Kindergarten on Reading Acquisition in First Grade

ABSTRACT Due to the diglossic nature of the Arabic language, Arabic-speaking children enter their first year of school with immaturity in literary language and word representations in their mental lexicon. This study examined the effects of interactive story reading in kindergarten on future reading skills and whether this effect can be generalized to reading skills of infrequent words. 102 Arabic-speaking kindergarten children were divided into two groups: 1) the intervention group was based on a first-grade textbook; 2) the control group was based on classic stories from the kindergarten library. The program was implemented for three months during the kindergarten year. The findings show that the intervention group’s scores were higher than the control group in measures of represented/practiced and unfamiliar items. Additionally, the gain of the intervention group in reading unfamiliar items provide evidence for a generalization effect of the intervention program. We conclude that the negative impact of diglossia on reading acquisition in Arabic can be reduced during kindergarten.

Text Vectorization Techniques Based on Wordnet

Abstract The utilization of text vectorization techniques has become essential for numerous classification tasks in present-day natural language processing. Word embedding methods commonly used today, such as Word2Vec, GloVe, etc., are based on the semantic similarity of words. WordNet, as a lexical database of words, provides a rich source of semantic information. In our article, we propose a text vectorization technique using extended text data with the data augmentation method, specifically by replacing words with their synonyms obtained from WordNet. The results obtained from text classification tasks using multiple classifiers demonstrate that expanding the corpus with this method leads to improved vector representations of words.

WITHDRAWN: Visual interpretation for contextualized word representation

WITHDRAWN: Visual interpretation for contextualized word representation

Do the US president's tweets better predict oil prices? An empirical examination using long short-term memory networks

ABSTRACT The price of oil is highly complex to predict as it is impacted by global demand and supply, geopolitical events, and market sentiment. The accuracy of such predictions, however, has far-reaching implications for supply chain performance, portfolio management, and expected stock market returns. This paper contributes to the oil price prediction literature by evaluating the predictive impact of the US President's communication on Twitter, while benchmarking various Natural Language Processing (NLP) techniques, including Term Frequency-Inverse Document Frequency (TF-IDF), Word2Vec, Doc2Vec, Global Vectors for Word Representation (GloVe), and Bidirectional Encoder Representations from Transformers (BERT). These techniques are combined with a deep neural network Long Short-Term Memory (LSTM) architecture using a five-day lag for both the oil price and the textual Twitter data. The data was collected during the term of US President Donald Trump, resulting in 1449 days of crude oil price prediction and a total of 16,457 tweets. The study is validated for Brent and West Texas Intermediate blends, using the daily price of a barrel of crude oil as the target variable. The results confirm that including the US President's tweets significantly increases the predictive power of oil price prediction models, and that an LSTM architecture with BERT as NLP technique has the best performance.

Texts Classification with the usage of Neural Network based on the Word2vec’s Words Representation

Assigning the submitted text to one of the predetermined categories is required when dealing with application-oriented texts. There are many different approaches to solving this problem, including using neural network algorithms. This article explores using neural networks to sort news articles based on their category. Two word vectorization algorithms are being used — The Bag of Words (BOW) and the word2vec distributive semantic model. For this work the BOW model was applied to the FNN, whereas the word2vec model was applied to CNN. We have measured the accuracy of the classification when applying these methods for ad texts datasets. The experimental results have shown that both of the models show us quite the comparable accuracy. However, the word2vec encoding used for CNN showed more relevant results, regarding to the texts semantics. Moreover, the trained CNN, based on the word2vec architecture, has produced a compact feature map on its last convolutional layer, which can then be used in the future text representation. I.e. Using CNN as a text encoder and for learning transfer.

Delta-Band Neural Responses to Individual Words Are Modulated by Sentence Processing.

To understand language, we need to recognize words and combine them into phrases and sentences. During this process, responses to the words themselves are changed. In a step toward understanding how the brain builds sentence structure, the present study concerns the neural readout of this adaptation. We ask whether low-frequency neural readouts associated with words change as a function of being in a sentence. To this end, we analyzed an MEG dataset by Schoffelen et al. (2019) of 102 human participants (51 women) listening to sentences and word lists, the latter lacking any syntactic structure and combinatorial meaning. Using temporal response functions and a cumulative model-fitting approach, we disentangled delta- and theta-band responses to lexical information (word frequency), from responses to sensory and distributional variables. The results suggest that delta-band responses to words are affected by sentence context in time and space, over and above entropy and surprisal. In both conditions, the word frequency response spanned left temporal and posterior frontal areas; however, the response appeared later in word lists than in sentences. In addition, sentence context determined whether inferior frontal areas were responsive to lexical information. In the theta band, the amplitude was larger in the word list condition ∼100 milliseconds in right frontal areas. We conclude that low-frequency responses to words are changed by sentential context. The results of this study show how the neural representation of words is affected by structural context and as such provide insight into how the brain instantiates compositionality in language.SIGNIFICANCE STATEMENT Human language is unprecedented in its combinatorial capacity: we are capable of producing and understanding sentences we have never heard before. Although the mechanisms underlying this capacity have been described in formal linguistics and cognitive science, how they are implemented in the brain remains to a large extent unknown. A large body of earlier work from the cognitive neuroscientific literature implies a role for delta-band neural activity in the representation of linguistic structure and meaning. In this work, we combine these insights and techniques with findings from psycholinguistics to show that meaning is more than the sum of its parts; the delta-band MEG signal differentially reflects lexical information inside and outside sentence structures.

Automatic text summarization using deep reinforced model coupling contextualized word representation and attention mechanism

Automatic text summarization using deep reinforced model coupling contextualized word representation and attention mechanism

Space Metaphor in the Representation of Concrete and Abstract Words: An Embodied Cognition Perspective

Research on embodied cognition is becoming increasingly crucial as society develops. To study embodied spatial metaphor, this study examines the relationship between perceived distance and understanding of abstract and concrete words. Concrete words are usually considered to be relatively close to peoples lives, while abstract words are not. Thirty-five participants between the ages of 18 and 29 participated in the study. They were asked to make either concrete or abstract responses quickly and accurately to words displayed on a screen, where words were presented to be either distance or in close proximity to the participant. The response time for correct answers was recorded. The results show that the reaction time of the abstract words with a short distance is not significantly longer than that of the abstract words with a long distance, while the results of the concrete words are consistent. According to the results, it can conclude that there is no significant relationship between spatial metaphor and the nature of words.

Evaluation of Accuracy Degradation Resulting from Concept Drift in a Fake News Detection System Using Emotional Expression

Fake news on social media has become a social problem. Fake news refers to false information that is deliberately intended to deceive people. Several studies have been conducted on automatic detection systems that reduce the damage caused by fake news. However, most studies address the improvements made in detection accuracy, and real-world operations are rarely discussed. As the contents and expressions of fake news change over time, a model with a high detection accuracy loses accuracy after a few years. This phenomenon is called concept drift. As most conventional methods employ word representations, these methods exhibit accuracy degradation resulting from changes in word fads and usage. However, methods using the sentiment information of words can identify inflammatory sentences, which is a characteristic of fake news, and may suppress performance degradation caused by concept drift. In this study, a model using vector representations obtained from an emotion dictionary was compared with a model using conventional word embedding. Subsequently, we verified the resistance of the model to performance degradation. The results revealed the method using sentiment representation is less susceptible to concept drift. Models and learning methods that can achieve both detection accuracy and resistance to accuracy degradation can enable further development of fake news detection systems.