Word Representation Method Research Articles

Predictors of explicit and implicit anthropomorphism in house facades.

Anthropomorphism describes the tendency to endow objects with human characteristics, with some individuals being more inclined to do this than others. In an ambiguous environment, this phenomenon can offer guidance. This study investigates the relationship between self-reported attribution and evoked anthropomorphism when viewing house facades. Data was collected from three countries (Germany, Denmark, Canada; N = 305). Implicit house anthropomorphism was measured using the Global Vectors for Word Representation method. Explicit anthropomorphism was assessed using the Individual Differences in Anthropomorphism Questionnaire (IDAQ) and a specific House Anthropomorphism Score (EHAS). No significant relationship was found between implicit and explicit house anthropomorphism. Individual IDAQ scores were significantly associated with EHAS across all participants, regardless of country. Additionally, a high degree of agreement in explicit ratings between countries suggests that cultural differences are rather negligible. When objects are given human personality traits and people interact with them because emotions are triggered, it is important to understand which aspects elicit positive and reactive behaviors. In particular, houses, which have high psychological significance as objects of self-expression, might contribute to well-being, so research in this area can provide important knowledge for urban design and architecture.

Sentiment analysis of coronavirus data with ensemble and machine learning methods

The coronavirus pandemic has distanced people from social life and increased the use of social media. People's emotions can be determined with text data collected from social media applications. This is used in many fields, especially in commerce. This study aims to predict people's sentiments about the pandemic by applying sentiment analysis to Twitter tweets about the pandemic using single machine learning classifiers (Decision Tree-DT, K-Nearest Neighbor-KNN, Logistic Regression-LR, Naïve Bayes-NB, Random Forest-RF) and ensemble learning methods (Majority Voting (MV), Probabilistic Voting (PV), and Stacking (STCK)). After vectorizing the tweets using two predictive methods, Word2Vec (W2V) and Doc2Vec, and two traditional word representation methods, Term Frequency-Inverse Document Frequency (TF-IDF) and Bag of Words (BOW), classification models built using single machine learning classifiers were compared to models built using ensemble learning methods (MV, PV and STCK) by heterogeneously combining single machine classifier algorithms. Accuracy (ACC), F-measure (F), precision (P), and recall (R) were used as performance measures, with training/test separation rates of 70%-30% and 80%-20%, respectively. Among these models, the ACC of ensemble learning models ranged from 89% to 73%, while the ACC of single classifier models ranged from 60% to 80%. Among the ensemble learning methods, STCK with Doc2Vec text representation/embedding method gave the best ACC result of 89%. According to the experimental results, ensemble models built with heterogeneous machine learning classifier algorithms gave better results than single machine learning classifier algorithms.

Investigating Computational Identity and Empowerment of The Students Studying Programming: A Text Mining Study

In this study, it is aimed to predict the data obtained from the answers given by the students who receive programming education to open-ended questions with text mining algorithms. Thus, text-based data on computational identity and programming empowement were analyzed and the performances of different algorithms were compared. The participants of the research consisted of 646 students whose age range was between 12-20 and who received programming education. An electronic form consisting of open-ended questions was prepared to collect the opinions of the students who received programming education. A total of six open-ended questions have been prepared about computational identity and (3 questions) and programming empowerment (3 questions). The text mining process was followed in the analysis of the data set. Analyzes were made in Python 3.8 program. In the study, the performance of Word2vec (W2v) and Term Frequency-Inverse Document Frequency (TF-IDF) word representation methods with five machine learning algorithms were compared: (a) Logistic regression, (b) Decision tree, (c) Support Vector Machines, (d) Random Forest, (e) Neural Network. Regarding computational identity, the highest prediction accuracy was found in artificial neural network (tf-idf) and logistic regression (tf-idf) algorithms. These algorithms have an accuracy rate of 93% regarding computational identity. It was determined that the logistic regression (tf-idf) method reached the highest accuracy prediction rate (96%) in programming empowement. Following this method, the accuracy rate of random forest (tf-idf), support vector machine (tf-idf) and artificial neural network (tf-idf) algorithms was 94%. The fact that these obtained values are above 90% indicates that the estimation performance is sufficient.

General Words Representation Method for Modern Language Model

This paper proposes a new word representation method emphasizes general words over specific words. The main motivation for developing this method is to address the weighting bias in modern Language Models (LMs). Based on the Transformer architecture, contemporary LMs tend to naturally emphasize specific words through the Attention mechanism to capture the key semantic concepts in a given text. As a result, general words, including question words are often neglected by LMs, leading to a biased word significance representation (where specific words have heightened weight, while general words have reduced weights). This paper presents a case study, where general words' semantics are as important as specific words' semantics, specifically in the abstractive answer area within the Natural Language Processing (NLP) Question Answering (QA) domain. Based on the selected case study datasets, two experiments are designed to test the hypothesis that "the significance of general words is highly correlated with its Term Frequency (TF) percentage across various document scales”. The results from these experiments support this hypothesis, justifying the proposed intention of the method to emphasize general words over specific words in any corpus size. The output of the proposed method is a list of token (word)-weight pairs. These generated weights can be used to leverage the significance of general words over specific words in suitable NLP tasks. An example of such task is the question classification process (classifying question text whether it expects factual or abstractive answer). In this context, general words, particularly the question words are more semantically significant than the specific words. This is because the same specific words in different questions might require different answers based on their question words (e.g. "How many items are on sale?" and "What items are on sale?" questions). By employing the general weight values produced by this method, the weightage of question and specific words can be heightened, making it easier for the classification system to differentiate between these questions. Additionally, the token (word)-weight pair list is made available online at https://www.kaggle.com/datasets/saliimiabbas/genwords-weight.

TRSAv1: A new benchmark dataset for classifying user reviews on Turkish e-commerce websites

The amount of data produced significantly increased with the development of Internet technologies. Accordingly, the importance of natural language processing studies increased, and this topic became one of the most studied artificial intelligence subjects. Even though it is a popular topic that is widely studied on, not enough studies have been conducted on the Turkish language. Even the studies conducted in Turkey are primarily on English and other natural languages instead of Turkish. The lack of a Turkish dataset is the most crucial reason for the lack of studies. Therefore, to create a solution, user reviews on e-commerce websites were collected and labelled reviews as positive, negative and neutral, and a new and unique dataset consisting of 150,000 reviews was created. This dataset was named TRSAv1, which was publicly shared with the researchers will contribute to the Turkish natural language processing studies; however, the effect of different word representation methods on algorithm performance was examined in detail, and the results were compared.

Unsupervised cross-lingual model transfer for named entity recognition with contextualized word representations.

Named entity recognition (NER) is one fundamental task in the natural language processing (NLP) community. Supervised neural network models based on contextualized word representations can achieve highly-competitive performance, which requires a large-scale manually-annotated corpus for training. While for the resource-scarce languages, the construction of such as corpus is always expensive and time-consuming. Thus, unsupervised cross-lingual transfer is one good solution to address the problem. In this work, we investigate the unsupervised cross-lingual NER with model transfer based on contextualized word representations, which greatly advances the cross-lingual NER performance. We study several model transfer settings of the unsupervised cross-lingual NER, including (1) different types of the pretrained transformer-based language models as input, (2) the exploration strategies of the multilingual contextualized word representations, and (3) multi-source adaption. In particular, we propose an adapter-based word representation method combining with parameter generation network (PGN) better to capture the relationship between the source and target languages. We conduct experiments on a benchmark ConLL dataset involving four languages to simulate the cross-lingual setting. Results show that we can obtain highly-competitive performance by cross-lingual model transfer. In particular, our proposed adapter-based PGN model can lead to significant improvements for cross-lingual NER.

Sentiment Classification Performance Analysis Based on Glove Word Embedding

Representation of words in mathematical expressions is an essential issue in natural language processing. In this study, data sets in different categories are classified as positive or negative according to their content. Using the Glove (Global Vector for Word Representation) method, which is one of the word embedding methods, the effect of the vector set based on the word similarities previously calculated on the classification performance has been analyzed. In this study, the effect of pretrained, embedded and deterministic word embedding classification performance has analyzed by using Long Short Term Memory (LSTM). The porposed LSTM based deep learning model has been tested on three different data sets and the results was evaluated.

LogClass: Anomalous Log Identification and Classification With Partial Labels

Logs are imperative in the management process of networks and services. However, manually identifying and classifying anomalous logs is time-consuming, error-prone, and labor-intensive. Additionally, rule-based approaches cannot tackle the challenges underlying anomalous log identification and classification resulting from new types of logs and partial labels. We propose LogClass, a framework to automatically and robustly identify and classify anomalous logs for network and service based on partial labels . LogClass combines a word representation method, a positive and unlabeled learning (PU learning) model, and a machine learning classifier. Besides, we propose a novel Inverse Location Frequency (ILF) method to weight the words of logs in feature construction properly. We evaluate the performance of LogClass based on 18 million+ real-world switch logs and six public log datasets. It achieves 99.56% and 98% F1 scores in anomalous log identification on switch logs and publicly available supercomputer logs, respectively, and very-close-to-one F1 score in anomalous log classification. Moreover, we have conducted extensive experiments to demonstrate LogClass’ superior performance in addressing partial labels and new types of logs.

Text Classification Based on Convolutional Neural Networks and Word Embedding for Low-Resource Languages: Tigrinya

This article studies convolutional neural networks for Tigrinya (also referred to as Tigrigna), which is a family of Semitic languages spoken in Eritrea and northern Ethiopia. Tigrinya is a “low-resource” language and is notable in terms of the absence of comprehensive and free data. Furthermore, it is characterized as one of the most semantically and syntactically complex languages in the world, similar to other Semitic languages. To the best of our knowledge, no previous research has been conducted on the state-of-the-art embedding technique that is shown here. We investigate which word representation methods perform better in terms of learning for single-label text classification problems, which are common when dealing with morphologically rich and complex languages. Manually annotated datasets are used here, where one contains 30,000 Tigrinya news texts from various sources with six categories of “sport”, “agriculture”, “politics”, “religion”, “education”, and “health” and one unannotated corpus that contains more than six million words. In this paper, we explore pretrained word embedding architectures using various convolutional neural networks (CNNs) to predict class labels. We construct a CNN with a continuous bag-of-words (CBOW) method, a CNN with a skip-gram method, and CNNs with and without word2vec and FastText to evaluate Tigrinya news articles. We also compare the CNN results with traditional machine learning models and evaluate the results in terms of the accuracy, precision, recall, and F1 scoring techniques. The CBOW CNN with word2vec achieves the best accuracy with 93.41%, significantly improving the accuracy for Tigrinya news classification.

FastText ve Kelime Çantası Kelime Temsil Yöntemlerinin Turistik Mekanlar İçin Yapılan Türkçe İncelemeler Kullanılarak Karşılaştırılması

Nowadays, with the increasing number and use of social media platforms, people now share their experiences about a product they have bought or a place they have been to on social media platforms more frequently. Considering the volume of data on social media platforms, it is considered that there is some meaningful information for institutions or companies in the reviews and experiences shared on social media platforms. As such, it is important to improve the methods of extracting meaningful information from the reviews and experiences shared on social media and to know which method is better. In this study, the classification successes of the bag of words and the fastText word representation methods, which are among the word representation methods in sentiment analysis methods mentioned above, were compared by using Turkish reviews performed for touristic places. Besides, while performing the comparison process, it was measured whether the process of separating the words into their roots and negation of the words, which is the preliminary stage of the sentiment analysis process, contributed to the classification success. In the study, both two-class (positive, negative) sentiment analysis and three-class (positive, negative, neutral) sentiment analysis were performed. Six data sets were created to carry out the mentioned comparison operations. The data sets were first classified using the Naive Bayes (NB), Multinomial Naive Bayes (MNB), k-Nearest Neighbor (k-NN) and Support Vector Machines (SVM) algorithms, which are frequently used in text mining, and based on bag of words word representation method, they were classified with WEKA program. After the test results of all data sets were obtained according to the bag of words word representation method, the tests of the fastText word representation method were carried out using the fastText library of the Python programming language. Classification procedures were carried out with 10-fold cross-validation methods, and f-score values of the classification processes were obtained. Finally, it was determined that bag of words word representation method performed a more successful classification than the fastText word representation method in two-class emotion analysis, while the fastText word representation method performed a more successful classification process than bag of words word representation method in three-class emotional analysis. It was observed that the process of separating the words into their roots and negating the words, which are the preliminary processes of sentiment analysis, did not contribute positively or negatively to the classification processes performed with the fastText word representation method. However, it was determined that it had a minor contribution to sentiment analysis processes performed by using bag of words word representation method. In the two-class sentiment analysis, the most successful classification result was achieved by using the machine learning model created with the SVM algorithm with the value of 0.91 f-score employing bag of words word representation method. In the three-class sentiment analysis, the most successful classification result was achieved with the machine learning model created using the fastText word representation method with the value of 0.78 f-score.

Defect Texts Mining of Secondary Device in Smart Substation with GloVe and Attention-Based Bidirectional LSTM

In the process of the operation and maintenance of secondary devices in smart substation, a wealth of defect texts containing the state information of the equipment is generated. Aiming to overcome the low efficiency and low accuracy problems of artificial power text classification and mining, combined with the characteristics of power equipment defect texts, a defect texts mining method for a secondary device in a smart substation is proposed, which integrates global vectors for word representation (GloVe) method and attention-based bidirectional long short-term memory (BiLSTM-Attention) method in one model. First, the characteristics of the defect texts are analyzed and preprocessed to improve the quality of the defect texts. Then, defect texts are segmented into words, and the words are mapped to the high-dimensional feature space based on the global vectors for word representation (GloVe) model to form distributed word vectors. Finally, a text classification model based on BiLSTM-Attention was proposed to classify the defect texts of a secondary device. Precision, Recall and F1-score are selected as evaluation indicators, and compared with traditional machine learning and deep learning models. The analysis of a case study shows that the BiLSTM-Attention model has better performance and can achieve the intelligent, accurate and efficient classification of secondary device defect texts. It can assist the operation and maintenance personnel to make scientific maintenance decisions on a secondary device and improve the level of intelligent management of equipment.

Learning variable-length representation of words

A standard word embedding algorithm, such as ‘word2vec’, embeds each word as a dense vector of a preset dimensionality, the components of which are learned by maximizing the likelihood of predicting the context around it. However, as an inherent linguistic phenomenon, it is evident that there is a varying degree of difficulty in identifying words from their contexts. This suggests that a variable granularity in word vector representation may be useful to obtain sparser and more compressed word representations, requiring less storage space. To that end, in this paper, we propose a word vector training algorithm that uses a variable number of components to represent words. Given a text collection of documents, our algorithm, similar to the skip-gram approach of word2vec, learns to predict the context of a word given the current instance of a word. However, in contrast to skip-gram, which uses a static number of dimensions for each word vector, we propose to dynamically increase the dimensionality as a stochastic function of the prediction error. Our experiments with standard test collections demonstrate that our word representation method is able to achieve comparable (and sometimes even better) effectiveness than skip-gram word2vec, using a significantly smaller number of parameters (achieving compression ratio of around 65%).

Contextual Word Representation and Deep Neural Networks-based Method for Arabic Question Classification

Contextual Word Representation and Deep Neural Networks-based Method for Arabic Question Classification

Enriching Word Embeddings with Global Information and Testing on Highly Inflected Language

In this paper we evaluate our new approach based on the Continuous Bag-of-Words and Skip-gram models enriched with global context information on highly inflected Czech language and compare it with English results. As a source of information we use Wikipedia, where articles are organized in a hierarchy of categories. These categories provide useful topical information about each article. Both models are evaluated on standard word similarity and word analogy datasets. Proposed models outperform other word representation methods when similar size of training data is used. Model provide similar performance especially with methods trained on much larger datasets.

Convolution–deconvolution word embedding: An end-to-end multi-prototype fusion embedding method for natural language processing

Existing unsupervised word embedding methods have been proved to be effective to capture latent semantic information on various tasks of Natural Language Processing (NLP). However, existing word representation methods are incapable of tackling both the polysemous-unaware and task-unaware problems that are common phenomena in NLP tasks. In this work, we present a novel Convolution–Deconvolution Word Embedding (CDWE), an end-to-end multi-prototype fusion embedding that fuses context-specific information and task-specific information. To the best of our knowledge, we are the first to extend deconvolution (e.g. convolution transpose), which has been widely used in computer vision, to word embedding generation. We empirically demonstrate the efficiency and generalization ability of CDWE by applying it to two representative tasks in NLP: text classification and machine translation. The models of CDWE significantly outperform the baselines and achieve state-of-the-art results on both tasks. To validate the efficiency of CDWE further, we demonstrate how CDWE solves the polysemous-unaware and task-unaware problems via analyzing the Text Deconvolution Saliency, which is an existing strategy for evaluating the outputs of deconvolution.

Chinese Event Detection Based on Multi-Feature Fusion and BiLSTM

With the rapid development of the Internet, the number of Internet users has grown rapidly, and the Internet has become more and more influential on people's lives. As a result, the amount of network text is increasing rapidly, and it is difficult to extract interested event information from it only by manual reading. Therefore, event extraction technique automatically extracting useful information from a large amount of unstructured texts becomes increasingly important. Event detection is the first step of event extraction task and plays a vital role in it. However, current event detection research lacks comprehensive consideration of the context of the trigger words. A Chinese event detection method based on multi-feature fusion and BiLSTM is proposed in this paper. The contextual information of word is divided into sentence-level and document-level in the method. The contextual information is captured based on BiLSTM model. At the same time, a word representation method suitable for trigger word classification tasks is proposed in this paper. The word representation incorporates semantic information, grammar information, and document-level context information of word. The word vectors in the sentence are sequentially inputted into BiLSTM model to obtain output vectors containing sentence-level contextual information. Finally, output vectors of BiLSTM are inputted into the Softmax classifier to realize the identification of the trigger words. The experimental results show that Chinese Event Detection Based on Multi-feature Fusion and BiLSTM method proposed in this paper has high accuracy.

Sentiment Analysis of Comment Texts Based on BiLSTM

With the rapid development of Internet technology and social networks, a large number of comment texts are generated on the Web. In the era of big data, mining the emotional tendency of comments through artificial intelligence technology is helpful for the timely understanding of network public opinion. The technology of sentiment analysis is a part of artificial intelligence, and its research is very meaningful for obtaining the sentiment trend of the comments. The essence of sentiment analysis is the text classification task, and different words have different contributions to classification. In the current sentiment analysis studies, distributed word representation is mostly used. However, distributed word representation only considers the semantic information of word, but ignore the sentiment information of the word. In this paper, an improved word representation method is proposed, which integrates the contribution of sentiment information into the traditional TF-IDF algorithm and generates weighted word vectors. The weighted word vectors are input into bidirectional long short term memory (BiLSTM) to capture the context information effectively, and the comment vectors are better represented. The sentiment tendency of the comment is obtained by feedforward neural network classifier. Under the same conditions, the proposed sentiment analysis method is compared with the sentiment analysis methods of RNN, CNN, LSTM, and NB. The experimental results show that the proposed sentiment analysis method has higher precision, recall, and F1 score. The method is proved to be effective with high accuracy on comments.

Enhanced news sentiment analysis using deep learning methods

We explore the predictive power of historical news sentiments based on financial market performance to forecast financial news sentiments. We define news sentiments based on stock price returns averaged over one minute right after a news article has been released. If the stock price exhibits positive (negative) return, we classify the news article released just prior to the observed stock return as positive (negative). We use Wikipedia and Gigaword five corpus articles from 2014 and we apply the global vectors for word representation method to this corpus to create word vectors to use as inputs into the deep learning TensorFlow network. We analyze high-frequency (intraday) Thompson Reuters News Archive as well as the high-frequency price tick history of the Dow Jones Industrial Average (DJIA 30) Index individual stocks for the period between 1/1/2003 and 12/30/2013. We apply a combination of deep learning methodologies of recurrent neural network with long short-term memory units to train the Thompson Reuters News Archive Data from 2003 to 2012, and we test the forecasting power of our method on 2013 News Archive data. We find that the forecasting accuracy of our methodology improves when we switch from random selection of positive and negative news to selecting the news with highest positive scores as positive news and news with highest negative scores as negative news to create our training data set.

Combining the Attention Network and Semantic Representation for Chinese Verb Metaphor Identification

Metaphor is the central issue of language and thinking. Metaphor identification plays a significant preliminary role in the field of machine translation, reading comprehension, and automatic summarization, making it a focus of natural language processing. Recently, research into Chinese verb metaphor identification has become a widespread concern. The main problem is that the usage of semantic resources is relatively simple, and there is a lack of deep semantic support. Therefore, this paper proposes a word representation method suitable for metaphor classification tasks, which combines the traditional word vector with structural information from the Synonym Thesaurus, so that the word vector can contain the abstraction degree of the word in the metaphor. On this basis, we propose a verb metaphor attention network based on subject–predicate and verb–object relationships, which gives full consideration to the global syntactic information when we perform LSTM encoding and calculate the weight of each word. At the same time, it can facilitate the understanding of literalness and non-literalness. The experimental results show that the identification effect improves on the existing results, indicating that word representation combining semantic resources and attention network can improve the verb metaphor identification performance.

A COMPARATIVE STUDY OF WORD REPRESENTATION METHODS WITH CONDITIONAL RANDOM FIELDS AND MAXIMUM ENTROPY MARKOV FOR BIO-NAMED ENTITY RECOGNITION

Bio-Named Entity Recognition (Bio-NER) is the process of identifying and semantically classifying biomedical technical terms and named entities in Biomedicine literature. Therefore, it is a major task in biomedical knowledge acquisition. Meanwhile, Natural Language Processing (NLP) plays an important role in Bio-NER in the biomedical domain. The first and most essential biomedical literature mining task incorporates biomedical entity recognition such as protein, gene, and chemicals. The most recent Bio-NER methods rely on predefined traditional features, which attempt to capture the specific surface properties of entity types. However, these empirically predefined feature sets differ between entity types and are manually constructed and complicated, which means developing them is costly. In this paper, we systematically present a comparative evaluation study of three methods, which are: the traditional feature representation method, the continuous bag-of-words (CBOW) model, and a new prototypical representation method with two popular sequence-labeling approaches (Conditional Random Fields (CRFs) and Maximum Entropy Markov Models (MEMM)). We evaluated these models with two major Bio-NER tasks, which involve the JNLPBA and GENETAG corpora. This paper examined the prototypical word representation method and found that Word2Vec can be successfully used for Bio-NER. Our results show that the new prototypical representation method improved the performance of the two machine learning models with different datasets. Also, the new prototypical representation method performed better than the traditional feature representation method and CBOW model for both datasets. Finally, our experiment proved that the CRF classifier with the new prototypical representation method achieved the best results when 90% data was used as training data, yielding overall F-measure values of 0.79% and 0.85% for the JNLPBA corpus and GENETAG corpus, respectively. In comparison, the results achieved using the ME classifier yielded overall F-measure values of 0.76% and 0.78% for the JNLPBA corpus and GENETAG corpus, respectively.