Vector Representations Of Words Research Articles

Label-Affinity Self-Adaptive Central Similarity Hashing for Image Retrieval

Due to the usage of global similarity, the hashing methods based on predefined hash centers have achieved more accurate retrieval results than the pairwise/triplet-based methods. Nevertheless, the fixed hash centers lack the perception of data distribution and are limited by the pre-determined Hadamard matrix, which consider neither the label semantic information nor the object scale size, resulting in sub-optimal retrieval performance and weak generalization ability. In this paper, we (1) adopt the label semantic information to generate self-adaptive hash centers and (2) propose the label-affinity coefficient ( <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">lac</i> ) that considers the scale size of each label/object appearing in the given image to calculate the real hash centroid for this image. Based on this, we propose <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Label-affinity Self-adaptive Central Similarity Hashing (LSCSH)</i> for image retrieval. LSCSH consists of a hash code generator module and a hash center adapter module. First, we obtain the label word vector ( <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">i.e.</i> , the word vector representation of each class label) via the Word2Vector technique to generate and update the hash centers that adapt to the distribution of both label word vectors and generated hash codes. Second, we learn <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">lac</i> to indicate the dominance of different labels corresponding to objects in each given image, which considers the unequal scales of each object (corresponding to a label) to calculate a more accurate hash centroid for each image. Last but not least, we design an asynchronous learning mechanism to enable each hash code and its corresponding hash centroid to adapt to each other dynamically. We conduct extensive experiments on 5 image datasets including CIFAR-10, ImageNet, VOC2012, MS-COCO and NUS-WIDE. The experimental results demonstrate that LSCSH can achieve the state-of-the-art visual retrieval performance on both single-label and multi-label image datasets. The code of this work is released at: <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/lzHZWZ/LSCSH_sourcecode.git</uri> .

Distance geometry for word representations and applications

Many machine learning methods used for the treatment of sequential data often rely on the construction of vector representations of unitary entities (e.g. words in natural language processing, or k-mers in bioinformatics). Traditionally, these representations are constructed with optimization formulations arising from co-occurrence based models. In this work, we propose a new method to embed these entities based on the Distance Geometry Problem: find object positions based on a subset of their pairwise distances or inner products. Considering the empirical Pointwise Mutual Information as a surrogate for the inner product, we discuss two Distance Geometry based algorithms to obtain word vector representations. The main advantage of such algorithms is their significantly lower computational complexity in comparison with state-of-the-art word embedding methods, which allows us to obtain word vector representations much faster. Furthermore, numerical experiments indicate that our word vectors behave quite well on text classification tasks in natural language processing as well as regression tasks in bioinformatics.

Algorithmic Bias and Non-Discrimination in Argentina

One of the major research problems related to artificial intelligence (AI) models at present is algorithmic bias. When an automated system “makes a decision” based on its training data, it can reveal biases similar to those inherent in the humans who provided the training data. Much of the data used to train the models comes from vector representations of words obtained from text corpuses, which can transmit stereotypes and social prejudices. AI system design focused on optimising processes and improving prediction accuracy ignores the need for new standards for compensating the negative impact of AI on the most vulnerable categories of peoples. An improved understanding of the relationship between algorithms, bias, and non-discrimination not only precedes any eventual solution, but also helps us to recognize how discrimination is created, maintained, and disseminated in the AI era, as well as how it could be projected into the future using various neurotechnologies. The opacity of the algorithmic decision-making process should be replaced by transparency in AI processes and models. The present work aims to reconcile the use of AI with algorithmic decision processes that respect the basic human rights of the individual, especially the principles of non-discrimination and positive discrimination. The Argentine legislation serves as the legal basis of this work.

Алгоритм пошуку кореферентних об’єктів в україномовних текстах з використанням дерев рішень

The paper examines the problem of coreference resolution in Ukrainian-language texts using decision trees. An application that uses vector representations of Elmo words and other characteristics for the automated formation of a decision tree has been developed. A set of prepared texts containing more than 360,000 words was used to form the decision tree and evaluate the accuracy of the algorithm. The decision tree created to determine whether a pair of objects is coreference was used to form clusters of coreference objects. Special metrics were used for comparison with the results obtained by other algorithms in the Ukrainian language.

Chinese Text Sentiment Classification Based on ERNIE and BiLSTM-AT

Chinese text sentiment classification is a sub-task of natural language processing. However, when text representation is carried out, the polysemy of a word cannot be processed when using the traditional language model to construct the word vector, and the long-distance text information cannot be fully extracted when extracting text features. To solve this problem, this paper proposes a text sentiment classification model combining ERNIE and BiLSTM-AT. First, the pre-training model ERNIE is used to obtain the word vector representation of the fused statement context. Then, the bidirectional long-short-term memory neural network is used to extract the context information and depth semantic information of the text. Then, the attention mechanism is used to assign the corresponding weights to the hidden layer vectors of each time step output by the BiLSTM layer, and the weighted summation is integrated into the sentence features. Finally, the softmax function is used to calculate the probability distribution of the emotional category of the text in the output layer. The results show that the proposed model can achieve high accuracy on both hotel reviews and takeaway reviews. Based on the pre-training model, adding bidirectional long-term and short-term memory network and attention mechanism is beneficial to improve the classification effect of the model, and has certain practicability in text sentiment classification tasks.

OpExHAN: opinion extraction using hierarchical attention network from unstructured reviews.

In recent decades, sellers and merchants have asked their customers to share their opinion on the products at online marketplace. Analysis of the massive amounts of reviews for a potential customer is an immense challenge, to decide whether to purchase a product or not. In this paper, a hierarchical attention network-based framework is presented to resolve this challenge. In this proposed framework, the Amazon’s Smartphone review dataset is preprocessed using NLP approaches and then applied Glove embedding to extract word vector representation of reviews which identifies contextual information of words. These word vectors are fed into the hierarchical attention network, which produces vectors at the word and sentence levels. Bi-GRU model encodes the words and sentences into hidden vectors. Finally, reviews are classified into five opinion classes like extremely positive, positive, extremely negative, negative and neutral. Furthermore, to perform the experiments with the proposed method, the dataset is divided into three parts such as 80% train, 10% validation and 10% test. Experiments reveal that the proposed framework outperforms baseline methods in terms of accuracy, precision and recall. The OpExHAN model achieved admirable results like 94.6% accuracy, 91% both precision and recall after a lot of hyper-parameter experimentation.

A Talent Training Model for Electrical Courses considering Diverse Constraint Models and Knowledge Recognition Algorithms

In order to improve the talent training effect of electrical courses, this paper proposes a talent training model for electrical courses considering diverse constraint models and knowledge recognition algorithms. In order to obtain better performance of traditional deep learning models, it is usually necessary to increase the parameter scale of traditional deep learning models. Pretrained language models can be trained unsupervised directly using unlabeled corpora to learn vector representations of words without using labeled datasets. In addition, this paper uses the knowledge base and alias dictionary to build a knowledge graph and constructs a teaching model for electrical courses considering diverse constraint models and knowledge recognition algorithms. Through the research, it can be seen that the experimental teaching model of electrical courses proposed in this paper considering diverse constraint models and knowledge recognition algorithms has a very good effect on talent training.

Do Semantic Vectors Contain Traces of Biophilic Connections Between Nature and Mental Health?

Natural environments have positive effects on mental health, but the nature of this relationship requires further understanding. There may be psychosocial aspects of this relationship that are reflected in the semantic structure of language. Natural language processing methods provide a tool to explore this possibility. In this study, machine learning-derived vector representations of words were provided by a neural network-based language model. This was combined with statistical analyses to test whether nature-related words have particularly strong positive versus negative connections to mental health. Statistically significant associations were indeed found between a range of nature-related words and positive-versus-negative word pairs related to mental health. The results thus confirm a semantic connection between nature and mental health as represented using computational methods trained on language usage. This raises the possibility that semantic associations could play a role in nature's influence on mental health, for instance through appraisal processes. The results provide a proof of principle of a methodological approach that could be used to further probe hypotheses on nature and well-being. Finally, the current results provide information on relationships between specific aspects of nature environments and mental health that may be of use to future research.

Impact of convolutional neural network and FastText embedding on text classification

Efficient word representation techniques (word embeddings) with modern machine learning models have shown reasonable improvement on automatic text classification tasks. However, the effectiveness of such techniques has not been evaluated yet in terms of insufficient word vector representation for training. Convolutional Neural Network has achieved significant results in pattern recognition, image analysis, and text classification. This study investigates the application of the CNN model on text classification problems by experimentation and analysis. We trained our classification model with a prominent word embedding generation model, Fast Text on publically available datasets, six benchmark datasets including Ag News, Amazon Full and Polarity, Yahoo Question Answer, Yelp Full, and Polarity. Furthermore, the proposed model has been tested on the Twitter US airlines non-benchmark dataset as well. The analysis indicates that using Fast Text as word embedding is a very promising approach.

Social Media Users' Perceptions of a Wearable Mixed Reality Headset During the COVID-19 Pandemic: Aspect-Based Sentiment Analysis.

BackgroundMixed reality (MR) devices provide real-time environments for physical-digital interactions across many domains. Owing to the unprecedented COVID-19 pandemic, MR technologies have supported many new use cases in the health care industry, enabling social distancing practices to minimize the risk of contact and transmission. Despite their novelty and increasing popularity, public evaluations are sparse and often rely on social interactions among users, developers, researchers, and potential buyers.ObjectiveThe purpose of this study is to use aspect-based sentiment analysis to explore changes in sentiment during the onset of the COVID-19 pandemic as new use cases emerged in the health care industry; to characterize net insights for MR developers, researchers, and users; and to analyze the features of HoloLens 2 (Microsoft Corporation) that are helpful for certain fields and purposes.MethodsTo investigate the user sentiment, we collected 8492 tweets on a wearable MR headset, HoloLens 2, during the initial 10 months since its release in late 2019, coinciding with the onset of the pandemic. Human annotators rated the individual tweets as positive, negative, neutral, or inconclusive. Furthermore, by hiring an interannotator to ensure agreements between the annotators, we used various word vector representations to measure the impact of specific words on sentiment ratings. Following the sentiment classification for each tweet, we trained a model for sentiment analysis via supervised learning.ResultsThe results of our sentiment analysis showed that the bag-of-words tokenizing method using a random forest supervised learning approach produced the highest accuracy of the test set at 81.29%. Furthermore, the results showed an apparent change in sentiment during the COVID-19 pandemic period. During the onset of the pandemic, consumer goods were severely affected, which aligns with a drop in both positive and negative sentiment. Following this, there is a sudden spike in positive sentiment, hypothesized to be caused by the new use cases of the device in health care education and training. This pandemic also aligns with drastic changes in the increased number of practical insights for MR developers, researchers, and users and positive net sentiments toward the HoloLens 2 characteristics.ConclusionsOur approach suggests a simple yet effective way to survey public opinion about new hardware devices quickly. The findings of this study contribute to a holistic understanding of public perception and acceptance of MR technologies during the COVID-19 pandemic and highlight several new implementations of HoloLens 2 in health care. We hope that these findings will inspire new use cases and technological features.

Analysis and Evaluation of the Impact of Integrating Mental Health Education into the Teaching of University Civics Courses in the Context of Artificial Intelligence

In higher education teaching work, college students not only need to master the professional knowledge and professional skills they learn during their school study but also need to improve their self-education and self-cultivation and constantly improve their comprehensive ability of learning. At present, there are differences and relationships between the education of college students in civic and mental normal education, and how to play the role of the integration of the two educations has become a problem that needs to be considered in the current work of college students’ training. The integration between civic education and mental normal education can make up for the shortcomings of monolithic civic education and mental normal education work and also optimize the teaching methods between them from a certain perspective to achieve the development goals of complementing each other and not being independent of each other, so that students can understand more learning methods and contents that promote the normal development of their own minds and minds. In response to the problem that mind-normal education cannot be automatically integrated into the teaching of university thought and political science courses, in the context of artificial intelligence, this paper proposes a multi-channel-based mind-normal and ideological and political information fusion model. The model has two channels, BERT+CNN and BERT and BiLSTM-Attention; firstly, the pretraining model BERT is used to obtain the word vector representation of the fused text context; then, the CNN network of channel one is used to enhance the ability of local feature extraction of the text, and the BiLSTM-Attention model of channel two enhances the ability of long sequence text processing and is key. Finally, the fused features of channel 1 and channel 2 are classified using a softmax excitation function. To verify the effectiveness of the proposed model, experiments are conducted on public datasets to demonstrate the effectiveness of the proposed method.

Dual embedding with input embedding and output embedding for better word representation

Recent &lt;span lang="EN-US"&gt;studies in distributed vector representations for words have variety of ways to represent words. We propose a various ways using input embedding and output embedding to better represent words than single model. We compared the performance in terms of word analogy and word similarity with each input and output embeddings and various dual embeddings which are the combination of those two embeddings. Performance evaluation results show that the proposed dual embeddings outperform each single embedding, especially with the way of simply adding input and output embeddings. We figured out two things in this paper, i) not only input embedding but also output embedding has such meaning to represent the words and ii) combining input embedding and output embedding as dual embedding outperforms the single embedding when we use input embedding and output embedding individually.&lt;/span&gt;

A Joint Learning Model to Extract Entities and Relations for Chinese Literature Based on Self-Attention

Extracting structured information from massive and heterogeneous text is a hot research topic in the field of natural language processing. It includes two key technologies: named entity recognition (NER) and relation extraction (RE). However, previous NER models consider less about the influence of mutual attention between words in the text on the prediction of entity labels, and there is less research on how to more fully extract sentence information for relational classification. In addition, previous research treats NER and RE as a pipeline of two separated tasks, which neglects the connection between them, and is mainly focused on the English corpus. In this paper, based on the self-attention mechanism, bidirectional long short-term memory (BiLSTM) neural network and conditional random field (CRF) model, we put forth a Chinese NER method based on BiLSTM-Self-Attention-CRF and a RE method based on BiLSTM-Multilevel-Attention in the field of Chinese literature. In particular, considering the relationship between these two tasks in terms of word vector and context feature representation in the neural network model, we put forth a joint learning method for NER and RE tasks based on the same underlying module, which jointly updates the parameters of the shared module during the training of these two tasks. For performance evaluation, we make use of the largest Chinese data set containing these two tasks. Experimental results show that the proposed independently trained NER and RE models achieve better performance than all previous methods, and our joint NER-RE training model outperforms the independently-trained NER and RE model.

Extraction of Joint Entity and Relationships with Soft Pruning and GlobalPointer

In recent years, scholars have paid increasing attention to the joint entity and relation extraction. However, the most difficult aspect of joint extraction is extracting overlapping triples. To address this problem, we propose a joint extraction model based on Soft Pruning and GlobalPointer, short for SGNet. In the first place, the BERT pretraining model is used to obtain the text word vector representation with contextual information, and then the local and non-local information of the word vector is obtained through graph operations. Specifically, to address the lack of information caused by the rule-based pruning strategies, we utilize the Gaussian Graph Generator and the attention-guiding layer to construct a fully connected graph. This process is called soft pruning for short. Then, to achieve node message passing and information integration, we employ GCNs and a thick connection layer. Next, we use the GlobalPointer decoder to convert triple extraction into quintuple extraction to tackle the problem of problematic overlapping triples extraction. The GlobalPointer decoder, unlike the typical feedforward neural network (FNN), can perform joint decoding. In the end, to evaluate the model performance, the experiment was carried out on two public datasets: the NYT and WebNLG. The experiments show that SGNet performs substantially better on overlapping extraction and achieves good results on two publicly available datasets.

Deep learning method for Chinese multisource point of interest matching

Multisource point of interest (POI) matching refers to the pairing of POIs that refer to the same geographic entity in different data sources. This also constitutes the core issue in geospatial data fusion and update. The existing methods cannot effectively capture the complex semantic information from a text, and the manually defined rules largely affect matching results. This study developed a multisource POI matching method based on deep learning that transforms the POI pair matching problem into a binary classification problem. First, we used three different Chinese word segmentation methods to segment the POI text attributes and used the segmentation results to train the Word2Vec model to generate the corresponding word vector representation. Then, we used the text convolutional neural network (Text-CNN) and multilayer perceptron (MLP) to extract the POI attributes' features and generate the corresponding feature vector representation. Finally, we used the enhanced sequential inference model (ESIM) to perform local inference and inference combination on each attribute to realize the classification of POI pairs. We used the POI dataset containing Baidu Map, Tencent Map, and Gaode Map from Chengdu to train, verify, and test the model. The experimental results show that the matching precision, recall rate, and F1 score of the proposed method exceed 98% on the test set, and it is significantly better than the existing matching methods.

WOVe: Incorporating Word Order in GloVe Word Embeddings

Word vector representations open up new opportunities to extract useful information from unstructured text. Defining a word as a vector made it easy for the machine learning algorithms to understand a text and extract information from. Word vector representations have been used in many applications such word synonyms, word analogy, syntactic parsing, and many others. GloVe, based on word contexts and matrix vectorization, is an effective vector-learning algorithm. It improves on previous vector-learning algorithms. However, the GloVe model fails to explicitly consider the order in which words appear within their contexts. In this paper, multiple methods of incorporating word order in GloVe word embeddings are proposed. Experimental results show that our Word Order Vector (WOVe) word embeddings approach outperforms unmodified GloVe on the natural language tasks of analogy completion and word similarity. WOVe with direct concatenation slightly outperformed GloVe on the word similarity task, increasing average rank by 2%. However, it greatly improved on the GloVe baseline on a word analogy task, achieving an average 36.34% improvement in accuracy.

EMDLP: Ensemble multiscale deep learning model for RNA methylation site prediction.

BackgroundRecent research recommends that epi-transcriptome regulation through post-transcriptional RNA modifications is essential for all sorts of RNA. Exact identification of RNA modification is vital for understanding their purposes and regulatory mechanisms. However, traditional experimental methods of identifying RNA modification sites are relatively complicated, time-consuming, and laborious.Machine learning approaches have been applied in the procedures of RNA sequence features extraction and classification in a computational way, which may supplement experimental approaches more efficiently. Recently, convolutional neural network (CNN) and long short-term memory (LSTM) have been demonstrated achievements in modification site prediction on account of their powerful functions in representation learning. However, CNN can learn the local response from the spatial data but cannot learn sequential correlations. And LSTM is specialized for sequential modeling and can access both the contextual representation but lacks spatial data extraction compared with CNN. There is strong motivation to construct a prediction framework using natural language processing (NLP), deep learning (DL) for these reasons.ResultsThis study presents an ensemble multiscale deep learning predictor (EMDLP) to identify RNA methylation sites in an NLP and DL way. It organically combines the dilated convolution and Bidirectional LSTM (BiLSTM), which helps to take better advantage of the local and global information for site prediction.The first step of EMDLP is to represent the RNA sequences in an NLP way. Thus, three encodings, e.g., RNA word embedding, One-hot encoding, and RGloVe, which is an improved learning method of word vector representation based on GloVe, are adopted to decipher sites from the viewpoints of the local and global information. Then, a dilated convolutional Bidirectional LSTM network (DCB) model is constructed with the dilated convolutional neural network (DCNN) followed by BiLSTM to extract potential contributing features for methylation site prediction. Finally, these three encoding methods are integrated by a soft vote to obtain better predictive performance. Experiment results on m1A and m6A reveal that the area under the receiver operating characteristic(AUROC) of EMDLP obtains respectively 95.56%, 85.24%, and outperforms the state-of-the-art models. To maximize user convenience, a user-friendly webserver for EMDLP was publicly available at http://www.labiip.net/EMDLP/index.php (http://47.104.130.81/EMDLP/index.php).ConclusionsWe developed a predictor for m1A and m6A methylation sites.Supplementary InformationThe online version contains supplementary material available at 10.1186/s12859-022-04756-1.

Building Better Machine Learning Models for Rhetorical Analyses: The Use of Rhetorical Feature Sets for Training Artificial Neural Network Models

ABSTRACT In this paper, we investigate two approaches to building artificial neural network models to compare their effectiveness for accurately classifying rhetorical structures across multiple (non-binary) classes in small textual datasets. We find that the most accurate type of model can be designed by using a custom rhetorical feature list coupled with general-language word vector representations, which outperforms models with more computing-intensive architectures.

Anomaly Detection in Log Files Using Selected Natural Language Processing Methods

In this article, we address the problem of detecting anomalies in system log files. Computer systems generate huge numbers of events, which are noted in event log files. While most of them report normal actions, an unusual entry may inform about a failure or malware infection. A human operator may easily miss such an entry; therefore, anomaly detection methods are used for this purpose. In our work, we used an approach known from the natural language processing (NLP) domain, which operates on so-called embeddings, that is vector representations of words or phrases. We describe an improved version of the LogEvent2Vec algorithm, proposed in 2020. In contrast to the original version, we propose a significant shortening of the analysis window, which both increased the accuracy of anomaly detection and made further analysis of suspicious sequences much easier. We experimented with various binary classifiers, such as decision trees or multilayer perceptrons (MLPs), and the Blue Gene/L dataset. We showed that selecting an optimal classifier (in this case, MLP) and a short log sequence gave very good results. The improved version of the algorithm yielded the best F1-score of 0.997, compared to 0.886 in the original version of the algorithm.

Vector based sentiment and emotion analysis from text: A survey

As a primary means of communication, texts are used to implicitly or explicitly reflect emotions. Emotion or sentiment detection from text has emerged as an important and expanding research area to more clearly understand the actual feelings of humans. Most of the word representation models, such as Word2Vec or GloVe, project the words in vector space such that if words have similar context, then their representations are also very similar. However, according to the recent studies, this approach limits the success of studies in areas such as emotion detection. For instance, love and happy are emotionally similar words, but they may have a lower similarity score than emotionally dissimilar word such as happy and sad which have high co-occurrence frequency, as they are in similar contexts. Recently, researchers propose some methods based on the addition of emotional or sentimental information to the original word vectors. These have improved the vector representation of words and achieved better results in emotion detection or classification tasks. In this survey, we analyze in detail such recent text-based studies in the literature. We summarize their methods used, emotion models, data sources, findings, and performances.